LlamaIndex

Part I: The mental model

readers (SimpleDirectoryReader, LlamaHub)   raw bytes -> [Document]
      |
      v
node_parser (SentenceSplitter, ...)         Document -> [TextNode] with metadata + relationships
      |
      v
embed_model.get_text_embedding_batch        each node -> a vector
      |
      v
VectorStoreIndex          StorageContext = docstore + index_store + vector_store
      |
      v
index.as_query_engine()   retriever + node_postprocessors + response_synthesizer
      |
      v
query_engine.query("...") wraps the string in a QueryBundle
      |
      v
retriever.retrieve()      embed query -> similarity search -> [NodeWithScore]
      |
      v
node_postprocessors       similarity cutoff, rerank, metadata replacement
      |
      v
response_synthesizer      pack nodes into prompts -> LLM -> Response(.response, .source_nodes)

The one-sentence identity. LlamaIndex is a data framework whose single load-bearing abstraction is the Node, a chunk of content that carries text, metadata, and typed relationships to other chunks, so that ingestion, indexing, retrieval, and answer synthesis are all just operations that produce or consume lists of nodes. A raw file becomes a Document, a document is split into TextNode objects, an index organizes nodes for lookup, a retriever returns nodes as NodeWithScore, and a response synthesizer turns those scored nodes plus the question into a prompt for the model. Once you see that every stage speaks nodes, the framework stops looking like a pile of classes and starts looking like one pipeline with swappable stages.

The second load-bearing idea is a clean separation that most people conflate. Retrieval and synthesis are two different jobs, and a query engine is precisely the composition of a retriever, a list of node postprocessors, and a response synthesizer. The retriever decides which nodes are relevant. The synthesizer decides how to fit those nodes into the model's context window and phrase the answer. Because the two are separate objects with stable interfaces, you can keep a vector retriever and change how answers are composed, or keep the synthesizer and swap in a keyword or hybrid retriever, without touching the rest. That factoring is the reason the same handful of pieces covers naive RAG, reranked RAG, multi-document routing, SQL question answering, and agentic tool use.

A note on the surface you will actually import. In the v0.10 release LlamaIndex split into a small llama-index-core package plus dozens of separately versioned integration packages, so the modern import root is llama_index.core and model or store integrations live under names like llama_index.llms.openai and llama_index.vector_stores.chroma. Older tutorials that write from llama_index import ... or use a ServiceContext predate that split, and I flag the migrations as they come up. Everything here is described against the current core API. The project moves quickly, so where a class has changed location or a detail is likely to shift I say so and stay at concept level rather than inventing a path.

Part II: Using it

LlamaIndex is a pure-Python library and runs anywhere Python does, including macOS, with no GPU required for the framework itself. The starter metapackage pulls in core plus a default OpenAI LLM and embedding model and the local file reader, which is enough for a first pipeline.

pip install llama-index
# core only, then add exactly the integrations you want:
pip install llama-index-core
pip install llama-index-llms-openai llama-index-embeddings-openai
export OPENAI_API_KEY=sk-...

The canonical first program is the five-line RAG loop. Put a few text or PDF files in a folder called data and run this.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)

That is the whole ceremony, and it hides four steps. Reading turns files into Document objects. from_documents runs the default node parser to split each document into nodes, calls the embedding model to vectorize every node, and stores the vectors in an in-memory SimpleVectorStore. as_query_engine assembles a retriever and a response synthesizer around the index. And query embeds the question, finds the most similar nodes, and asks the model to answer using them. The Response object prints as its answer text but also carries response.source_nodes, the exact nodes that grounded the answer, which is the first thing to inspect when an answer looks wrong.

Building the index costs embedding calls, so you do not want to rebuild it every run. Persist it once and reload it.

from llama_index.core import StorageContext, load_index_from_storage

index.storage_context.persist(persist_dir="storage")

# later, in a fresh process
storage_context = StorageContext.from_defaults(persist_dir="storage")
index = load_index_from_storage(storage_context)

The global Settings object is where you change the models and chunking defaults for everything downstream. This is the modern replacement for the deprecated ServiceContext, and it is a plain singleton, so set it once near the top of your program.

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 512
Settings.chunk_overlap = 32

When you want control over the middle of the pipeline you build it explicitly. The IngestionPipeline makes the transform chain a first-class object with a cache, so re-running over unchanged documents skips the embedding calls.

from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=512, chunk_overlap=32),
        OpenAIEmbedding(),
    ]
)
nodes = pipeline.run(documents=documents)   # returns embedded TextNode objects
index = VectorStoreIndex(nodes)

And the query side comes apart the same way. Assembling the query engine by hand is how you add a similarity cutoff, a reranker, or a non-default response mode.

from llama_index.core import get_response_synthesizer
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import SimilarityPostprocessor

retriever = VectorIndexRetriever(index=index, similarity_top_k=5)
synth = get_response_synthesizer(response_mode="compact")
query_engine = RetrieverQueryEngine(
    retriever=retriever,
    response_synthesizer=synth,
    node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.7)],
)
response = query_engine.query("Summarize the argument in section 3.")

Now the mistakes people make early. First, forgetting that from_documents silently calls the embedding model, so a missing or wrong OPENAI_API_KEY surfaces as an authentication error deep in indexing rather than at import. Second, treating similarity_top_k as free. Retrieving more nodes means a longer prompt, more tokens, and often a worse answer as the relevant chunk gets buried, so tune it deliberately rather than raising it to paper over a retrieval miss. Third, mixing embedding models between indexing and querying. The query vector must come from the same model that produced the stored vectors or the cosine scores are meaningless, and this bites people who set Settings.embed_model in one script and forget it in another. Fourth, reaching for an agent when a plain query engine would do. If your task is answer-a-question-over-docs, a query engine is faster, cheaper, and far easier to debug than a tool-calling loop.

Part III: When it is the right tool

LlamaIndex is the right tool when the center of gravity of your application is your data. You have documents, tables, wikis, or a database, and you want a language model to answer over them accurately and with citations. It gives you readers for hundreds of source formats through LlamaHub, a well-worn set of chunking strategies, a dozen index and retriever types, response synthesis that respects the context window, and increasingly good observability into which nodes produced an answer. If you are learning applied RAG, it is also one of the clearest codebases to read, which is why it pairs naturally with the applied generative AI material.

The honest cases for alternatives. LangChain and its graph successor LangGraph cover a broader surface of chains, agents, and integrations, and if your app is mostly orchestration and tool use with retrieval as one step among many, that ecosystem may fit the shape of your problem better. Haystack, from deepset, is a strong retrieval-first pipeline framework with a similar node-and-pipeline flavor and a production bent. DSPy takes a different stance entirely, treating prompts and pipelines as programs to be compiled and optimized against a metric rather than composed by hand, which is the better choice when you want to optimize a pipeline rather than wire one. And for a small, well-understood corpus you can skip frameworks altogether. A vector library like FAISS plus a few hundred lines of your own glue and a direct model SDK call is a legitimate and very debuggable stack, and LlamaIndex earns its keep only once the variety of readers, index types, and query strategies starts to outpace what you want to maintain yourself.

The shape-of-the-problem warning is about retrieval quality, which is where RAG systems actually live or die. The framework makes it trivial to stand up a naive vector index, and a naive vector index over badly chunked documents with no reranking is the NFS-mounted-SQLite of this domain. It looks fine in a demo and fails quietly on real questions.

fragile:   chunk blindly -> embed -> top_k cosine -> stuff into prompt
   symptom: confidently wrong answers, right facts never retrieved,
            the relevant sentence split across two chunks

sturdier:  chunk with structure (sentences, headings, windows)
           -> embed -> retrieve wider top_k
           -> rerank with a cross-encoder, cut low scores
           -> synthesize with citations, inspect source_nodes

None of that is exotic, and every piece of it is a swappable stage in the pipeline above. But the framework will happily let you ship the fragile version, and the failure mode is not an error, it is an answer that sounds right. Treat retrieval evaluation as part of the build, not an afterthought.

Part IV: The full life of one query

The specimen is a single call, response = query_engine.query("What did the author do growing up?"), against a VectorStoreIndex built over a folder of essays with the default in-memory store. Most of the machinery runs the same whether the store is SimpleVectorStore or a managed database, and where the path forks I follow both briefly.

Stage 1: query() and the QueryBundle

as_query_engine returned a RetrieverQueryEngine. Its base class wraps your string in a QueryBundle, an object that carries the query string and can also carry custom embeddings or embedding strings for advanced cases, then fires the query-start callback for whatever observability handlers are installed and dispatches to the engine's internal _query. The QueryBundle exists so that every stage downstream receives the same rich query object rather than a bare string, which is what lets a retriever reuse a precomputed query embedding or a router inspect the query before deciding where to send it.

Stage 2: retrieve, embed, and search

RetrieverQueryEngine first calls its retriever, a VectorIndexRetriever. The retriever asks the embedding model for the query's vector, then constructs a VectorStoreQuery holding that embedding and similarity_top_k, and hands it to the vector store's query method. With the default SimpleVectorStore the search is honest brute force. It computes cosine similarity between the query vector and every stored node vector in memory with numpy and returns the top k. With an external store the same call delegates to that store's index, an HNSW or IVF structure in something like FAISS or a managed vector database, which is the retrieval math covered in the databases and FAISS write-ups. The store returns node ids and similarity scores, the retriever looks the full nodes up in the docstore, and it wraps each one as a NodeWithScore, a node paired with its retrieval score. This list of scored nodes is the currency for everything that follows.

Stage 3: node postprocessors

Before synthesis the engine runs the retrieved nodes through its node_postprocessors in order. Each postprocessor takes a list of NodeWithScore and returns a possibly shorter, reordered, or rewritten list. A SimilarityPostprocessor drops nodes below a score cutoff. An LLMRerank or a cross-encoder SentenceTransformerRerank re-scores the candidates with a stronger relevance model and keeps the best. A MetadataReplacementPostProcessor swaps each node's text for a wider window stored in its metadata, which is how the sentence-window strategy retrieves precisely but synthesizes with context. If you set none, this stage is a pass-through. This is the seam where naive RAG becomes good RAG, and it is deliberately its own stage so you can add reranking without touching retrieval or synthesis.

Stage 4: response synthesis

The engine now calls response_synthesizer.synthesize(query, nodes). The synthesizer's job is to fit possibly more node text than the context window allows into one or more prompts and produce a coherent answer. The default mode, compact, packs as many node chunks as fit into a single question-answering prompt using the text_qa_template, and if the nodes overflow one prompt it continues in refine style, feeding the running answer and the next chunk into a refine_template so the model updates its answer with each additional piece of context. Each template is a real, inspectable prompt with placeholders for the context and the query. The synthesizer makes one or more LLM calls, collects the text, and returns a Response.

Stage 5: the Response and its sources

The Response object holds the answer string in response.response, the grounding nodes in response.source_nodes, and a metadata map. Printing it shows the answer, but the source nodes are the point. They are the exact chunks the model was shown, with their scores and their document metadata, so you can verify that the answer came from your data and not from the model's memory. If you had asked for a streaming response the synthesizer would instead return a generator you consume token by token, and if you had chosen response_mode="no_text" it would return the retrieved nodes with no LLM call at all, which is the fastest way to debug whether a bad answer is a retrieval problem or a synthesis problem. That closes the loop of one query. A string in, an embedding and a similarity search, a reranked list of nodes, a packed prompt, and a grounded answer out.

Part V: Internals deep dives

Deep dive: the Node, and why everything is one

The schema module defines the vocabulary. A Document is the source-level container, a whole file or record with text and metadata. A TextNode is a chunk of that document, and both descend from a common BaseNode that gives every node a stable node_id, a metadata dict, an embedding slot, and a set of relationships. The relationships are typed by NodeRelationship, and the important ones are SOURCE back to the parent document, PREVIOUS and NEXT to sibling chunks, and PARENT and CHILD for hierarchical splits.

Document "essay.txt"
   |  SOURCE
   +-- TextNode 0  --NEXT-->  TextNode 1  --NEXT-->  TextNode 2
       node_id, text, metadata{file_name, page_label, ...}, embedding, relationships

Two design choices here do a lot of work. First, metadata travels with the node and can be included in or excluded from what the embedding model and the LLM see, controlled per field, so a file_name can help the model cite without polluting the similarity vector. Second, the relationships mean a retrieved chunk knows its neighbors, which is what lets a PrevNextNodePostprocessor pull in surrounding context or a hierarchical retriever climb from a small matched chunk to its larger parent. Because every stage produces or consumes these nodes, adding a capability usually means writing one new transform, retriever, or postprocessor rather than changing the pipeline, and that uniformity is the whole reason the framework composes.

Deep dive: readers and node parsers

Ingestion has two halves. Readers, also called data connectors, turn an external source into Document objects. SimpleDirectoryReader handles local files and picks a parser by extension, and LlamaHub hosts hundreds more for databases, APIs, Notion, Slack, and the rest, each now shipped as a llama-index-readers-* package. For messy PDFs the LlamaParse service, exposed as a reader, does layout-aware parsing into clean markdown, which upstream node parsers then split well.

Node parsers turn documents into nodes, and the choice matters more than beginners expect because retrieval can only be as good as the chunks. The default SentenceSplitter keeps sentences intact while targeting a chunk size, which avoids cutting a fact in half at a token boundary. TokenTextSplitter splits on a raw token count when you want strict sizes. SentenceWindowNodeParser makes each node a single sentence but stashes a surrounding window in metadata, so you retrieve precisely and, with the metadata-replacement postprocessor, synthesize with context. SemanticSplitterNodeParser places boundaries where the embedding similarity between adjacent sentences drops, chunking by meaning rather than length. HierarchicalNodeParser produces parents and children at several sizes for auto-merging retrieval, and MarkdownNodeParser and CodeSplitter respect document structure. All of them are just transforms from a list of nodes to a list of nodes, which is why the ingestion pipeline can chain them with an embedding step.

Deep dive: the indices, vector, summary, keyword

An index is a data structure over nodes that makes a particular kind of retrieval cheap, and the three foundational ones embody three different answers to what should be retrieved.

VectorStoreIndex is the one you reach for by default. It embeds every node and stores the vectors, and its retriever embeds the query and returns the nearest nodes. Its behavior is dominated by the vector store behind it, from the in-memory SimpleVectorStore for prototypes to Chroma, Qdrant, Weaviate, pgvector, or FAISS in production, all exposed through the same VectorStore interface so the index code does not change when the backend does.

SummaryIndex, once called the list index, keeps nodes in a plain ordered list and by default retrieves all of them. That sounds naive, and for question answering it is, but for summarize-this-whole-document it is exactly right, because you want every chunk fed through a tree_summarize synthesis rather than a similarity filter that might drop half the content. It also offers embedding and LLM-based retrievers when you want a subset. KeywordTableIndex extracts keywords from each node and inverts them into a keyword-to-nodes table. The plain KeywordTableIndex uses the LLM to extract keywords, while SimpleKeywordTableIndex and RAKEKeywordTableIndex use cheap regex and RAKE extraction. At query time it pulls keywords from the question and returns the nodes those keywords point to, which shines for exact identifiers and codes that embeddings blur together. Beyond these three sit TreeIndex, DocumentSummaryIndex, and the graph indices KnowledgeGraphIndex and the newer PropertyGraphIndex, all of which are still just different organizations of the same nodes.

Deep dive: retrievers and the query engine

A retriever implements one method, take a QueryBundle, return a list of NodeWithScore, and that narrow contract is why retrievers compose. Every index exposes one through as_retriever, so a VectorStoreIndex gives a VectorIndexRetriever and a SummaryIndex gives its all-nodes or selective retriever. Because they share the interface you can wrap them. A QueryFusionRetriever fires several retrievers and fuses their results for hybrid search, an AutoMergingRetriever walks the parent and child relationships to merge small hits into their larger parent, and a RecursiveRetriever follows references from one node to another index. None of these know or care which concrete retriever they wrap.

A query engine adds the answer. The base RetrieverQueryEngine is exactly retriever plus postprocessors plus synthesizer, as Part IV traced, but the family is larger. A RouterQueryEngine uses the LLM to pick which sub-engine should handle a question, sending summary questions to a summary index and factual ones to a vector index. A SubQuestionQueryEngine decomposes a complex question into sub-questions, answers each against a chosen tool, and combines the results, which is how you answer questions that span several documents. A CitationQueryEngine splits sources so the answer can cite them by number, and the SQL engines like NLSQLTableQueryEngine translate a natural-language question into SQL against a real database and answer from the rows. The applied generative AI material builds several of these by hand, and reading them here is the fastest way to see how modest the machinery underneath a RouterQueryEngine actually is.

Deep dive: response synthesis modes

Response synthesis is where the context window becomes the binding constraint, and the response modes are the different strategies for living within it. Choosing the wrong one is a common cause of truncated or expensive answers.

The mechanism behind refine and compact is worth internalizing. refine feeds the first node with a question-answering template, then for each remaining node feeds the current answer plus the new node with a refine template that asks the model to improve the answer only if the new context helps. compact is the same loop but it first stuffs as many nodes as fit into each prompt, so a five-node answer that would take five calls in refine might take one or two. tree_summarize instead builds a bottom-up tree, summarizing batches of nodes in parallel and then summarizing those summaries until one remains, which is why it scales to large corpora where refine's sequential chain would be slow. The response mode is not a cosmetic setting, it is the algorithm that decides how your token budget is spent, and it is a one-line change precisely because synthesis is a separate stage from retrieval.

Deep dive: storage, and Settings over ServiceContext

The StorageContext bundles the four stores an index leans on. The docstore holds the actual node objects keyed by id, the index store holds the index's own structure such as the summary index's node ordering or the keyword table, the vector store holds embeddings, and a graph or property-graph store holds triples for the graph indices. StorageContext.from_defaults gives you in-memory versions of all four, and swapping any one for a persistent backend, a Redis docstore, a Postgres index store, a real vector database, is a constructor argument, not a rewrite. Persisting an index writes these stores to disk, and load_index_from_storage reads them back and reconstructs the index object, which is why the reload in Part II does not need to know the index type in advance.

The configuration story has a sharp edge worth calling out. Older LlamaIndex threaded a ServiceContext object holding the LLM, the embedding model, the node parser, and the prompt helper through nearly every constructor. Current versions replace it with the global Settings singleton, and ServiceContext is deprecated and removed in recent releases. The practical upshot is that most tutorials and Stack Overflow answers written before 2024 will not run as printed, and the fix is almost always to delete the service_context= argument and set the corresponding Settings field once. Any object that needs a non-default model still accepts it directly, for example passing llm= to a query engine, and that local argument wins over the global default.

Deep dive: the agent and workflow layer

The newest layer sits on top of everything above and answers a different question, how do you orchestrate several steps and tools rather than run one query. The foundation is Workflows, an event-driven engine where you subclass Workflow and write @step methods that receive an event and return the next event. A step's input event type is declared by its type hint, and the engine wires steps together by matching who emits an event to who consumes it, so the control flow is defined by the events, not by a hand-drawn graph. Execution starts from a StartEvent and ends when a step returns a StopEvent, and a shared Context object carries state between steps and supports fan-out and fan-in with send_event and collect_events.

from llama_index.core.workflow import (
    Workflow, step, StartEvent, StopEvent, Event, Context,
)
from llama_index.core import get_response_synthesizer

class RetrieveEvent(Event):
    nodes: list

class RAGWorkflow(Workflow):
    @step
    async def retrieve(self, ctx: Context, ev: StartEvent) -> RetrieveEvent:
        retriever = ev.index.as_retriever(similarity_top_k=5)
        nodes = await retriever.aretrieve(ev.query)
        await ctx.store.set("query", ev.query)
        return RetrieveEvent(nodes=nodes)

    @step
    async def synthesize(self, ctx: Context, ev: RetrieveEvent) -> StopEvent:
        query = await ctx.store.get("query")
        synth = get_response_synthesizer()
        response = await synth.asynthesize(query, nodes=ev.nodes)
        return StopEvent(result=response)

You run it with await RAGWorkflow().run(index=index, query="..."), and because the engine knows every step and event it can draw the graph with draw_all_possible_flows for inspection. The workflow context API has evolved, older code used ctx.set and ctx.get where current code uses ctx.store, and the workflow engine itself has been factored toward a standalone package re-exported from llama_index.core.workflow, so treat the import as stable and the exact storage spelling as a detail that may drift.

Agents are built on this engine. A FunctionAgent wraps a function-calling model in a workflow that loops, calling tools until the model produces a final answer, and a ReActAgent does the same with the reason-act-observe prompting pattern for models without native tool calling. Tools are the bridge back to everything in this chapter, because a QueryEngineTool turns any query engine into a callable the agent can invoke, so an agent can reason over when to search your documents and when to do something else.

from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.tools import QueryEngineTool
from llama_index.llms.openai import OpenAI

tool = QueryEngineTool.from_defaults(
    query_engine=query_engine,
    name="essays",
    description="Answers questions about the author's essays.",
)
agent = FunctionAgent(tools=[tool], llm=OpenAI(model="gpt-4o"))
response = await agent.run("What advice does the author give founders, with sources?")

An AgentWorkflow goes one level up and orchestrates several agents that hand off to each other. The older ReActAgent.from_tools and OpenAIAgent APIs still exist under llama_index.core.agent and the OpenAI agent package, and the workflow-based agents in llama_index.core.agent.workflow are the current direction, so new code should prefer the latter. The layering is the point. An agent is a workflow, a workflow step can call a query engine, a query engine is a retriever and a synthesizer, and a retriever returns nodes. It is nodes all the way down.

Part VI: Reading the repository

The run-llama/llama_index repository is a monorepo of many packages rather than one library, and knowing the top-level layout saves hours. The exact folder names are stable but individual file paths inside llama-index-core do move, so I name modules by their role.

Stage 0, the layout. At the root, the framework lives under llama-index-core, every model, store, and reader integration lives under llama-index-integrations as an independently versioned package, pre-packaged applications called LlamaPacks live under llama-index-packs, unstable pieces sit in llama-index-experimental, the command-line tool is in llama-index-cli, and prose docs are in docs. Questions to hold. Why does an import from llama_index.llms.openai resolve even though the code is in a separate package, and what does the top-level llama-index metapackage actually install?

Stage 1, the schema. Read the core schema module that defines Document, BaseNode, TextNode, NodeWithScore, NodeRelationship, and RelatedNodeInfo. This is the vocabulary the rest of the codebase speaks. Questions. What is the difference between a document and a node, and how does metadata get included in or excluded from the embedding versus the LLM prompt?

Stage 2, ingestion. Read the readers module around SimpleDirectoryReader, then the node_parser package, starting with SentenceSplitter, then the ingestion package with IngestionPipeline and its cache. Questions. What is the common transform interface that lets a splitter and an embedding model chain in one pipeline, and what does the cache key on?

Stage 3, indices and retrievers. Read the indices package, one index per sitting, in the order vector store, summary or list, then keyword table, and for each read its paired retriever. Then read the vector_stores module to see the SimpleVectorStore brute-force search that the external stores replace. Questions. Where does from_documents call the node parser and the embedding model, and what exactly does as_retriever return for each index?

Stage 4, the query path. Read the query_engine package with RetrieverQueryEngine as the destination, then the postprocessor package, then the response_synthesizers package with Refine and TreeSummarize. Read them against Part IV. Questions. In what order do retrieve, postprocess, and synthesize run, and how does compact differ from refine in the code, not just the docs?

Stage 5, storage and settings. Read the storage package with StorageContext and the docstore and index store, and the settings module. Questions. What are the four stores, what does persist write, and how does a Settings field become the default an index picks up?

Stage 6, the frontier. Read the workflow engine and the agent package, especially agent/workflow with FunctionAgent and AgentWorkflow, and the tools package with QueryEngineTool. Questions. How does the engine decide which step runs next from event types alone, and how does an agent invoke a query engine as a tool?

Where not to start. The graph and property-graph indices add a store and an extraction pipeline that only make sense after the dense vector story is solid, the llama-index-experimental tree is unstable by definition, and the older QueryPipeline DAG API is being superseded by Workflows, so learn Workflows first and meet the query pipeline only when you hit older code that uses it.

Part VII: Hands-on labs

None of these need a GPU. They need an API key for whichever model you configure, and a small folder of documents. Lab outputs vary with the fast pace of the project.

Lab 1: the five-line pipeline, then look inside. Concept: the query life of Part IV.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

docs = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(docs)
resp = index.as_query_engine(similarity_top_k=3).query("Give me the main claim.")
print(resp)
for n in resp.source_nodes:
    print(round(n.score, 3), n.node.metadata.get("file_name"), n.node.text[:80])

Read the source nodes before the answer. Confirm the top node actually contains the claim, then rerun with similarity_top_k=1 and 10 and watch how the answer and its grounding change. This is the fastest way to feel that retrieval, not the model, sets the ceiling on answer quality.

Lab 2: swap the response mode. Concept: synthesis as a separate, swappable stage.

qe = index.as_query_engine(response_mode="tree_summarize")
print(qe.query("Summarize the entire document."))

qe2 = index.as_query_engine(response_mode="no_text")
print([n.node.node_id for n in qe2.query("anything").source_nodes])

Notice that tree_summarize gives a fuller summary than the default on a summarization prompt, and that no_text returns nodes with no answer, which is the debugging trick from Part IV. Ask a factual question in tree_summarize mode and see why it is the wrong mode for lookup.

Lab 3: build the ingestion pipeline by hand. Concept: transforms as a chain, and the cache.

from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding

pipe = IngestionPipeline(transformations=[
    SentenceSplitter(chunk_size=256, chunk_overlap=16),
    OpenAIEmbedding(),
])
nodes = pipe.run(documents=docs)
print(len(nodes), "nodes; first embedding dim:", len(nodes[0].embedding))
nodes_again = pipe.run(documents=docs)   # served from cache, no new embed calls

Change chunk_size to 1024 and watch the node count drop and the retrieved context per node grow. This is the single most important knob in RAG and it lives in the parser, not the index.

Lab 4: add a reranker. Concept: node postprocessors turning naive RAG into good RAG.

from llama_index.core.postprocessor import SentenceTransformerRerank

rerank = SentenceTransformerRerank(top_n=3, model="cross-encoder/ms-marco-MiniLM-L-2-v2")
qe = index.as_query_engine(similarity_top_k=10, node_postprocessors=[rerank])
resp = qe.query("A question whose answer is not in the top vector hit.")
print(resp)

Retrieve a wide top-10, then let the cross-encoder pick the best 3. Compare the answer and the surviving source nodes against the same query with no reranker at similarity_top_k=3. The gap is the value of the postprocessing stage.

Lab 5: two indices, one router. Concept: retrieval and synthesis strategies as composable engines.

from llama_index.core import SummaryIndex
from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.tools import QueryEngineTool

vector_qe = index.as_query_engine()
summary_qe = SummaryIndex.from_documents(docs).as_query_engine(response_mode="tree_summarize")

router = RouterQueryEngine.from_defaults(query_engine_tools=[
    QueryEngineTool.from_defaults(vector_qe, description="Specific factual questions."),
    QueryEngineTool.from_defaults(summary_qe, description="Whole-document summaries."),
])
print(router.query("Summarize the document."))     # routes to the summary engine
print(router.query("What is the exact date in section 2?"))   # routes to the vector engine

Watch the router pick a different sub-engine for a summary question than for a factual one. This is the mental model of Part V made concrete, a summary index for global questions and a vector index for local ones, chosen by the model.

Lab 6: a minimal workflow. Concept: the event-driven layer under the agents.

from llama_index.core.workflow import Workflow, step, StartEvent, StopEvent
from llama_index.utils.workflow import draw_all_possible_flows

class Echo(Workflow):
    @step
    async def go(self, ev: StartEvent) -> StopEvent:
        return StopEvent(result=f"you said: {ev.msg}")

# import asyncio; print(asyncio.run(Echo().run(msg="hi")))
draw_all_possible_flows(Echo, filename="echo.html")

Run it, then add a second step and a custom Event class between start and stop, and redraw the graph to see the engine wire the two steps together purely from their event type hints. That wiring-by-types is the idea the whole agent layer stands on.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is LlamaIndex, in one sentence?

A data framework for LLM applications whose core abstraction is the Node, a chunk of content with metadata and relationships, so that ingestion, indexing, retrieval, and answer synthesis are all operations that produce or consume lists of nodes.

2. What is a Node and how does it differ from a Document?

A Document is a whole source item, a file or record. A TextNode is a chunk of a document produced by a node parser. Both share a base type that gives a stable id, a metadata dict, an embedding slot, and typed relationships such as source, previous, next, and parent-child, and those relationships let retrieval reach a chunk's neighbors or its larger parent.

3. What are the pieces of a query engine, and why split them?

A retriever, an ordered list of node postprocessors, and a response synthesizer. Splitting them means you can rerank without touching retrieval, change the synthesis strategy without touching either, or swap a vector retriever for a hybrid one, because each stage has a stable node-in, node-or-answer-out contract.

4. Trace the life of one query().

The string is wrapped in a QueryBundle. The retriever embeds it and asks the vector store for the top-k most similar nodes, returning them as NodeWithScore. Node postprocessors cut low scores or rerank. The response synthesizer packs the surviving nodes into one or more prompts and calls the LLM, returning a Response with the answer and the source nodes.

5. When do you use a summary index instead of a vector index?

When the task is summarize-the-whole-thing rather than answer-a-specific-question. A summary index keeps all nodes and by default retrieves all of them for a tree_summarize synthesis, whereas a vector index filters to the few most similar nodes, which is wrong for a global summary and right for a local fact.

6. What does a keyword table index buy you over embeddings?

Exact-term matching. Embeddings blur rare identifiers, codes, and names together, so a keyword table that maps extracted keywords to node ids retrieves the node containing a precise term reliably where cosine similarity might not. The tradeoff is that it misses semantic paraphrases, which is why hybrid retrieval combines the two.

7. Explain the difference between compact and refine response modes.

refine makes one LLM call per node, each call updating a running answer with the next chunk of context. compact first packs as many node chunks as fit into each prompt and then refines across whatever prompts remain, so it makes far fewer calls for the same nodes. compact is the default because it is cheaper with essentially the same quality.

8. Why is inspecting source_nodes the first debugging move?

Because it separates a retrieval failure from a synthesis failure. If the source nodes do not contain the answer, the retriever is the problem and no prompt change will help. If they do contain it but the answer is wrong, the problem is synthesis or the model. Running with response_mode="no_text" gives you the nodes with no LLM call at all.

9. Why did old tutorials break, and what replaced ServiceContext?

The v0.10 refactor split the library into llama-index-core plus separate integration packages, so flat imports changed, and it replaced the per-call ServiceContext with the global Settings singleton, now the place to set the LLM, embedding model, and chunking defaults. The fix for old code is usually to drop the service_context= argument and set the matching Settings field once.

10. What is a Workflow, and how do steps connect?

An event-driven orchestration engine. You write @step methods that receive an event and return the next event, and the engine connects a step that emits an event to the step that consumes that event type, so control flow comes from event types rather than an explicit graph. It runs from a StartEvent to a StopEvent with a shared Context for state.

11. How does an agent reuse the RAG pipeline from this chapter?

Through a QueryEngineTool, which wraps any query engine as a callable tool. A FunctionAgent or ReActAgent loops, letting the model decide when to invoke that tool, so an agent that searches your documents is just a workflow whose step calls a query engine, which retrieves and synthesizes over nodes.

12. When would you choose LangChain, Haystack, DSPy, or nothing over LlamaIndex?

LangChain or LangGraph when the app is mostly orchestration and tool use with retrieval as one step among many, Haystack for a retrieval-first production pipeline, DSPy when you want to optimize a pipeline against a metric rather than wire one by hand, and a plain vector library like FAISS with your own glue when the corpus is small and you value a fully debuggable stack over breadth of connectors.

13. A RAG answer is confidently wrong. Name three suspects from this chapter.

Bad chunking that split the relevant fact across two nodes so neither is retrieved cleanly, too small a similarity_top_k or a missing reranker so the right node ranks just outside the cut, and a model or embedding mismatch between index time and query time. The source nodes and a no_text run tell you which one it is.

14. Why is the vector store an interface rather than baked in?

So the index code never changes when the backend does. The default SimpleVectorStore does brute-force cosine in memory for prototypes, and Chroma, Qdrant, Weaviate, pgvector, or FAISS implement the same VectorStore query contract for production scale, so moving from a laptop demo to a database is a constructor argument.

Part IX: Design lessons

Pick one abstraction and make everything speak it. By funneling ingestion, indexing, retrieval, and synthesis through the Node, LlamaIndex turns adding a feature into writing one transform or retriever rather than modifying a pipeline. A single well-chosen data type that every stage produces and consumes is the same instinct behind Unix's byte stream and a compiler's single intermediate representation.

Separate the two hard jobs. Retrieval and synthesis fail for different reasons and improve with different techniques, so the framework makes them different objects with a clean seam between them, the postprocessor stage. Whenever two concerns have different failure modes, giving each its own boundary lets you debug and optimize them independently, which is why the source-nodes trick even works.

Make the backend an interface from day one. The same vector-store contract covers an in-memory numpy search and a managed database, so prototypes and production share code and differ only in a constructor. Designing to the interface rather than the fast path is what lets a demo grow up without a rewrite, the same lesson as coding to a storage abstraction rather than to one database.

Configuration should be global default plus local override. Moving from a threaded ServiceContext to a Settings singleton with per-object overrides removed a parameter from nearly every constructor while keeping full control where it is needed. Sensible global defaults with a local escape hatch beat threading configuration through every call, a pattern shared by logging frameworks and dependency injection done well.

Let control flow emerge from types. The workflow engine wires steps by matching emitted and consumed event types instead of asking you to draw the graph, so the graph is derivable and drawable rather than hand-maintained and drift-prone. Deriving structure from declarations rather than duplicating it shows up in build systems that infer the dependency graph from rules and in reactive systems that connect producers to consumers by message type.

Own the fact that RAG fails silently. The framework's best affordance is not a clever index, it is response.source_nodes and the no_text mode, which make an invisible failure visible. Systems whose worst outcome is a plausible wrong answer earn their trust by making the evidence inspectable, the same reason good numerical code reports residuals and good search reports scores.

Part X: Memorization framework

The one-sentence summary. LlamaIndex loads sources into Documents, splits them into Nodes, indexes the nodes for a kind of lookup, and answers a query by retrieving scored nodes, postprocessing them, and synthesizing an answer within the context window, and the agent and workflow layer is just this pipeline wrapped as tools inside an event-driven loop.

read -> parse into nodes -> embed -> index (vector | summary | keyword)
  -> as_query_engine = retriever + postprocessors + response_synthesizer
  -> query: embed -> similarity search -> rerank/cut -> pack -> LLM -> Response
  -> wrap the query engine as a QueryEngineTool inside a Workflow/agent

The chain mapped to the codebase:

read              readers/ (SimpleDirectoryReader), LlamaHub packages
parse             node_parser/ (SentenceSplitter, SentenceWindow, Semantic)
ingest            ingestion/ (IngestionPipeline + cache)
index             indices/ (VectorStoreIndex, SummaryIndex, KeywordTableIndex)
store             storage/ (docstore, index_store) + vector_stores/
retrieve          retrievers/ (VectorIndexRetriever) -> [NodeWithScore]
postprocess       postprocessor/ (SimilarityPostprocessor, rerankers)
synthesize        response_synthesizers/ (Refine, TreeSummarize)
engine            query_engine/ (RetrieverQueryEngine, Router, SubQuestion)
orchestrate       workflow/ + agent/workflow/ (FunctionAgent, AgentWorkflow)

Memorize these blocks:

• The node: Document to TextNode via a node parser, each node carrying id, text, metadata, embedding, and relationships, and every stage speaks lists of nodes.
• Query engine equation: retriever plus ordered node postprocessors plus response synthesizer, run in that order on every query.
• Three indices: vector for semantic lookup, summary for whole-corpus summarization, keyword table for exact-term matching.
• Response modes: compact by default, refine per node, tree_summarize for corpora, no_text to debug retrieval.
• Config: global Settings replaced ServiceContext, and v0.10 split core from separately versioned integration packages.
• Top layer: Workflows are event-driven steps wired by event type, and agents are workflows that call query engines as tools.

Part XI: Papers and further reading

The ideas this framework packages come from a short list of papers, and each one rewards a direct read. Where this site derives the same idea in depth, the companion link points there.

1. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. Named and shaped the retrieve-then-generate pattern that every query engine in this chapter runs. The applied generative AI class on this site builds the same pipeline by hand.
2. Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, 2020. Showed that a learned dense embedding beats sparse lookup for passage retrieval, which is the bet behind VectorStoreIndex.
3. Reimers and Gurevych, Sentence-BERT, Sentence Embeddings using Siamese BERT-Networks, 2019. The sentence-embedding recipe behind most local embedding models you would plug into Settings.embed_model, covered in the sentence-transformers walkthrough.
4. Nogueira and Cho, Passage Re-ranking with BERT, 2019. The cross-encoder reranking idea that SentenceTransformerRerank applies at the postprocessor stage.
5. Malkov and Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs, 2016. The HNSW graph that most production vector stores run underneath the retriever, with the surrounding index structures covered in the databases material.
6. Johnson et al., Billion-scale Similarity Search with GPUs, 2017. The FAISS paper, and FAISS is one of the swappable vector stores here, covered in the FAISS walkthrough.
7. Liu et al., Lost in the Middle, How Language Models Use Long Contexts, 2023. The evidence that models neglect mid-context passages, which is why raising similarity_top_k can bury the relevant chunk instead of helping.
8. Yao et al., ReAct, Synergizing Reasoning and Acting in Language Models, 2022. The reason-act-observe loop that ReActAgent implements over query-engine tools.
9. Gao et al., Retrieval-Augmented Generation for Large Language Models, A Survey, 2023. A map of naive, advanced, and modular RAG that places every technique in this chapter. Evaluating these pipelines is the subject of the data pipelines and evaluation class.

Part XII: Final takeaway

If the retrieval math under the vector index is the gap, the databases material and the FAISS chapter derive the nearest-neighbor structures the store delegates to, and the applied generative AI material builds the routers, rerankers, and agents this framework packages. When you deploy, the model behind Settings.llm is usually served by something like vLLM, so LlamaIndex is the data-and-retrieval half of a stack whose other half is high-throughput inference. Then come back and read query_engine.query() once more. It will read like a for loop over nodes, which is the entire point.