GPT Researcher

Part I: The mental model

cli.py "query"  /  GPTResearcher(query)  /  server (uvicorn main:app)
      |
      v
GPTResearcher            agent.py: one orchestrator, many skills
      |
      v
choose_agent(query)      actions/agent_creator.py -> a role and an agent prompt
      |
      v
plan_research()          skills/researcher.py: STRATEGIC_LLM writes N sub-queries
      |
      v
asyncio.gather(...)      one executor coroutine per sub-query, all in flight
      |                  each: retrieve -> scrape -> compress
      v
retrievers               actions/retriever.py factory (tavily, google, ...)
      |
      v
scrapers                 scraper/ (bs4, browser, pdf, tavily_extract, ...)
      |
      v
ContextManager           embedding similarity, top-k passages, dedup by visited_urls
      |
      v
SourceCurator (optional) an LLM re-ranks the surviving sources
      |
      v
generate_report()        SMART_LLM streams one long cited markdown report

The one-sentence identity: GPT Researcher turns one question into a cited report by separating a planner that writes sub-questions from executors that answer them, fanning the executors out across web retrievers and scrapers in parallel, compressing the scraped pages down to the passages that matter, and letting a single writer model synthesize the aggregate into prose with inline sources. The design is deliberately not a general autonomous agent that loops on a scratchpad until it decides it is done. It is a fixed, legible pipeline with one planning step and one writing step bracketing a wide parallel research phase, and almost everything interesting happens in that middle band.

Two ideas carry the whole system. The first is the planner and executor split. A reasoning-oriented model reads your question and emits a short list of sub-queries, each a concrete web search that attacks one facet of the question. Those sub-queries are then researched independently and concurrently, which is what makes the agent both fast and broad, and only at the end does a second model see the pooled findings and write. The second idea is everything external is pluggable behind a tiny factory. A retriever is just a class that maps a query string to result URLs, a scraper is just a class that maps a URL to text, and the orchestrator picks both by name from configuration. Swapping Tavily for Google, or BeautifulSoup for a headless browser, or OpenAI for a local model, is a config change, not a code change.

A third property is worth naming because it is why the repo reads so well. The orchestrator owns no logic of its own. The GPTResearcher object in agent.py is mostly a bag of state plus a set of skill objects, and each skill is a small class with one job, planning, browsing, managing context, curating sources, writing. To understand the agent you read seven or eight short files, not one giant loop. Everything below is verified against the master branch in mid 2026. The project moves quickly and supports many LLM providers, so where a default model name or an exact path is likely to drift I say so and stay at the level of the role a component plays.

Part II: Using it

GPT Researcher runs anywhere Python 3.11 or newer runs, including macOS, since the heavy lifting is web requests and LLM API calls rather than local GPUs. The one hard requirement is API keys, an LLM provider and a search provider. The defaults expect an OpenAI key and a Tavily key, though both are swappable. Install from pip or from source:

pip install gpt-researcher

# or from source, which also gives you the server and frontend
git clone https://github.com/assafelovic/gpt-researcher.git
cd gpt-researcher
pip install -r requirements.txt

export OPENAI_API_KEY=sk-...
export TAVILY_API_KEY=tvly-...

The fastest way to see it work is the command-line entry point at the repo root. It takes the query as a positional argument and a report type, writes the report to disk, and by default also exports PDF and DOCX:

python cli.py "What is the state of small language models in 2026?" \
  --report_type research_report

# a longer report that researches subtopics and concatenates them
python cli.py "Compare RAG and long-context approaches" \
  --report_type detailed_report --tone analytical

The report types are a real enumeration and worth knowing. research_report is the default, a summary of roughly a couple of minutes. detailed_report plans subtopics and researches each as its own report, then merges them, so it runs longer and produces more words. resource_report and outline_report produce a curated source list and an outline respectively, subtopic_report is the unit the detailed pipeline composes, and deep_research triggers the recursive tree-shaped exploration described later. --report_source selects where evidence comes from, web by default, or local and hybrid to research your own documents.

For programmatic use the pip package exposes one class. The API is two async calls, research first, then write, which mirrors the planner and executor split exactly:

import asyncio
from gpt_researcher import GPTResearcher

async def main():
    researcher = GPTResearcher(
        query="Why is Nvidia's data-center revenue growing?",
        report_type="research_report",
        report_format="APA",     # citation style for the writer
        tone="objective",
    )
    # phase 1: plan sub-queries, retrieve, scrape, aggregate context
    context = await researcher.conduct_research()
    # phase 2: synthesize the pooled context into a cited report
    report = await researcher.write_report()

    print(report)
    print(researcher.get_research_sources())   # titles, urls, snippets
    print(researcher.get_costs())              # token spend for the run

asyncio.run(main())

The split between conduct_research and write_report is not cosmetic. After research returns you can inspect the gathered context, sources, and images, feed your own custom prompt into the writer, or call helpers like write_report_conclusion and get_subtopics. The context is the durable artifact, the report is one rendering of it.

To run the web application from a source checkout, start the FastAPI server, which streams the report token by token over a WebSocket to a small frontend. It is built on the same stack as my FastAPI notes:

python -m uvicorn main:app --reload
# then open http://localhost:8000

Now the configuration you will actually reach for. GPT Researcher is model-agnostic, and it names models with a provider:model string. There are three distinct LLM roles, and they exist because the three jobs have different needs:

# the planner and other reasoning-heavy steps
export STRATEGIC_LLM="openai:o4-mini"
# the writer, which composes the final long report
export SMART_LLM="openai:gpt-4.1"
# cheap high-volume calls, summarization and small utilities
export FAST_LLM="openai:gpt-4o-mini"
# how scraped pages are ranked for relevance
export EMBEDDING="openai:text-embedding-3-small"
# which search engine the retriever uses
export RETRIEVER="tavily"

Those default names change over time as better models ship, so treat the exact strings above as a snapshot and the three roles as the stable idea. Because the provider prefix is honored by the LLM layer, pointing the whole agent at a local model is a one-line change, for example SMART_LLM="ollama:llama3.1" against a local Ollama server, or an OpenAI-compatible endpoint served by vLLM. The retriever and embedding provider are equally swappable.

The mistakes newcomers make are mostly about keys and providers. First, forgetting the search key. An LLM key alone gets you a planner and a writer with nothing to research, and the run fails when the retriever cannot authenticate. Second, assuming research_report reads dozens of pages. It aims for a focused summary from a handful of sub-queries, and the way to get depth is detailed_report or the deep research mode, not a bigger prompt. Third, expecting conduct_research to return a string. It returns the aggregated context and stores state on the researcher, and the readable report only comes from write_report. Fourth, running blocking code, the API is async throughout because the entire point is to have many sub-queries and many page fetches in flight at once.

Part III: When it is the right tool

GPT Researcher is the right tool when you want a report, a structured, cited, multi-paragraph answer to a real question, produced automatically from live web sources. That covers market and competitive research, literature scans, due diligence, and "explain the current state of X" prompts, and it covers research over your own document sets through the local and hybrid sources. It is also, frankly, one of the best codebases to read if you want to learn how a production research agent is actually wired, because the architecture is small and honest.

The honest cases for alternatives. Stanford's STORM attacks a similar goal, generating grounded long-form articles, but frames the work as simulated multi-perspective question asking and outline building, and reads more like a paper artifact than a deployable service. LangChain's Open Deep Research and Hugging Face's smolagents-based open Deep Research are close cousins that lean on those teams' own agent frameworks, so they fit best if you are already committed to that stack. General autonomous agents like AutoGPT or BabyAGI will also research and write, but they are open-ended loops without the fixed research pipeline, so they are less predictable and harder to make reliably cite. And the commercial products, Perplexity, OpenAI's and Google's deep research features, are polished and closed, so you reach for GPT Researcher precisely when you want the pipeline open, forkable, local-model-capable, and yours.

It is worth being clear about what GPT Researcher is not. It is not a general agent framework in the sense of LangGraph or AutoGen, where you assemble arbitrary graphs of tools and actors. The core agent is a purpose-built pipeline. Interestingly, the repository also ships a multi_agents package that is built on LangGraph, an editorial team of a chief editor, researchers, reviewers, revisers, a writer, and a publisher, for when you want the heavier, review-loop workflow. So the project spans both, a tight single-agent core and an optional multi-agent orchestration on top, and choosing between them is really choosing how much process you want around the same underlying research skill.

One architecture-shaped warning about cost and rate limits. The whole value proposition is fan-out, many sub-queries, each scraping several pages, each page summarized or embedded. That multiplies token spend and outbound requests fast, and the failure mode at scale is not a crash but a slow, expensive run that trips a provider rate limit mid-flight. The knobs that govern the blast radius, the number of sub-queries, results per query, and scraper workers, are the ones to understand before pointing this at a large batch of questions, and they are exactly the config values examined in Part V.

Part IV: The full life of one research task

The specimen: one research_report for the question "Why is Nvidia's data-center revenue growing?", run against the default web source. I follow it from the entry point through the planner, the parallel executors, retrieval, scraping, aggregation, and the writer, naming the module that owns each stage.

Stage 1: the entry point and the orchestrator

Whether you enter through cli.py, the pip package, or the server, everything converges on constructing one GPTResearcher in gpt_researcher/agent.py and awaiting conduct_research(). The constructor is wide, it accepts the query, report type and format, tone, source selection, optional source_urls and document_urls, a vector store, and MCP settings, but it does almost nothing heavy. Its real job is to build the skill objects the orchestrator delegates to, roughly a ResearchConductor, a ReportGenerator, a ContextManager, a BrowserManager, a SourceCurator, a Memory for embeddings, and optionally an ImageGenerator and a DeepResearchSkill. It also loads a Config and initializes bookkeeping, notably a visited_urls set that will enforce deduplication for the entire run.

Stage 2: choosing an agent role

The first thing conduct_research does, if you did not supply a role, is decide who should be answering. It calls into actions/agent_creator.py, which asks the LLM to pick an appropriate expert persona for the query and to write the matching system prompt. For our Nvidia question it might return a financial-analyst agent with a role prompt about objective market analysis. This is a small but load-bearing move, the same question answered as a "financial analyst" versus a "hardware engineer" pulls different sub-queries and a different report. The chosen role prompt rides along as the system message for the rest of the run.

Stage 3: the planner writes sub-queries

Now the planner half of the split. The ResearchConductor in skills/researcher.py calls its plan_research, which delegates to plan_research_outline and generate_sub_queries in actions/query_processing.py. This step uses the strategic LLM, the reasoning model, and it is fed the original query, the report type, and, importantly, a quick preliminary search over the original question so the planner has some real context about what the web actually contains before it decides how to slice the problem. The model returns a short JSON list of sub-queries, defaulting to a small number rather than dozens, each a standalone search string. For our question the plan might be data-center GPU demand, hyperscaler capex trends, Nvidia quarterly segment results, and competitive position versus AMD. The function is defensive here, it runs the response through JSON repair and falls back across models if the strategic call errors, because a malformed plan would sink the whole run.

Stage 4: executors fan out in parallel

This is the heart of the design. With sub-queries in hand, the conductor launches one executor coroutine per sub-query and awaits them together:

# the shape of the executor fan-out, paraphrased from skills/researcher.py
context = await asyncio.gather(*[
    self._process_sub_query(sub_query, scraped_data)
    for sub_query in sub_queries
])

Because each _process_sub_query is fully independent, searching, scraping, and compressing its own slice, the wall-clock time of the research phase is roughly the time of the slowest single sub-query rather than the sum. This is ordinary Python concurrency doing exactly what it is good at, overlapping I O-bound waits, the same kind of structured fan-out discussed in my parallel computing notes, just applied to network calls instead of GPU kernels. Each executor's job has three steps, retrieve, scrape, compress, which are Stages 5 through 7.

Stage 5: retrieval, the factory picks a search engine

Inside an executor, _search_relevant_source_urls asks the configured retrievers for URLs. Retriever selection is a clean factory in actions/retriever.py. get_retrievers reads the config, headers, or defaults, and get_retriever maps a name string like "tavily" or "google" to the matching class in gpt_researcher/retrievers/. There are many, Tavily, Google, Bing, DuckDuckGo, SearX, Serper, SerpAPI, SearchAPI, plus academic sources like arXiv, PubMed Central, and Semantic Scholar, and an MCP retriever for custom data sources. Each is a small class that returns a list of results with URLs. The executor collects those URLs, filters out any already in the run's visited_urls set, and caps the count with MAX_SEARCH_RESULTS_PER_QUERY. Deduplication starts here, a URL already fetched for another sub-query is never fetched twice.

Stage 6: scraping, URLs become text

The surviving URLs go to the BrowserManager in skills/browser.py, whose browse_urls calls into the scraper subsystem in gpt_researcher/scraper/. Scraping runs across a worker pool bounded by MAX_SCRAPER_WORKERS, and the concrete scraper is chosen by config, bs for the default BeautifulSoup path, or a headless browser for JavaScript-heavy pages, plus specialized extractors for PDFs through PyMuPDF, arXiv, Firecrawl, and Tavily's own extract endpoint. The manager returns a list of content dictionaries, one per page, holding the raw text and any images. It also collects images and keeps the top few by relevance, deduplicating them by a content hash so the same figure scraped from two sites is not embedded twice. Successfully scraped URLs are recorded, both as research sources on the researcher and into the visited_urls set.

Stage 7: compression, keeping only what matters

A scraped page is mostly noise for any one sub-query, navigation, boilerplate, unrelated sections. The ContextManager in skills/context_manager.py fixes that. Its get_similar_content_by_query takes the sub-query and the scraped pages, splits the pages into chunks, embeds both the chunks and the query using the configured embedding model held by the researcher's Memory, and keeps only the chunks whose similarity clears a threshold, returning the top matches. This is nearest-neighbor retrieval over embeddings, the same idea behind a vector index like FAISS, applied at the granularity of one research page set. The similarity threshold and result cap are config values, so each executor hands back a tight, on-topic block of text rather than a pile of raw HTML. When you research your own documents instead, the sibling method routes the same query through a real vector store.

Stage 8: aggregation and optional curation

asyncio.gather returns the per-sub-query context blocks, and the conductor concatenates them into one aggregated context, which is what conduct_research returns and stores on the researcher. If source curation is enabled, the SourceCurator in skills/curator.py gets a pass first, its curate_sources hands the collected sources to an LLM with a curation prompt and a low temperature, asking it to rank them by relevance and reliability and keep the best, falling back to the unranked set if the model misbehaves. Aggregation plus deduplication plus optional curation is how a dozen overlapping pages across four sub-queries become one clean, non-redundant evidence pool.

Stage 9: the writer synthesizes a cited report

Finally the executor half yields to the writer. write_report calls the ReportGenerator, which calls generate_report in actions/report_generation.py. This selects a prompt template by report type, builds a single large user message from the aggregated context, the query, the report format, and the agent role, and calls create_chat_completion with the smart LLM, a moderate temperature, and streaming enabled so tokens flow to the CLI or WebSocket as they are produced. The prompt instructs the model to write in the requested citation style and to ground claims in the provided sources with inline links, which is why the output reads as a cited report rather than a freeform essay. For a detailed report this whole pipeline runs once per subtopic and the pieces are concatenated with an introduction and conclusion. That closes the loop of one task, question in, a plan, parallel research across the web, compressed evidence, a written report with sources out.

Part V: Internals deep dives

Deep dive: the orchestrator and its skills

The cleanest thing about this codebase is the separation between the orchestrator and the skills. GPTResearcher in agent.py is a facade. It holds run state, the query, config, context, sources, images, costs, and the visited_urls set, and it holds one instance of each skill. Its public methods are thin, conduct_research calls the conductor, write_report calls the report generator, get_research_sources reads a list. The skills in gpt_researcher/skills/ are where behavior lives, and each is a small class named for a verb or role, researcher.py plans and conducts, browser.py scrapes, context_manager.py compresses, curator.py ranks, writer.py composes, deep_research.py recurses, image_generator.py makes figures.

Beneath the skills sits gpt_researcher/actions/, a layer of stateless functions the skills call, query planning in query_processing.py, retriever selection in retriever.py, scraping helpers in web_scraping.py, agent selection in agent_creator.py, and report composition in report_generation.py. The pattern is a facade over stateful skills over stateless actions, and it is why you can hold the whole agent in your head, each layer is small and only knows about the one below it. If you ever wonder where a behavior lives, the verb tells you, a skill for the workflow step, an action for the pure function it delegates to.

Deep dive: the planner and executor split, and its parallelism

The planner and executor split is the load-bearing idea, so it deserves a careful look at why it is shaped this way. The planner is a single reasoning call that converts an open question into a small set of closed search queries. Making it a reasoning model on the STRATEGIC_LLM role is deliberate, decomposing a question well is the step where a stronger model pays off most, and it runs only once, so its cost is bounded. The executors are the opposite, many cheap, uniform, I O-bound coroutines, and their number is data-dependent, one per sub-query. Keeping them independent is what lets asyncio.gather run them concurrently with no coordination, and it is why adding a sub-query adds breadth without adding much wall-clock time.

The controls on the fan-out are config values worth memorizing, because they set both quality and cost. Their defaults as of this writing:

Read that table as a budget. Sub-queries times results per query times pages, bounded by the worker pool, is the outbound request count and roughly the token spend, and the similarity threshold is what keeps the writer's prompt from ballooning with off-topic text. The whole system's cost and quality live in these few numbers, not in the prompts. A detailed report multiplies the picture by subtopics, which is why it is minutes rather than seconds. The deep research mode generalizes the fan-out into a tree, each sub-query can spawn its own sub-queries down to a configured depth and breadth, trading a lot more cost for much wider coverage, and it lives in skills/deep_research.py so the core pipeline stays simple.

Deep dive: retrievers and scrapers as pluggable factories

Retrieval and scraping are the two places the agent touches the outside world, and both are behind the same simple factory pattern so neither leaks into the orchestrator. A retriever is a class exposing a search that returns result URLs for a query, and actions/retriever.py resolves a config name to that class. The list is long on purpose, general web engines like Tavily, Google, Bing, DuckDuckGo, and SearX, API services like Serper, SerpAPI, and SearchAPI, and domain retrievers like arXiv, PubMed Central, and Semantic Scholar for academic work. You can also pass a comma-separated list of retrievers and the agent will query several, or restrict searches to specific domains with query_domains.

Scrapers mirror this. A scraper turns a URL into text, and the config SCRAPER chooses which, defaulting to bs for BeautifulSoup. The catalog in gpt_researcher/scraper/ covers the cases the web actually throws at you, static HTML with BeautifulSoup, JavaScript pages with a real browser, PDFs with PyMuPDF, arXiv, a Firecrawl integration, and Tavily's extract endpoint. scraper.py dispatches to the right one and runs them across the bounded worker pool. The pattern to notice is that neither the conductor nor the browser manager knows or cares which concrete retriever or scraper is active. The interface is a query in, results out, and a URL in, text out, and that narrowness is exactly what makes the catalog extensible. Adding a new search backend or a new scraper is writing one class and registering its name, which is why the community has contributed so many.

Deep dive: source aggregation, deduplication, and compression

The research phase produces a lot of overlapping, redundant, noisy text, and three mechanisms turn it into a clean pool. First, URL deduplication through the run-wide visited_urls set, a URL retrieved for one sub-query is filtered out of every later sub-query and never scraped twice, so overlap between related sub-queries costs nothing. Second, relevance compression in the ContextManager, each page set is chunked, embedded, and filtered against the sub-query by cosine similarity with the SIMILARITY_THRESHOLD cutoff, so only passages that actually bear on the sub-query survive into the context. Third, image deduplication by content hash in the browser manager, so identical figures collapse to one.

The compression step is the subtle one, and it is why the writer can be handed the pooled evidence in a single prompt without overflowing the context window. Consider the arithmetic, four sub-queries, five URLs each, is up to twenty pages, and a page can be thousands of tokens. Concatenated raw that is far too much, and most of it is irrelevant to any specific claim. By embedding each chunk and keeping only near neighbors of the sub-query, the manager shrinks twenty pages to a few hundred relevant lines. This is retrieval-augmented generation applied inside the agent, the same embed-and-nearest-neighbor mechanism you would build over FAISS, run per sub-query rather than over a static index. The correctness trap to know, the threshold is a blunt instrument, set it too high and genuinely relevant but differently-worded passages get dropped, set it too low and the writer drowns in noise, and it interacts with the embedding model you choose.

Deep dive: three LLM roles and config as environment

The three-model design is easy to miss and central to both cost and quality. STRATEGIC_LLM is the planner, reasoning over the question to produce sub-queries and, in detailed reports, subtopics. SMART_LLM is the writer, a strong general model that sees the pooled context and composes the long cited report. FAST_LLM is the workhorse for high-volume small jobs like summarizing a scraped page or the many utility calls, where a cheap fast model is the right trade. Matching model strength to job is the same instinct as matching parallelism to interconnect, spend the capable model where a task is hard and rare, spend the cheap model where a task is easy and frequent. A run that feels expensive is usually a SMART_LLM or STRATEGIC_LLM pointed at a top-tier model doing work a smaller one would handle.

Configuration itself is layered and worth understanding because it is how you retune the agent without touching code. Defaults live in the config package under gpt_researcher/config/, every one of them is an environment variable, and a JSON config file or constructor arguments can override them per run. The provider-prefixed model strings mean the LLM layer, not the agent, resolves which SDK to call, so anthropic:, google:, ollama:, and OpenAI-compatible endpoints all work through the same interface. This is why running the entire agent against local models, an Ollama server or a vLLM endpoint, changes nothing about the pipeline, only the strings. The design lesson is that pushing all of provider, model, retriever, scraper, and thresholds into configuration is what lets one small pipeline serve wildly different deployments.

Deep dive: the multi-agent layer on LangGraph

The multi_agents/ package is a second, heavier way to use the same research skill, and it is instructive to see the contrast. Where the core agent is a fixed pipeline, this is an editorial team modeled as a LangGraph state machine, a chief editor that plans and orchestrates, a researcher that is essentially the core GPTResearcher, an editor that outlines, and then per-subtopic loops of reviewer and reviser that critique and rewrite drafts, before a writer assembles the whole and a publisher exports to PDF, DOCX, and Markdown. A task.json configures the query, models, sections, and guidelines, and langgraph.json wires the deployment. The reason to reach for it is when you want an explicit review-and-revise loop around the research, at the cost of more calls and more latency. The reason it stays a separate package is that the core pipeline should not pay for that machinery when you do not want it, the two layers share the research skill but not the orchestration.

Part VI: Reading the repository

The repository is large because it ships a package, a server, a frontend, a multi-agent layer, docs, and infrastructure, but the part that matters for understanding the agent is small and reads in an afternoon. Paths below are current on master in mid 2026 and may shift.

Stage 0, orientation. Read the top-level README.md and the getting-started docs, then skim cli.py and main.py at the repo root. Questions to hold, what are the three ways in, what does cli.py pass to the agent, and what does the server stream over its WebSocket?

Stage 1, the orchestrator. Read gpt_researcher/agent.py top to bottom. It is a facade, so read it as an index, which skills exist, what state the run holds, and which method calls which skill. Questions, what does the constructor actually build, why are there separate conduct_research and write_report phases, and what is visited_urls for?

Stage 2, the planner and executor. Read skills/researcher.py, the ResearchConductor, with actions/query_processing.py open beside it. This is the core loop. Find plan_research, then the asyncio.gather over _process_sub_query, then follow one sub-query through search, scrape, and compress. Questions, why does the planner run a preliminary search first, what makes the executors safe to run concurrently, and where does a sub-query's context get produced?

Stage 3, the outside world. Read actions/retriever.py for the retriever factory, then a couple of concrete retrievers in gpt_researcher/retrievers/, then skills/browser.py and gpt_researcher/scraper/scraper.py. Questions, how is a config string turned into a retriever class, where is MAX_SEARCH_RESULTS_PER_QUERY applied, and which scraper handles a JavaScript page versus a PDF?

Stage 4, aggregation and writing. Read skills/context_manager.py for compression, skills/curator.py for optional ranking, and actions/report_generation.py with skills/writer.py for composition. Questions, how does the context manager decide what to keep, what does the curator add, and how does the writer's prompt produce inline citations?

Stage 5, configuration and the frontier. Read the gpt_researcher/config/ package to see how defaults, environment, and file config compose, then the multi_agents/ LangGraph team and skills/deep_research.py. Questions, how does a provider-prefixed model string get routed, and how does the recursive deep research generalize the flat fan-out into a tree?

Where not to start, the frontend/, backend/ server internals, terraform/, docker-compose.yml, and the MCP server are important for deployment but tell you nothing about how research works, and multi_agents/ and deep_agents/ only make sense after the single-agent core is solid.

Part VII: Hands-on labs

All labs need only a machine with Python, an OpenAI key, and a Tavily key, no GPU. Output formats and log lines vary with the fast pace of master.

Lab 1: a first report and its artifacts. Concept: the pipeline of Part IV end to end.

python cli.py "What changed in open-source LLM training in 2026?" \
  --report_type research_report

Watch the console. You should see the chosen agent role, the planned sub-queries, a burst of scraping, and then the report streaming in. Open the written output and note the inline citations. Match each console phase to a stage in Part IV, agent selection, planning, the parallel research, and the writer.

Lab 2: watch the fan-out from code. Concept: the planner and executor split.

import asyncio, logging
from gpt_researcher import GPTResearcher

logging.basicConfig(level=logging.INFO)

async def main():
    r = GPTResearcher(query="How do vector databases index embeddings?",
                      report_type="research_report", verbose=True)
    await r.conduct_research()
    print("sub-topics:", await r.get_subtopics())
    print("sources:", len(r.get_research_sources()))
    print("context chars:", len(str(r.get_research_context())))
    print("costs:", r.get_costs())
    print(await r.write_report())

asyncio.run(main())

With verbose=True the sub-queries and per-source progress print as they happen. Note that the sources count is larger than the sub-query count, several URLs per sub-query, and that the context is far smaller than the raw pages would be, because compression already ran. That gap is Stage 7 made visible.

Lab 3: change the fan-out budget. Concept: the config knobs that set cost and breadth.

# widen retrieval, then compare runtime, source count, and cost
export MAX_SEARCH_RESULTS_PER_QUERY=8
export MAX_ITERATIONS=5
python cli.py "State of humanoid robotics startups" --report_type research_report

Run once with defaults and once with the values above, and compare the number of sources, the wall-clock time, and get_costs. You are directly feeling the budget from the Part V table. Then set MAX_SEARCH_RESULTS_PER_QUERY=1 and watch the report get thinner.

Lab 4: swap the retriever and the writer. Concept: everything external is pluggable.

export RETRIEVER="duckduckgo"        # no Tavily key needed for this one
export SMART_LLM="openai:gpt-4o-mini"  # cheaper writer
python cli.py "Recent advances in state-space models" --report_type research_report

The pipeline is unchanged, only the search backend and the writer model differ. If you have a local model, try SMART_LLM="ollama:llama3.1" and confirm the same run works against an Ollama server, proving the provider string is the only thing that moved.

Lab 5: research your own documents. Concept: the local and hybrid source, and the vector path in the context manager.

# point at a folder of PDFs/txt/csv and research over them
export DOC_PATH=./my-docs
python cli.py "Summarize the key risks in these filings" \
  --report_type research_report --report_source local

Now the retriever and scraper are bypassed and the context manager embeds and searches your documents instead of web pages. Switch to --report_source hybrid and watch it combine your documents with fresh web results, which is the same aggregation step fed from two sources.

Lab 6: detailed versus deep. Concept: subtopic composition and recursive research.

python cli.py "Compare the major open-source vector databases" \
  --report_type detailed_report

Observe that a detailed report first plans subtopics, then runs the whole Part IV pipeline once per subtopic, and concatenates them with an introduction and conclusion, so it is noticeably longer and slower. Compare against --report_type deep_research and note the recursive, tree-shaped exploration, and its larger cost in get_costs. The arithmetic of breadth times depth is the lesson.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is GPT Researcher, in one sentence?

An autonomous research agent that splits a planner, which turns a question into sub-queries, from executors, which research those sub-queries in parallel over pluggable web retrievers and scrapers, then aggregates and compresses the findings and has a writer model synthesize a single cited report.

2. What is the planner and executor split, and why does it matter?

The planner is one reasoning call that decomposes the question into a small set of concrete search queries. The executors are many independent coroutines, one per sub-query, each retrieving, scraping, and compressing its slice. It matters because decomposition is where a strong model pays off and it runs once, while research is I O-bound and embarrassingly parallel, so separating them gives both quality and speed.

3. How are the sub-queries researched concurrently?

The conductor builds one _process_sub_query coroutine per sub-query and awaits them together with asyncio.gather. Because each is independent, the research phase takes roughly the time of the slowest sub-query rather than the sum, and adding a sub-query adds breadth without adding much latency.

4. Why are there three LLM roles?

Because the three jobs differ. The strategic model plans, a hard, rare, reasoning task worth a strong model. The smart model writes the final report, which needs a capable general model. The fast model handles high-volume small jobs like summarization where a cheap model is the right trade. Matching model strength to job is the main cost lever.

5. How does the agent avoid drowning the writer in text?

The context manager chunks each scraped page set, embeds the chunks and the sub-query, and keeps only chunks whose similarity clears a threshold, returning the top matches. So twenty raw pages become a few hundred relevant lines per sub-query, and the pooled context fits in one writer prompt. It is retrieval-augmented generation run inside the agent per sub-query.

6. Where does deduplication happen?

In three places. URLs are deduplicated against a run-wide visited_urls set, so a page fetched for one sub-query is never fetched again. Relevance compression drops off-topic chunks. And images are deduplicated by content hash. The URL set is the important one, it makes overlapping sub-queries cheap.

7. How are retrievers and scrapers chosen?

By a factory. A config name like tavily or google is mapped to a retriever class in actions/retriever.py, and the SCRAPER config picks a scraper class. Each retriever maps a query to URLs, each scraper maps a URL to text, and the orchestrator never knows which concrete one is active, which is what makes both catalogs extensible.

8. What is the difference between a research report and a detailed report?

A research report runs the pipeline once and summarizes a handful of sub-queries into a focused report. A detailed report first plans subtopics, runs the full pipeline once per subtopic, and concatenates the subtopic reports with an introduction and conclusion, so it is longer and slower. Deep research goes further, recursing the fan-out into a tree.

9. Which config values govern cost, and how?

Sub-query count, results per query, and scraper workers set the volume of searches, fetches, and summaries, and therefore token spend and outbound requests. The similarity threshold bounds how much text reaches the writer. Total words and subtopic count size the writing. The system's cost lives in these numbers, not in the prompts.

10. How does the writer produce citations?

generate_report builds a prompt from the aggregated context, the query, and the requested report format, and instructs the smart model to ground claims in the provided sources with inline links in that citation style. The sources are present in the context because scraping recorded them, so the model can attribute each claim to a URL rather than inventing references.

11. When would you use the multi-agent layer instead of the core agent?

When you want an explicit review-and-revise loop around the research. The multi_agents LangGraph team adds a chief editor, reviewers, and revisers around the same research skill, producing a more edited report at the cost of more model calls and latency. The core agent is the right default, the team is for when process matters more than speed.

12. How do you run the whole agent against a local model?

Change the provider prefix on the model config, for example SMART_LLM="ollama:llama3.1" or an OpenAI-compatible endpoint served by vLLM. The LLM layer resolves the SDK from the prefix, so the pipeline is untouched and only the strings change. The retriever and embedding provider are swappable the same way.

13. Why is the API split into conduct_research and write_report?

Because the aggregated context is the durable artifact and the report is one rendering of it. Splitting the phases lets you inspect the context, sources, and images after research, supply a custom writer prompt, or generate several report shapes from one research pass without paying to research again.

14. A run is slow and expensive with no error. Name three suspects from this chapter.

A strong model on the smart or strategic role doing work a cheaper one would handle, a widened fan-out, high results per query or many sub-queries multiplying scrapes and summaries, or a report type like detailed or deep that multiplies the pipeline by subtopics or recursion. The cost knobs and the model roles distinguish them.

Part IX: Design lessons

Separate planning from doing. One reasoning call decomposes the problem, many uniform workers execute the pieces. This is the same shape as a query planner over an execution engine, or a build system that plans a graph then runs it in parallel. Deciding what to do and doing it have different cost profiles, so give them different machinery.

Make the wide phase embarrassingly parallel. The executors share no state and communicate through nothing but their returned context, so asyncio.gather parallelizes them for free. Whenever the expensive phase of a system is I O-bound and independent, the win is designing the units to need no coordination, then concurrency is a one-liner.

Put the outside world behind a two-method interface. A retriever is query in, URLs out, a scraper is URL in, text out, and both are chosen by name. That narrowness is what let the community contribute a dozen search backends and half a dozen scrapers without touching the core. Narrow interfaces at the system boundary are what make a project extensible.

Match model strength to task frequency. A strong model where the task is hard and rare, a cheap model where it is easy and frequent. Three named roles instead of one model make that trade explicit and tunable, and they are the main reason two runs of the same pipeline can differ tenfold in cost.

Push every choice into configuration. Provider, model, retriever, scraper, thresholds, and word budgets are all environment or file config, resolved at run time. That is why one small pipeline serves web research, private-document research, cloud and local models, and many search engines, without forks. The invariant is the pipeline, everything else is data.

Keep the orchestrator logic-free. The agent object is a facade over skills over stateless actions, so understanding it is reading a handful of small files rather than one large loop. A readable agent is one where each layer is small and only knows the layer below, and this repo is a good argument that clarity and capability are not at odds.

Part X: Memorization framework

The one-sentence summary: GPT Researcher chooses an expert role, has a strategic model plan a few sub-queries, runs one executor per sub-query in parallel to retrieve and scrape and compress the web, deduplicates and aggregates the findings into a compact context, and has a smart model stream a cited report from it.

query -> GPTResearcher (agent.py)
  -> choose_agent -> role + agent prompt
  -> plan_research: STRATEGIC_LLM -> sub-queries
  -> asyncio.gather over sub-queries:
       retrieve (factory) -> scrape (workers) -> compress (embeddings)
  -> dedup by visited_urls -> aggregate context -> curate (optional)
  -> generate_report: SMART_LLM streams a cited report

The chain mapped to source:

entry             cli.py, main.py, GPTResearcher(agent.py)
role              actions/agent_creator.py
plan              skills/researcher.py + actions/query_processing.py
executors         skills/researcher.py (_process_sub_query, asyncio.gather)
retrieve          actions/retriever.py -> gpt_researcher/retrievers/
scrape            skills/browser.py -> gpt_researcher/scraper/
compress          skills/context_manager.py (embeddings, similarity)
curate            skills/curator.py
write             skills/writer.py -> actions/report_generation.py
config            gpt_researcher/config/ (three LLM roles, thresholds)

Memorize these blocks:

• The split: one strategic planning call writes sub-queries, many parallel executors answer them, one smart call writes the report.
• The three roles: STRATEGIC_LLM plans, SMART_LLM writes, FAST_LLM does high-volume small jobs, each a provider:model string.
• The factories: a retriever maps query to URLs, a scraper maps URL to text, both chosen by config name, both a long extensible catalog.
• Dedup and compress: visited_urls kills repeat fetches, embedding similarity above SIMILARITY_THRESHOLD keeps only on-topic chunks, image hashes kill duplicate figures.
• The cost knobs: MAX_ITERATIONS (sub-queries), MAX_SEARCH_RESULTS_PER_QUERY, MAX_SCRAPER_WORKERS, MAX_SUBTOPICS, TOTAL_WORDS govern breadth, latency, and spend.

Part XI: Papers and further reading

The ideas this agent is built from live in a short list of papers, and each one rewards a direct read. Where this site develops the same idea in depth, the companion link points there.

1. Wang et al., Plan-and-Solve Prompting, Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models, 2023. The clearest statement of the plan-then-execute pattern, one call that devises the sub-tasks and separate calls that carry them out, which is the shape of this agent's core loop.
2. Yao et al., ReAct, Synergizing Reasoning and Acting in Language Models, 2022. The interleaved reason-and-act loop that open-ended agents like AutoGPT run, the design GPT Researcher deliberately trades away for a fixed pipeline. The broader family of agent patterns is covered in the applied generative AI class on this site.
3. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. The retrieve-then-generate idea the context manager applies per sub-query, run over freshly scraped pages instead of a static index.
4. Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, 2020. The dense-embedding retrieval that compression is built on, and the nearest-neighbor machinery behind it is worked through in the FAISS walkthrough.
5. Nakano et al., WebGPT, Browser-assisted question-answering with human feedback, 2021. The ancestor of web research agents, a model taught to search, browse, and quote its sources.
6. Shao et al., Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models, 2024. The STORM system, the closest research cousin, which grounds long-form writing in simulated multi-perspective question asking.
7. Shinn et al., Reflexion, Language Agents with Verbal Reinforcement Learning, 2023. The self-reflection line behind the reviewer and reviser loop in the multi-agent layer, surveyed alongside its relatives in the self-improving agents class.
8. Madaan et al., Self-Refine, Iterative Refinement with Self-Feedback, 2023. The same critique-and-rewrite idea in single-model form, a model improving its own draft from its own feedback.
9. Wu et al., AutoGen, Enabling Next-Gen LLM Applications via Multi-Agent Conversation, 2023. The multi-agent conversation framing the editorial team echoes, and the graph runtime that team actually runs on is covered in the LangGraph walkthrough.
10. Gao et al., Enabling Large Language Models to Generate Text with Citations, 2023. Why grounded inline citations are hard to get and how to measure them, the very property the writer prompt works to enforce.

Part XII: Final takeaway

If the retrieval-and-embeddings machinery underneath the context manager is the unfamiliar part, the nearest-neighbor ideas are built up in my FAISS chapter, the local model backends the agent can drive live in vLLM and Ollama, and the server it streams through is plain FastAPI. Then come back and read agent.py and skills/researcher.py once more. They will read like a short, honest description of exactly what the agent does, which is the whole point.