Most "scrape Reddit" tutorials give you a script that does exactly one thing: fetch posts matching a query, dump them to a file, done. You get data, but you still have to read all of it, decide what matters, chase the interesting threads into their comments, and write up what you learned. The scraping was the easy 10%. The research was the other 90%, and the script didn't touch it.

An agent closes that gap. Instead of a fixed script, you give Claude a research question and a set of Reddit tools, and let it decide what to search, which threads to open, which comment trees to read, and when it has enough to answer. It runs the loop a human researcher would run — search, read, follow the thread, refine the query, repeat — and hands back a synthesis instead of a data dump. The difference between a scraper and a research agent is the difference between "here are 200 posts" and "here's what people actually think about X, with the receipts."

This guide builds that agent two ways. The fast way uses ScrapeBadger's MCP server, which lets Claude call Reddit tools directly during inference — you write almost no integration code. The transparent way builds the tool-use loop by hand, so you can see and control exactly what the agent does at each step. Both use the same ScrapeBadger Reddit endpoints underneath.

What "Agent" Actually Means Here

Before the code, the concept — because "agent" is an overloaded word and the mechanism is simpler than the hype suggests.

A plain API call is one-shot: you ask, the model answers from what it already knows. An agent is a loop. You give the model a goal and a set of tools, and on each turn the model either calls a tool or gives a final answer. When it calls a tool, you run that tool, hand back the result, and let the model decide what to do next. It keeps going — search, read the results, decide it needs the comments on the third post, read those, decide it needs a different search — until it judges the goal met and stops. That is the entire idea. The model is the reasoning; the tools are its hands; the loop is what turns a single answer into actual research.

For Reddit research, the tools are the Reddit operations: search posts across the site or within a subreddit, pull a subreddit's recent posts, and read the full comment tree of a specific thread. Give Claude those three, plus a question, and it can conduct genuine multi-step research — the same moves a person would make, executed in seconds.

Setup

You need an Anthropic API key and a ScrapeBadger API key.

bash

pip install anthropic httpx python-dotenv

env

ANTHROPIC_API_KEY=sk-ant-...
SCRAPEBADGER_API_KEY=your_scrapebadger_key

Approach 1: The MCP Connector (Least Code)

ScrapeBadger runs a remote MCP server. Anthropic's Messages API can connect to a remote MCP server directly through the mcp_servers parameter — Claude calls the tools on ScrapeBadger's server during inference, and you never write a tool handler. This is the whole agent in one API call.

A few facts worth knowing before you use it, because they shape what this approach can and can't do. The MCP connector currently supports tool calls only — not MCP resources or prompts. The server must be reachable over public HTTP, which ScrapeBadger's is. And it requires a beta header. That's the real shape of the feature as of writing; verify the current details in Anthropic's MCP connector docs, since this part of the API is still evolving.

python

# agent_mcp.py
import os
from anthropic import Anthropic

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

def research(question: str) -> str:
    response = client.beta.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=4096,
        messages=[{
            "role": "user",
            "content": (
                f"You are a Reddit research analyst. Research this question by "
                f"searching Reddit, reading the most relevant threads and their "
                f"comments, and synthesizing what real users actually say. "
                f"Cite specific subreddits and paraphrase representative opinions. "
                f"\n\nQuestion: {question}"
            ),
        }],
        mcp_servers=[{
            "type": "url",
            "url": "https://scrapebadger.com/mcp",
            "name": "scrapebadger",
            "authorization_token": os.environ["SCRAPEBADGER_API_KEY"],
            "tool_configuration": {
                "enabled": True,
                # Limit the agent to the Reddit tools for this task
                "allowed_tools": [
                    "reddit_search_posts",
                    "reddit_get_subreddit_posts",
                    "reddit_get_post_comments",
                    "reddit_search_subreddits",
                ],
            },
        }],
        betas=["mcp-client-2025-04-04"],
    )

    # The response contains the final synthesis plus a record of the tool calls
    # Claude made along the way. Pull out the text blocks for the answer.
    return "\n".join(
        block.text for block in response.content if block.type == "text"
    )


if __name__ == "__main__":
    answer = research(
        "What do people building indie SaaS actually think about using "
        "Reddit for customer acquisition? Is it worth it or a waste of time?"
    )
    print(answer)

That is a working Reddit research agent. Claude receives the question, sees the Reddit tools available on ScrapeBadger's server, and runs its own loop — searching, reading threads, pulling comments — entirely on Anthropic's side, then returns a synthesis. The allowed_tools list is worth setting deliberately: it constrains the agent to the tools relevant to the task, which keeps it focused and predictable.

The trade-off of this approach is control. You get the answer and a record of what the agent did, but the loop runs inside the API call — you can't intervene between steps, add custom scoring, cache results, or branch the logic. For many research tasks that is completely fine. When you need that control, build the loop yourself.

Approach 2: The Manual Tool-Use Loop (Full Control)

Here you define the Reddit tools to Claude explicitly, run the agent loop yourself, and execute each tool call by hitting ScrapeBadger's REST API. More code, but you see and control every step — which matters when you want to cache aggressively, add your own filtering between turns, log the reasoning, or enforce hard limits on how far the agent ranges.

First, thin wrappers over the ScrapeBadger Reddit endpoints:

python

# reddit_tools.py
import os
import httpx

BASE = "https://scrapebadger.com/v1"
HEADERS = {"x-api-key": os.environ["SCRAPEBADGER_API_KEY"]}


def search_posts(q: str, subreddit: str = None, sort: str = "relevance",
                 t: str = "month", limit: int = 15) -> dict:
    params = {"q": q, "sort": sort, "t": t, "limit": limit}
    if subreddit:
        params["subreddit"] = subreddit
    r = httpx.get(f"{BASE}/reddit/search", params=params, headers=HEADERS, timeout=30)
    r.raise_for_status()
    return r.json()


def get_subreddit_posts(subreddit: str, sort: str = "hot", limit: int = 15) -> dict:
    params = {"sort": sort, "limit": limit}
    r = httpx.get(f"{BASE}/reddit/r/{subreddit}", params=params, headers=HEADERS, timeout=30)
    r.raise_for_status()
    return r.json()


def get_post_comments(post_id: str, sort: str = "top", limit: int = 30) -> dict:
    params = {"sort": sort, "limit": limit}
    r = httpx.get(f"{BASE}/reddit/comments/{post_id}", params=params, headers=HEADERS, timeout=30)
    r.raise_for_status()
    return r.json()


# Map tool names to functions for the agent loop
TOOL_FUNCTIONS = {
    "search_posts": search_posts,
    "get_subreddit_posts": get_subreddit_posts,
    "get_post_comments": get_post_comments,
}

Next, describe those tools to Claude in the format the Messages API expects. The descriptions matter — they are how the model knows when to reach for each tool, so write them as if briefing a new analyst:

python

# tool_schemas.py
TOOLS = [
    {
        "name": "search_posts",
        "description": (
            "Search Reddit posts by keyword. Use this to find threads discussing "
            "a topic. Optionally scope to a subreddit. Returns post titles, bodies, "
            "IDs, scores, and comment counts. Use the returned post 'id' with "
            "get_post_comments to read what people said."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "q": {"type": "string", "description": "Search keywords"},
                "subreddit": {"type": "string", "description": "Optional: limit to this subreddit"},
                "t": {"type": "string", "enum": ["day", "week", "month", "year", "all"],
                      "description": "Time window for results"},
            },
            "required": ["q"],
        },
    },
    {
        "name": "get_subreddit_posts",
        "description": (
            "Get recent posts from a specific subreddit. Use this to survey what a "
            "community is currently discussing, or after search_subreddits identifies "
            "a relevant community."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "subreddit": {"type": "string"},
                "sort": {"type": "string", "enum": ["hot", "new", "top"]},
            },
            "required": ["subreddit"],
        },
    },
    {
        "name": "get_post_comments",
        "description": (
            "Read the comment tree of a specific post by its ID. This is where the "
            "real opinions are — always read comments on the most relevant posts "
            "rather than relying on titles alone."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "post_id": {"type": "string", "description": "The post 'id' from search results"},
            },
            "required": ["post_id"],
        },
    },
]

Now the loop itself — the piece that makes it an agent. Claude responds; if it wants a tool, you run the tool and feed the result back; repeat until it stops asking for tools and gives its synthesis:

python

# agent_manual.py
import os
import json
from anthropic import Anthropic
from reddit_tools import TOOL_FUNCTIONS
from tool_schemas import TOOLS

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

SYSTEM = (
    "You are a Reddit research analyst. Given a question, research it by "
    "searching Reddit, surveying relevant subreddits, and reading the comment "
    "trees of the most relevant threads. Do multiple searches and read multiple "
    "threads before concluding — the first search is rarely enough. When done, "
    "synthesize what real users actually say: the consensus, the disagreements, "
    "and representative views, citing specific subreddits. Paraphrase; never "
    "reproduce long quotes verbatim."
)


def research(question: str, max_turns: int = 12) -> str:
    messages = [{"role": "user", "content": question}]

    for turn in range(max_turns):
        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=4096,
            system=SYSTEM,
            tools=TOOLS,
            messages=messages,
        )

        # Record Claude's turn (may contain text and/or tool-use requests)
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason != "tool_use":
            # No more tools requested — Claude has given its final synthesis
            return "".join(
                b.text for b in response.content if b.type == "text"
            )

        # Execute every tool Claude asked for this turn
        tool_results = []
        for block in response.content:
            if block.type == "tool_use":
                fn = TOOL_FUNCTIONS[block.name]
                print(f"  [turn {turn+1}] {block.name}({block.input})")
                try:
                    result = fn(**block.input)
                    # Trim to keep the context lean — send the fields that matter
                    payload = json.dumps(result)[:6000]
                except Exception as e:
                    payload = f"Error: {e}"
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": payload,
                })

        messages.append({"role": "user", "content": tool_results})

    return "Reached max turns without a final synthesis."


if __name__ == "__main__":
    print(research(
        "What are the most common complaints about Notion from people who "
        "switched away from it? Which tools did they switch to?"
    ))

Run it and you can watch the agent think:

  [turn 1] search_posts({'q': 'switched away from Notion', 't': 'year'})
  [turn 2] search_posts({'q': 'Notion alternative why I left', 't': 'year'})
  [turn 3] get_post_comments({'post_id': '1abc234'})
  [turn 4] get_post_comments({'post_id': '1def567'})
  [turn 5] search_posts({'q': 'Notion too slow', 'subreddit': 'productivity'})
  [turn 6] get_post_comments({'post_id': '1ghi890'})

[final synthesis: common complaints — performance on large workspaces,
offline reliability, and the learning curve; most-cited switch targets
were Obsidian for local-first users and Linear-adjacent tools for teams...]

That trace is the whole value proposition. The agent ran six research steps — refining its search twice, reading three comment trees, scoping one search to a specific subreddit — and synthesized a real answer. You wrote the tools and the loop; Claude supplied the judgment about what to do next.

Which Approach to Use

The MCP connector is the right default. It is less code, the loop is handled for you, and for the majority of research tasks — "what do people think about X," "summarize the debate on Y," "what are the complaints about Z" — it does the job with a single API call. Reach for it first.

Build the manual loop when you need control the connector doesn't give you: caching tool results to cut costs on repeated research, inserting your own scoring or filtering between the agent's turns, logging every step for auditability, enforcing hard limits on how far the agent ranges, or combining Reddit tools with other data sources in one custom loop. The manual version is more moving parts, but every part is yours to shape.

A useful pattern is to prototype with the MCP connector to confirm the agent produces good research on your questions, then, only if you hit a real need for control, port to the manual loop. Don't build the complex version until the simple version proves insufficient.

Making the Agent Actually Good

The mechanics above give you a working agent. A few things separate a good one from a mediocre one, and they're mostly about the prompt and the guardrails rather than the code.

Push it to search more than once. The single biggest failure mode is an agent that runs one search, reads two posts, and concludes. The system prompt above explicitly tells it the first search is rarely enough — that instruction measurably improves depth. For important research, tell it a minimum: survey at least three searches and read at least three comment trees before synthesizing.

Give it the comment-reading instinct. Titles and post bodies are the question; the comments are the answer. An agent that only reads post titles produces shallow summaries. The tool description for get_post_comments and the system prompt both push toward reading comments on the most relevant threads, because that is where the real opinions live.

Keep the context lean. Reddit responses are verbose, and feeding full raw JSON back to the model burns tokens and dilutes attention. The manual loop trims tool results to the fields that matter before returning them. In production, parse each result down to title, score, and body for posts, and author-plus-text for comments, before handing it back.

Bound the loop. The max_turns cap is a real safety measure, not a formality — it stops a confused agent from looping expensively. Set it to the depth your task genuinely needs and no more.

Respect the copyright line. The system prompt instructs the agent to paraphrase and never reproduce long verbatim quotes. Keep that instruction. The output you want is synthesis in your own words — the consensus and the disagreements — not a wall of copied comments.

From here, the same pattern extends naturally. Swap in ScrapeBadger's other endpoints — Twitter, YouTube, Amazon, Google — and the same loop becomes a multi-source research agent that cross-references what people say on Reddit against what they say on Twitter, or against product reviews. The Reddit agent is the template; the research surface is up to you.

Free trial at scrapebadger.com/reddit-scraper — 1,000 credits, no card. MCP docs at docs.scrapebadger.com.

Common Questions

Do I need to know MCP to build this? For the connector approach, no — you point the mcp_servers parameter at ScrapeBadger's server URL and Claude handles the rest. You don't implement any MCP client yourself. For the manual loop you don't touch MCP at all; you're using Claude's standard tool-use API with plain REST calls underneath. MCP knowledge helps if you later build your own server, but it isn't required to use one.

Which is cheaper to run, the connector or the manual loop? The Claude token cost is similar — both run the same reasoning loop. The manual loop can be made cheaper in practice because you control the context: trimming verbose tool results before returning them, and caching repeated searches, both cut token usage. The connector is simpler but gives you fewer levers to optimize cost. For occasional research, the connector's simplicity wins; for high-volume automated research, the manual loop's control pays off.

Can the agent read entire comment threads, not just top comments? Yes. The comment endpoint returns the comment tree, and you can request more comments or deeper sorting. Be deliberate about how much you feed back to the model, though — a huge thread returned in full will blow your context budget and dilute the model's attention. Trimming to the most relevant comments before returning is usually better than dumping everything.

How do I stop the agent from going off on tangents? Three levers: the allowed_tools list (connector) or the tool set you define (manual loop) constrains what it can do; the system prompt constrains what it should do; and max_turns bounds how far it can range. A focused tool set plus a clear system prompt plus a sensible turn cap keeps it on task.

Can I use a bigger model for deeper analysis? Yes — swap the model string for a more capable model when the synthesis quality matters more than speed or cost. A common pattern is running the tool-use loop on a fast model to gather the raw material, then doing the final synthesis with a more capable model in one last call. Check the current model names in Anthropic's documentation, since the lineup changes.