Firecrawl: Live Web Research
- Add the Firecrawl MCP server using an API key from firecrawl.dev and verify it connects
- Distinguish between the scrape and search tools and choose the right one for a given research task
- Apply Firecrawl to real developer workflows: documentation scraping, competitive analysis, and news monitoring
The documentation page that Claude does not know about
Claude's training data has a cutoff. Documentation that was written or updated after that cutoff, changelogs for recent library versions, blog posts announcing new API behavior, and any site that Claude simply was not trained on — none of that is in Claude's knowledge. Asking Claude about it produces confident answers built on outdated or incomplete information.
Firecrawl solves this by giving Claude the ability to fetch live web content. Instead of recalling what it learned in training, Claude scrapes the current page, reads its actual content, and responds based on what the page says right now.
Setting up Firecrawl
Firecrawl runs as a local stdio server via npx and requires an API key from firecrawl.dev. The free tier provides 10 scrapes per minute and 5 searches per minute with no credit card required — enough for most development workflows.
Get an API key at firecrawl.dev (sign up, go to your dashboard, copy the key). It starts with fc-.
Add the server:
claude mcp add --env FIRECRAWL_API_KEY=fc-YOUR_KEY --transport stdio firecrawl -- npx -y firecrawl-mcp
Verify with claude mcp list. The server should show as ✓ Connected. On first use, npx downloads the package, so the first connection may take a moment.
Scrape versus search
Firecrawl provides two primary tools, and choosing the right one saves rate limit budget:
Extracts the content of one specific URL you provide. Handles JavaScript-rendered pages, strips ads and navigation, and returns clean markdown. Use this when you know exactly which page you need and want its full content.
Takes a query string, finds the most relevant URLs for that query, and scrapes them. Returns content from multiple sources. Use this when you need to discover which pages are relevant before reading them.
If you have the URL, use scrape — it is faster and more accurate. Use search when you are exploring a topic and do not know which specific page has the answer.
Documentation workflows
The most common use case for developers is reading documentation Claude does not have in training. A library just released a new version with a new API — scrape the changelog:
Scrape https://docs.somelib.dev/changelog and tell me what changed in the last two versions. Then update our usage to use the new API.
Claude scrapes the changelog, reads what actually changed, and updates the code accordingly — rather than inventing behavior based on an earlier version it was trained on.
For exploring an unfamiliar tool's documentation:
Search for how to configure retry logic in the Temporal workflow SDK. Then implement retries in our existing workflow file.
Claude searches, finds the relevant docs pages, reads them, and implements the feature from the actual documentation.
Research and analysis workflows
Firecrawl is also effective for structured research tasks:
Competitive analysis:
Scrape our three main competitors' pricing pages and summarize how they structure their plans compared to ours.
Claude fetches each page, extracts the pricing structure, and produces a comparison. The data reflects what the pages say today, not what Claude guessed from training.
News monitoring:
Search for the latest news about the OpenAI API in the last week. Give me a summary of any pricing or model changes.
Claude searches for recent results, reads the content, and summarizes only what the actual articles report.
Structured extraction:
Scrape this job posting page and extract the required skills, salary range, and company size in JSON format.
Firecrawl returns clean markdown from the page, and Claude parses it into the structured format you requested.
Rate limits and responsible use
The free tier's limits (10 scrapes/min, 5 searches/min) are sufficient for focused research sessions. They apply per API key, so large batch scraping operations will hit them. For heavy use, Firecrawl's paid tiers raise these limits significantly.
Firecrawl handles JavaScript rendering, bot detection, and content cleaning automatically. Sites that block headless browsers or require login will not be scrapable — Firecrawl returns an error for these, and Claude will report that the page was not accessible.
- Firecrawl gives Claude access to live web content — pages published after its training cutoff, recent changelogs, and any site not in its training data become readable through tool calls.
- Use scrape when you have a specific URL; use search when you need to discover which pages are relevant to a topic — scrape is faster and more focused.
- The free tier (10 scrapes/min, 5 searches/min, no credit card) covers most development research workflows; heavy batch use requires a paid tier.
- Documentation scraping is the most reliable developer use case: fetch the actual current changelog or API reference rather than relying on Claude's training-data recollection of a library.
- Store the API key in a shell environment variable and reference it with <code>--env FIRECRAWL_API_KEY=${FIRECRAWL_API_KEY}</code> rather than hardcoding it in the add command.