Getting Clean Text Into Your AI: Firecrawl, Crawl4AI, MarkItDown and Docling

Language models are good at reasoning over text and bad at getting it. The web is HTML wrapped in navigation, cookie banners, ads and scripts; your archive is PDFs, spreadsheets and slide decks. Feeding either to a model means converting messy source material into clean text first.

That conversion step is where most AI projects quietly fail. Not the model, not the prompt — the input. Garbage in, confident garbage out.

These are the tools that do it properly.

Star counts are from GitHub at the time of writing.

Firecrawl — the polished option

mendableai/firecrawl · ~176,000 stars · TypeScript · AGPL-3.0

Firecrawl takes a URL and returns clean markdown. It renders JavaScript, follows links to crawl whole sites, handles pagination, and strips the boilerplate that would otherwise eat your context window.

It handles the annoying cases well — dynamic content, single-page apps, sites that only render after scroll. Structured extraction is built in: describe the fields you want and it returns JSON rather than prose.

Read this before adopting it: Firecrawl is AGPL-3.0. That's the strongest copyleft licence in common use, and unlike GPL it extends to network use — if you run modified Firecrawl as a service others reach over a network, you may be obliged to publish your source. For internal use it's unproblematic. For a commercial product, understand this properly or use their hosted service, which is what the licence choice is designed to encourage.

Crawl4AI — permissive and purpose-built

unclecode/crawl4ai · ~81,000 stars · Python · Apache-2.0

Crawl4AI does much the same job under Apache-2.0, which for most commercial situations makes it the more comfortable choice.

Built specifically for LLM pipelines: markdown output tuned for context windows, chunking strategies, CSS or LLM-driven extraction, async crawling for volume. It handles JavaScript through a real browser and supports session reuse for pages behind a login.

If you're building something you intend to sell, start here rather than with Firecrawl. The capability gap is smaller than the licence gap.

ScrapeGraphAI — describe what you want

ScrapeGraphAI/Scrapegraph-ai · ~30,000 stars · Python · MIT

Different approach: instead of writing selectors, you describe in plain language what to extract, and it uses an LLM to build the extraction graph.

The appeal is resilience. Traditional scrapers break when a site changes its markup — you wrote .product-title, they renamed it, everything stops. A model looking for "the product name" survives that.

The trade is cost and speed: every page costs tokens, and it's slower. Sensible for scraping many differently-structured sites; wasteful for pulling one field from a thousand pages of the same template. Use selectors where the structure is stable, models where it isn't.

MarkItDown — for everything that isn't a web page

microsoft/markitdown · ~178,000 stars · Python · MIT

MarkItDown converts documents to markdown: PDF, Word, Excel, PowerPoint, images, audio, HTML, CSV, ZIP archives.

It solves the unglamorous problem that eats time in every real project. Your source material is rarely web pages — it's a decade of PDFs and spreadsheets. This turns them into text a model can read, in one call, with a sane API.

Microsoft-maintained, MIT licensed, actively developed. If you're building anything that ingests documents, you will end up using this or Docling.

Docling — when document structure matters

docling-project/docling · ~66,000 stars · Python · MIT

Docling goes deeper than conversion: it does layout analysis. Tables stay tables. Reading order is preserved in multi-column academic papers. Figures and captions stay associated.

That matters more than it sounds. A financial report converted naively turns its tables into an unreadable stream of numbers with no column relationships — and a model reading that will confidently misattribute figures. Docling preserves the structure, so the numbers keep their meaning.

Heavier and slower than MarkItDown because it's doing more. Use MarkItDown for bulk text extraction; use Docling when the documents have tables you need to be right.

Choosing, quickly

Web pages, commercial project → Crawl4AI (Apache-2.0). Web pages, internal only, want polish → Firecrawl. Many differently-structured sites → ScrapeGraphAI. Documents, bulk conversion → MarkItDown. Documents with tables that must be accurate → Docling.

Most real pipelines use two: one for the web, one for documents.

Things that will bite you

Context windows are smaller than the web. A crawled site is easily hundreds of thousands of tokens. Clean, chunk and retrieve selectively rather than pasting everything into a prompt — that's what a vector store is for.

Tables are where extraction goes wrong. Always spot-check a converted table against the original. This is the most common source of confidently wrong answers in document pipelines, and it's invisible unless you look.

Scraped content is untrusted input. A page can contain text written to manipulate a model that reads it — instructions disguised as content. If your pipeline scrapes a page and then acts on what it says, you have an injection path. Treat retrieved text as data to reason about, never as instructions to follow, and be especially careful when an agent can both fetch pages and run commands.

Respect the rules. Check robots.txt and terms of service. Rate-limit yourself. Identify your crawler honestly. Scraping public pages is generally fine; hammering a small site until it falls over is not, and it's how you get blocked.

Cache aggressively. Pages change slowly. Re-crawling the same URL on every run wastes time, money and the goodwill of whoever's server you're hitting.

The unglamorous conclusion

Nobody demos the extraction layer. It's the least interesting part of an AI project and the part most likely to determine whether the output is any good.

If your model is giving vague or wrong answers about your own documents, the odds are strongly that the problem is upstream — the PDF converted badly, the table lost its structure, the chunking split a paragraph mid-sentence. Check the text your model is actually receiving before you touch the prompt.

It's usually the input.