OneCrawler¶

from onecrawler import Settings, SiteMap, Scraper
from onecrawler.utils import writter


async def main():
    settings = Settings(include_link_patterns=["/news/*"], concurrency=10)

    sitemap = SiteMap(settings)
    urls = await sitemap.run("https://example.com")

    async with Scraper(settings) as scraper:
        results = await scraper.run(urls)  # [{"url": ..., "result": ...}, ...]

    writter.dump_json(results, "output.json")

Why OneCrawler¶

Many crawling projects start as small scripts and become hard to maintain once they need retries, concurrency, browser rendering, sitemap fallback, URL filtering, and structured extraction. OneCrawler gives those concerns a shared settings model and a few focused async engines, so production scripts stay readable.

Choosing the right tool¶

The full workflow¶

Start with sitemap discovery whenever possible — it is the fastest, cleanest, and least expensive way to collect URLs. If the site has no useful sitemap, use LinkExtractor (shallow for one page, deep for recursive traversal), then pass the final URL list to Scraper.

from onecrawler import Settings, LinkExtractor, Scraper
from onecrawler.utils import writter


async def main():
    settings = Settings(
        link_extraction_limit=500,
        include_link_patterns=["/news/*"],
        concurrency=10,
    )

    async with LinkExtractor(settings) as extractor:
        urls = await extractor.run("https://example.com")
        print(f"Found {len(urls)} URLs")

    async with Scraper(settings) as scraper:
        results = await scraper.run(urls)
        print(f"Scraped {len(results)} pages")

    writter.dump_json(results, "output.json")


if __name__ == "__main__":
    import asyncio

    asyncio.run(main())

Production principles¶

• Prefer sitemaps before crawling pages. They are faster, friendlier to target sites, and usually more complete than what a browser can discover from navigation.
• Constrain every job. Set link_extraction_limit, include_link_patterns, concurrency, request_timeout, and max_retries explicitly so a crawl behaves predictably when the target site changes.
• Use heuristic scraping by default. It is cheaper and more repeatable; move to GenAI only when you need semantic interpretation, field normalization, or a typed Pydantic schema.
• Treat browser crawling as the heavier tool. It is useful for JavaScript-rendered pages and dynamic link discovery, but has more moving parts than sitemap parsing or direct HTTP fetching.

Where to next¶