Scraping Engine Package¶

The scraping package provides content extraction engines for scraping web pages: deterministic heuristic extraction, whole-page Markdown conversion, and AI-powered structured extraction.

Classes¶

Scraper¶

Main scraping engine that supports heuristic, markdownify, and GenAI content extraction strategies.

from onecrawler import Settings, Scraper

async with Scraper(settings) as scraper:
    item = await scraper.run("https://example.com/article")
    print(item["url"], item["result"])

Parameters¶

• settings (Settings): Configuration for scraping behavior

Methods¶

• run(url: str) -> dict: Extract content from the given URL, returning {"url": str, "result": Any} — result's shape depends on scraping_strategy/scraping_output_format (a dict, plain text, or a GenAI output_schema model instance)
• run(urls: List[str]) -> List[dict]: Extract content from multiple URLs, returning one {"url": str, "result": Any} dict per successful extraction

Strategies¶

• Heuristic: Fast, rule-based extraction using trafilatura; isolates article-like content and strips boilerplate, but can return little or nothing on non-article pages (e-commerce, dashboards, listings)
• Markdownify: Faithful whole-page HTML-to-Markdown conversion with no content extraction; never returns None for a rendered page, so it's the fallback for pages heuristic extraction can't handle
• GenAI: AI-powered extraction with structured output

Heuristic strategy¶

Rule-based content extraction using the trafilatura library. Select it with Settings(scraping_strategy="heuristic") — see Heuristic Scraping below.

Features¶

• Fast extraction: No model calls, deterministic results
• Multiple formats: Markdown, JSON, XML, and XML-TEI output
• Language detection: Automatic language identification
• Content cleaning: Removes boilerplate and navigation

Markdownify strategy¶

Whole-page HTML-to-Markdown conversion using html-to-markdown, with no content extraction. Select it with Settings(scraping_strategy="markdownify").

Features¶

• Never empty: Converts the whole rendered page, so it works on pages heuristic extraction returns little or nothing for
• Faithful conversion: Keeps navigation, footers, and other page chrome — no boilerplate removal and no extracted metadata
• Deterministic filtering: Set settings.exclude_selectors (e.g. ["nav", "footer", ".cookie-banner"]) to strip known chrome before conversion, at no LLM cost

GenAI strategy¶

AI-powered content extraction using language models. Select it with Settings(scraping_strategy="genai", genai=LLMSettings(...)) — see GenAI Scraping below.

Features¶

• Structured output: Pydantic schema-based extraction
• Semantic understanding: Context-aware content extraction
• Field normalization: Consistent data formatting
• Custom schemas: Define your own output structure

Usage Examples¶

Heuristic Scraping¶

from onecrawler import Settings, Scraper

async def scrape_heuristic():
    settings = Settings(
        scraping_strategy="heuristic",
        scraping_output_format="json",
        concurrency=10,
        request_timeout=15
    )

    async with Scraper(settings) as scraper:
        item = await scraper.run("https://example.com/article")

    return item["result"]

if __name__ == "__main__":
    import asyncio
    asyncio.run(scrape_heuristic())

GenAI Scraping¶

from pydantic import BaseModel
from onecrawler import Settings, LLMSettings, Scraper

class Article(BaseModel):
    title: str
    author: str
    content: str
    published_date: str

async def scrape_with_ai():
    settings = Settings(
        scraping_strategy="genai",
        scraping_output_format="json",
        genai=LLMSettings(
            provider="openai",
            model_name="gpt-4o-mini",
            api_key="your-api-key",
            output_schema=Article
        ),
        concurrency=2,
        request_timeout=30
    )

    async with Scraper(settings) as scraper:
        item = await scraper.run("https://example.com/article")

    return item["result"]  # an Article instance

if __name__ == "__main__":
    import asyncio
    asyncio.run(scrape_with_ai())

Batch Scraping¶

async def scrape_multiple():
    urls = [
        "https://example.com/article1",
        "https://example.com/article2",
        "https://example.com/article3"
    ]

    settings = Settings(
        scraping_strategy="heuristic",
        concurrency=5,
        max_retries=3
    )

    async with Scraper(settings) as scraper:
        results = await scraper.run(urls)  # [{"url": ..., "result": ...}, ...]

    return results

Configuration¶

Scraping behavior is controlled through Settings:

Output Formats¶

Heuristic Strategy¶

• JSON: Structured data with metadata
• Markdown: Clean text formatting
• XML: Original TEI-agnostic XML structure
• XML-TEI: TEI-conformant XML output

Markdownify Strategy¶

• Markdown only: scraping_output_format is ignored; always returns a plain Markdown string of the whole page, with no extracted metadata

GenAI Strategy¶

• JSON: Structured output matching Pydantic schema
• Custom formats: Based on your schema definition

Performance Considerations¶

Heuristic Scraping¶

• Fast: No model calls, deterministic timing
• Lightweight: Lower memory and CPU usage
• Scalable: Higher concurrency possible
• Consistent: Predictable performance

GenAI Scraping¶

• Slower: Model response time (10-30 seconds)
• Expensive: API costs per request
• Limited concurrency: Lower parallelism
• Variable: Response time depends on model

Best Practices¶

When to Use Heuristic¶

• Bulk content extraction: Large numbers of pages
• Fast processing: Time-sensitive operations
• Cost efficiency: Budget-conscious projects
• Simple content: Articles, blog posts, documentation

When to Use GenAI¶

• Structured data: Specific fields required
• Complex content: Mixed or unstructured pages
• Normalization: Consistent data formatting
• Semantic extraction: Understanding context

General Tips¶

1. Start with heuristic: Faster and cheaper
2. Filter URLs: Only scrape relevant pages
3. Set timeouts: Handle slow pages gracefully
4. Monitor errors: Track failure rates
5. Batch processing: Group similar requests

Error Handling¶

The scraper handles various error conditions:

• Network errors: Automatic retries with backoff
• Parsing errors: Graceful degradation
• Timeout errors: Configurable timeouts
• Rate limiting: Respects server limits
• Model errors: Fallback strategies for GenAI

Integration Examples¶

Save to File¶

from onecrawler import Settings, Scraper
from onecrawler.utils import writter

async def scrape_and_save():
    settings = Settings(scraping_strategy="heuristic")

    async with Scraper(settings) as scraper:
        item = await scraper.run("https://example.com/article")

    writter.dump_json(item, "output.json")  # includes both "url" and "result"

Database Integration¶

from onecrawler import Settings, Scraper

async def scrape_to_database():
    settings = Settings(scraping_strategy="heuristic")

    async with Scraper(settings) as scraper:
        urls = ["https://example.com/page1", "https://example.com/page2"]
        results = await scraper.run(urls)

    # Save to database
    for item in results:
        await save_to_database(item["url"], item["result"])

Troubleshooting¶

Common Issues¶

1. Empty results: Check URL accessibility and content structure
2. Timeout errors: Increase request_timeout or reduce concurrency
3. GenAI failures: Verify API keys and model availability
4. Memory issues: Reduce concurrency for large batches
5. Rate limits: Implement delays between requests

Debug Mode¶

Enable detailed logging for troubleshooting:

settings = Settings(
    logging_level="DEBUG"
)