Scrapyard

Self-hosted web search and page extraction for AI agents.

Run it on your own machine. Use it over REST or MCP. No account, API key, metered plan, or telemetry.

Install

python3.12 -m pip install scrapyard
python3.12 -m playwright install chromium
scrapyard

What it does

Scrapyard searches the web, turns static pages, JavaScript pages, and PDFs into clean Markdown, discovers published sitemap URLs, and can search and extract in one bounded request. Responses keep the extraction method, pipeline steps, upstream status, and final URL. A daily canary detects broken search adapters and produces a repair brief for your coding agent; the patch still has to pass the security suite, canary, and live verification gate.

Limits: requests come from your IP, so heavily defended sites may block them. There is no proxy pool or site-wide crawl. Concurrency is 3. Google search is deliberately absent because it challenges non-browser clients.