Here are 18 public repositories matching this topic...

firecrawl / firecrawl

🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

markdown crawler scraper ai html-to-markdown web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction ai-agents web-search ai-search web-data llm ai-crawler ai-scraping

Updated Mar 16, 2026
TypeScript

ScrapeGraphAI / Scrapegraph-ai

Sponsor

Python scraper based on AI

Updated Mar 15, 2026
Python

oxylabs / ai-crawler-py

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

ai web-crawler ai-agents ai-crawler ai-studio ai-scraping ai-web-crawler crawl-agent

Updated Oct 13, 2025

oxylabs / oxylabs-ai-studio-py

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM agents with fresh data. AI Studio python SDK for intelligent web data gathering.

web-scraping ai-search proxy-scraper python-ai ai-tools web-scraping-python ai-crawler web-scraping-api ai-web-scraper ai-scraping ai-scraper web-scraping-ai

Updated Dec 4, 2025
Python

vakra-dev / reader

Open-source, production-grade web scraping engine built for LLMs. Scrape and crawl the entire web, clean markdown, ready for your agents.

Updated Feb 2, 2026
TypeScript

BrowserCash / teracrawl

High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative

html-to-markdown web-scraper web-scraping data-extraction browser-automation ai-agents web-search google-serp ai-search serpapi browser-agent ai-crawler antibot-bypass ai-scraping crawl4ai firecrawl-mcp firecrawl-api firecrawl-alternative

Updated Dec 3, 2025
TypeScript

oxylabs / oxylabs-ai-studio-js

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM agents with fresh data. AI Studio JS SDK for intelligent web data gathering.

web-scraping ai-search proxy-scraper python-ai aistudio ai-tools web-scraping-python ai-crawler ai-studio ai-web-scraper ai-scraping ai-scraper web-scraping-ai ai-map

Updated Dec 8, 2025
TypeScript

lennyerik / crawl4ai-proxy

A simple proxy server to integrate crawl4ai with OpenWebUI

adapter crawler machine-learning ai proxy web-crawler inference gpt llm ai-crawler llm-ui openwebui llm-crawler crawl4ai

Updated Nov 20, 2025
Go

kr3t3n / smolagents-video-script-generator

A sophisticated system that uses multiple AI agents to research, create, and polish video scripts for social media platforms. The system employs specialized agents for research, script writing, polishing, and evaluation to ensure high-quality, engaging content.

ai-agents ai-crawler ai-writer smolagents

Updated Jan 2, 2025
Python

JerryZhi / AI-Crawler-Detector

Tool for Fast Detection of Website/Server AI Crawler Blocking Policies（Not robots.txt）

seo geo robots ai-crawler ai-seo-tool

Updated Aug 20, 2025
Python

crawlcore / scp-protocol

A collection-based format for serving clean, structured web content to AI training systems and search engines through pre-generated collections.

web-standards specification internet-draft data-format ai-training ai-crawler web-protocol ai-crawling ietf-draft

Updated Jan 4, 2026
Python

kr3t3n / documentation-crawler

A powerful tool that crawls documentation websites and generates a clean, well-formatted markdown document. Built with FastAPI and support for multiple LLM providers (DeepSeek and Groq).

documentation crawler ai crawl crawler-python ai-crawler ai-documentation

Updated Jan 1, 2025
Python

breadrock1 / news-rss

There is simple project to scrape and collect news using rss and llm API based on rust.

rss crawler scraper rss-feed html2text llm ai-crawler

Updated May 15, 2025
Rust

ramonclaudio / firecrawl-toolkit

The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities.

map sitemap web-crawler web-scraper crawl scrape ai-toolkit sitemap-crawler ai-crawler sitemap-scraper firecrawl fire-crawl batch-scrape ai-scraper ai-batch-scrape firecrawl-ai

Updated Dec 21, 2024
Python

sonhm3029 / Auto-Crawl-Any-Page-with-Large-Language-Model

Auto Crawl Any Page with Large Language Model

crawler ai-agents large-language-models ai-crawler llm-crawler

Updated Oct 4, 2024
Jupyter Notebook

pixeldownward / ai-social-media-agent

🤖 Generate high-quality social media posts effortlessly with this AI agent that researches, drafts, critiques, and finalizes content for you.

python sales social-media mcp cryptocurrency learn web3 1password multi-agent-systems claude solana supabase ai-crawler ai-writer openai-integration

Updated Mar 17, 2026

MubeenAk47 / ai-ready-website

🚀 Analyze your website's AI readiness and optimize for performance with real-time scoring, recommendations, and detailed metrics.

markdown crawler html-to-markdown web-crawler scraping web-scraping agents ai-search web-data multi-modality multi-modal-imaging huggingface llm ai-crawler chatgpt tree-of-thoughts ai-scraping ai-website-builder

Updated Mar 17, 2026
TypeScript

lxgicstudios / robots-gen

Generate robots.txt with smart defaults. Allow or block AI crawlers (GPTBot, ClaudeBot) based on your strategy. Framework presets for Next.js and Gatsby. Staging lockdown. Zero dependencies.

cli seo robots-txt npx ai-crawler gptbot

Updated Feb 10, 2026
JavaScript