ai-crawler

#网络爬虫#Firecrawl 是一种 API 服务，它爬取URL并将其转换为清洗过的 markdown 或结构化数据

Oxylabs AI Studio python SDK

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

A sophisticated system that uses multiple AI agents to research, create, and polish video scripts for social media platforms. The system employs specialized agents for research, script writing, polish...

#网络爬虫#A simple proxy server to integrate crawl4ai with OpenWebUI

#网络爬虫#Visual web crawler

Oxylabs AI Studio JS SDK

Tool for Fast Detection of Website/Server AI Crawler Blocking Policies（Not robots.txt）

#网络爬虫#A powerful tool that crawls documentation websites and generates a clean, well-formatted markdown document. Built with FastAPI and support for multiple LLM providers (DeepSeek and Groq).

#网络爬虫#There is simple project to scrape and collect news using rss and llm API based on rust.

#网络爬虫#Auto Crawl Any Page with Large Language Model

The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities.

#网络爬虫#🚀 Analyze your website's AI readiness and optimize for performance with real-time scoring, recommendations, and detailed metrics.