XML sitemap parser designed to extract and process millions of URLs while bypassing most modern anti-bot protections. Supports plain and compressed XML, unlimited nested sitemaps, multi-threading, multiple inputs, CloudScraper integration, fingerprint randomization, proxy/user agent rotation, auto stealth mode, and detailed monitoring.

A blazing fast CLI application that processes sitemaps in golang.

YiraBot: Simplifying Web Scraping for All. A user-friendly tool for developers and enthusiasts, offering command-line ease and Python integration. Ideal for research, SEO, and data collection.

Go library for parsing Sitemaps

Crawl sitemap of a given website and export metadata of its pages recursively into CSV format.

A versatile Python-based web scraper that extracts content from single URLs or entire sitemaps, organizing data into structured text files. Features include sitemap parsing, content grouping by URL structure, and an easy-to-use command-line interface. Ideal for data extraction, content analysis, and web research tasks.

Python tool to extract URLs from XML sitemaps (including nested sitemap indexes) and automatically submit them to Google Indexing API for faster indexing. Supports bulk processing, rate limiting, and detailed progress reporting.

A Python script to automate URL update and deletion notifications to Google using the Indexing and Search Console APIs. It fetches URLs from an XML sitemap or processes batch URLs from a JSON file, handling Google API authentication and indexing requests.

Quick lookup for HTTP Status codes of a Sitemap!

An advanced, auto-pilot web scraping framework for building clean text datasets. Features recursive sitemap parsing, smart media filtering, and content extraction from complex page builders (like Elementor). Supports auto-resume and multithreading.

🕷️ Automate web scraping with OmniCrawler, a powerful tool that builds large datasets by discovering and downloading relevant content effortlessly.

Parse robots.txt and sitemaps using dotnet

nostrax is a fast, async web crawler and URL extraction toolkit for mapping, analyzing, and auditing websites.

Atlas — Web-scraping toolkit for Swedish authorities. Automates data collection with pagination, PDF crawling, metadata extraction, and multi-format export.

This repository provides practical examples of website link scraping using Python and Node.js.

Высокопроизводительный асинхронный веб-парсер на Python с поддержкой Sitemap, рендерингом JS (Playwright) и дашбордом в реальном времени на Streamlit.

🗺️ Harvest URLs and metadata from website sitemaps efficiently with this fast Python tool. Get organized insights for your digital projects.

🌐 Automate URL extraction from XML sitemaps and submit to Google Indexing API for faster indexing and improved SEO performance.