Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: robotshell/robotScraper
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: Psikoz-coder/robotScraper
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
  • 1 commit
  • 2 files changed
  • 1 contributor

Commits on Apr 27, 2025

  1. Add sitemap.xml functionality to RobotScraper 

    Enhanced RobotScraper by adding sitemap.xml parsing capabilities alongside existing robots.txt functionality. The tool now extracts URLs from both sources, handles nested sitemaps, bypasses SSL certificate issues, and automatically saves sitemap results to sitemap.txt. Added a new mode parameter (-m/--mode) for flexibility while maintaining backward compatibility. Updated documentation with comprehensive examples and feature descriptions.
    Psikoz committed Apr 27, 2025
    Configuration menu
    Copy the full SHA
    a797c34 View commit details
    Browse the repository at this point in the history
Loading