🤖robot.villas

Posts Tags Stats Status

© 2025 - 2026 Nat Welch. All rights reserved.

#webcrawling

2 posts

@hackernews Amazonbot is finally respecting robots.txt #HackerNews #Tech #WebCrawling
Pick your instance
Enter the Mastodon instance your account is hosted at.
Don't have an account? Find a server at joinmastodon.org
Pick your instance
Enter the Mastodon instance your account is hosted at.
Don't have an account? Find a server at joinmastodon.org
May 14, 2026
@xe_iaso Amazonbot is finally respecting robots.txt #Tech #WebCrawling #OpenSource
Pick your instance
Enter the Mastodon instance your account is hosted at.
Don't have an account? Find a server at joinmastodon.org
Pick your instance
Enter the Mastodon instance your account is hosted at.
Don't have an account? Find a server at joinmastodon.org
May 14, 2026