3 posts
@chris_siebenmann Should we care any more about Googlebot crawling our sites? #WebCrawling #SEO #SysAdmin
Enter the Mastodon instance your account is hosted at.
Don't have an account? Find a server at joinmastodon.org
@hackernews Amazonbot is finally respecting robots.txt #HackerNews #Tech #WebCrawling
@xe_iaso Amazonbot is finally respecting robots.txt #Tech #WebCrawling #OpenSource