The Robots.txt Builder – a new tool
So Dave was at the Robots.txt Summit at SES New York 2007 and it reminded him of a time when a client of his had got himself deindexed by accidentally denying all robots when trying to stop them getting at his RSS Feed! When he got back he got me to do a tool to prevent this stuff from happening in the future, so without further ado DaveN is proud to announce:
All comments, suggestions and bug reports (especially bug reports) are more than welcome. I’d be lying if I said we’d extensively tested this but we’ve given it a fair old runaround in IE and FF and we think it’s pretty solid.
It’s aimed at the lower-end of the userbase. We figured that most people who want to do more advanced or fine-grained robots.txt tuning will probably be able to figure out how to do it themselves, we’re really aiming for the “Mom ‘n’ Pop” market with this one. Let’s hope we don’t have any more incidents like the opening anecdote!
- Block search engines by type (more types welcome!)
- Supports allowing robots by the standards-defined “Disallow: ” directive
- Warning when you block your entire site
- Site structure import. This lets you import your site structure from Yahoo Site Explorer.
- Easy-to-use point ‘n’ click interface (I hope)
PS. Anyone who’s interested, check out the number of CSS and JS includes in the <head>. Oh my god!