The Robots.txt Builder – a new tool
So Dave was at the Robots.txt Summit at SES New York 2007 and it reminded him of a time when a client of his had got himself deindexed by accidentally denying all robots when trying to stop them getting at his RSS Feed! When he got back he got me to do a tool to prevent this stuff from happening in the future, so without further ado DaveN is proud to announce:
All comments, suggestions and bug reports (especially bug reports) are more than welcome. I’d be lying if I said we’d extensively tested this but we’ve given it a fair old runaround in IE and FF and we think it’s pretty solid.
It’s aimed at the lower-end of the userbase. We figured that most people who want to do more advanced or fine-grained robots.txt tuning will probably be able to figure out how to do it themselves, we’re really aiming for the “Mom ‘n’ Pop” market with this one. Let’s hope we don’t have any more incidents like the opening anecdote!
Features:
- Block search engines by type (more types welcome!)
- Supports allowing robots by the standards-defined “Disallow: ” directive
- Warning when you block your entire site
- Site structure import. This lets you import your site structure from Yahoo Site Explorer.
- Easy-to-use point ‘n’ click interface (I hope)
- More JavaScript than you can shake a stick at
Big love out to Yahoo! for their excellent Site Explorer and TreeView YUI widget. You rock, guys.
Enjoy!
PS. Anyone who’s interested, check out the number of CSS and JS includes in the <head>. Oh my god!
Dudibob 2253 days ago
http://www.dudibob.comwow, looks amazing! off to test it
and that’s alot of JSw
Peter 2252 days ago
you really hate google, however neither fuckin wonder when Matt Cutts mentions you on every second blog post without giving you a link. However I dont think you should punish google by sucking up to msn and yahoo. They are bastards aswell.
Rob Haswell 2252 days ago
Dude for a start Dave didn’t write this post, I (one of his programmers) did.
And for the record, I totally suck up to Yahoo because they totally deserve it. Google pulls their search APIs while Yahoo make theirs better and give me access to stuff like their mail! I mean, if you were a programmer, who would you prefer?!
Dave doesn’t really have a habit of writing in the third person, you know.