Twitter adds more Nofollows and robots.txt

by David Naylor
Bronco - Digital Marketing Agency
See Dave Speak SMX London

Today Twitter added Nofollows to trending topics

twitter-trending

and they have INCORRECTLY added a Robots.txt to m.twitter.com to stop external links been indexed,

m-twitter

you can clearly see the web link is not nofollowed

so what they have done is added :

# Repel the robot invasion of sub-domains
User-agent: *
Disallow: /

which is pretty dumb! what they should have done is :

m.twitter.com
All of Twitter's content is currently available in the organic rankings via twitter.com and m.twitter.com. they could have corrected this so the mobile version of the site is only available from Google Mobile and also that Twitter.com isn't available from Google Mobile.

The solution is to block GoogleBot-Mobile indexing twitter.com by placing the following in the "twitter.com/robots.txt" file:

User-agent: Googlebot-Mobile
Disallow: /

Then they should block GoogleBot from m.twitter.com by placing the following in the "m.twitter.com/robots.txt", but still allow the Googlebot-Mobile:

User-agent: Googlebot
Disallow: /

User-agent: Googlebot-Mobile
Allow: /

Dave

Bronco - Digital Marketing Agency
Making your inbox more interesting
Looking to keep up to date, or find out those things we can’t mention on the blog? Then sign up to our semi-regular newsletter. Don’t worry, we won’t spam you.

18 Comments

Get in Touch

Things are better when they’re made simpler. That’s why the David Naylor blog is now just that; a blog. No sales pages, no contact form - just interesting* info about SEO.

If you’d like to find out more about the Digital Marketing services we do provide then head over to Bronco (our main company website) to get in touch.

Get in Touch Today * Interestingness not guaranteed
Part of the Bronco family