Twitter adds more Nofollows and robots.txt

by David Naylor
Bronco - Digital Marketing Agency

Today Twitter added Nofollows to trending topics


and they have INCORRECTLY added a Robots.txt to to stop external links been indexed,


you can clearly see the web link is not nofollowed

so what they have done is added :

# Repel the robot invasion of sub-domains
User-agent: *
Disallow: /

which is pretty dumb! what they should have done is :
All of Twitter's content is currently available in the organic rankings via and they could have corrected this so the mobile version of the site is only available from Google Mobile and also that isn't available from Google Mobile.

The solution is to block GoogleBot-Mobile indexing by placing the following in the "" file:

User-agent: Googlebot-Mobile
Disallow: /

Then they should block GoogleBot from by placing the following in the "", but still allow the Googlebot-Mobile:

User-agent: Googlebot
Disallow: /

User-agent: Googlebot-Mobile
Allow: /


Bronco - Digital Marketing Agency
Making your inbox more interesting
Looking to keep up to date, or find out those things we can’t mention on the blog? Then sign up to our semi-regular newsletter. Don’t worry, we won’t spam you.


Get in Touch

Things are better when they’re made simpler. That’s why the David Naylor blog is now just that; a blog. No sales pages, no contact form - just interesting* info about SEO.

If you’d like to find out more about the Digital Marketing services we do provide then head over to Bronco (our main company website) to get in touch.

Get in Touch Today * Interestingness not guaranteed
Part of the Bronco family