Twitter adds more Nofollows and robots.txt
Today Twitter added Nofollows to trending topics
and they have INCORRECTLY added a Robots.txt to m.twitter.com to stop external links been indexed,
you can clearly see the web link is not nofollowed
so what they have done is added :
# Repel the robot invasion of sub-domains
User-agent: *
Disallow: /
which is pretty dumb! what they should have done is :
m.twitter.com
All of Twitter's content is currently available in the organic rankings via twitter.com and m.twitter.com. they could have corrected this so the mobile version of the site is only available from Google Mobile and also that Twitter.com isn't available from Google Mobile.
The solution is to block GoogleBot-Mobile indexing twitter.com by placing the following in the "twitter.com/robots.txt" file:
User-agent: Googlebot-Mobile
Disallow: /
Then they should block GoogleBot from m.twitter.com by placing the following in the "m.twitter.com/robots.txt", but still allow the Googlebot-Mobile:
User-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Allow: /
Dave
18 Comments
Becky - http://www.beckynaylor.co.uk
Who is giving twitter all this incorrect and half thought out advice? Do they actually have an “seo expert” on board or are they just making this strategy up as they go along?
Allan Stewart - http://www.fireflyseo.com
Owch. I bet their inhouse SEO feels like a right cock now.
Allan Stewart - http://www.fireflyseo.com
Also, I am not 100% convinced that they need…
Disallow: /*?
Disallow: /*/with_friends
In the G, Y! and MSN if they have a generic User-agent: * which “at a guess” will probably sweep all search engines with those Disallows. Code repetition perhaps?
Dave/Team, can you advise?
David Fairhurst - http://www.intelligentretail.co.uk
LOL… Probably are more worried about spammers / crackers while you’re constantly outing their app’s security flaws Dave. Definitely a bit of a schoolboy error! @fireflyseo still makes me giggle with his totally un-PC comments… brilliant!
Bronson - http://www.bronsonharrington.com
Nice one Dave, you rock.
Well spotted and great to see an adequate, workable resolution.
Also, thanks for the optimization tips when it comes to mobile vs. web domains.
James
Looks like (as the comment in there implies) it’s across a bunch of their subdomains:
http://assets0.twitter.com/robots.txt
http://assets1.twitter.com/robots.txt
http://assets2.twitter.com/robots.txt
http://jobs.twitter.com/robots.txt
http://api.twitter.com/robots.txt
Were a few I found the same. http://apiwiki.twitter.com/robots.txt has the interesting line:
Sitemap: http://twitterapiapiwiki.twitter.com/sitemap.xml
Which seems to be wrong….
Brent - http://einfo.blogspot.com
Quote from Biz on their search optimization strategy… ” our neighbor kid’s doing it for us.”
John - http://www.twitter.com/omfg_followme
“Owch. I bet their inhouse SEO feels like a right cock now.” That’s what she said.
David W
lol John I like your twitter username m8
John - http://www.twitter.com/omfg_followme
Cheers. It seems to be an effective CTA. 🙂
Twitter adds more Nofollows and robots.txt | Complete Twitter - pingback
[…] here to read the rest: Twitter adds more Nofollows and robots.txt Tweet This Or Share Through Other Social […]
SearchCap: The Day In Search, September 17, 2009 - pingback
[…] Twitter adds more Nofollows and robots.txt, David Naylor […]
Gareth James - http://www.seo-doctor.co.uk/
Thats 2 major things you’ve pointed out in about 2 weeks. Listen out for that Headhunter Dave, Twitter is coming for you.
Jason - http://motiv-designs.com
Wow that is a pretty huge oversight, or someone is doing something odd. The nofollowing of those trending topics seems odd also. This might be a stretch, but perhaps they are trying to make people have to search for things on their site and just taking other engines out of the picture.
Bryan Casson - http://seocowboy.co.za
Once again you make perfect sense David.
Combson's Yachts - http://www.yachts123.com
More and more services are joining the bandwagon of using nofollow links. Twitter joined Digg and some other big authority websites…
Search & Social Weekly Wrap-Up 9/25 - pingback
[…] similar news, Twitter has added the no-follow tag to its “trending topics” links while also adding robots.txt commands to disallow the indexing of most of its subdomains […]
Lorenz - http://onlinedesignbureau.com
Well spotted. Evan and Biz should put you on their payroll.