Today Twitter added Nofollows to trending topics
and they have INCORRECTLY added a Robots.txt to m.twitter.com to stop external links been indexed,
you can clearly see the web link is not nofollowed
so what they have done is added :
# Repel the robot invasion of sub-domains
User-agent: *
Disallow: /
which is pretty dumb! what they should have done is :
m.twitter.com
All of Twitter’s content is currently available in the organic rankings via twitter.com and m.twitter.com. they could have corrected this so the mobile version of the site is only available from Google Mobile and also that Twitter.com isn’t available from Google Mobile.
The solution is to block GoogleBot-Mobile indexing twitter.com by placing the following in the “twitter.com/robots.txt” file:
User-agent: Googlebot-Mobile
Disallow: /
Then they should block GoogleBot from m.twitter.com by placing the following in the “m.twitter.com/robots.txt”, but still allow the Googlebot-Mobile:
User-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Allow: /
Dave
18 Comments
-
- 2
Owch. I bet their inhouse SEO feels like a right cock now.
- 3
Also, I am not 100% convinced that they need…
Disallow: /*?
Disallow: /*/with_friendsIn the G, Y! and MSN if they have a generic User-agent: * which “at a guess” will probably sweep all search engines with those Disallows. Code repetition perhaps?
Dave/Team, can you advise?
- 4
LOL… Probably are more worried about spammers / crackers while you’re constantly outing their app’s security flaws Dave. Definitely a bit of a schoolboy error! @fireflyseo still makes me giggle with his totally un-PC comments… brilliant!
- 5
Nice one Dave, you rock.
Well spotted and great to see an adequate, workable resolution.
Also, thanks for the optimization tips when it comes to mobile vs. web domains.
- 6
Looks like (as the comment in there implies) it’s across a bunch of their subdomains:
http://assets0.twitter.com/robots.txt
http://assets1.twitter.com/robots.txt
http://assets2.twitter.com/robots.txt
http://jobs.twitter.com/robots.txt
http://api.twitter.com/robots.txtWere a few I found the same. http://apiwiki.twitter.com/robots.txt has the interesting line:
Sitemap: http://twitterapiapiwiki.twitter.com/sitemap.xml
Which seems to be wrong….
- 7
Quote from Biz on their search optimization strategy… ” our neighbor kid’s doing it for us.”
- 8
“Owch. I bet their inhouse SEO feels like a right cock now.” That’s what she said.
- 9
lol John I like your twitter username m8
- 10
Cheers. It seems to be an effective CTA.
- 11
[...] here to read the rest: Twitter adds more Nofollows and robots.txt Tweet This Or Share Through Other Social [...]
- 12
[...] Twitter adds more Nofollows and robots.txt, David Naylor [...]
- 13
Thats 2 major things you’ve pointed out in about 2 weeks. Listen out for that Headhunter Dave, Twitter is coming for you.
- 14
Wow that is a pretty huge oversight, or someone is doing something odd. The nofollowing of those trending topics seems odd also. This might be a stretch, but perhaps they are trying to make people have to search for things on their site and just taking other engines out of the picture.
- 15
Once again you make perfect sense David.
- 16
More and more services are joining the bandwagon of using nofollow links. Twitter joined Digg and some other big authority websites…
- 17
[...] similar news, Twitter has added the no-follow tag to its “trending topics” links while also adding robots.txt commands to disallow the indexing of most of its subdomains [...]
- 18
Well spotted. Evan and Biz should put you on their payroll.
Who is giving twitter all this incorrect and half thought out advice? Do they actually have an “seo expert” on board or are they just making this strategy up as they go along?