Twitter adds more Nofollows and robots.txt
Today Twitter added Nofollows to trending topics
and they have INCORRECTLY added a Robots.txt to m.twitter.com to stop external links been indexed,
you can clearly see the web link is not nofollowed
so what they have done is added :
# Repel the robot invasion of sub-domains
User-agent: *
Disallow: /
which is pretty dumb! what they should have done is :
m.twitter.com
All of Twitter’s content is currently available in the organic rankings via twitter.com and m.twitter.com. they could have corrected this so the mobile version of the site is only available from Google Mobile and also that Twitter.com isn’t available from Google Mobile.
The solution is to block GoogleBot-Mobile indexing twitter.com by placing the following in the “twitter.com/robots.txt” file:
User-agent: Googlebot-Mobile
Disallow: /
Then they should block GoogleBot from m.twitter.com by placing the following in the “m.twitter.com/robots.txt”, but still allow the Googlebot-Mobile:
User-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Allow: /
Dave
Becky 1372 days ago
http://www.beckynaylor.co.ukWho is giving twitter all this incorrect and half thought out advice? Do they actually have an “seo expert” on board or are they just making this strategy up as they go along?
Allan Stewart 1372 days ago
http://www.fireflyseo.comOwch. I bet their inhouse SEO feels like a right cock now.
Allan Stewart 1372 days ago
http://www.fireflyseo.comAlso, I am not 100% convinced that they need…
Disallow: /*?
Disallow: /*/with_friends
In the G, Y! and MSN if they have a generic User-agent: * which “at a guess” will probably sweep all search engines with those Disallows. Code repetition perhaps?
Dave/Team, can you advise?
David Fairhurst 1372 days ago
http://www.intelligentretail.co.ukLOL… Probably are more worried about spammers / crackers while you’re constantly outing their app’s security flaws Dave. Definitely a bit of a schoolboy error! @fireflyseo still makes me giggle with his totally un-PC comments… brilliant!
Bronson 1372 days ago
http://www.bronsonharrington.comNice one Dave, you rock.
Well spotted and great to see an adequate, workable resolution.
Also, thanks for the optimization tips when it comes to mobile vs. web domains.
James 1372 days ago
Looks like (as the comment in there implies) it’s across a bunch of their subdomains:
http://assets0.twitter.com/robots.txt
http://assets1.twitter.com/robots.txt
http://assets2.twitter.com/robots.txt
http://jobs.twitter.com/robots.txt
http://api.twitter.com/robots.txt
Were a few I found the same. http://apiwiki.twitter.com/robots.txt has the interesting line:
Sitemap: http://twitterapiapiwiki.twitter.com/sitemap.xml
Which seems to be wrong….
Brent 1372 days ago
http://einfo.blogspot.comQuote from Biz on their search optimization strategy… ” our neighbor kid’s doing it for us.”
John 1372 days ago
http://www.twitter.com/omfg_followme“Owch. I bet their inhouse SEO feels like a right cock now.” That’s what she said.
David Whitehouse 1372 days ago
lol John I like your twitter username m8
John 1372 days ago
http://www.twitter.com/omfg_followmeCheers. It seems to be an effective CTA.
Twitter adds more Nofollows and robots.txt | Complete Twitter 1372 days ago
[...] here to read the rest: Twitter adds more Nofollows and robots.txt Tweet This Or Share Through Other Social [...]
SearchCap: The Day In Search, September 17, 2009 1371 days ago
[...] Twitter adds more Nofollows and robots.txt, David Naylor [...]
Gareth James 1371 days ago
http://www.seo-doctor.co.uk/Thats 2 major things you’ve pointed out in about 2 weeks. Listen out for that Headhunter Dave, Twitter is coming for you.
Jason 1370 days ago
http://motiv-designs.comWow that is a pretty huge oversight, or someone is doing something odd. The nofollowing of those trending topics seems odd also. This might be a stretch, but perhaps they are trying to make people have to search for things on their site and just taking other engines out of the picture.
Bryan Casson 1367 days ago
http://seocowboy.co.zaOnce again you make perfect sense David.
Combson's Yachts 1366 days ago
http://www.yachts123.comMore and more services are joining the bandwagon of using nofollow links. Twitter joined Digg and some other big authority websites…
Search & Social Weekly Wrap-Up 9/25 1364 days ago
[...] similar news, Twitter has added the no-follow tag to its “trending topics” links while also adding robots.txt commands to disallow the indexing of most of its subdomains [...]
Lorenz 1358 days ago
http://onlinedesignbureau.comWell spotted. Evan and Biz should put you on their payroll.