Google ignores Bings robots.txt ?

I’ve seen Google “ignore” robots.txt exclusions on more than one occasioned when the target is heavily linked (e.g., Bing). I have an entirely excluded subdomain that’s a PR5!

Robots.txt is not the REP which excludes indexing.

In fact excluding robots using robots.txt is the worst thing to do if you want to stop it being indexed.

If they had used X-ROBOTS-HEADERS or a meta robots tag then it would a fair point, but since the only example with a snippet is the url not excluded using robots.txt it appears that Google absolutely IS obeying bing’s REP

You know the drill if Google wants something it gets it!

I do find it funny that it follows the robot text for “doubleclick ad sever” http://ad.doubleclick.net/robots.txt .

Maybe it just a Bing thing!

Are you still digging Bing Dave?

Yep – Bing has used Robots.txt to prevent Googlebot from reading the contents of their search result pages, hence there being no titles or descriptions in your screenshot. Bing should use meta noindex if they want to keep their search results out of the index.

I agreew with manley that google is obaying the rules but in this case the should filter the pages out of the search results as ranking another search engines result at number 11 for a key phrase isn’t a great result for a searcher on Google.

Simlar thing happened with wolfram alpha results in google – http://www.aukseo.com/wolfman-alpha-results-appearing-in-google-results-462/

Google’s stated in the past that they will include URLs in their index if others link to them and they are blocked by robots.txt.

As Manley pointed out, the fact that there’s just the URL and no description is typical of this behavior.

The only way to exclude these URLs from displaying in the index is to tell Google to NOINDEX. Of course, Google has to be able to access & read the page to get the header or meta instructions.

Apparently even with and Google can still find your pages. The way to avoid them being indexed is using the new rel=canonical tag.

@auskeo – But then Google and Wolfram have a special relationship – Sergey Brin was an intern there.

@Lou_geek the canonical link element is not going to work as an REP. Yes, you can suggest the exclusion of a duplicate folder or sub domain, but it does not work across domains and there has to be a canonical page. Even if all the /search?q= pages had a canonical link element pointing back at /search, there very little hope that the context of the pages linking into or the content of those pages will be similar enough for Google to consider that the webmaster included indicator is accurate enough to apply it.

Another possibility is that Bing just now noticed Google indexing all those pages and threw up the robots.txt

This is what Google calls “an uncrawled reference” 🙂

If the pages were also in DMOZ (which they obviously aren’t) then Google would even show a snippet based on that data. Making a perfect search result and nobody would know the difference 😉

You can’t disable Google index pages only with robots.txt – you have to remove all links to page too. It you don’t you can found some external links in Google SERP.

User is on the first place for Google – so user gets the page if place right query 🙂

Very innovative! thanks for sharing the tips which will prove to be very helpful in future as well.
Thanks

how to block robots bings? in htaccess ?

Google ignores Bings robots.txt ?

18 Comments

Get in Touch