Google Dupe Content

I have seen a lot of sites get washed recently due to Googles dupe content filters, the most common one is the MOD_Rewrite, yer go figure.. the amount of times I have told people to use a mod_rewrite, was it bad advice ?? well it didn’t used to be, So now I have added this statement to my “use a Mod_Rewrite”..

Ok Mr Matt Cutts .. first what you need to do is get rid of those Urls that look like this:
http://www.mattcutts.com/blog/?p=16 and replace them with
http://www.mattcutts.com/blog/up-up-up-up-up/ .. but Matt.. what you must do is have a robots.txt .. in that robots. txt file add this little line in ..

User-agent: *
Disallow: /?p

I think that’s right … i have never really wanted to stop SE’s spiders getting in before.. also watch out for “PRINT Article” links if they go to the same content with a different CSS.. you could get into trouble with that too.. oh hum

DaveN

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • Live
  • StumbleUpon
  • Facebook
  • Google
  • Reddit
  • Technorati
  • Mixx

5 Comments | Leave a comment »

  1. 1. Matt Cutts | October 7th 2005 @ 4:47 pm

    I *want* to see how well engines handle it, and how to fix it when I notice a problem. :)

  2. 2. BroadProspect | October 8th 2005 @ 6:47 pm

    IMHO, this is a huge problem right now, there are TONS of scrapper sites out there, targeting most of the times the top SERPs, so it may be that the real solution is to cloack pages and give GBot and yBot and msBot diffrent content then the one we give the user …

  3. 3. Mikkel deMib Svendsen | October 11th 2005 @ 11:40 pm

    > so it may be that the real solution is to cloack pages and give GBot and yBot and msBot diffrent content then the one we give the user …

    Thats called “personalization” and is very comon these days :)

  4. 4. Joeduck | October 19th 2005 @ 4:35 am

    Matt! Throw me a bone on fixing OHWY.com which appears to be killed by (unfair!) dupe filtering or we’ll have to hire … DAVE!

    JoeDuck/Joe Hunkins

  5. 5. brad | October 23rd 2005 @ 2:38 pm

    Go get ‘em Dave ;)

    This continues to be an issue with blogging software, and needs to be addressed.

Leave a Reply

required

required, hidden

+ Advertise Here