Robots.txt
How do you stop Google indexing you robots.txt ?
I don’t want users poking around in the Folders I have banned Google from :
take webmasterworld ( actually I’m with Brett on this one )
Brett cloaks his robots.txt :
http://www.webmasterworld.com/robots.txt
But Google has to get the real one :
Google Cached Copy of WMW robots.txt
How do I stop Google showing my Robots.txt to the world
DaveN
TheShaneDJShow 2105 days ago
http://www.shanedcan you not password protect it in th .htaccess file at all? or will that also block the spiders
quadszilla 2105 days ago
http://seoblackhat.comDave – why don’t you just cloak? or are you more asking in the general sense for less sophisticated webmaster?
JohnMu 2105 days ago
X-Robots HTTP tag ?
DaveN 2105 days ago
@quadszilla yer more general webmasters and also if you mention Cloaking to a Corp Webmaster they tend to poo their pants a little,
DaveN
Shimon Sandler 2105 days ago
http://www.shimonsandler.comPerhaps disallow robots.txt file in the robots.txt file so the robots.txt file isn’t indexed;-)
JohnMu 2105 days ago
It’s not cloaking, it’s personalization!
Rob Haswell 2105 days ago
The only way to stop Google indexing stuff without exposing yourself is to properly protect the areas you’re hiding.
That means in library folders use with “deny from all” in a .htaccess, or use some sort of login system (possibly HTTP Basic), or restrict by IP.
Richard Hearne 2105 days ago
http://www.redcardinal.ieIf memory serves me correct then oracle.com do simple cloaking based on user agent. Pretty good example to give to corporate client I reckon
Couldn’t you just use x-robots tag in the header to stop Google indexing the file?
g1smd 2104 days ago
Double blind also works.
Use
Disallow: /foo
in the robots.txt file.
Inside that folder set DirectoryIndex to OFF so that none of the filenames can be listed.
Inside that folder create a folder: /foo/bar/
Again set DirectoryIndex to off for that sub-folder.
Put the stuff you don’t want to be found inside that sub-folder.
g1smd 2104 days ago
In robots.txt be moe clever as to what you disallow, in how you define it.
You have a folder called /scripts/.
Normally you would put:
Disallow: /scripts
in the robots.txt file.
Instead, call the folder /scripting462782/ or some such number.
Make sure that is the only folder name that begins with /scr
In the robots.txt file use
Disallow: /scr
That will disallow any URL that begins /scr without exposing the real path name.
Rob Haswell 2102 days ago
g1smd: http://en.wikipedia.org/wiki/Security_through_obscurity
Click Input 2102 days ago
http://www.clickinput.comSo anyone find a solution to this?
g1smd 2102 days ago
Is that agreeing with me, or pointing to a problem?
DWR 2098 days ago
Just put:
Options -Indexes
into your htaccess file and it will block folks from getting into any directory that doesn’t have an index file in it.
DWR 2098 days ago
http://deletedThat doesn’t keep your robots file from being Googled, but it does solve the problem of folks “poking around in your directories”, so the initial desire is fullfilled.
Igor The Troll 2093 days ago
Use the if’s in the htaccess if condition user agent blah blah serve the file blah blah…
Maybe you guys can look up the apache syntax?
Igor The Troll 2093 days ago
But a good proxy will by pass it, so if you want to see Bert’s robot.txt just write a proxy to pretend to be Google…
Guess JohnMu has one to lend you..
Igor The Troll 2093 days ago
Dave, forget about asking JohnMu for a proxy script, I have just learned he is one of them now…
So all his hacking has finally payed off and he has been rewarded as the new Googler…
Well lose some win some.
Maybe he will make sure none of our sites are deindexed inadvertently by Google.
But if you are interested in reading a cloked robots.txt with a proxy, look into curl_init in PHP
SeLvesTr 2051 days ago
http://newyorkforum.usSo what is the verdict? Did anyone test and find a perfect solution to this?
JohnMu 2025 days ago
You can disallow your robots.txt in your robots.txt:
user-agent: *
disallow: /robots.txt
That will keep it from getting crawled and will prevent a “cached” link from appearing (if it is shown in the index anyway). Google will still be able to access it. If you want to prevent users from seeing it as well, you can use the bot identification setups available for the major search engine bots and only serve them the real file.
led display 1818 days ago
http://www.chipshow.comMay Just use a 301 redirect.
GreenWithEnvy 1568 days ago
http://www.console-addicts.com/I don’t know if you are still looking for the answer to this, but I think i know what he does.
He enabled the robots.txt as a php or asp script. This is down through the .htaccess file as seen below:
SetHandler application/x-httpd-php
This allows PHP code in the robots.txt file to be executed. He simply has a PHP script check to see if the user requesting the file is indeed a google bot, and if so display the actual robots.txt data. If not, go ahead and pull the news posts from the databse and display them for users like you to be confused =P
I hope this helps! Please pay me a visit or send an email if you appreciated this answer.
Matt