How do you stop Google indexing you robots.txt ?
I don’t want users poking around in the Folders I have banned Google from :
take webmasterworld ( actually I’m with Brett on this one )
Brett cloaks his robots.txt :
http://www.webmasterworld.com/robots.txt
But Google has to get the real one :
Google Cached Copy of WMW robots.txt
How do I stop Google showing my Robots.txt to the world
DaveN
Firewall Script I'm going to do a full write up on this, but I have been testing this out on a blog, that's been hacked and DDOS a few times over the last 6 months, why do people feel I want to host their Malware, why don't I just update WP time and effort I guess
Anyway I had been looking for a solution that i can install on a shared box that that I didn't have root access on.
Anyway My test is coming to an end, and I going to try and score a deal on a multiple domain license, or even try and buy a lump of the company lol
21 Comments | Leave a comment »
can you not password protect it in th .htaccess file at all? or will that also block the spiders
Dave - why don’t you just cloak? or are you more asking in the general sense for less sophisticated webmaster?
X-Robots HTTP tag ?
@quadszilla yer more general webmasters and also if you mention Cloaking to a Corp Webmaster they tend to poo their pants a little,
DaveN
Perhaps disallow robots.txt file in the robots.txt file so the robots.txt file isn’t indexed;-)
It’s not cloaking, it’s personalization!
The only way to stop Google indexing stuff without exposing yourself is to properly protect the areas you’re hiding.
That means in library folders use with “deny from all” in a .htaccess, or use some sort of login system (possibly HTTP Basic), or restrict by IP.
If memory serves me correct then oracle.com do simple cloaking based on user agent. Pretty good example to give to corporate client I reckon :)
Couldn’t you just use x-robots tag in the header to stop Google indexing the file?
Double blind also works.
Use
Disallow: /foo
in the robots.txt file.
Inside that folder set DirectoryIndex to OFF so that none of the filenames can be listed.
Inside that folder create a folder: /foo/bar/
Again set DirectoryIndex to off for that sub-folder.
Put the stuff you don’t want to be found inside that sub-folder.
In robots.txt be moe clever as to what you disallow, in how you define it.
You have a folder called /scripts/.
Normally you would put:
Disallow: /scripts
in the robots.txt file.
Instead, call the folder /scripting462782/ or some such number.
Make sure that is the only folder name that begins with /scr
In the robots.txt file use
Disallow: /scr
That will disallow any URL that begins /scr without exposing the real path name.
g1smd: http://en.wikipedia.org/wiki/Security_through_obscurity
So anyone find a solution to this?
Is that agreeing with me, or pointing to a problem?
Just put:
Options -Indexes
into your htaccess file and it will block folks from getting into any directory that doesn’t have an index file in it.
That doesn’t keep your robots file from being Googled, but it does solve the problem of folks “poking around in your directories”, so the initial desire is fullfilled.
Use the if’s in the htaccess if condition user agent blah blah serve the file blah blah…
Maybe you guys can look up the apache syntax?
But a good proxy will by pass it, so if you want to see Bert’s robot.txt just write a proxy to pretend to be Google…
Guess JohnMu has one to lend you..
Dave, forget about asking JohnMu for a proxy script, I have just learned he is one of them now…
So all his hacking has finally payed off and he has been rewarded as the new Googler…
Well lose some win some.
Maybe he will make sure none of our sites are deindexed inadvertently by Google. :)
But if you are interested in reading a cloked robots.txt with a proxy, look into curl_init in PHP
So what is the verdict? Did anyone test and find a perfect solution to this?
You can disallow your robots.txt in your robots.txt:
user-agent: *
disallow: /robots.txt
That will keep it from getting crawled and will prevent a “cached” link from appearing (if it is shown in the index anyway). Google will still be able to access it. If you want to prevent users from seeing it as well, you can use the bot identification setups available for the major search engine bots and only serve them the real file.
May Just use a 301 redirect.