Topic: robots.txt and security

[Danll]

It occurs to me that the robots.txt file, which is intended, in part, to tell search engines NOT to catalog certain files, is publicly available (which is how the search engines get it).

So anyone can download my robots.txt file, and determine which files I don't want advertised on search engines. Them I suppose, they can go get those files.

In other words, it's trivial for people to figure out exactly which files I don't want widely advertised. So that means that, if someone knows where your site is, they know what you don't want the search engines to catalog.

That does seem a bit amusing, and pretty much assures that the robots.txt file doesn't have anything to do with what one might consider "security".

[glsmith]

No, nor is security it's intention. Security by obscurity has never been good security to begin with.

[Danll]

Yes, "security by obscurity" (I like that phrase!) is a poor construct.

Of course, it is kind of funny that by telling search engines what you don't want people to see, you're telling people what you don't want them to see.

What you're really telling search engines is that you don't want people to be directed to certain things. But by doing so, you're advertising what those certain things are, and where people can find them.

[glsmith]

Come to think of it, we're in the 21st century and this exact concern is probably what brought about some changes to search engines. For instance say you have subdirectories "this", "that" and "other". All you want indexed is "this", "other" and your main index page. Today many of the major search engines accept this;

[James Blond]

it is a good strategie to block google for folder that contains images which shall not be indexed. All other stuff like downloads, folder, etc should be protected via apache config and or your software / code.
Often bad people find your stuff by taking a look at the robots.txt
With many files from a CMS I wouldn't go the strategie from glsmith.

[Danll]

[glsmith]

James' last line of his comment did get me thinking, especially since most CMS' use search engine friendly URLs. With a little config ingenuity, one could have a dynamic robots.txt file.

http://funwithrobots.linkpc.net:88

Hit the robots.txt file in your browser and see what it gives you. If you have Firefox and the User Agent Switcher extension (or similar), go again telling the server you are Googlebot. You will certainly see something different. Find the prize hidden inside.

Of course, there is still no real security here as anyone can do same. If you must completely secure something, don't put it on the server at all.

Password protect behind https at least if it must be available. No spider will index it then

[James Blond]

Well a dynamic robots.txt will cause a sinking in google ranking if not kicking. Sometimes the google bot does not come with the google IP nor the google bot User Agent to check cloaking.

BUT with the CMS it will be possible to create a white list. That is an idea I like. Disallow everything and have white list from the CMS.

--- EDIT ---
Since more or less it is a modified sitemap.