Saturday, November 26, 2005

Robots.txt

Was searching for something and randomly came across the White House's robot.txt file which discourages search engines and archive.org from searching/providing a cache of the web pages in it. No idea if this is a new exciting discovery.

Interesting.