> If I cannot block all robots, I want to at least be able to identify them to prevent my script from counting robot-generated page views as being from real people Nginx & Apache2 log the user agent, ip and page accessed. Web crawlers usually add things like "bing bot" to their headers. I still respect your other justifications for blocking the crawlers. edit: OP is using Lighttpd

maybe it was somebody trying to archive the site. I think Wayback Machine also crawls and archives websites

> Unfortunately, I do not record the user agents of robots that do not contain one of the following special strings: "bot", "Bot", "rawl", slurp", "pider", "oogle", "rchive", "acebook", "earch", or "http". This is because I don't log user agents that I think are coming from real people. This speeds up my script and reduces the wear on my storage media Wow you’ve potentially saved literal microseconds and increased the lifetime of your drives by the same amount.