13 20 May 2015 01:04 by u/gaftra
By trying to hide them from robots, they even made them more visible. Who would guess a random doc file name?
Not an issue if you properly utilize permissions on your web server. But yes, I actually made a search engine in college that scraped robots.txt files.
2 comments
1 u/Lordchump 25 May 2015 14:52
By trying to hide them from robots, they even made them more visible. Who would guess a random doc file name?
1 u/hashbrowns 25 May 2015 21:11
Not an issue if you properly utilize permissions on your web server. But yes, I actually made a search engine in college that scraped robots.txt files.