0 posts · 1 comment · 1 total
Active in: v/programming (1)
Not an issue if you properly utilize permissions on your web server. But yes, I actually made a search engine in college that scraped robots.txt files.
Not an issue if you properly utilize permissions on your web server. But yes, I actually made a search engine in college that scraped robots.txt files.