{"id":340,"date":"2008-08-09T16:38:20","date_gmt":"2008-08-09T20:38:20","guid":{"rendered":"http:\/\/blogs.law.harvard.edu\/zeroday\/2008\/08\/09\/the-never-ending-robotstxt\/"},"modified":"2008-08-09T16:38:20","modified_gmt":"2008-08-09T20:38:20","slug":"the-never-ending-robotstxt","status":"publish","type":"post","link":"https:\/\/archive.blogs.harvard.edu\/zeroday\/2008\/08\/09\/the-never-ending-robotstxt\/","title":{"rendered":"The never ending robots.txt"},"content":{"rendered":"<p>While looking over logs for a server of mine I decided to write some code that would help me deter someone sniffing my server for weaknesses.  The first thing I decided to write was a robots.txt file that had a few different qualities.<br \/>\n1) It would never end<br \/>\n2) It would not bog down the CPU<br \/>\n3) It would not repeat<br \/>\n4) It would be a valid robots.txt file<\/p>\n<p>At the time I was using PHP for another project and so it was already configured on my server.  I reused a password generation function and stuck it in a time delayed infinite loop.  Then I changed the .txt handler on the server to be PHP.<br \/>\n<code><\/p>\n<p>@header(\"Content-Type: text\/plain\");<br \/>\n@header(\"Pragma: no-cache\");<br \/>\n@header(\"Expires: 0\");<br \/>\n$standardStatement = \"User-agent: * \\n\";<br \/>\nprint $standardStatement;<\/p>\n<p>function randpass() {<br \/>\n $chars = \"1234567890abcdefGHIJKLMNOPQRSTUVWxyzABCDEFghijklmnopqrstuvwXYZ1234567890\";<br \/>\n $thepass = '';<br \/>\n for($i=0;$i&lt;11;$i++)<br \/>\n {<br \/>\n  $thepass .= $chars{rand() % 39};<br \/>\n }<\/p>\n<p> return $thepass;<br \/>\n}<\/p>\n<p>while (true):<br \/>\n        $newpath = randpass();<br \/>\n        print \"Disallow: \/$newpath\\n\";<br \/>\n        usleep(6000);<br \/>\nendwhile;<br \/>\n<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"<p>While looking over logs for a server of mine I decided to write some code that would help me deter someone sniffing my server for weaknesses. The first thing I decided to write was a robots.txt file that had a few different qualities. 1) It would never end 2) It would not bog down the [&hellip;]<\/p>\n","protected":false},"author":214,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[273],"tags":[],"class_list":["post-340","post","type-post","status-publish","format-standard","hentry","category-non-sequiter"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/archive.blogs.harvard.edu\/zeroday\/wp-json\/wp\/v2\/posts\/340","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive.blogs.harvard.edu\/zeroday\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive.blogs.harvard.edu\/zeroday\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/zeroday\/wp-json\/wp\/v2\/users\/214"}],"replies":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/zeroday\/wp-json\/wp\/v2\/comments?post=340"}],"version-history":[{"count":0,"href":"https:\/\/archive.blogs.harvard.edu\/zeroday\/wp-json\/wp\/v2\/posts\/340\/revisions"}],"wp:attachment":[{"href":"https:\/\/archive.blogs.harvard.edu\/zeroday\/wp-json\/wp\/v2\/media?parent=340"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/zeroday\/wp-json\/wp\/v2\/categories?post=340"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/zeroday\/wp-json\/wp\/v2\/tags?post=340"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}