A robots.txt file is a way for a site administrator to give instructions to web crawlers (robots) that are crawling the site.
Posted on July 27, 2010, at 4:35 p.m. ET
They are not usually anywhere near this funny.
Editorial Director, God King of BuzzFeed
Contact Jack Shepherd at firstname.lastname@example.org.
Got a confidential tip? Submit it here.