BuzzFeed News

Reporting To You

tech

The Best robots.txt File

A robots.txt file is a way for a site administrator to give instructions to web crawlers (robots) that are crawling the site.

Posted on July 27, 2010, at 4:35 p.m. ET

ADVERTISEMENT
ADVERTISEMENT
ADVERTISEMENT
ADVERTISEMENT
ADVERTISEMENT