Robots.txt is used as part of the Robots Exclusion Standard, a voluntary protocol the Internet Archive respects that disallows bots from indexing certain pages delineated by its creator as off-limits. As a result, the Internet Archive has rendered unavailable a number of web sites that now are inaccessible through the Wayback Machine. Currently, the Internet Archive applies robots.txt rules retroactively; if a site blocks the Internet Archive, such as Healthcare Advocates, any previously archived pages from the domain are rendered unavailable as well. In cases of blocked sites, only the robots.txt file is archived.

What kind of protocol is the Robots Exclusion Standard?