Google Flexes Robots Exclusion Protocol (Part 2)
Fully removing something from Google still requires the URL removal tool, found as one of Google’s Webmaster Central tools.
Google also extended some control over assets beyond web pages to webmasters. Those who publish PDF, audio, video, or other file types can direct the crawler on how Google should manage access to them from its index.
“We’ve extended our support for META tags so they can now be associated with any file,” said Crow. “Simply add any supported META tag to a new X-Robots-Tag directive in the HTTP Header used to serve the file.”
Supported META tags include options like noarchive, nosnippet, noindex, and unavailable_after. Google sees these as offering enough flexibility to satisfy site publishers; we imagine they have organizations like AFP and Copiepresse in mind here.
Filed under: Webmasters
