To stay away from unwanted material during the research indexes, webmasters can instruct spiders to not crawl selected files or directories throughout the common robots.txt file in the foundation Listing of the domain. On top of that, a web site can be explicitly excluded from a online search engine's databases by making use of a meta tag particula