Robots.txt Is really a file that exists on the foundation directory of each Site and can be used to instruct search engines like google and yahoo on which directories/documents of the website they might crawl and incorporate of their index. This is often what search engines like yahoo use to https://news.rafeeg.ae