Among the ignored fields, the example of crawl-delay is hungary phone number given . What do these four fields do? User-agent is used to identify which crawler the rules that are then listed refer to. The documentation, for example, reminds us that this field is not case-sensitive.
The allow and disallow fields must be completed with a path regarding the contents that should or should not be made accessible to crawlers. What is in these two fields is instead, and the documentation always reminds us, susceptible to the use of uppercase and lowercase letters .
Google Returns to Talk About the Robots.txt File – sos-wp.it
Finally, there is the sitemap field which is also case sensitive and is supported by the vast majority of search engines. If you have other fields entered and counted beyond these four, know that Google bots ignore them. Among the other fields that are therefore ignored are nofollow , which Google has never officially declared to support, and noindex , which the big G company has always discouraged the use of.
Best Practices for a Working Robots.txt File
The clarification provided by Google also helps us better understand how to build a robots.txt file that makes the most of the supported fields and that works for SEO. As we have already mentioned, its use allows bots to index only what you actually want to be indexed. Indexed means that it then emerges if a user performs a certain online search.
The need to block access to some sections of your site can occur for various reasons. For example, it helps to manage when there are pages that could be perceived as duplicates, because they have the same structure, same elements but slightly different language. A good robots.txt file also has the advantage of allowing you to manage bot traffic .
If there is too much attention from bots on your site, you could exceed your budget and then have problems when real users try to navigate. But be careful how you decide to manage links to resources that you do not want indexed. Those that are blocked in turn block the potential value of other links that are within the content.