A robots.txt is a text file that instructs web robots on how to crawl pages on a website. The robots.txt file belongs to the robots exclusion protocol (REP), which is a group of web standards that oversee how robots crawl the web, including accessing and indexing content.

The robots.txt indicates to user agents if they can or cannot crawl certain areas of a website. These crawling parameters are indicated through “allow” or “disallow”. Below is an example of what a robots.txt file looks like:

How Do robots.txt and Search Engine Optimization Correlate?

When a web robot comes to a website to crawl and index the site it will first look for a robots.txt file. The robots.txt serves as a road map to bots on how they should crawl and index the website and if there are any resources on the website that should be excluded. Therefore, a robots.txt file provides the necessary information for web bots to effectively and efficiently crawl and index a website.

Where Should A robots.txt File Be Placed On A Website?

It’s imperative that a robots.txt file gets added to the top-level category. An example of this would be: https://www.ecreativeworks.com/robots.txt. It’s also important to know that robots.txt files are case sensitive and have to be exactly as follows: /robots.txt.

Crawling Parameters of robots.txt Files

As mentioned earlier, there are two types of crawling parameters for robots.txt:

• Disallow: prevents web robots from crawling specific resources on a website
  • Example: Disallow: /admin
• Allow: indicates to web robots that specific resources should be included when crawling and indexing the website
  • Example: Allow:/images

Crawling parameters exist so there is no ambiguity when a web robot comes to crawl a website. The Disallow: /admin example above is a prime example of this. A web robot crawling the admin page of a website is not going to help SEO efforts and won’t serve users any valuable information by crawling it. Therefore, by disallowing the /page resource a web robot will not waste valuable time crawling unnecessary resources on the website.

The other important item that should be included in a robots.txt file is the URL for the website’s XML sitemap. Since the robots.txt provides the road map for what resources a web bot should crawl and index on the website it’s crucial to have the XML sitemap(s) listed so there’s a clear navigation pathway for what pages should be crawled. In the example below there are seven sitemaps listed in the robots.txt file.

How To Optimize a robots.txt file for Search Engine Optimization

When just starting on the long journey of SEO for a website the robots.txt is one of the first places that should be checked and tested. If your site has critical resources that are marked as disallowed in the robots.txt that is a problem because web bots can’t crawl them.

A quick way to test for blocked resources within the robots.txt is the ‘Inspect URL’ tool in Google Search Console. This tool will show what resources are being blocked. If there are resources that are being blocked and shouldn’t be, it’s crucial to update the robots.txt file so a web bot has full access to the resources they need to accurately crawl and index the website.

Contact Ecreative

Our ECW team provides comprehensive digital marketing strategies that help business owners navigate the ever-changing requirements of Google and other search engines. Contact us for more information on our digital marketing program and options – and how we can help optimize your website’s robots.txt.