Robots.txt Generator

Robots.txt is important for SEO because it guides search engine crawlers on which pages or sections of a website to crawl and index, helping to manage crawl budget and protect sensitive or duplicate content from being indexed.

Preview

Help

This tool helps you generate a `robots.txt` file to control the crawling of web pages by search engine robots. Follow the instructions below to create your `robots.txt` file.

Instructions

Add Rules: Click on the "Add Rule" button to add a new rule for a specific user-agent. A user-agent is a specific search engine bot, such as Googlebot or Bingbot. You can also use * to target all bots.
Fill in Details: For each rule, specify the user-agent, and the paths you want to allow or disallow. The 'Allow' field lets you specify which pages can be crawled, while the 'Disallow' field lets you specify which pages cannot be crawled.
Remove Rules: If you need to remove a rule, click on the red "Remove Rule" button associated with that rule.
Generate Preview: Click on the "Generate" button to see a preview of your `robots.txt` file in the preview section on the right.
Download File: Once you are satisfied with your `robots.txt` file, click on the "Download" button to download the file.

Example

Below is an example of a `robots.txt` file:

User-agent: *
Disallow: /private/
Allow: /public/

User-agent: Googlebot
Disallow: /no-google/
Allow: /

In this example:

All user-agents are disallowed from accessing the `/private/` directory but can access the `/public/` directory.
Googlebot is disallowed from accessing the `/no-google/` directory but can access all other directories.