Robots.txt Generator
Create a robots.txt file for your website.
robots.txt Settings
Generated robots.txt
User-agent: * Disallow: /cgi-bin/
Advertisement
About Robots.txt Generator
Create a robots.txt file for your website to control search engine crawlers. Easily allow or disallow specific bots and directories.
How to Use
- 1
Add Rules
Add Allow and Disallow paths for the user-agents.
- 2
Save
Copy the output and save it as a robots.txt file.
Frequently Asked Questions
What is a robots.txt file?
It is a text file that tells search engine spiders which pages they should and should not crawl on your site.
Where do I upload it?
The robots.txt file must be uploaded to the root directory of your website (e.g., example.com/robots.txt).