Use the form below to generate a robots.txt file for your website. This file gives search engine crawlers instructions about which sections of your site they may access and which paths should be excluded from crawling.
Use these directives carefully. Blocking a page in robots.txt does not guarantee that the URL will disappear from search results, and an incorrect rule can prevent important content from being crawled. In most cases, an Allow: / directive is unnecessary because crawling is allowed by default.
A robots.txt file helps manage how search engine crawlers access different areas of a website. You can use it to discourage crawling of administrative sections, duplicate URLs, internal search results, development areas, or other pages that do not need to be crawled.
You may also include the location of your XML sitemap so search engines can find it more easily. However, robots.txt rules control crawling, not indexing. A blocked URL may still appear in search results if search engines discover it through links or other sources. Use a noindex directive on an accessible page when appropriate, and protect truly private content with authentication rather than relying on robots.txt.
