Where should I upload the robots.txt file?

The robots.txt file must be placed in the top-level directory (root) of your website host. For example: https://example.com/robots.txt.

Does robots.txt hide my pages from Google?

No. Robots.txt only prevents crawling. If a page is already indexed or linked to by other sites, it may still appear in search results. To truly hide a page, use a 'noindex' meta tag.

Can I have multiple robots.txt files?

No, a website can only have one robots.txt file located at the root. Bots will only look for the file at that specific location.

Robots.txt Generator

Default User-agent (Global)

Crawl Delay (Seconds)

Sitemap URL (Optional)

Restricted Directories (Disallow Paths)

User-agent: * Disallow:

What is a robots.txt file?

A robots.txt file tells search engine crawlers (like Googlebot) which pages or files they can or can't request from your site. This is used mainly to avoid overloading your site with requests.

Common Rules:

User-agent: * means the rule applies to all crawlers.
Disallow: /admin/ prevents crawlers from indexing your admin dashboard.
Sitemap: helps crawlers find your sitemap quickly.