Stay organized with collections
Save and categorize content based on your preferences.
Update your robots.txt file
To update the rules in your existing robots.txt file, download a copy of your robots.txt file
from your site and make the necessary edits.
Download your robots.txt file
You can download your robots.txt file various ways, for example:
Navigate to your robots.txt file, for example https://example.com/robots.txt
and copy its contents into a new text file on computer. Make sure you follow the guidelines
related to the
file format
when creating the new local file.
Download an actual copy of your robots.txt file with a tool like cURL. For example:
curl https://example.com/robots.txt -o robots.txt
Use the
robots.txt report
in Search Console to copy the content of your robots.txt file, which you can then paste into a
file on your computer.
Edit your robots.txt file
Open the robots.txt file you downloaded from your site in a text editor and make the necessary
edits to the rules. Make sure you use the
correct syntax
and that you save the file with UTF-8 encoding.
Upload your robots.txt file
Upload your new robots.txt file to the root directory of your site as a text file named
robots.txt. The way you upload a file to your site is highly platform and server dependent. Check
out our tips for finding help with
uploading a robots.txt file to your site.
Refresh Google's robots.txt cache
During the automatic crawling process, Google's crawlers notice changes you made to your
robots.txt file and update the cached version every 24 hours. If you need to update the
cache faster, use the Request a recrawl function of the
robots.txt report.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2024-10-31 UTC."],[[["The robots.txt file controls which parts of your website search engines can crawl and index."],["You can edit your robots.txt file by downloading it, making changes, and uploading it back to your website's root directory."],["Website hosting services may offer alternative ways to manage search engine crawling, such as through their settings pages."],["Google automatically updates its cached version of your robots.txt file, but you can request a faster refresh using Search Console."]]],[]]
