Add a Robots.txt File and Sitemap
Search engines (such as Google) are constantly indexing and crawling any new website that comes up on the internet. However, sometimes it may index pages in error or take a long time to show new pages in search results.
To speed up the process, you can submit Robots.txt and sitemap files to Google. These files tell Google what pages on your website to index and how your website is structured.
IN THIS ARTICLE
Add a Robots.txt File
A Robots.txt file instructs web robots (such as search engine crawlers) how to crawl the pages on your website. You use it to tell web crawling software what parts of your website to crawl or not to crawl. You can use it to block a page, a subdirectory, or your entire site.
Why Would You Want to Use a Robots.txt File?
In general, most agents want search engines to index their website. There are a few specific reasons why you may want to exclude specific page, such as:
- You don't want your site to be indexed until you've done a compliance review.
- You want to keep specific pages, such as "agent only" or "for existing customers" pages, hidden unless you direct people to them.
- You have duplicate pages on your site and you want to make sure one page ranks.
The bottom line: you only need a robots.txt file if you want to exclude certain pages on your website from being crawled
Set Up Your Robots.txt File
We've designed the AgentMethods platform to make robots.txt files easy to create and use. However, this is a technical aspect of SEO so feel free to reach out to use with any questions or if you run into trouble.
To set up your robots.txt file, start by signing into your AgentMethods account. Click on Website on the left-side menu, and then select Site Settings.
Scroll to the bottom of the page and find the section titled Custom Robots File.
You'll see three options, "Allow", "Block", or "Custom". Choose the one that best fits your needs.
Allow: I want search engines like Google to index all of the pages on my website (If you don't want to hide any pages, select this one!)
Block: I don't want Google to index my website at all (we do not recommend this option: without Google indexing your website, you won't appear in search results!)
Custom: I want to block Google from indexing a few pages on my website that I select
Don't forget to select Save Site Settings at the bottom of the page to save your changes.
Add a Custom Robots.txt file
If you selected "Create a custom robots.txt file", a blank box will appear. In this box, you will input your robots.txt file: a text file that you write containing rules about which parts of your website you'd like Google to crawl.
This part is technical, and we recommend referring to Google's Robots.txt documentation for guidance on how to create your text file.
Once you've input your file, don't forget to select Save Site Settings at the bottom of the page to save your changes.
Warning: Be aware some crawlers may choose to ignore your robots.txt file. This is especially common with more nefarious crawlers like malware robots or email address scrapers.
Add a Sitemap
You can help make sure Google's web crawler finds all of the content on your site by submitting a sitemap. While a robots.txt file tells Google what not to crawl on your website, a sitemap gives Google a detailed list of your pages and how they are structured. This gets your site in search engine results faster and more accurately.
At AgentMethods, we automatically generate a sitemap for you. You can find it at www.yourdomain.com/sitemap.
Submitting your sitemap to Google is easy and only takes a few minutes!
Start by logging into your Google Webmaster Account. If you do not currently have a Google Webmaster account, we strongly recommend you create one. You will need to verify your website ownership through Google's verification process.
Once you've logged in to your Google Webmaster Account, make sure you're on the properties section for your website. Then select Sitemaps on the left-hand sidebar.
Click on the red Add/Test Sitemap button and enter "sitemap" into the field. Then click submit!
That's it! Google will update you with the last date it processed your sitemap, how many pages it submitted, and how many of those pages it added to its index.
While we can't offer a detailed level of Google support, our support team would be happy to point you in the right direction if you get stuck! Contact us and we'd be happy to help.