Crawling, indexing and re-indexing are highly important terms in SEO. If you have had a website for a couple of months now you are probably familiar with the terms. However, for those who are not aware:
- Crawling is when Google visits your website with Googlebot to track changes in your website
- Indexing is when Google acknowledges those changes and puts them in Google Index (web search)
Most websites want a high indexing because it means that their content is constantly up to date. Nevertheless, that is not the case for most websites out there.
As a website owner, you may have noticed that google takes quite a while to re-index your pages in search. Whenever you make changes in your website, you have to wait for crawlers to detect those changes. That time, however, is undefined due to a large number of websites. It may even take weeks for some websites.
Why does that happen?
According to Mueller, a Senior Webmaster Trends Analyst at Google, the reason why it takes an undefined time to re-index a website is that of a large number of websites on the internet.
If more crawlers were to re-index pages during a short period of time, then the Google servers would be strained. As such, Googlebot only checks a limited number of websites that are assigned to it.
What pages are crawled more often?
Since it is a time-consuming process, Google gives priority to specific pages and categories when it comes to re-indexing them. Those pages and categories are usually the most visited ones on your website. That would include your home page and high-level category pages. That is because new content is more likely to be posted on the homepage or those high-level category pages. Therefore, if you make changes in these websites, chances are they will be noticed faster from Googlebot crawlers rather than an old blog post.
What affects re-indexing?
There are a number of factors that can have a say on the frequency of re-indexing. One of the most important criteria is to have a high ranked website. Factors that assist ranking include:
- Domain Name – is of high importance when it comes to re-indexing. Google will usually prioritize domain names that contain a keyword in them. This means they post quality content relevant to that keyword.
- Number of Backlinks – Backlinks are essential to establish the trustworthiness of your website. If you have a high number of backlinks, Google will consider you as a website that posts high-quality content thus, rank you higher and crawl into your content more often. The opposite will be true if you have a low number of backlinks.
- Internal Linking – which is also known as deep linking is a great way to maintain active visitors in your website. This is widely practiced among websites to help with deep crawling.
- XML Sitemap – helps your website to become auto-generated. This will inform Google of any change that could have been made thus, it will crawl in your website more often.
- Duplicate Content – affects re-indexing negatively. In fact, it can even get your website banned if used continuously as crawlers notice that. Therefore, keeping duplicate content at a minimum is best for you.
- Meta Tags – Non-competitive Meta tags are a great way to reach a higher ranking.
- Pinging – If you have a WordPress site, you have to add all of your main pings in there. There are plugins that can help you with that.
If you follow the above-mentioned indexing factors, Google will for sure crawl in your website more often and index it.