Indexing large websites on Google takes a lot of time. Before new websites are indexed to a large extent, they are first crawled. Google then waits for signals before deciding to index the newly created page further.
John Müller (Senior Webmaster Trends Analyst at Google) announced some information about indexing and crawling new websites at Google in the Google Search Central SEO Office Hours on January 8th. First, however, it should be clarified who John Müller is and what indexing means.
Who is John Müller?
John Müller, actually Johannes Müller, is the Senior Webmaster Trends Analyst at Google. Müller occasionally gives interviews or appears in YouTube videos in which he gives small tips or solutions to problems. Sometimes he also gives ideas on how he would solve a problem. Müller regularly uploads videos on the “Google Webmaster Central Blog” with news on crawling and indexing
What is indexing?
Indexing is a method of collecting information. In the context of the Internet, a document is included in the index, i.e. the register, of a search engine.
For example, the “Google Index” is the entirety of all websites recognized by Google. So-called crawlers are used to search the web.
These work independently and help with the indexing of new websites in the index. A crawler moves from website to website via hyperlinks, saves them and inserts them into a list of all URLs. If a website violates the Google guidelines, it will be removed and no longer displayed. Indexing is very important for search engine optimization. The higher the quality of the content on your own page, the higher the page will be listed in the search engine.
However, indexing is no guarantee that users will see the website on the first Google page. It is also important for indexing and crawling that your website is structured in such a way that the crawler can easily find all internal links and add them to the index. An SEO-optimized page therefore also makes indexing easier.
Can I start indexing myself?
It is possible to start indexing your own website yourself. To do this, you can contact Google and submit a request for indexing. To do this, you must visit https://www.google.de/addurl/. If you have a Google account, you can use it to send Google the necessary data. Alternatively, you can also send sitemaps (list of websites) directly to Google via the “Search Console”. This is then crawled within 24 hours. If new content and links have been added to your website, you can resubmit them to Google for crawling in order to keep your website up to date in the index. However, if you do not submit your website to Google personally, this is not a problem at all. Since Google’s crawler works continuously, new pages are added to the index very quickly.
How does indexing work at Google?
For the operators of a new website, it is usually not quick enough to have their own site crawled and indexed. John Müller said that large websites are only partially indexed at the beginning, even if Google tries to index as much as possible. He was referring to the request for a website with several million URLs that are supposed to lead to various products, which has been in existence for three to four months. Google is trying to stay on the safe side here, says John Müller. Indexing websites takes a lot of time and memory. In order not to overload the servers and thus avoid problems, the indexing speed is somewhat slower at the beginning.
According to John Müller, Google wants to ensure that the indexing of a website is also profitable. This should show Google that it is worth indexing the page because it contains something that users are looking for. If it turns out over time that the website is well integrated into the web, they increase the speed and more content is added. Google recognizes the usefulness of the website and indexes further URLs when several users call it up and recommend it to others. According to John Müller, it is normal for particularly large websites to be indexed slowly at the beginning. Google is not able to index millions of URLs at once.
Müller refers to various signals, such as the emergence of backlinks or new websites, which Google has to collect bit by bit. It can take several six to nine months for the rankings of a new website to stabilize.
Müller recommends concentrating on a few but important pages, for example. This means Google has to index less. For example, you could limit yourself to a few categories or important product pages at the beginning. You should focus on something at the beginning and then expand your site from there.
Indexing (especially for large websites) therefore takes time. Patience is required from the owners of the website. However, you should also remember to limit your new site to the most important things at the beginning and then expand it bit by bit. This allows Google to index more easily and quickly.