.png){kind=logo}
There’s no guarantee every page on the web will be indexed, so the ‘Discovered – Currently not indexed’ status in Search Console’s Index Coverage report could last forever – Google stated last week in the office-hours hangout.
When asked about how long we can wait for a new page on the website to be crawled and indexed, Google’s Search Advocate John Mueller said:
That can be forever. It’s something where we just don’t crawl and index all pages and it’s completely normal for any website, that we don’t have everything indexed.
And especially with a newer website, if you have a lot of content, then I would assume that it’s expected that a lot of the new content for a while would be discovered and not indexed. And then, over time, usually it kind of shifts over and it’s actually crawled, or it’s actually indexed when we see that there’s actually value in focusing more on the website itself, but it’s not guaranteed.
So, from that point of view, it’s not that I would say you should just wait for a little bit, and then suddenly things would get better with crawling and indexing, it’s more that like, continue working on the website and make sure that our systems recognize that there’s value in crawling and indexing more and then over time we will crawl and index more.
Despite knowing that a page exists, Google won’t crawl it for several reasons, like:
Technical reasons – Google might have tried to crawl the URL but the site was overloaded. If that’s the case, Google will come back another day and try to crawl into it.
Quality reasons – The website doesn’t meet a certain quality threshold by Google. Websites considered valuable and of high quality are given priority in Google’s crawling of the internet.
What Should You Do?
Make sure that it’s easy for Google to recognize important content. In this case, it’s quality over quantity, so your content doesn’t have to be long, it has to be good.
Internal linking – that way Google can see what is important for you. For example, if things that are linked from the home page can usually help Google see that you think those pages are important.
External links – when Google sees that these pages are important to people, they should see them as important too.
Site maps and RSS feeds also help from a technical perspective – they help Google understand if the pages were added recently.
Read more about this topic:
Could “Discovered – but currently not indexed” put a URL in some sort of ‘blacklist’? Thought I’d share something strange and interesting that happened w/a few blog posts of a client.. (1/5) (I hate doing threads but this needs a little detail) 👇🏻 — Dan Shure (@dan_shure) November 8, 2021
