.png){kind=logo}
A fifth of the pages on your website aren’t getting indexed by Google – should you panic? Not exactly, a rep from Google recently explained.
Google Webmaster Trends Analyst John Mueller discussed the indexing process with a concerned viewer on a recent livestream.
Indexing is one of the three main steps Google takes to push your web pages onto the result page of a search query. Specifically, indexing involves analyzing a page’s content, cataloguing its elements, and, essentially, trying to ‘understand’ the page. It is then stored in Google’s massive index of pages.
If you want visitors to find your website through Google searches – and, let’s face it, you probably do – you’ll need Google to index your pages.
But what if Google doesn’t index them?
A viewer explained to Mueller that about 20 percent of their site’s pages weren’t being indexed, but they weren’t sure why. They speculated that the problem was due to an overloaded server and an accompanying 500 error response.
“Twenty percent of my pages are not getting indexed,” the viewer said. “It says they’re ‘discovered’ but not crawled. Does this have anything to do with the fact that it’s not crawled because of potential overload of my server, or does it have to do with the quality of the page?”
Mueller explained that both server capacity and page quality could be relevant. A small site, though, would not typically reach crawling capacity, he explained.
Mueller added that the way a website is crawled and indexed is important to consider.
“If you have five pages that are not indexed at the moment, it’s not that those five pages are the ones we would consider low quality,” he said.
“It’s more that …overall, we consider this website maybe to be a little bit lower quality, and therefore we won’t go off and index everything on this site. If we don’t have that page indexed, then we’re not really going to know if that’s high quality or low quality.”
Mueller offered one more piece of advice: if your site uses a common content management system (CMS), you’ll almost never have to worry about this issue.
