16.1% of Small Business Web Pages Can't Appear in Google Search

A roofing company pays for a new service page about storm damage repairs. It looks fine in the browser, the owner shares the link, and months later it has never appeared in a single Google search. The cause is one line of code left over from the build that tells Google not to index the page. Nobody sees it, because it is invisible to visitors.

Google can only rank pages it is allowed to add to its index, the catalogue of pages it draws search results from. A page that carries a "do not index" signal earns no search traffic no matter how good it is. Some pages should carry that signal, such as a checkout or a login screen, but a service page, a location page or a blog post that carries it is wasted work.

Across 5,120 small business websites, 16.1% of pages cannot be indexed. The problem is concentrated: 28.9% of websites have at least one such page, and 6.6% hide half or more of their pages from Google. For those businesses, a large part of the website does not exist in search.

On September 23 and 24, 2026 we checked the websites of 5,120 US small businesses listed on Google across 56 industries, from plumbers and roofers to florists and fashion stores. We read up to 20 pages per website, 73,202 pages in all, and recorded whether each page was allowed into Google's index. We report two figures: the share for the average website, where each website counts once, and the share across all pages together, where larger problems on some websites weigh more.

A page counts as non-indexable when it carries a signal that keeps it out of Google. There are four: a robots.txt rule (a file at the root of a website that tells search engines which addresses to stay away from), a noindex tag (a line in the page, or in the server's response, that asks search engines not to list it), a canonical tag pointing to a different address (which tells Google another page is the main version, so this one is left out), and a redirect loop, where the page sends visitors in a circle and never loads.

The two headline figures differ for a reason. On the average small business website, 8% of pages cannot be indexed. Count all 73,202 pages together and the share doubles to 16.1%. That gap means the problem is lopsided: most websites hide few pages or none, and a smaller group hides a great many, enough to pull the overall figure up.

Most of the problem sits in that minority. 28.9% of small business websites have at least one page Google is told to skip, and about one in 15 has half or more of its pages hidden. For a business in that group, the pages Google can see are a fraction of the website it paid for, and every hidden service or location page is a search it cannot win.

A robots.txt rule is the biggest single reason small business pages stay out of Google, blocking 9.8% of all pages. Noindex tags account for 3.8% and canonical tags pointing to another address for 2.5%. A robots.txt rule can block a whole folder of pages at once, so a single misplaced line can hide dozens of addresses, which is why it outweighs the other causes combined.

The most frequently blocked address is /wp-json, the data feed that WordPress websites expose, with 1,589 blocked pages. Store paths such as /collections, where online shops list their product ranges, follow with 684. Both show how these blocks are usually set: by a website platform, a theme or a plugin, not by the owner, who may never know the rule exists.

Retailers keep the largest share of their websites out of Google. The average fashion store website hides 17.5% of its pages and the average mattress store 17.4%, followed by optometrists at 16.5% and restaurants and florists at 13.1%. Online stores generate many addresses for product ranges, filters and sorting options, and their platforms often block those addresses by default, which catches product and category pages along the way.

Service businesses sit at the other end. The average funeral home website hides 1% of its pages, plumbers 2.9% and orthodontists 3.1%. Their websites are smaller and simpler, with a handful of service pages and no product catalogue, so there are fewer addresses for a stray rule to catch.

Open Google Search Console, the free tool Google gives every website owner, and go to the Pages report. It lists every address Google found but did not index, with the reason. Check the entries marked "Excluded by noindex tag" and "Blocked by robots.txt". If a service page, location page or blog post appears there, remove the block and ask Google to index the page again.

Then read your robots.txt file at yourdomain.com/robots.txt. A single "Disallow: /" line blocks the whole website, and it often survives from the build stage when a new website goes live. Keep blocks on checkout, cart, login and account pages, and remove them everywhere else. If you run an online store, check that your product and category pages are not caught by rules meant for filters and sorting.

Repeat the check after every redesign, migration or plugin change, because that is when these rules appear. Our technical seo services include an indexing check on every page that should earn search traffic.

Embed this research

Cite this study