47% of the news websites Google News shows most often block GPTBot, the crawler OpenAI uses to collect training material for ChatGPT, against 17.6% of all news publisher websites. Far fewer block the bots ChatGPT uses to fetch and cite pages when it answers.
Analysis of 178 top Google News websites and 52,326 news publisher websites · Google News results and the public HTTP Archive crawl, September 2026
Key Findings
- 1.47% of the top news websites in Google News block OpenAI's GPTBot training crawler, against 17.6% of all news publisher websites.
- 2.61.2% of top news websites block at least one AI crawler, and 54.5% block at least one of OpenAI's three bots.
- 3.21.3% of top news websites block OAI-SearchBot, the bot behind ChatGPT's search results, less than half the 47% that block GPTBot.
- 4.Among top news websites that also rank in the 10,000 most visited sites on the web, 55.2% block GPTBot.
- 5.Reuters blocks ChatGPT's search and browsing bots yet is ChatGPT's most-cited news outlet, in 250 news answers: a robots.txt block does not by itself remove an outlet from ChatGPT's answers.
Summary
Every website can post a short file, robots.txt, that tells automated visitors what they may read. Since AI companies began collecting the web to train their models, newsrooms have been using that file to say no. The New York Times, USA Today, NBC News, NPR and CNN all turn away every OpenAI bot.
That choice has two sides for a publisher. Blocking the training crawler keeps articles out of the material future AI models learn from, which protects the value of the reporting. Blocking the bots that fetch pages for live answers can also keep the outlet out of ChatGPT's citations, and so out of a growing source of readers. For brands and PR teams, it decides which outlets' coverage an AI engine can read and quote.
The biggest news brands block far more than the rest. 47% of the news websites that appear most often in Google News block GPTBot, against 17.6% of all 52,326 news publisher websites. But most of them treat training and search differently: 21.3% block OAI-SearchBot, the bot behind ChatGPT's search results.
What we measured
We took the 200 websites that appeared most often in 2,000 Google News result pages collected on September 23, 2026, and matched 178 of them to their homepage in the September 2026 HTTP Archive crawl, a public record of millions of websites. We read each site's robots.txt file and checked which AI crawlers it blocks. For comparison, we ran the same check on all 52,326 websites that identify themselves as news publishers in their page code.
A crawler, or bot, is a program that visits web pages automatically. OpenAI runs three: GPTBot collects pages to train its models, ChatGPT-User fetches a page when a ChatGPT user asks about it, and OAI-SearchBot gathers pages for ChatGPT's search results. A site blocks a bot when its robots.txt file tells that bot to stay off the whole site.
Nearly half of top news sites block GPTBot
84 of the 178 top news websites turn GPTBot away. The share rises to 54.5% for at least one of OpenAI's three bots, and to 61.2% for at least one AI crawler of any kind. 6.7% block other AI companies' crawlers while leaving all of OpenAI's bots alone.
Source: Suff Digital analysis of 178 top Google News websites and 52,326 news publisher websites · Google News results and the public HTTP Archive crawl, September 2026
Across all news publisher websites the picture is very different: 17.6% block GPTBot. The top outlets block it at more than two and a half times that rate. Blocking AI is mostly a big-publisher decision.
News sites block training more than live answers
Top news websites are far more willing to block training than search. 47% block GPTBot, 42.1% block ChatGPT-User and 21.3% block OAI-SearchBot. Many publishers keep their articles out of model training while leaving the door open for ChatGPT to find and cite them in search.
| Outlet | Blocks GPTBot | Blocks ChatGPT-User | Blocks OAI-SearchBot |
|---|---|---|---|
| nytimes.com | Yes | Yes | Yes |
| usatoday.com | Yes | Yes | Yes |
| nbcnews.com | Yes | Yes | Yes |
| npr.org | Yes | Yes | Yes |
| cnn.com | Yes | Yes | Yes |
| apnews.com | Yes | No | No |
| cbsnews.com | Yes | No | No |
| reuters.com | No | Yes | Yes |
| foxnews.com | No | No | No |
| theguardian.com | No | No | No |
| washingtonpost.com | No | No | No |
The best-known outlets split into clear camps. The New York Times, USA Today, NBC News, NPR, CNN, People and Yahoo block all three OpenAI bots. AP News, CBS News and the Los Angeles Times block GPTBot only, so they stay reachable for ChatGPT's answers. Reuters does the reverse, allowing GPTBot and blocking the other two. Fox News, the Guardian and the Washington Post block none of the three.
The biggest publishers block the most
The more popular the site, the more likely it blocks. Among top news websites that also rank in the 1,000 most visited sites on the web, 8 of 14 block GPTBot. In the top 5,000 it is 24 of 42, and in the top 10,000, 32 of 58, or 55.2%. Across all news publisher websites, the rate falls to 17.6%.
For a brand, this means the outlets with the largest audiences are the ones most likely to be closed to OpenAI's training crawler. Coverage in those outlets still reaches readers, but whether ChatGPT can read and quote it depends on which of the three bots the outlet blocks.
Blocking does not keep outlets out of ChatGPT
Robots rules do not map neatly onto who gets cited. In a separate Suff Digital study of ChatGPT's news answers, Reuters was ChatGPT's most-cited outlet, in 250 answers, even though its robots.txt blocks ChatGPT-User and OAI-SearchBot. AP News was cited in 165 ChatGPT answers while blocking GPTBot. ChatGPT can cite a page it found through a search index or partner feed without fetching it, so a robots.txt block does not by itself remove an outlet from its answers.
A robots.txt rule controls how a bot collects pages. On its own, it does not decide whether an outlet's name and stories appear in ChatGPT's answers, so a publisher can protect its archive from training and still be a leading source.
What this means for publishers
Decide about training and search separately. Blocking GPTBot keeps your content out of OpenAI's model training; blocking OAI-SearchBot and ChatGPT-User affects whether ChatGPT can fetch and cite your pages when it answers. Most top news sites that block one bot do not block all three, and each choice has a different cost in visibility.
Review your robots.txt file at least once a quarter. New AI crawlers appear often, and many sites' rules name only a few of them, so the policy on paper can drift from the policy you want. Our technical seo services include a review of AI crawler rules for publishers and brands, matched to where you want to be cited.
Embed this research
Paste this on your site to embed the charts. It links back to the source automatically.
<iframe id="sd-news-sites-block-gptbot" src="https://www.suffdigital.com/embed/data-studies/news-sites-block-gptbot" width="100%" height="600" style="width:100%;border:1px solid #E5E7EB;border-radius:12px" loading="lazy" title="47% of Top News Sites Block ChatGPT's Training Crawler vs 17.6% of All News Sites - Suff Digital"></iframe>
<script>window.addEventListener("message",function(e){if(e&&e.data&&e.data.sdEmbed==="news-sites-block-gptbot"&&e.data.height){var f=document.getElementById("sd-news-sites-block-gptbot");if(f){f.style.height=e.data.height+"px";}}});</script>
<p style="font:14px/1.5 system-ui,sans-serif">Source: <a href="https://www.suffdigital.com/resources/data-studies/news-sites-block-gptbot">47% of Top News Sites Block ChatGPT's Training Crawler vs 17.6% of All News Sites - Suff Digital</a></p>
Cite this study
Suff Digital. (2026). 47% of Top News Sites Block ChatGPT's Training Crawler vs 17.6% of All News Sites. https://www.suffdigital.com/resources/data-studies/news-sites-block-gptbot
47% of Top News Sites Block ChatGPT's Training Crawler vs 17.6% of All News Sites - Suff Digital - https://www.suffdigital.com/resources/data-studies/news-sites-block-gptbot
Frequently asked questions
Related studies
- AI SEOReuters Is AI's Most-Cited News Source, in 328 AI News Answers
- AI SEOChatGPT Cites OpenAI's News Partners 2.2x as Often as Google News Does
- AI SEOPaywalls in AI: Paywalled Outlets Are Cited in 16.7% of AI News Answers
- AI SEOBrands That Block GPTBot Still Get Named by ChatGPT: 31.5 Mentions vs 20.7 for Brands That Allow It