{"id":4255,"date":"2026-09-25T11:52:34","date_gmt":"2026-09-25T11:52:34","guid":{"rendered":"https:\/\/projectfifty4.com\/ai-crawler-access-llms-txt-energy-b2b\/"},"modified":"2026-10-05T19:58:36","modified_gmt":"2026-10-05T19:58:36","slug":"ai-crawler-access-llms-txt-energy-b2b","status":"publish","type":"post","link":"https:\/\/projectfifty4.com\/fr\/ai-crawler-access-llms-txt-energy-b2b\/","title":{"rendered":"AI Crawler Access Control for Energy B2B: What Google Says You Can Stop Doing"},"content":{"rendered":"<p><strong>Most energy companies are running an AI visibility strategy built on advice that the platforms themselves contradict. Google&#x27;s own documentation, updated 10 July 2026, tells site owners not to build llms.txt, not to chunk content for AI, and not to add special schema for AI answers. Meanwhile the decision that genuinely governs whether an assistant can cite you, which crawler you allow, is being made by accident in robots.txt files written before the fleets split. This dossier separates the controls that work from the ones being sold.<\/strong><\/p>\n<h2>Do you need an llms.txt file to be cited by AI assistants?<\/h2>\n<p>No. Google states in its own generative AI optimisation guide, updated 10 July 2026, that you do not need to create AI text files, special markup or Markdown to appear in Google Search including its AI features, because Google Search does not use them and ignores them. No major AI provider has publicly committed to reading llms.txt from third party websites. Ahrefs measured every one of the 137,210 domains in its Web Analytics panel in May 2026 and found that 28 percent publish an llms.txt file, that 97 percent of those files received zero requests, and that AI retrieval bots, the ones that decide citations, accounted for just 1.1 percent of requests to the small remainder that were fetched at all. The control that does decide whether an assistant can cite you is robots.txt, and specifically which of a provider&#8217;s three separate crawlers you allow: the training crawler, the retrieval crawler and the user triggered fetcher.<\/p>\n<h2>Points cl\u00e9s \u00e0 retenir<\/h2>\n<ul>\n<li>Blocking AI crawlers is three decisions, not one. Every major provider now runs a separate training crawler, retrieval crawler and user triggered fetcher, each controlled by its own robots.txt token. Blocking the training bot costs no visibility. Blocking the retrieval bot removes you from that assistant&#8217;s answers.<\/li>\n<li>Google is explicit that llms.txt does nothing for Google Search. Its AI optimisation guide says creating one will neither harm nor help visibility, because Google Search ignores it. The measured evidence agrees: 97 percent of published llms.txt files were never requested once in May 2026.<\/li>\n<li>Google-Extended is a trap. It is not a crawler, it is a usage flag on data Googlebot has already taken, and blocking it does not remove you from AI Overviews or AI Mode. It does remove you from Gemini grounding. Publishers who blocked it to stay out of AI answers got the cost without the benefit.<\/li>\n<li>A client rendered site can rank in Google and be blank to ChatGPT. Google and Applebot render JavaScript. OpenAI, Anthropic and Perplexity say nothing about rendering in their documentation, and the only substantial log study found that no major AI crawler executes JavaScript.<\/li>\n<li>The market has already converged on the sensible split. Cloudflare reports that fewer than 1 percent of sites on its network block search bots while 17 percent block training. Publishers are refusing training and welcoming retrieval, which is exactly the distinction most robots.txt files fail to make.<\/li>\n<li>For B2B technology specifically, organic rank still predicts AI citation better than the cross industry average suggests. BrightEdge measured 71.0 percent overlap between AI Overview citations and organic results in B2B Tech, third highest of nine industries and up 32.4 points in sixteen months.<\/li>\n<\/ul>\n<h2>The fleets split, and most robots.txt files did not<\/h2>\n<p>The structural change since 2024 is that every major AI operator has divided its crawler fleet by purpose, and each user agent is controlled independently in robots.txt. A robots.txt file written before that split treats a provider as a single entity. That is now a category error with a measurable cost.<\/p>\n<p>OpenAI&#8217;s own documentation is unambiguous that <a href=\"https:\/\/developers.openai.com\/api\/docs\/bots\" target=\"_blank\" rel=\"noopener nofollow\">each setting is independent of the others<\/a>, and that a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot. GPTBot collects content that may be used in training. OAI-SearchBot builds the index ChatGPT retrieves from, and OpenAI states plainly that sites opted out of it will not be shown in ChatGPT search answers. ChatGPT-User fetches a page because a named person asked, and OpenAI notes that because those actions are initiated by a user, robots.txt rules may not apply.<\/p>\n<p>Anthropic publishes the same three way split with an unusually plain account of the consequences. Its <a href=\"https:\/\/support.claude.com\/en\/articles\/8896518\" target=\"_blank\" rel=\"noopener nofollow\">crawler documentation<\/a> states that disabling Claude-SearchBot prevents indexing for search and may reduce a site&#8217;s visibility and accuracy in user search results, while restricting ClaudeBot signals that future materials should be excluded from model training. Anthropic is also the only major provider on the documented record that carves out no user triggered exception at all: it states its bots honour industry standard robots.txt directives and will not attempt to bypass anti circumvention technologies, with no agent side exemption.<\/p>\n<p><a href=\"https:\/\/docs.perplexity.ai\/guides\/bots\" target=\"_blank\" rel=\"noopener nofollow\">Perplexity<\/a> is the outlier in the other direction. PerplexityBot is explicitly not used to crawl content for foundation models, which means Perplexity runs no training crawler on the public web at all. For an energy company whose objection is training rather than retrieval, Perplexity presents nothing to block. Perplexity-User, by contrast, generally ignores robots.txt, which Perplexity states in its own documentation.<\/p>\n<p>Two inheritance rules deserve attention from anyone writing tight rules. <a href=\"https:\/\/support.apple.com\/en-gb\/119829\" target=\"_blank\" rel=\"noopener nofollow\">Apple<\/a> states that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. <a href=\"https:\/\/developer.amazon.com\/amazonbot\" target=\"_blank\" rel=\"noopener nofollow\">Amazon<\/a> applies the same logic to Amzn-SearchBot against other search bots. A rule written for one crawler can silently govern several.<\/p>\n<p>The practical consequence for energy B2B is narrow and specific. Blocking training costs nothing in visibility and is purely a rights decision. Blocking retrieval removes you from an assistant&#8217;s answers. Most of the blanket blocks deployed in 2024 and 2025 did both.<\/p>\n<h2>No, and three years of publishers have been wrong about this<\/h2>\n<p>Google-Extended is the most misunderstood control in the category, and Google&#8217;s own documentation resolves it without ambiguity. It is not a crawler. Google states that <a href=\"https:\/\/developers.google.com\/crawling\/docs\/crawlers-fetchers\/google-common-crawlers\" target=\"_blank\" rel=\"noopener nofollow\">Google-Extended does not have a separate HTTP request user agent string<\/a>, that crawling is done with existing Google user agent strings, and that the robots.txt token is used in a control capacity. It is a flag on data Googlebot has already collected.<\/p>\n<p>What it controls is whether that content may be used for training future Gemini models and for grounding, meaning supplying content from the Search index to the model at prompt time. Both, not just training. What it does not control is Search: Google states directly that Google-Extended does not impact a site&#8217;s inclusion in Google Search and is not used as a ranking signal.<\/p>\n<p>The trap is the next step. There is no AI Overviews crawler and no AI Mode crawler. Google&#8217;s <a href=\"https:\/\/developers.google.com\/search\/docs\/appearance\/ai-features\" target=\"_blank\" rel=\"noopener nofollow\">AI features documentation<\/a> states that AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot are the control for site owners managing how their sites are crawled for Search. The only lever that removes you from AI Overviews is blocking Googlebot, which removes you from Google entirely.<\/p>\n<p>Publishers who blocked Google-Extended in 2023 and 2024 believing it would keep them out of AI answers were mistaken, and have been mistaken ever since. They did not avoid AI Overviews. They did remove themselves from Gemini grounding. That is a real cost with none of the intended benefit, and it is still sitting in a large number of robots.txt files that nobody has revisited.<\/p>\n<p>Below the nuclear option, the controls that genuinely exist are nosnippet, data-nosnippet, max-snippet and noindex. Apple offers a parallel and slightly finer instrument: its documentation describes nosnippet as opting content out of broad world knowledge answers in Siri and Search, separately from training.<\/p>\n<h2>The measurement is in, and it is not close<\/h2>\n<p>Le <a href=\"https:\/\/llmstxt.org\/\" target=\"_blank\" rel=\"noopener nofollow\">llms.txt proposal<\/a> was published by Jeremy Howard of Answer.AI on 3 September 2024, and now carries a v2 modified 10 August 2026. It proposes a markdown file giving a project name, a summary and curated links to machine friendly content. Howard is honest about what it is: the specification states that llms.txt information is used on demand, when an agent needs information about a topic while assisting a user. Despite the filename, it is not an access control mechanism. It blocks nothing and permits nothing.<\/p>\n<p>The decisive question is whether the systems that decide citations consume it. Google has answered in primary documentation. Its <a href=\"https:\/\/developers.google.com\/search\/docs\/fundamentals\/ai-optimization-guide\" target=\"_blank\" rel=\"noopener nofollow\">generative AI optimisation guide<\/a>, updated 10 July 2026, carries a section headed mythbusting generative AI search, and states that you do not need to create new machine readable files, AI text files, markup or Markdown to appear in Google Search including its generative AI capabilities, because Google Search itself does not use them. It adds that maintaining such files for other systems will neither harm nor help visibility in Google Search, as Google Search ignores them.<\/p>\n<p>OpenAI, Anthropic and Perplexity say nothing about llms.txt in their crawler documentation, in either direction. No major provider has publicly committed to reading it from the open web.<\/p>\n<p>The empirical picture is the same. Ahrefs <a href=\"https:\/\/ahrefs.com\/blog\/llmstxt-study\/\" target=\"_blank\" rel=\"noopener nofollow\">analysed all 137,210 domains<\/a> in its Web Analytics panel that received traffic in May 2026, checked each root for an llms.txt returning HTTP 200, screened out soft 404s, and then examined every request made to those paths. Twenty eight percent of domains publish one, a figure Ahrefs flags as an upper bound because its customer base skews SEO aware. Ninety seven percent of those files received zero requests. Of the three percent that saw any traffic, 96 percent came from bots, and 77 percent of those bots were not AI tools. Slackbot fetched llms.txt files more often than PerplexityBot did.<\/p>\n<p>The honest verdict for an energy B2B publisher: if the goal is citation in ChatGPT, Perplexity, Claude or Google&#8217;s AI features, llms.txt is decoration. It is cheap and harmless, and it is almost certainly inert. The one evidence backed exception is coding agents, where Claude Code out fetched every AI retrieval bot in Ahrefs&#8217; dataset. If you sell a developer facing product, an llms.txt over your documentation path is defensible. An energy operator&#8217;s corporate site is not that case.<\/p>\n<p>Google is not internally consistent here and it is worth saying so. Days after the Search team published the mythbusting guidance, the Chrome team shipped an llms.txt check in Lighthouse&#8217;s experimental agentic browsing audits. Both positions are live inside the same company. The Search position is the one that governs Search.<\/p>\n<h2>Google renders. The AI native labs are silent, and the one study says no<\/h2>\n<p>This is the most under priced technical risk in the category, because the failure is invisible from inside Google Analytics. A client rendered React or Vue site can rank perfectly well in Google Search while returning an empty shell to ChatGPT, Claude and Perplexity.<\/p>\n<p>Apple is the only AI adjacent operator that documents rendering officially. Its <a href=\"https:\/\/support.apple.com\/en-gb\/119829\" target=\"_blank\" rel=\"noopener nofollow\">Applebot documentation<\/a>, published 5 September 2026, states that Applebot may render the content of a website within a browser, and that if JavaScript, CSS and other resources are blocked via robots.txt it may not be able to render the content properly, including XHR. Google&#8217;s position is documented by implication: its AI guide states that Google can process content within JavaScript as long as it is not blocked, and Google serves AI features from the same index and the same Googlebot control surface as classic Search.<\/p>\n<p>OpenAI, Anthropic and Perplexity say nothing. All three crawler documentation pages are silent on rendering in either direction. OAI-SearchBot&#8217;s user agent string advertises a Chrome version, but a user agent string is cosmetic and is not evidence of rendering.<\/p>\n<p>The empirical basis for the common claim rests substantially on one piece of work: the <a href=\"https:\/\/vercel.com\/blog\/the-rise-of-the-ai-crawler\" target=\"_blank\" rel=\"noopener nofollow\">Vercel and MERJ log study<\/a> of 17 December 2024, which monitored server logs across the Vercel network and concluded that none of the major AI crawlers render JavaScript. It is credible and it is uncontradicted, but it is now 21 months old, it predates Claude-SearchBot and Meta-WebIndexer entirely, it ran substantially on one site, and both authors sell products whose value rises with the finding. No independent replication at comparable scale has been published. Treat it as the best available evidence, not as settled fact.<\/p>\n<p>Vercel&#8217;s own caveat is the part most coverage drops, and it matters operationally: content present in the initial HTML response, including inline JSON, remains readable even though it is not rendered. So client side rendering is invisible. Any use of JavaScript is not.<\/p>\n<p>During the research for this dossier, two documentation pages returned nothing to a non rendering fetch. Microsoft&#8217;s Bing Webmaster Tools help site returned only a message asking the reader to enable JavaScript. That is a sharper illustration of the risk than any vendor study: Microsoft&#8217;s own webmaster guidance is invisible to a crawler that does not render.<\/p>\n<p>The test takes ten minutes and beats every study. Fetch your own highest value pages with curl and no JavaScript, and read what comes back. If your capability pages, your case studies and your technical specifications are not in that response, the assistants your buyers are asking cannot see them.<\/p>\n<h2>Google&#x27;s answer is that it is still SEO, and the data does not disagree<\/h2>\n<p>Before reaching for vendor correlation studies, it is worth reading what Google has published directly. Its <a href=\"https:\/\/developers.google.com\/search\/docs\/fundamentals\/ai-optimization-guide\" target=\"_blank\" rel=\"noopener nofollow\">AI optimisation guide<\/a> states that generative AI features are rooted in core Search ranking and quality systems, using retrieval augmented generation against the Search index and query fan out. It then says something most agencies do not quote: that creating content people find unique, compelling and useful will likely influence a site&#8217;s presence in generative AI search more than any of the other suggestions in the guide.<\/p>\n<p>The distinction Google draws is commodity versus non commodity content, and the example translates directly into energy B2B. Commodity content is a listicle of procurement tips. Non commodity content is what your engineers learned on a specific project that nobody else can write. Google&#8217;s own framing is that optimising for generative AI search is optimising for the search experience, and thus still SEO.<\/p>\n<p>Google&#8217;s mythbusting list is worth reading as a budget document. It says explicitly that there is no requirement to break content into tiny chunks for AI, that you do not need to write in a specific way just for generative AI search, that seeking inauthentic mentions is not as helpful as it might seem, and that structured data is not required for generative AI search with no special schema markup to add. It keeps one instruction on schema that is genuinely useful: make sure your structured data matches the visible text on the page.<\/p>\n<p>The independent evidence is consistent with this. The only peer reviewed controlled work in the field, <a href=\"https:\/\/arxiv.org\/abs\/2311.09735\" target=\"_blank\" rel=\"noopener nofollow\">Aggarwal and colleagues at KDD 2024<\/a>, tested interventions across roughly 10,000 queries and reports visibility gains of up to 40 percent. The highest performing interventions were adding quotations from authoritative sources, relevant statistics and cited sources. Not markup, not keyword density. That is a description of good research writing.<\/p>\n<p>Rank still matters, moderately. Ahrefs measured a <a href=\"https:\/\/ahrefs.com\/blog\/does-ranking-higher-on-google-mean-youll-get-cited-in-ai-overviews\/\" target=\"_blank\" rel=\"noopener nofollow\">Spearman correlation of 0.347<\/a> between top ten ranking and top three AI Overview citation at URL level across a million keywords. No published study has produced a URL level rank to citation coefficient above roughly 0.45. The strongest measured correlates in Ahrefs&#8217; <a href=\"https:\/\/ahrefs.com\/blog\/ai-brand-visibility-correlations\/\" target=\"_blank\" rel=\"noopener nofollow\">75,000 brand study<\/a> are not on page factors at all: YouTube mentions at roughly 0.737 and branded web mentions at 0.664 to 0.709 sit well above Domain Rating at 0.266 to 0.326. Note the tension with Google&#8217;s warning about inauthentic mentions. The defensible reading is that earned mention volume correlates with citation because both are downstream of actually being a notable company.<\/p>\n<p>The single most useful figure for this sector comes from <a href=\"https:\/\/www.brightedge.com\/resources\/weekly-ai-search-insights\/rank-overlap-after-16-months-of-aio\" target=\"_blank\" rel=\"noopener nofollow\">BrightEdge&#8217;s industry breakdown<\/a>: B2B Tech shows 71.0 percent overlap between AI Overview citations and organic results, third highest of nine industries and up 32.4 points over sixteen months, against 22.9 percent for e-commerce. For technical B2B, organic rank remains a materially better predictor of AI citation than cross industry averages suggest. The implication is not that AI visibility is a separate discipline requiring separate spend. It is that the organic programme you already fund is the AI visibility programme.<\/p>\n<p>One caution on the numbers in this section. With the exception of the peer reviewed KDD work and <a href=\"https:\/\/www.pewresearch.org\/short-reads\/2025\/07\/22\/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results\/\" target=\"_blank\" rel=\"noopener nofollow\">Pew&#8217;s browsing panel study<\/a>, effectively every quantitative source on AI citation is published by a vendor selling AI visibility software. Read the methodology before you read the headline.<\/p>\n<h2>Six decisions, in order of how much they cost to get wrong<\/h2>\n<p>First, audit robots.txt against the three way split. Separate the training tokens from the retrieval tokens and decide each one deliberately. If your organisation&#8217;s position is that it objects to training but wants to be found, the file should say GPTBot disallow, ClaudeBot disallow, Meta-ExternalAgent disallow, and allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer. Cloudflare&#8217;s network data suggests this is where the market has already landed: it reports that fewer than 1 percent of sites block search bots while <a href=\"https:\/\/blog.cloudflare.com\/accountable-mixed-use-ai-crawlers\/\" target=\"_blank\" rel=\"noopener nofollow\">17 percent block training<\/a>.<\/p>\n<p>Second, remove any Google-Extended block that was added to avoid AI Overviews. It never did that. It only removed you from Gemini grounding. If the objection to Gemini training is genuine, keep it deliberately; if it was defensive, it is costing you.<\/p>\n<p>Third, test rendering. Curl your ten highest value commercial pages with no JavaScript and read the output. Server render or statically generate main content, metadata, navigation and internal links. Reserve client side rendering for genuine enhancements. Get critical content into the initial HTML response rather than a post hydration fetch.<\/p>\n<p>Fourth, stop paying for llms.txt work unless you are selling to developers. If you already have one, leave it: it is harmless. Version control it, restrict edit rights, and keep the content to plain links and descriptions with nothing instruction shaped in it. Ahrefs noted that the largest single research crawler in its dataset identified itself as a prompt injection survey. Agents are designed to ingest and trust that file.<\/p>\n<p>Fifth, redirect the budget to what the evidence supports. Quotations from named sources, real figures with citations, and first hand operational detail are the interventions with controlled evidence behind them. In energy B2B that means project specifics, commissioning data, regulatory reading and named engineers, not another explainer on the energy transition.<\/p>\n<p>Sixth, know the OAI-AdsBot exists before you write a blanket OpenAI block. It validates landing pages submitted as ChatGPT ads and uses landing page content to judge relevance. If your organisation buys or intends to buy ChatGPT placement, a blanket block breaks your own media.<\/p>\n<p>The strategic point underneath all six is that the platforms have been more honest than the agencies. Google has published, in plain language, a list of things it says you can stop doing. Most energy marketing teams are still funding at least three of them.<\/p>\n<p><strong>Separate the three tokens<\/strong> Training, retrieval and user triggered fetchers are independent controls. Decide each one. A blanket block buys a rights position you may not want at a visibility cost you did not intend.<\/p>\n<p><strong>Get the content into the HTML<\/strong> Google and Applebot render. The AI native labs are undocumented and the one log study says they do not. Server rendered content is the only version that is safe across all of them.<\/p>\n<p><strong>Fund evidence, not format<\/strong> The only controlled study in the field found that quotations, statistics and cited sources drive visibility. That is a research standard, not a markup schema.<\/p>\n<h2>FAQ<\/h2>\n<h3>Does llms.txt help you get cited by AI?<\/h3>\n<p>There is no evidence that it does, and Google states directly that Google Search ignores it and that creating one will neither harm nor help visibility. No major AI provider has publicly committed to reading llms.txt from third party sites. Ahrefs measured 137,210 domains in May 2026 and found 97 percent of published llms.txt files received zero requests, with AI retrieval bots accounting for 1.1 percent of requests to the small share that were fetched. The one defensible use case is documentation for coding agents.<\/p>\n<h3>Should I block GPTBot?<\/h3>\n<p>Blocking GPTBot is purely a rights decision and costs nothing in visibility. It stops content being used to train OpenAI&#8217;s foundation models and has no effect on whether you appear in ChatGPT search answers. The bot that governs that is OAI-SearchBot, which is controlled separately. OpenAI states in its own documentation that each setting is independent of the others.<\/p>\n<h3>Does blocking Google-Extended remove me from AI Overviews?<\/h3>\n<p>No. Google-Extended is not a crawler, it is a usage control token applied to data Googlebot has already collected, and it governs Gemini training and grounding. Google states that it does not impact inclusion in Google Search. There is no separate AI Overviews crawler, so the only way to be excluded from AI Overviews is to block Googlebot, which removes you from Google Search entirely.<\/p>\n<h3>Can ChatGPT read a JavaScript heavy website?<\/h3>\n<p>OpenAI does not document its rendering capability in either direction. The best available evidence, a server log study by Vercel and MERJ published in December 2024, found that no major AI crawler executed JavaScript. Content present in the initial HTML response remains readable. The practical implication is that a client rendered site can rank normally in Google, which does render, while returning an empty page to AI assistants that do not. Testing your own pages with curl and no JavaScript settles it in ten minutes.<\/p>\n<h3>What actually drives AI citation for a B2B energy company?<\/h3>\n<p>Ranking well organically, publishing first hand non commodity content with named sources and real figures, keeping it fresh, and earning genuine third party mentions. Google states that generative AI features run on its core Search ranking systems and that optimising for them is still SEO. The only peer reviewed controlled study in the field found the highest performing interventions were adding quotations from authoritative sources, statistics and citations. BrightEdge measures 71.0 percent overlap between AI Overview citations and organic results in B2B Tech, so the organic programme is the AI visibility programme.<\/p>","protected":false},"excerpt":{"rendered":"<p>Most energy companies are running an AI visibility strategy built on advice that the platforms themselves contradict. Google&#8217;s own documentation, updated 10 July 2026, tells site owners not to build llms.txt, not to chunk content for AI, and not to add special schema for AI answers. Meanwhile the de<\/p>","protected":false},"author":12,"featured_media":4251,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"p54_article_data":"{\"meta\": {\"kicker\": \"Insight \u00b7 Specialism\", \"topics\": [\"Strategy\", \"Energy\"], \"title\": \"AI Crawler Access Control for Energy B2B: What Google Says You Can Stop Doing\", \"dek\": \"Most energy companies are running an AI visibility strategy built on advice that the platforms themselves contradict. Google's own documentation, updated 10 July 2026, tells site owners not to build llms.txt, not to chunk content for AI, and not to add special schema for AI answers. Meanwhile the decision that genuinely governs whether an assistant can cite you, which crawler you allow, is being made by accident in robots.txt files written before the fleets split. This dossier separates the controls that work from the ones being sold.\", \"date\": \"25 September 2026\", \"readTime\": \"14 min read\", \"author\": \"Project 54, Research & Strategy\"}, \"quickAnswer\": {\"q\": \"Do you need an llms.txt file to be cited by AI assistants?\", \"a\": \"No. Google states in its own generative AI optimisation guide, updated 10 July 2026, that you do not need to create AI text files, special markup or Markdown to appear in Google Search including its AI features, because Google Search does not use them and ignores them. No major AI provider has publicly committed to reading llms.txt from third party websites. Ahrefs measured every one of the 137,210 domains in its Web Analytics panel in May 2026 and found that 28 percent publish an llms.txt file, that 97 percent of those files received zero requests, and that AI retrieval bots, the ones that decide citations, accounted for just 1.1 percent of requests to the small remainder that were fetched at all. The control that does decide whether an assistant can cite you is robots.txt, and specifically which of a provider's three separate crawlers you allow: the training crawler, the retrieval crawler and the user triggered fetcher.\"}, \"takeaways\": [\"Blocking AI crawlers is three decisions, not one. Every major provider now runs a separate training crawler, retrieval crawler and user triggered fetcher, each controlled by its own robots.txt token. Blocking the training bot costs no visibility. Blocking the retrieval bot removes you from that assistant's answers.\", \"Google is explicit that llms.txt does nothing for Google Search. Its AI optimisation guide says creating one will neither harm nor help visibility, because Google Search ignores it. The measured evidence agrees: 97 percent of published llms.txt files were never requested once in May 2026.\", \"Google-Extended is a trap. It is not a crawler, it is a usage flag on data Googlebot has already taken, and blocking it does not remove you from AI Overviews or AI Mode. It does remove you from Gemini grounding. Publishers who blocked it to stay out of AI answers got the cost without the benefit.\", \"A client rendered site can rank in Google and be blank to ChatGPT. Google and Applebot render JavaScript. OpenAI, Anthropic and Perplexity say nothing about rendering in their documentation, and the only substantial log study found that no major AI crawler executes JavaScript.\", \"The market has already converged on the sensible split. Cloudflare reports that fewer than 1 percent of sites on its network block search bots while 17 percent block training. Publishers are refusing training and welcoming retrieval, which is exactly the distinction most robots.txt files fail to make.\", \"For B2B technology specifically, organic rank still predicts AI citation better than the cross industry average suggests. BrightEdge measured 71.0 percent overlap between AI Overview citations and organic results in B2B Tech, third highest of nine industries and up 32.4 points in sixteen months.\"], \"sections\": [{\"id\": \"three\", \"q\": \"Why is blocking AI crawlers three decisions, not one?\", \"h\": \"The fleets split, and most robots.txt files did not\", \"p\": [\"The structural change since 2024 is that every major AI operator has divided its crawler fleet by purpose, and each user agent is controlled independently in robots.txt. A robots.txt file written before that split treats a provider as a single entity. That is now a category error with a measurable cost.\", \"OpenAI's own documentation is unambiguous that <a href=\\\"https:\/\/developers.openai.com\/api\/docs\/bots\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">each setting is independent of the others<\/a>, and that a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot. GPTBot collects content that may be used in training. OAI-SearchBot builds the index ChatGPT retrieves from, and OpenAI states plainly that sites opted out of it will not be shown in ChatGPT search answers. ChatGPT-User fetches a page because a named person asked, and OpenAI notes that because those actions are initiated by a user, robots.txt rules may not apply.\", \"Anthropic publishes the same three way split with an unusually plain account of the consequences. Its <a href=\\\"https:\/\/support.claude.com\/en\/articles\/8896518\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">crawler documentation<\/a> states that disabling Claude-SearchBot prevents indexing for search and may reduce a site's visibility and accuracy in user search results, while restricting ClaudeBot signals that future materials should be excluded from model training. Anthropic is also the only major provider on the documented record that carves out no user triggered exception at all: it states its bots honour industry standard robots.txt directives and will not attempt to bypass anti circumvention technologies, with no agent side exemption.\", \"<a href=\\\"https:\/\/docs.perplexity.ai\/guides\/bots\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">Perplexity<\/a> is the outlier in the other direction. PerplexityBot is explicitly not used to crawl content for foundation models, which means Perplexity runs no training crawler on the public web at all. For an energy company whose objection is training rather than retrieval, Perplexity presents nothing to block. Perplexity-User, by contrast, generally ignores robots.txt, which Perplexity states in its own documentation.\", \"Two inheritance rules deserve attention from anyone writing tight rules. <a href=\\\"https:\/\/support.apple.com\/en-gb\/119829\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">Apple<\/a> states that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. <a href=\\\"https:\/\/developer.amazon.com\/amazonbot\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">Amazon<\/a> applies the same logic to Amzn-SearchBot against other search bots. A rule written for one crawler can silently govern several.\", \"The practical consequence for energy B2B is narrow and specific. Blocking training costs nothing in visibility and is purely a rights decision. Blocking retrieval removes you from an assistant's answers. Most of the blanket blocks deployed in 2024 and 2025 did both.\"], \"table\": {\"cols\": [\"Bot\", \"Job\", \"Blocking stops training?\", \"Visibility cost if blocked\"], \"rows\": [[\"GPTBot\", \"Training\", \"Yes\", \"None\"], [\"OAI-SearchBot\", \"Retrieval\", \"No\", \"Removed from ChatGPT search answers\"], [\"ChatGPT-User\", \"User triggered\", \"No\", \"OpenAI says robots.txt may not apply\"], [\"OAI-AdsBot\", \"Ad landing page validation\", \"No\", \"Breaks your own ChatGPT ads\"], [\"ClaudeBot\", \"Training\", \"Yes\", \"None\"], [\"Claude-SearchBot\", \"Retrieval\", \"No\", \"Reduced visibility in Claude search\"], [\"PerplexityBot\", \"Retrieval only\", \"No training use\", \"Removed from Perplexity results\"], [\"Google-Extended\", \"Usage control token\", \"Yes, for Gemini\", \"None in Search. See below\"], [\"Googlebot\", \"Search and all AI features\", \"No\", \"Catastrophic. Removes you from Search\"], [\"Meta-WebIndexer\", \"Retrieval\", \"No\", \"Removed from Meta AI citations\"], [\"Amzn-SearchBot\", \"Retrieval\", \"No\", \"Removed from Alexa and Amazon search\"]]}}, {\"id\": \"extended\", \"q\": \"Does blocking Google-Extended keep you out of AI Overviews?\", \"h\": \"No, and three years of publishers have been wrong about this\", \"p\": [\"Google-Extended is the most misunderstood control in the category, and Google's own documentation resolves it without ambiguity. It is not a crawler. Google states that <a href=\\\"https:\/\/developers.google.com\/crawling\/docs\/crawlers-fetchers\/google-common-crawlers\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">Google-Extended does not have a separate HTTP request user agent string<\/a>, that crawling is done with existing Google user agent strings, and that the robots.txt token is used in a control capacity. It is a flag on data Googlebot has already collected.\", \"What it controls is whether that content may be used for training future Gemini models and for grounding, meaning supplying content from the Search index to the model at prompt time. Both, not just training. What it does not control is Search: Google states directly that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.\", \"The trap is the next step. There is no AI Overviews crawler and no AI Mode crawler. Google's <a href=\\\"https:\/\/developers.google.com\/search\/docs\/appearance\/ai-features\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">AI features documentation<\/a> states that AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot are the control for site owners managing how their sites are crawled for Search. The only lever that removes you from AI Overviews is blocking Googlebot, which removes you from Google entirely.\", \"Publishers who blocked Google-Extended in 2023 and 2024 believing it would keep them out of AI answers were mistaken, and have been mistaken ever since. They did not avoid AI Overviews. They did remove themselves from Gemini grounding. That is a real cost with none of the intended benefit, and it is still sitting in a large number of robots.txt files that nobody has revisited.\", \"Below the nuclear option, the controls that genuinely exist are nosnippet, data-nosnippet, max-snippet and noindex. Apple offers a parallel and slightly finer instrument: its documentation describes nosnippet as opting content out of broad world knowledge answers in Siri and Search, separately from training.\"]}, {\"id\": \"llmstxt\", \"q\": \"Does anyone actually read llms.txt?\", \"h\": \"The measurement is in, and it is not close\", \"p\": [\"The <a href=\\\"https:\/\/llmstxt.org\/\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">llms.txt proposal<\/a> was published by Jeremy Howard of Answer.AI on 3 September 2024, and now carries a v2 modified 10 August 2026. It proposes a markdown file giving a project name, a summary and curated links to machine friendly content. Howard is honest about what it is: the specification states that llms.txt information is used on demand, when an agent needs information about a topic while assisting a user. Despite the filename, it is not an access control mechanism. It blocks nothing and permits nothing.\", \"The decisive question is whether the systems that decide citations consume it. Google has answered in primary documentation. Its <a href=\\\"https:\/\/developers.google.com\/search\/docs\/fundamentals\/ai-optimization-guide\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">generative AI optimisation guide<\/a>, updated 10 July 2026, carries a section headed mythbusting generative AI search, and states that you do not need to create new machine readable files, AI text files, markup or Markdown to appear in Google Search including its generative AI capabilities, because Google Search itself does not use them. It adds that maintaining such files for other systems will neither harm nor help visibility in Google Search, as Google Search ignores them.\", \"OpenAI, Anthropic and Perplexity say nothing about llms.txt in their crawler documentation, in either direction. No major provider has publicly committed to reading it from the open web.\", \"The empirical picture is the same. Ahrefs <a href=\\\"https:\/\/ahrefs.com\/blog\/llmstxt-study\/\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">analysed all 137,210 domains<\/a> in its Web Analytics panel that received traffic in May 2026, checked each root for an llms.txt returning HTTP 200, screened out soft 404s, and then examined every request made to those paths. Twenty eight percent of domains publish one, a figure Ahrefs flags as an upper bound because its customer base skews SEO aware. Ninety seven percent of those files received zero requests. Of the three percent that saw any traffic, 96 percent came from bots, and 77 percent of those bots were not AI tools. Slackbot fetched llms.txt files more often than PerplexityBot did.\", \"The honest verdict for an energy B2B publisher: if the goal is citation in ChatGPT, Perplexity, Claude or Google's AI features, llms.txt is decoration. It is cheap and harmless, and it is almost certainly inert. The one evidence backed exception is coding agents, where Claude Code out fetched every AI retrieval bot in Ahrefs' dataset. If you sell a developer facing product, an llms.txt over your documentation path is defensible. An energy operator's corporate site is not that case.\", \"Google is not internally consistent here and it is worth saying so. Days after the Search team published the mythbusting guidance, the Chrome team shipped an llms.txt check in Lighthouse's experimental agentic browsing audits. Both positions are live inside the same company. The Search position is the one that governs Search.\"], \"table\": {\"cols\": [\"Who requested llms.txt files in May 2026\", \"Share of requests to the 3% that were fetched\"], \"rows\": [[\"SEO audit tools\", \"21.7%\"], [\"AI agents and agentic infrastructure\", \"10.5%\"], [\"GEO and AEO tools, llms.txt scanners, research bots\", \"12.1%\"], [\"AI training crawlers\", \"5.3%\"], [\"AI assistants\", \"2.5%\"], [\"AI retrieval bots, which decide citations\", \"1.1%\"]]}}, {\"id\": \"render\", \"q\": \"Can an AI assistant actually see a JavaScript website?\", \"h\": \"Google renders. The AI native labs are silent, and the one study says no\", \"p\": [\"This is the most under priced technical risk in the category, because the failure is invisible from inside Google Analytics. A client rendered React or Vue site can rank perfectly well in Google Search while returning an empty shell to ChatGPT, Claude and Perplexity.\", \"Apple is the only AI adjacent operator that documents rendering officially. Its <a href=\\\"https:\/\/support.apple.com\/en-gb\/119829\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">Applebot documentation<\/a>, published 5 September 2026, states that Applebot may render the content of a website within a browser, and that if JavaScript, CSS and other resources are blocked via robots.txt it may not be able to render the content properly, including XHR. Google's position is documented by implication: its AI guide states that Google can process content within JavaScript as long as it is not blocked, and Google serves AI features from the same index and the same Googlebot control surface as classic Search.\", \"OpenAI, Anthropic and Perplexity say nothing. All three crawler documentation pages are silent on rendering in either direction. OAI-SearchBot's user agent string advertises a Chrome version, but a user agent string is cosmetic and is not evidence of rendering.\", \"The empirical basis for the common claim rests substantially on one piece of work: the <a href=\\\"https:\/\/vercel.com\/blog\/the-rise-of-the-ai-crawler\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">Vercel and MERJ log study<\/a> of 17 December 2024, which monitored server logs across the Vercel network and concluded that none of the major AI crawlers render JavaScript. It is credible and it is uncontradicted, but it is now 21 months old, it predates Claude-SearchBot and Meta-WebIndexer entirely, it ran substantially on one site, and both authors sell products whose value rises with the finding. No independent replication at comparable scale has been published. Treat it as the best available evidence, not as settled fact.\", \"Vercel's own caveat is the part most coverage drops, and it matters operationally: content present in the initial HTML response, including inline JSON, remains readable even though it is not rendered. So client side rendering is invisible. Any use of JavaScript is not.\", \"During the research for this dossier, two documentation pages returned nothing to a non rendering fetch. Microsoft's Bing Webmaster Tools help site returned only a message asking the reader to enable JavaScript. That is a sharper illustration of the risk than any vendor study: Microsoft's own webmaster guidance is invisible to a crawler that does not render.\", \"The test takes ten minutes and beats every study. Fetch your own highest value pages with curl and no JavaScript, and read what comes back. If your capability pages, your case studies and your technical specifications are not in that response, the assistants your buyers are asking cannot see them.\"]}, {\"id\": \"citation\", \"q\": \"What actually earns an AI citation?\", \"h\": \"Google's answer is that it is still SEO, and the data does not disagree\", \"p\": [\"Before reaching for vendor correlation studies, it is worth reading what Google has published directly. Its <a href=\\\"https:\/\/developers.google.com\/search\/docs\/fundamentals\/ai-optimization-guide\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">AI optimisation guide<\/a> states that generative AI features are rooted in core Search ranking and quality systems, using retrieval augmented generation against the Search index and query fan out. It then says something most agencies do not quote: that creating content people find unique, compelling and useful will likely influence a site's presence in generative AI search more than any of the other suggestions in the guide.\", \"The distinction Google draws is commodity versus non commodity content, and the example translates directly into energy B2B. Commodity content is a listicle of procurement tips. Non commodity content is what your engineers learned on a specific project that nobody else can write. Google's own framing is that optimising for generative AI search is optimising for the search experience, and thus still SEO.\", \"Google's mythbusting list is worth reading as a budget document. It says explicitly that there is no requirement to break content into tiny chunks for AI, that you do not need to write in a specific way just for generative AI search, that seeking inauthentic mentions is not as helpful as it might seem, and that structured data is not required for generative AI search with no special schema markup to add. It keeps one instruction on schema that is genuinely useful: make sure your structured data matches the visible text on the page.\", \"The independent evidence is consistent with this. The only peer reviewed controlled work in the field, <a href=\\\"https:\/\/arxiv.org\/abs\/2311.09735\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">Aggarwal and colleagues at KDD 2024<\/a>, tested interventions across roughly 10,000 queries and reports visibility gains of up to 40 percent. The highest performing interventions were adding quotations from authoritative sources, relevant statistics and cited sources. Not markup, not keyword density. That is a description of good research writing.\", \"Rank still matters, moderately. Ahrefs measured a <a href=\\\"https:\/\/ahrefs.com\/blog\/does-ranking-higher-on-google-mean-youll-get-cited-in-ai-overviews\/\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">Spearman correlation of 0.347<\/a> between top ten ranking and top three AI Overview citation at URL level across a million keywords. No published study has produced a URL level rank to citation coefficient above roughly 0.45. The strongest measured correlates in Ahrefs' <a href=\\\"https:\/\/ahrefs.com\/blog\/ai-brand-visibility-correlations\/\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">75,000 brand study<\/a> are not on page factors at all: YouTube mentions at roughly 0.737 and branded web mentions at 0.664 to 0.709 sit well above Domain Rating at 0.266 to 0.326. Note the tension with Google's warning about inauthentic mentions. The defensible reading is that earned mention volume correlates with citation because both are downstream of actually being a notable company.\", \"The single most useful figure for this sector comes from <a href=\\\"https:\/\/www.brightedge.com\/resources\/weekly-ai-search-insights\/rank-overlap-after-16-months-of-aio\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">BrightEdge's industry breakdown<\/a>: B2B Tech shows 71.0 percent overlap between AI Overview citations and organic results, third highest of nine industries and up 32.4 points over sixteen months, against 22.9 percent for e-commerce. For technical B2B, organic rank remains a materially better predictor of AI citation than cross industry averages suggest. The implication is not that AI visibility is a separate discipline requiring separate spend. It is that the organic programme you already fund is the AI visibility programme.\", \"One caution on the numbers in this section. With the exception of the peer reviewed KDD work and <a href=\\\"https:\/\/www.pewresearch.org\/short-reads\/2025\/07\/22\/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results\/\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">Pew's browsing panel study<\/a>, effectively every quantitative source on AI citation is published by a vendor selling AI visibility software. Read the methodology before you read the headline.\"]}, {\"id\": \"action\", \"q\": \"What should an energy B2B team change this quarter?\", \"h\": \"Six decisions, in order of how much they cost to get wrong\", \"p\": [\"First, audit robots.txt against the three way split. Separate the training tokens from the retrieval tokens and decide each one deliberately. If your organisation's position is that it objects to training but wants to be found, the file should say GPTBot disallow, ClaudeBot disallow, Meta-ExternalAgent disallow, and allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer. Cloudflare's network data suggests this is where the market has already landed: it reports that fewer than 1 percent of sites block search bots while <a href=\\\"https:\/\/blog.cloudflare.com\/accountable-mixed-use-ai-crawlers\/\\\" target=\\\"_blank\\\" rel=\\\"noopener\\\">17 percent block training<\/a>.\", \"Second, remove any Google-Extended block that was added to avoid AI Overviews. It never did that. It only removed you from Gemini grounding. If the objection to Gemini training is genuine, keep it deliberately; if it was defensive, it is costing you.\", \"Third, test rendering. Curl your ten highest value commercial pages with no JavaScript and read the output. Server render or statically generate main content, metadata, navigation and internal links. Reserve client side rendering for genuine enhancements. Get critical content into the initial HTML response rather than a post hydration fetch.\", \"Fourth, stop paying for llms.txt work unless you are selling to developers. If you already have one, leave it: it is harmless. Version control it, restrict edit rights, and keep the content to plain links and descriptions with nothing instruction shaped in it. Ahrefs noted that the largest single research crawler in its dataset identified itself as a prompt injection survey. Agents are designed to ingest and trust that file.\", \"Fifth, redirect the budget to what the evidence supports. Quotations from named sources, real figures with citations, and first hand operational detail are the interventions with controlled evidence behind them. In energy B2B that means project specifics, commissioning data, regulatory reading and named engineers, not another explainer on the energy transition.\", \"Sixth, know the OAI-AdsBot exists before you write a blanket OpenAI block. It validates landing pages submitted as ChatGPT ads and uses landing page content to judge relevance. If your organisation buys or intends to buy ChatGPT placement, a blanket block breaks your own media.\", \"The strategic point underneath all six is that the platforms have been more honest than the agencies. Google has published, in plain language, a list of things it says you can stop doing. Most energy marketing teams are still funding at least three of them.\"], \"pillars\": [{\"n\": \"01\", \"t\": \"Separate the three tokens\", \"d\": \"Training, retrieval and user triggered fetchers are independent controls. Decide each one. A blanket block buys a rights position you may not want at a visibility cost you did not intend.\"}, {\"n\": \"02\", \"t\": \"Get the content into the HTML\", \"d\": \"Google and Applebot render. The AI native labs are undocumented and the one log study says they do not. Server rendered content is the only version that is safe across all of them.\"}, {\"n\": \"03\", \"t\": \"Fund evidence, not format\", \"d\": \"The only controlled study in the field found that quotations, statistics and cited sources drive visibility. That is a research standard, not a markup schema.\"}]}], \"media\": {\"image\": {\"src\": \"\/wp-content\/uploads\/2026\/09\/ai-crawler-access-control-process-pipework-1.jpg\", \"label\": \"Access control is a routing problem. Three crawlers, three decisions, one file that usually treats them as one.\", \"credit\": \"Project 54\"}, \"infographicLabel\": \"The three way crawler split: a training crawler whose block costs no visibility, a retrieval crawler whose block removes you from an assistant's answers, and a user triggered fetcher most providers exempt from robots.txt entirely.\", \"pdf\": {\"href\": \"https:\/\/projectfifty4.com\/wp-content\/uploads\/2026\/09\/ai-crawler-access-llms-txt-energy-b2b.pdf\", \"title\": \"AI Crawler Access Control for Energy B2B, Slide Deck\", \"meta\": \"PDF \u00b7 briefing deck\"}}, \"poll\": {\"q\": \"What does your robots.txt actually say about AI crawlers right now?\", \"options\": [{\"id\": \"a\", \"label\": \"We block all of them\", \"insight\": \"This is the most common position and the most expensive. It buys a training opt out you could have had for free by blocking GPTBot and ClaudeBot alone, and pays for it by removing you from ChatGPT, Claude and Perplexity answers.\"}, {\"id\": \"b\", \"label\": \"We allow all of them\", \"insight\": \"Defensible, and it costs you nothing in visibility. It is a rights decision rather than a marketing one, and it is worth making consciously rather than by omission, because it is the default in the absence of a file.\"}, {\"id\": \"c\", \"label\": \"We block training and allow retrieval\", \"insight\": \"This is where Cloudflare's network data says the market has converged: under 1 percent of sites block search bots, 17 percent block training. It requires knowing which token does which job, which is why so few files get there by accident.\"}, {\"id\": \"d\", \"label\": \"Nobody has looked at it since 2024\", \"insight\": \"Then it almost certainly predates the fleet split, and very likely contains a Google-Extended block added to avoid AI Overviews, which never worked. Both are ten minute fixes with real upside.\"}], \"note\": \"Responses are anonymous and are used to shape future Project 54 research on AI visibility in energy B2B.\"}, \"faq\": [{\"q\": \"Does llms.txt help you get cited by AI?\", \"a\": \"There is no evidence that it does, and Google states directly that Google Search ignores it and that creating one will neither harm nor help visibility. No major AI provider has publicly committed to reading llms.txt from third party sites. Ahrefs measured 137,210 domains in May 2026 and found 97 percent of published llms.txt files received zero requests, with AI retrieval bots accounting for 1.1 percent of requests to the small share that were fetched. The one defensible use case is documentation for coding agents.\"}, {\"q\": \"Should I block GPTBot?\", \"a\": \"Blocking GPTBot is purely a rights decision and costs nothing in visibility. It stops content being used to train OpenAI's foundation models and has no effect on whether you appear in ChatGPT search answers. The bot that governs that is OAI-SearchBot, which is controlled separately. OpenAI states in its own documentation that each setting is independent of the others.\"}, {\"q\": \"Does blocking Google-Extended remove me from AI Overviews?\", \"a\": \"No. Google-Extended is not a crawler, it is a usage control token applied to data Googlebot has already collected, and it governs Gemini training and grounding. Google states that it does not impact inclusion in Google Search. There is no separate AI Overviews crawler, so the only way to be excluded from AI Overviews is to block Googlebot, which removes you from Google Search entirely.\"}, {\"q\": \"Can ChatGPT read a JavaScript heavy website?\", \"a\": \"OpenAI does not document its rendering capability in either direction. The best available evidence, a server log study by Vercel and MERJ published in December 2024, found that no major AI crawler executed JavaScript. Content present in the initial HTML response remains readable. The practical implication is that a client rendered site can rank normally in Google, which does render, while returning an empty page to AI assistants that do not. Testing your own pages with curl and no JavaScript settles it in ten minutes.\"}, {\"q\": \"What actually drives AI citation for a B2B energy company?\", \"a\": \"Ranking well organically, publishing first hand non commodity content with named sources and real figures, keeping it fresh, and earning genuine third party mentions. Google states that generative AI features run on its core Search ranking systems and that optimising for them is still SEO. The only peer reviewed controlled study in the field found the highest performing interventions were adding quotations from authoritative sources, statistics and citations. BrightEdge measures 71.0 percent overlap between AI Overview citations and organic results in B2B Tech, so the organic programme is the AI visibility programme.\"}], \"related\": [{\"title\": \"What Is Answer Engine Optimization, and What Changes for Energy B2B\", \"topic\": \"Strategy\", \"href\": \"https:\/\/projectfifty4.com\/what-is-answer-engine-optimization\/\"}, {\"title\": \"Generative Engine Optimization for Energy B2B\", \"topic\": \"Strategy\", \"href\": \"https:\/\/projectfifty4.com\/generative-engine-optimization-energy-b2b\/\"}, {\"title\": \"AI Search Visibility for Energy Companies\", \"topic\": \"Strategy\", \"href\": \"https:\/\/projectfifty4.com\/ai-search-visibility-energy\/\"}, {\"title\": \"AI Share of Voice in Energy B2B\", \"topic\": \"Strategy\", \"href\": \"https:\/\/projectfifty4.com\/ai-share-of-voice-energy-b2b\/\"}, {\"title\": \"The Energy Findability Index, Edition 1\", \"topic\": \"Strategy\", \"href\": \"https:\/\/projectfifty4.com\/energy-findability-index\/\"}, {\"title\": \"Proof Without Permission: Selling When Clients Cannot Be Named\", \"topic\": \"Sales Enablement\", \"href\": \"https:\/\/projectfifty4.com\/energy-b2b-proof-assets-confidentiality\/\"}, {\"title\": \"Your Search Impressions Are Not Buyer Demand\", \"topic\": \"Analytics & Attribution\", \"href\": \"https:\/\/projectfifty4.com\/search-impressions-not-demand-energy-b2b\/\"}, {\"title\": \"AI Crawl to Referral: The Energy B2B Measurement Gap\", \"topic\": \"AI Visibility\", \"href\": \"https:\/\/projectfifty4.com\/ai-crawl-to-referral-measurement-energy-b2b\/\"}], \"newsletter\": {\"kicker\": \"The Energy Growth Brief\", \"title\": [\"Intelligence,\", \"to your inbox\"], \"body\": \"Join energy and industrial leaders getting our marketing, AI-growth and revenue-architecture intelligence, direct, no filler.\", \"placeholder\": \"you@company.com\", \"cta\": \"Subscribe\", \"note\": \"No spam. Unsubscribe anytime. We read every reply.\"}}","p54_faq":"","p54_media":"","p54_comments_enabled":"","footnotes":""},"categories":[92,125],"tags":[],"class_list":["post-4255","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analysis","category-strategy"],"acf":[],"_links":{"self":[{"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/posts\/4255","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/comments?post=4255"}],"version-history":[{"count":1,"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/posts\/4255\/revisions"}],"predecessor-version":[{"id":4263,"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/posts\/4255\/revisions\/4263"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/media\/4251"}],"wp:attachment":[{"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/media?parent=4255"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/categories?post=4255"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/projectfifty4.com\/fr\/wp-json\/wp\/v2\/tags?post=4255"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}