{"id":10253,"date":"2026-08-31T08:05:59","date_gmt":"2026-08-31T06:05:59","guid":{"rendered":"https:\/\/www.al-hostings.com\/arquitectura-web-ia\/"},"modified":"2026-08-31T08:25:57","modified_gmt":"2026-08-31T06:25:57","slug":"ai-oriented-architecture","status":"publish","type":"post","link":"https:\/\/www.al-hostings.com\/en\/ai-oriented-architecture\/","title":{"rendered":"AI-oriented web architecture: 5 technical pillars"},"content":{"rendered":"
\n

<\/p>\n

Quick answer<\/span><\/p>\n

An AI-oriented website architecture is one that lets answer-engine crawlers (GPTBot, ClaudeBot, PerplexityBot) crawl, understand and extract your content with the minimum obstacles. Its five pillars are: robots.txt with explicit permissions for AI bots, redirect chains of at most 2 hops, clean semantic hierarchy with correct headings, entity-oriented schema (Organization + Person), and up-to-date XML sitemaps. llms.txt is trendy but optional: bot adoption is still low and limited in 2026.<\/p>\n<\/div>\n

<\/p>\n

This is the most technical article in the silo, and it doesn’t beat around the bush. It’s aimed at anyone who wants to understand exactly how to structure their website so AI engines like ChatGPT, Perplexity and Claude can crawl, process and cite it. Many sites that rank well in Google are invisible<\/em> to AI because of architecture errors that Googlebot tolerates and AI bots don’t.<\/p>\n

One data point to show why this matters: in 2024, 35.7% of the top 1,000 websites blocked GPTBot<\/a> \u2014 and many without knowing. It’s the most common error: blocking AI by accident and wondering why nobody cites you.<\/p>\n

<\/p>\n

\n

The 5 pillars of an AI-oriented architecture<\/h2>\n

<\/p>\n

\n
1<\/div>\n
\n

robots.txt: the real access lever<\/h3>\n

This is the file that decides whether AI bots can enter or not. Not optional. If your robots.txt has no rules specific to AI bots, they all follow the User-agent: *<\/code> directive. If that default is restrictive, you’re blocking them without meaning to. I develop this in the next section with specific configuration.<\/p>\n<\/div>\n<\/div>\n

<\/p>\n

\n
2<\/div>\n
\n

Short redirect chains (max 2 hops)<\/h3>\n

This is the technical error that kills the most well-ranked-in-Google sites when it comes to AI. Captain DNS data (2026)<\/a>: Googlebot tolerates up to 10 redirect hops; real-time AI crawlers (OAI-SearchBot, Claude-SearchBot, Perplexity-User) give up at the third hop without retrying<\/strong>. A chain that passes for Google removes your page from AI answers with no error signal. The target must be 1-2 hops max.<\/p>\n<\/div>\n<\/div>\n

<\/p>\n

\n
3<\/div>\n
\n

Clean semantic hierarchy (H1 \u2192 H2 \u2192 H3)<\/h3>\n

AI engines parse HTML just like Google does, but with less tolerance for disorder. One H1 per page, H2 for sections, H3 for subsections. No skips (never H1 \u2192 H3). Headings must actually describe the content that follows \u2014 AI extracts fragments using this structure as a guide. A vague heading (“Our proposal”) performs worse than a descriptive one (“How we structure an SEO audit in 4 phases”).<\/p>\n<\/div>\n<\/div>\n

<\/p>\n

\n
4<\/div>\n
\n

Entity-oriented schema (not just page-level)<\/h3>\n

AI builds knowledge graphs, not lists of pages. Priority: Organization<\/code> with sameAs<\/code> to LinkedIn, Wikidata and real profiles; Person<\/code> for authors with credentials; knowsAbout<\/code> with your topics. The goal is for AI to identify your brand as an entity<\/em>, not as a collection of pages. I develop this in structured data: how they help SEO and AI<\/a>.<\/p>\n<\/div>\n<\/div>\n

<\/p>\n

\n
5<\/div>\n
\n

Up-to-date and accessible XML sitemap<\/h3>\n

Still the most efficient way to tell a crawler what pages you have and when they were updated. Declaration in robots.txt (Sitemap: https:\/\/yoursite.com\/sitemap.xml<\/code>), real <lastmod><\/code> dates (not faked), and all important URLs included. With WPML or multilingual sites: one sitemap per language or a sitemap index.<\/p>\n<\/div>\n<\/div>\n<\/div>\n

<\/p>\n

Practical robots.txt configuration for AI<\/h2>\n

Here’s the important nuance almost nobody explains: there are two types of AI bots<\/strong> and each is handled differently. Training bots<\/em> (GPTBot, Google-Extended, CCBot, anthropic-ai) collect content to train future models. Retrieval bots<\/em> (ChatGPT-User, Claude-User, PerplexityBot, OAI-SearchBot) read your site in real time when someone asks a question that mentions your category.<\/p>\n

The most common strategy in 2026: block training, allow retrieval. You block your content from entering future models (an intellectual property decision) while letting AI cite you in real-time answers (a visibility decision). Example:<\/p>\n

\n
# Training bots \u2014 BLOCK<\/div>\n
User-agent: GPTBot<\/div>\n
User-agent: Google-Extended<\/div>\n
User-agent: CCBot<\/div>\n
User-agent: anthropic-ai<\/div>\n
Disallow: \/<\/div>\n
# Retrieval bots \u2014 ALLOW<\/div>\n
User-agent: ChatGPT-User<\/div>\n
User-agent: OAI-SearchBot<\/div>\n
User-agent: Claude-User<\/div>\n
User-agent: Claude-SearchBot<\/div>\n
User-agent: PerplexityBot<\/div>\n
User-agent: Perplexity-User<\/div>\n
Allow: \/<\/div>\n
# Sitemap<\/div>\n
Sitemap: https:\/\/yoursite.com\/sitemap.xml<\/div>\n<\/div>\n

If you want maximum AI visibility with no restrictions<\/em> (recommended if your priority is being cited and you don’t care about training use), the setup simplifies: allow everything explicitly.<\/p>\n

<\/p>\n

\n

Important CDN warning<\/p>\n

If you use Cloudflare or another CDN, the CDN layer can override your robots.txt. Cloudflare has implemented default blocks for AI bots in several configurations, and many sites end up blocking crawlers at the CDN while their robots.txt says “allow.” Both layers must agree: check the CDN dashboard, not just the server’s robots.txt.<\/p>\n<\/div>\n

<\/p>\n

llms.txt: between the hype and real usefulness<\/h2>\n

llms.txt is a Markdown file proposed by Jeremy Howard (Answer.AI) in September 2024 that lives at the domain root and gives LLMs a curated overview of your content. It has gained huge media attention in recent months. And here we need to be honest.<\/p>\n

What’s true:<\/strong> Anthropic references it in its documentation, Google has included it in A2A protocol experiments, and some AI crawlers visit the file when it exists.<\/p>\n

What’s also true:<\/strong> no major AI provider has officially stated it processes it in production, adoption across domains is around 0.3% of the top 1,000 websites (March 2026), and independent analyses conclude that no material effect on AI retrieval is attributed to it in 2026<\/em>.<\/p>\n

My practical recommendation: if you have 1-2 hours, implement it \u2014 it’s low-risk, low-effort and functions as an act of informational hygiene for the day it does get read. But don’t prioritize it over robots.txt, schema or headings<\/strong>, which do have measurable impact today. And don’t treat it as “access control”: it isn’t. It’s a suggested guide, not a technical barrier.<\/p>\n

<\/p>\n

How to structure HTML so AI extracts it well<\/h2>\n

Access architecture (robots.txt, redirects) lets AI enter<\/em>. Semantic architecture makes it want<\/em> to cite you. Five concrete rules:<\/p>\n

\n
\u2713<\/div>\n
\n

Self-contained direct answer in the first paragraph<\/h3>\n

AI extracts literal citations preferentially from the first 40-60 words of a section. If your first paragraph is introductory or context without data, no citation. Every H2 should be followed by a paragraph that directly answers the implicit question of that heading.<\/p>\n<\/div>\n<\/div>\n

\n
\u2713<\/div>\n
\n

Frequent tables and numbered lists<\/h3>\n

LLMs extract tables and lists much better than long paragraphs. In articles over 1,000 words, include at least one comparison table or numbered list. Don’t force them \u2014 use them when the content naturally calls for one (comparisons, sequences, checklists).<\/p>\n<\/div>\n<\/div>\n

\n
\u2713<\/div>\n
\n

Visible dates: publication and update<\/h3>\n

AI values recency. Publish both the publication date and the last-updated date in the HTML (with datePublished<\/code> and dateModified<\/code> in Article schema). A 2026 article will be cited before a 2022 one on the same topic, even if the second has more authority.<\/p>\n<\/div>\n<\/div>\n

\n
\u2713<\/div>\n
\n

No blocking JavaScript for critical content<\/h3>\n

AI crawlers today are less capable than Googlebot at rendering JavaScript. If your main content only loads after JS runs, for many AI bots it doesn’t exist<\/em>. Server-side rendering, content in the HTML of the first response, or hybrid rendering. A React site without SSR is architecture hostile to AI.<\/p>\n<\/div>\n<\/div>\n

\n
\u2713<\/div>\n
\n

Internal links with descriptive anchor text<\/h3>\n

“Click here” or “more info” anchors tell AI nothing. Descriptive anchors with the destination’s keyword (“complete guide to schema for AI”) help AI understand your content map and build related citation chains.<\/p>\n<\/div>\n<\/div>\n

<\/p>\n

Architectural errors that make your site invisible to AI<\/h2>\n

The five I see most in well-ranked-in-Google sites that don’t<\/em> appear in ChatGPT or Perplexity:<\/p>\n

\n\n\n\n\n\n\n\n\n\n
Error<\/th>\nHow to detect and fix it<\/th>\n<\/tr>\n<\/thead>\n
robots.txt blocks GPTBot by default<\/td>\nVisit yoursite.com\/robots.txt<\/code> and look for rules for GPTBot\/ClaudeBot\/PerplexityBot. If they’re missing or there’s a general Disallow: \/<\/code>, check it. Many security plugins block them by default.<\/td>\n<\/tr>\n
Redirect chains of 3+ hops<\/td>\nUse curl or tools like httpstatus.io. Typical problem chains: HTTP \u2192 HTTPS \u2192 www \u2192 canonical folder. Collapse into a single redirect.<\/td>\n<\/tr>\n
JavaScript-dependent content<\/td>\nDisable JS in your browser and visit your site. If the main content disappears or goes blank, you have a serious AI-accessibility problem.<\/td>\n<\/tr>\n
Multiple H1s or broken hierarchy<\/td>\nExtensions like HeadingsMap (Chrome) show the visual hierarchy. One H1 per page, no H1 \u2192 H3 skips with no H2 in between.<\/td>\n<\/tr>\n
Blocked from CDN (Cloudflare, etc.)<\/td>\nCheck your CDN dashboard. Cloudflare has “Block AI Scrapers” and automatic settings that may be active without you having configured them explicitly.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

<\/p>\n

\n

Architecture checklist for AI<\/h2>\n

Ten concrete checks. With five or more \u274c, your site probably doesn’t appear in AI answers even if it ranks in Google.<\/p>\n