Outdated Technology May Be Reducing Your Visibility in AI Search

There's no single universal process. Different platforms use crawlers, search indexes, retrieval systems, and user-triggered visits. Still, the general flow can be understood in four stages:

• AI bots have their own rules: OpenAI uses OAI-SearchBot to surface pages in ChatGPT search results — that access can be managed separately from GPTBot, which relates to the possible use of content for model training. Perplexity documents a similar role for PerplexityBot. In both cases, the official guidance points to robots.txt, IP ranges, and security settings, not a universal two- or three-second reading limit. [1][2]

• Google sticks to SEO fundamentals: At Google, generative experiences remain built on Search's systems and index. The official guidance is to keep the fundamentals of SEO in place: crawlable pages, clear technical structure, useful and trustworthy content, and a good user experience. [3][4]

The age of the code, by itself, says little. An HTML page built years ago can be extremely fast and easy to crawl. Likewise, a freshly launched app can be heavy, unstable, and dependent on JavaScript in a way that makes content hard to access.

• Rule of thumb: Evaluate the site's behavior, not the technology's label. Version, support, security, accessibility, stability, and implementation quality matter far more than the name of the framework or CMS.

Access, structure, and content failures rarely show up alone — it's the combination of them that makes a company harder to find, understand, and trust:

• Crawlers blocked by robots.txt, a firewall, or a CDN: A site can be public to regular users and still block legitimate bots through robots.txt rules, anti-bot services, IP lists, or JavaScript challenges, which directly affects the discovery and processing of pages. To improve: review the rules applied to OAI-SearchBot, PerplexityBot, and Googlebot; confirm that your firewall or CDN allows the IP ranges published by the providers; analyze server logs to tell blocks, 403 errors, challenges, and network failures apart; and avoid treating robots.txt as a security mechanism — private content should require authentication. [1][2][8]

• Instability, HTTP errors, and slow server responses: Speed doesn't work like a switch where AI reads a page at 2.9 seconds and abandons one at 3.1 — but 5xx errors, timeouts, unstable DNS, and high latency make access less reliable and hurt both crawlers and users. To improve: monitor uptime, TTFB, 4xx/5xx errors, and DNS failures; use caching, a CDN, compression, and infrastructure that can handle traffic spikes; and fix redirect chains and external dependencies that block loading.

• Poor loading experience: Core Web Vitals measure real aspects of the experience — loading, interactivity, and visual stability. As a benchmark, Google recommends an LCP of 2.5 seconds or less, an INP of 200 milliseconds or less, and a CLS of 0.1 or less, measured at the 75th percentile; these metrics don't guarantee ranking or citation, but they help identify pages that frustrate users. To improve: prioritize main content and reduce render-blocking resources; compress and size images, using modern formats where appropriate; cut unnecessary JavaScript, excessive fonts, and third-party scripts; and use field data, not just isolated lab tests. [6]

• Main content dependent on fragile JavaScript rendering: JavaScript isn't incompatible with search engines — the problem arises when titles, text, links, or essential information only appear after calls that fail, require interaction, depend on a session, or lack a proper fallback. Google processes JavaScript, but documents specific steps and limitations for this type of page. To improve: deliver critical content in the initial HTML via SSR, SSG, or another suitable strategy; use real links with crawlable URLs and correct HTTP statuses; test pages with JavaScript disabled and in URL inspection tools; and make sure API errors don't result in empty pages. [5]

• Confusing semantic structure and navigation: Automated systems rely on structural cues: a clear H1, coherent subheadings, focused paragraphs, lists, tables, and descriptive internal links all help both users and machines understand the page. To improve: use a consistent heading hierarchy, without jumps made purely for looks; create dedicated pages for services, solutions, industries served, and real questions; connect related content through contextual internal links; and avoid hiding important information exclusively inside images or interactive components.

• Missing, incorrect, or contradictory structured data: Structured data gives explicit clues about articles, organizations, products, local businesses, and other entity types — it doesn't replace content and doesn't guarantee visibility, but it helps engines interpret the page, and the markup must match what the user actually sees. To improve: implement only applicable types, such as Article, Organization, BreadcrumbList, Product, or LocalBusiness; keep name, address, author, dates, prices, and availability consistent with the visible content; validate the implementation with the Rich Results Test and the Schema Markup Validator; and don't use Schema.org to declare information that doesn't appear on the page. [7]

• Neglected mobile experience and accessibility: Sites that are hard to use on small screens also tend to have navigation, interaction, hierarchy, and comprehension problems — accessibility benefits people and makes components clearer to automated agents too. To improve: use semantic HTML before adding ARIA; add clear labels to forms, buttons, menus, and interactive elements; ensure contrast, keyboard focus, tap target sizes, and proper readability on mobile devices; and avoid pop-ups and overlays that hide the main content.

• Generic, outdated, or unsubstantiated content: Technical access is just the entry point: a site can be perfectly crawlable and still not get chosen as a source because it offers shallow content that's indistinguishable from competitors or hard to trust. To improve: answer real audience questions with context, examples, and depth; show authorship, experience, sources, review dates, and verifiable criteria; update information when the subject changes, not just to bump the page's date; and produce original knowledge, such as case studies, methodology, comparisons, and data.

No technology offers an automatic advantage. The outcome depends on implementation, infrastructure, and content.

• There's no magic framework: A modern site can be bad, and an old site can be excellent. The decision to rebuild should come from evidence: security risks, maintenance cost, performance, ability to evolve, and real crawling or conversion obstacles.

Without inventing universal percentages, it's possible to map measurable business impacts. The relationship should be validated in each company's own context.

Mark each item "yes", "partial", or "no". The diagnosis should prioritize critical blockers and problems with proven impact, not an arbitrary score.

• Stable HTTP: Important pages consistently return HTTP 200.

• Valid HTTPS: The site has valid HTTPS and shows no security warnings.

• AI bots allowed: OAI-SearchBot, PerplexityBot, and Googlebot aren't blocked by mistake.

• Firewall and CDN configured: Firewall, CDN, and anti-bot protection are configured for legitimate crawlers.

• Main content accessible: The main content is available in the HTML or renders reliably.

• Clear titles and descriptions: URLs have clear titles, H1s, subheadings, and descriptions.

• Descriptive internal links: Internal links use crawlable URLs and descriptive text.

• Accessible pages: Relevant pages aren't isolated or buried too deep.

• Correct sitemap: The sitemap contains only canonical, indexable URLs.

• Robots.txt without improper blocks: robots.txt doesn't unnecessarily block CSS, JavaScript, or essential pages.

• Core Web Vitals monitored: Core Web Vitals are tracked using field data.

• Third-party resources controlled: Images, fonts, and third-party scripts are kept under control.

• Mobile works: The site works properly on mobile devices.

• Accessible components: Forms and interactive components are accessible.

• Valid structured data: Structured data is valid and matches the visible content.

• Consistent information: Company, service, author, and contact information is consistent.

• Evidence-backed content: Relevant content has authorship, dates, and verifiable evidence.

• Review process: There's a review process for information that changes over time.

• AI access measurement: Logs and analytics are used to measure access and traffic from AI platforms.

• Tracking queries: There's a fixed set of queries used to track presence and citations over time.

• How to run the migration: When a rebuild is necessary, the current site should stay online while the new version is developed in a staging environment. The deploy requires preserving URLs whenever possible and 301 redirects for pages that change address.

There's no universal citation-per-load-second rate, nor a guaranteed timeline for results. The reliable path is to record a baseline and compare indicators before and after changes.

• Avoid the anecdotal test: Asking a question once on ChatGPT or Perplexity isn't an audit. Use a documented set of queries, dates, platforms, and results, plus logs and business metrics.

• If my site is slow, why does it still show up on Google? Because speed is just one part of the experience and doesn't work as a standalone indexing or ranking criterion. Relevance, quality, structure, authority, and many other signals are part of the process. That said, severe slowness, instability, and errors can hurt users and limit access.

• Does modernizing the site fix everything? No. Technical modernization removes obstacles, but it doesn't replace useful content, reputation, authority, distribution, and value proposition. The outcome depends on technology, content, and digital presence working together.

• Which technology is best for GEO? There's no universal winner. The best option is whichever delivers understandable HTML, stable URLs, a good experience, security, governance, and the ability to publish quality content without unnecessary complexity.

• Do I need to create an llms.txt file? For Google Search, no. Official guidance states that special files like llms.txt aren't required to appear in Search's generative experiences. For other platforms, there's no universal standard that replaces crawlability, content, and official crawler settings. [3]

• Does structured data guarantee my company will get cited? No. It helps engines understand the meaning of the page and can enable rich experiences, but it doesn't guarantee indexing, visibility, or citation. The markup needs to be valid and reflect the visible content.

• How often do I need to update content? Whenever the information changes or the page stops serving the user's intent well. A stable institutional page can remain accurate for a long time; a pricing, legal, or technology page may need frequent review.

• How long does it take to see results? There's no universal timeline. Access fixes can be noticed after the next crawl; changes in content, authority, and conversion need to be tracked over weeks or months. Treat the timeline as a hypothesis, and measure it.

The technical claims in this article were reviewed based on the official documentation below. Accessed on August 5, 2026.

• [1] OpenAI — Overview of OpenAI Crawlers
• [2] Perplexity — Perplexity Crawlers
• [3] Google Search Central — Guide to Optimizing for Generative AI Features
• [4] Google Search Central — AI Features and Your Website
• [5] Google Search Central — Understand JavaScript SEO Basics
• [6] web.dev — Web Vitals
• [7] Google Search Central — Introduction to Structured Data Markup
• [8] Google Search Central — Introduction to robots.txt
• [9] OpenAI Help Center — Publishers and Developers FAQ
• [10] Google Search Central — Creating Helpful, Reliable, People-First Content