Technical SEO Checklist for AI Products (Step-by-Step)

Updated 2026-09-06 · checklist · SEO, technical, robots, sitemap

Here's a painful truth we learned the hard way: a website can be fully open, fast, and functional — and still invisible to Google.

We once shipped a product where the most important conversion page was hidden from Google and from the AI crawlers we'd whitelisted — for three months. Humans could open it fine. Crawlers couldn't. To the search engine, that page simply didn't exist.

The problem wasn't our code. It was the boring, unglamorous stuff nobody explains: robots rules, rendering, sitemaps, and how crawlers actually walk your site.

This is the checklist that fixes it. If you build AI products, work through it in order.

Why "it works in my browser" isn't enough

Technical work needs access, dependency mapping, and approval rules from day one. This SEO service onboarding assets guide pairs with the technical SEO checklist. Technical packages should be scoped from repeatable evidence; this SEO pricing models guide helps turn audit findings into priced workstreams.

Launch assets need technical readiness as well as copy. This feature announcement SEO guide pairs with the technical SEO checklist before release.

Technical fixes are easier to fund when they are framed as risk reduction and revenue enablement. This SEO budget and ROI reporting guide shows how to include remediation spend in a credible investment model. Crawlers are not browsers. They:

• follow links and rules, not your clicks
• often don't execute JavaScript (or only partially)
• read what the server returns directly, before any client-side rendering
• stop when a rule or an error says stop

That's why a page can be perfect in a browser and invisible to search. The funnel is:

Can it be found?  →  Can it be read?  →  Can it be indexed?  →  Will it rank?

Fix each layer in order. If a layer fails, everything below it is wasted.

Level 1: Can it be found? (rules & crawl access)

1.1 Audit your robots.txt — the #1 trap

This step now has a second audience: the AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and friends) that read the same robots.txt — often with different interpretations than Googlebot. A rule that looks fine for Google can be a hard stop for an AI crawler, which is exactly how you end up invisible to the AI engines while ranking fine on Google. We keep a full rundown in the AI crawlers guide.

A single bad line can block your best pages. Common traps:

• Disallow: / — blocks the entire site (yes, people ship this)
• Disallow: /checkout or /pricing — accidentally hiding money pages
• wildcard rules that are stricter than you think
• different crawlers reading the same rule differently — a rule one tool treats as a "green light" can be a hard stop for another

The fix: list every page that must be public, then confirm nothing in robots.txt blocks it. Test with the "as Googlebot" view in Search Console.

1.2 Submit a correct sitemap

A sitemap doesn't force indexing, but it makes discovery fast and explicit. Rules that actually matter:

1.3 Use the right internal links

• only include pages you want indexed (leave out admin, staging, thank-you pages) — and remember that paginated and faceted pages multiply fast, so control them deliberately; the pagination and faceted navigation guide shows how.
• use absolute URLs matching your canonical domain
• keep it under 50,000 URLs / 50MB per sitemap
• reference it from robots.txt and submit it in Search Console

Crawlers find pages by walking links. If a page is only reachable through a button that requires JavaScript, crawlers may never see it. Make sure important pages are reachable via plain, server-rendered <a href> links in your navigation or content.

Level 2: Can it be read? (rendering & content)

During a migration, apply those checks through the site migration SEO playbook with staged templates and release gates.

2.1 Server-render your important pages

If your content is rendered client-side (JavaScript builds the DOM), a crawler that doesn't run JS sees an empty shell. For critical pages:

• Prefer server-side rendering (SSR) or static generation (SSG)
• or use prerendering for crawler requests
• or ensure critical content is in the initial HTML response

Test: disable JavaScript in a browser and load your page. Is the core content still there? If not, that's what many crawlers see. For the full deep-dive on render strategies, SPAs, and prerendering, see JavaScript rendering and SEO for AI products. Remember the crawler checks the mobile version too — under mobile-first indexing, the mobile experience must contain the same full content (mobile-first SEO for AI products).

2.2 Give every page a unique title & description

2.3 Use clear headings and structure

2.4 Structured data (schema)

• one unique <title> per page (include the keyword naturally, keep it under ~60 chars)
• one unique meta description (under ~155 chars, written for humans)
• no "Untitled", no duplicated boilerplate across pages
• exactly one <h1> per page, containing the main topic
• logical h2 / h3 hierarchy after that
• semantic HTML (<article>, <nav>, <main>) — crawlers and AI engines both parse this

Add JSON-LD structured data where it applies:

• Organization / WebSite for the site
• Article / FAQPage for content pages (FAQ schema also feeds AI answers)
• Product / SoftwareApplication for your product

Structured data tells both Google and AI engines exactly what your page is. It's one of the highest-leverage, lowest-effort wins on this list — and it's the enabling layer for the rich results covered in the SERP features guide.

2.5 AI-crawler readiness: llms.txt and engine crawl budgets

Your pages are no longer read only by Google. By 2026, a real share of your referral traffic arrives as citations inside ChatGPT, Claude, Perplexity or Gemini — and those engines retrieve from pages their own crawlers fetched. Two 2026 additions to the technical checklist:

• Ship an llms.txt — a plain-text index of your most important pages, in the site root. AI engines check it before deciding what to trust and read; think of it as a sitemap specifically for AI crawlers. We cover the format and rules in the llms.txt guide.
• Mind each engine's crawl budget. GPTBot and friends fetch fewer pages than Googlebot, so the pages that matter most (your docs, your top guides) should be reachable from the homepage in one or two clicks and listed in both the sitemap and llms.txt. Depth in the site structure costs you AI visibility even when Google finds everything.

Add these two to your checklist and the invisible-to-AI-engines failure mode largely disappears — which is the same foundation our GEO guide builds on.

Level 3: Can it be indexed? (health signals)

Deprecation can create duplicate or blocked paths; the deprecation and docs-churn SEO guide keeps status and canonical rules clean.

3.1 HTTPS everywhere (non-negotiable)

Secure connection is a ranking signal and a trust signal. No exceptions in 2026.

3.2 Fast on mobile

Google indexes the mobile version of your site. If the mobile page is slow or broken, that's the version that gets judged:

3.3 Watch your crawl stats

• compress images (WebP/AVIF, lazy-load below the fold)
• minimize JS/CSS; avoid heavy frameworks for marketing pages
• target under ~2.5s on a mid-range phone

In Search Console, watch for:

Level 4: Will it rank? (content depth)

• a sudden drop in "discovered, not indexed" pages — investigate
• rising "server error" counts
• pages "excluded by noindex" you didn't intend

Technical SEO gets you in the game. Content depth decides whether you win.

The practical workflow

• Word count is not the goal, but thin content loses. In a test, our 550-word conversion page couldn't compete with competitors at 1500–3000 words. More useful, complete content outperforms.
• Answer the intent, not just the keyword. Cover the question fully: definition, how-to, examples, caveats, next steps.
• Update stale pages. Freshness is a ranking signal, and it's also what AI engines trust.

Sites and workflows keep changing, so technical work often becomes a monthly system; see SEO and AI Service Retainers for scope models.

Report technical fixes as outputs and signals, not promises; see Client Reporting for SEO and AI Services for the scorecard format.

Access and approval delays are delivery risks; see Client Onboarding for AI and SEO Services for the onboarding checklist.

Turn the audit workflow into a testable scope; see AI Service Proposals for deliverables, exclusions, and pricing.

A technical audit can become a scoped service offer; see Service Pages for AI Products for deliverable and intake structure.

The bottom line

1. Run an audit — use a technical SEO checker (like geo-seo-claude or a crawler tool) on your live site.
2. Save a baseline report — before you change anything. Note the numbers: indexed pages, discovered-not-indexed, average render time.
3. Fix Level 1 first — findability is the foundation. Fix rules, sitemap, links.
4. Add AI-crawler readiness in the same pass — confirm llms.txt exists, key pages are in it, and nothing in robots.txt blocks GPTBot/ClaudeBot/PerplexityBot from the pages you want cited.
5. Move down the funnel — rendering, then index health, then content.
6. Re-run the audit in 2–4 weeks — compare against baseline. The change over time is the real signal, not the single snapshot.

The vast majority of "my AI product doesn't get traffic" problems are Level 1 and Level 2 issues, not bad content. Before you write another article, make sure Google can actually find and read your site.

Fix the funnel in order, measure the before/after, and repeat. That's the entire game — boring, but it works.

Technical recommendations need an implementation owner and release window; this SEO SOW guide assigns those responsibilities before work starts.

Technical checks should preserve crawl paths, rendering, and depth rules defined in the service business site architecture.

Use this checklist to test crawlability, rendering, indexation, and speed during the SEO discovery phase.

The maintenance calendar handed over in SEO client offboarding and win-back should include the technical SEO checklist.

Next: how GEO builds on this foundation to get your content cited by AI engines.

FAQ