404 Pages, Soft 404s, and Link Rot: Error Handling for SEO

Updated 2026-09-06 · guide · technical, SEO

A 404 page is not a small detail — it's where your crawl budget, your link equity, and your AI citations can quietly drain away. Most sites get the basics wrong: they return soft 404s (a 200 status with a "page not found" body), leave dead links rotting in their content, or let a removed page return an empty error instead of a redirect. This guide covers correct error handling for the search-and-AI era: proper status codes, error pages that earn trust instead of destroying it, and the link-rot workflow that protects everything you've built.

Every link on the internet is a promise, and every 404 is a broken promise. For classic SEO, that costs you crawl budget and link equity. For AI-era SEO — where engines and agents decide which pages to trust and cite — an error experience that returns a misleading 200 or a dead end actively signals "this site is unreliable." Getting error handling right is cheap, mostly one-time work, and it protects everything downstream.

Why error handling matters more for AI product sites

Two things make error handling a bigger deal for AI product sites than for a typical brochure site:

1. AI engines treat broken signals as untrustworthiness. A soft 404 — a page that returns a 200 status but shows "not found" content — is exactly the kind of sloppy signal AI crawlers learn to discount. The engines that decide whether to cite you are watching for inconsistency, and error pages are where sloppiness shows up.
2. Your content gets linked to from places you don't control. When other sites, directories, and AI-generated answers point at your URLs, those links arrive with no warning. Link rot — the slow decay of old or moved URLs — is something you inherit as your site ages, and it only compounds.

The stakes are easy to miss because error pages are, by definition, the pages you never look at. But the crawler looks at every one. That's the asymmetry that makes this worth your time.

The status-code fundamentals (get these right)

Error handling starts with returning the correct HTTP status code. Everything else is decoration. The three codes that matter:

The classic mistakes live in the middle of that table:

• Soft 404s. A page returns 200 OK but the body says "Not Found." This is the most common and most damaging error: crawlers index a useless page, wasting index slots, and users see a broken page that looks like success. Fix: your app's not-found handler must return a real 404 status code, not just render a "not found" template with a 200.
• 404 instead of 301. When you move a page, a 404 throws away the accumulated link equity and any AI citations pointing at the old URL. A 301 tells everyone — including the AI engines that cached the old page — where the content went. The canonicals, redirects, and migrations guide has the full rules for when 301 beats canonical and vice versa.
• 200 for removed pages. If you delete content and return a generic 200 homepage, you've told the crawler "this page exists and is the homepage," splitting signals across a redirect the crawler has to figure out.

The rule of thumb: make the status code match reality. If the page is gone, say gone (404/410). If it moved, say where (301). If it exists, say exists (200) — and use noindex for pages that shouldn't rank.

Build a 404 page that earns trust

The 404 status code is non-negotiable, but the content of your 404 page is where you can turn a failure into a small win. A good 404 page does three jobs:

1. Admits the error honestly. No "this page has mysteriously vanished" euphemisms. Plain language: "That page isn't here (or has moved)."
2. Gives the visitor a next step in one click. A search box, a link to your most important pages (docs, pricing, home), and ideally a "popular articles" list. The goal is to keep a human from bouncing — every second of confusion is a lost session.
3. Stays on-brand and consistent. A jarring, broken-looking error page reads as neglect. A clean, on-brand 404 that's clearly designed reads as competence — which is exactly the signal an AI engine evaluating your site's quality picks up.

For an AI product site specifically, your 404 page should link to the pages you most want cited: the docs home, the main product page, and your best guides. You're not just rescuing a human visitor — you're handing a crawler that hit a dead end a map to your real content.

Soft 404s: the silent index-killer

Soft 404s deserve their own section because they're the most common error that's also invisible in your analytics. A soft 404 happens when a page returns a 200 status with content that says "not found" — or worse, returns the homepage. Crawlers then index a page that:

• has almost no useful content (an empty "not found" shell), or
• is a duplicate of your homepage (the "we redirect everything to home" anti-pattern).

Both pollute the index. The empty-shell variant wastes index slots with thin pages; the redirect-to-home variant splits your homepage's signals across dozens of URLs. Either way, your crawl budget gets spent on pages that help no one.

The fix is systematic, not one-off. Three checks to run on your site:

Link rot: the slow leak you inherit

1. Crawl for "200 with 'not found' in the title." Fetch a few known-dead URLs and inspect the response status. If your CMS or framework returns 200 for anything routed to a 404 template, fix the status code at the router level — this is a framework configuration issue, not a per-page one.
2. Check your search console's "not found" report. Google (and the AI engines that expose reports) list pages returning 404. The ones returning 404 are actually fine — what you want to hunt for is the missing ones: URLs that should 404 but instead return a 200.
3. Verify your redirects land on a real 200. A chain of 301 → 301 → soft-404 is a failure you'll never see unless you follow the whole chain. Test your redirects end-to-end, not just the first hop.

Link rot is the process by which URLs decay — pages move, get deleted, or silently change — and it's worse for AI product sites because your content accumulates links from sources you don't control. Every external site pointing at an old URL, every AI answer that cites a version of your page that no longer exists, is a leak.

The workflow that contains it:

1. Find the rot. Use your search console's coverage report and a crawl tool to list every URL returning 404 or 410 that still has inbound links — either from your own content or from external sites. Dead URLs with no links pointing at them are harmless; dead URLs with links pointing at them are the ones that matter.
2. Decide each URL's fate. For each dead URL, one of three outcomes:

• Redirect (301) if there's a clear successor page — this preserves link equity and AI citations.
• Restore if the content still has value and you simply removed it by accident (this happens more than teams admit).
• Let it 404 if nothing reasonably replaces it and no meaningful links point at it.

1. Fix your own outgoing links. The rot you control is the rot you're responsible for: every internal link to a dead URL in your own content. This connects directly to the internal linking strategy guide — healthy internal linking includes periodic link-hygiene sweeps, because dead internal links bleed authority from your site just like dead external ones do.

Link rot isn't a one-time cleanup; it's a recurring discipline. A quarterly sweep that redirects the dead, fixes the broken internal links, and re-checks the status codes is enough to keep the leak closed. It's also a natural moment to run content pruning — deciding which dead URLs deserve a redirect and which are simply dead weight that should stay gone (pruning and consolidation guide).

What this has to do with AI citations

Here's the part that's specific to AI-era SEO: AI engines and agents cache and cite your URLs. When an answer references your guide, that reference is a link into your site — and if it lands on a soft 404, a dead end, or a page that's been moved without a redirect, several bad things happen:

• The citation is wasted. An answer pointing at a broken page doesn't convert — the reader clicks, hits a dead end, and the trust you earned is spent on a 404.
• The engine learns your URLs are unreliable. Every broken or mislabeled URL that a crawler follows and gets a soft 404 from is a small vote against your site's dependability. Over time that's exactly the kind of signal that keeps you out of AI answers.
• The fix is the same as the classic fix. Keep status codes honest, redirect moved pages, keep a clean index. The GEO discipline of "be findable, be citeable, be quotable" starts with not letting your citations land in a hole.

So error handling is not a hygiene afterthought — it's part of your backlink and citation strategy. Every link you earn is a promise; error handling is how you keep it.

A practical audit (30 minutes)

Run these five checks and you'll know exactly where your error handling stands:

1. Fetch 10 known-dead URLs. Confirm each returns 404/410, not a soft 200.
2. Follow your redirect chains. Pick five 301s and verify they land on a real 200 with the right content — no chains longer than two hops, no loops.
3. Review your search-console coverage report. Note how many URLs are "not found (404)" and which of those still have inbound links.
4. Scan your internal links for dead targets. A crawl or a simple link checker will surface the internal links pointing at 404s.
5. Open your 404 page as a user. Does it clearly explain the error, and does it offer a one-click next step to your most valuable pages?

Most sites fail checks 1 and 4 — soft 404s and internal dead links. Fix those first; they're the highest-leverage and the cheapest.

Common mistakes

Bottom line

• Soft 404s everywhere. The framework returns 200 for "not found." This is the #1 error and it's a router-level bug, not a content problem. Fix the status code, not the page.
• Redirecting everything to the homepage. A blanket redirect-to-home is a soft-404-in-disguise that splits your homepage's signals across every dead URL. Redirect to a specific successor or let it 404.
• 404 pages that are dead ends. An error page with no links is a guaranteed bounce — for humans and for the crawler that hit it. Always offer a next step.
• Never auditing link rot. Dead URLs with inbound links accumulate silently. A quarterly sweep is cheap and prevents the slow bleed.
• Forgetting the AI crawlers. AI engines hit the same dead URLs your users do — often more thoroughly, because they crawl at scale. Make sure your error handling serves GPTBot, ClaudeBot, and PerplexityBot exactly as it serves Googlebot and your users.

Error handling is where SEO discipline shows up when nobody's looking: honest status codes, 404 pages that rescue visitors instead of losing them, and a link-rot workflow that redirects or restores the dead before it bleeds your equity and citations. The soft 404 is your number-one enemy — fix it at the router, not the template. Your single next action: fetch ten known-dead URLs on your site and check the status codes; fix any that return 200.

FAQ