I build and repair websites for real estate agents for a living, and over the past year I’ve audited dozens of production IDX sites – takeovers from other platforms, sites agents asked me to diagnose, sites that “should” have been ranking – the way a search engine sees them: no browser, no JavaScript, just raw HTML. What I found should worry anyone who has ever paid for an “SEO-optimized” IDX website:
Most of them were serving completely empty pages to Google.
Not thin pages. Not poorly optimized pages. Empty pages – a title tag and a blank div. One site had 66 blog posts listed in its sitemap, every single one returning zero readable content to a crawler. The agent had been publishing consistently for months and could not figure out why nothing ranked.
This article explains why this happens to IDX real estate websites specifically, how to test whether it’s happening to yours, and exactly how to fix it on the static hosting platforms most agent sites run on. Everything here comes from problems I’ve personally found and fixed on real production sites – this is field-repair experience, not theory.
Why IDX websites are uniquely vulnerable
IDX (Internet Data Exchange) websites pull live MLS listings into an agent’s site. Because the listing data changes constantly and the search experience needs to feel like an app – filters, maps, instant results – almost every modern IDX platform is built as a single-page application (SPA) using React, Vue, or a similar JavaScript framework.
Here’s the problem. When a SPA is client-side rendered (CSR), the server sends the browser a nearly empty HTML file:
The actual content – the agent bio, the neighborhood guides, the blog posts, the listings – only exists after the browser downloads and executes that JavaScript bundle. A human visitor never notices. A crawler that doesn’t execute JavaScript sees exactly what’s above: nothing.
Googlebot can render JavaScript, but as Google’s own JavaScript SEO documentation explains, rendering is a second-pass process with its own queue and budget – it can silently fail on script errors or timeouts, and it delays indexing. And as we’ll cover later, the AI crawlers that now drive a growing share of discovery – the bots behind ChatGPT, Perplexity, and Claude – largely don’t execute JavaScript at all. For them, a client-side rendered site simply does not exist.
Real estate agents get hit hardest by this because the content that should be their SEO moat – hyper-local neighborhood pages and consistent blogging – is exactly the content locked inside the JavaScript bundle. All the fundamentals of real estate SEO still apply, but none of them matter if the crawler can’t read the page. And the industry’s baseline crawl health is already poor: SEO Sandwitch’s own real estate SEO statistics found that 61% of real estate sites have crawl errors affecting at least 10% of their pages.
How to test what Googlebot actually sees
Don’t trust your eyes, and don’t trust “View Source” alone. Here are the tests I run on every site, in order of increasing rigor.
1. Fetch the page like a crawler
The fastest test is a terminal command. This fetches a page with no JavaScript execution, the way a non-rendering bot sees it:
If your blog post’s headline comes back, the page is crawlable. If you get nothing – or you grep for div id=”root” and find it sitting empty – the page is invisible to non-rendering crawlers.
Run this against your money pages: the homepage, your about page, two or three neighborhood pages, and at least three blog posts. Don’t just test one URL and generalize. In my audits I’ve found sites where the homepage was fine but every blog post was empty, and one site where it was the reverse.
2. Disable JavaScript in your browser
In Chrome DevTools: Settings → Preferences → Debugger → Disable JavaScript, then reload the page. This is the same test in visual form, and it’s the one to show a client or a skeptical vendor, because a blank white page makes the argument for you.
3. Google Search Console URL Inspection
Search Console’s URL Inspection → View Crawled Page shows you the HTML Google actually stored. Check the rendered HTML and the “More info” panel for JavaScript errors. This is the ground truth for Google specifically – but remember it tells you nothing about the AI crawlers that don’t render.
The trap to avoid: testing only with tools that render JavaScript (Lighthouse, most “SEO audit” browser extensions, or your own browser). They will cheerfully report a healthy page that no non-rendering crawler can read. Always include at least one raw-HTML test.
The fix: prerendering on static hosting
The standard advice is “migrate to Next.js” – and if you’re starting from scratch, server-side rendering or static-site generation is the right call. But most agent sites are already built, already deployed on static hosts like AWS Amplify, Netlify, or Vercel, and a full migration is rarely in the budget. The good news: you can retrofit crawlability onto an existing SPA with build-time prerendering, and it works entirely within static hosting.
The recipe has three parts.
Part 1: Generate real HTML at build time
For every public route, produce an actual HTML file during the build. For content that lives in a CMS or API (like blog posts), write a build script that fetches the published content and writes one static file per post – complete with the real headline in an <h1>, the full article body, a canonical tag, and structured data. The output looks like:
build/
index.html ← homepage with real content
about/index.html
neighborhood/maple-heights/index.html
blog/
index.html ← blog index listing every post
why-spring-is-selling-season.html
first-time-buyer-guide.html
Two hard-won practical notes on the build script itself:
- Bound your concurrency. One site I remediated had 205 blog posts, and its generator fetched all of them (plus an image each) simultaneously. The build exhausted its network sockets and failed – which meant deploys failed, which meant the fix made things worse before it made them better. Batch the fetches (I use a concurrency of 8) and let one bad post skip rather than abort the run.
- Don’t fail the whole deploy if the content API hiccups. Log the error and exit cleanly. A site with yesterday’s blog posts is better than a site that didn’t deploy.
Part 2: Rewrite rules that serve the static files
Here’s the part almost everyone gets wrong. Generating the HTML files does nothing if your hosting still routes every request to the SPA shell. Static hosts use a catch-all rewrite (/* → /index.html) to make client-side routing work – and that catch-all will happily swallow your prerendered pages too.
The rewrite set needs to serve static files first and fall through to the SPA last. For blog posts, use a wildcard pair rather than one rule per post:
1. /blog/<*>.html → /blog/<*> (301: canonicalize away the .html)
2. /blog/<*> → /blog/<*>.html (200: serve the static file)
3. /<everything else> → /index.html (200: SPA catch-all, LAST)
Three details that each cost me a debugging session:
- Rule order matters. The .html-to-clean 301 must come before the clean-to-.html serve rule. Get them backwards and a request for post.html rewrites to post.html.html – an infinite redirect loop.
- The catch-all must exclude .html (and .xml, images, etc.). Most catch-all regexes exclude common static extensions but forget html itself, so the rewritten /blog/post.html request falls into the catch-all and serves the empty shell anyway.
- Wildcards, never per-post rules. A setup I’ve encountered more than once generates one rewrite rule per blog post, which has to be re-pasted into the hosting console on every publish. Nobody does that reliably, which is exactly how sites end up with a crawlable blog from March and invisible posts ever since. Two wildcard rules cover every post forever.
Part 3: Match your slugs to your sitemap – exactly
This one is subtle, and I caught it on a remediation the same week I’m writing this. The static files were named from a slug derived from the post title: lowercase, strip punctuation, spaces to hyphens. The backend’s sitemap generator derived its URLs from the same titles – but with one difference: it didn’t trim trailing hyphens. A title with an accidental trailing space became …/custom-home-in-tulsa- in the sitemap, while the file generator wrote …/custom-home-in-tulsa.
Result: 9 out of 65 posts had a sitemap URL that didn’t match any static file, fell through to the SPA catch-all, and served an empty shell – on a site that looked fully fixed on a spot check. Titles with apostrophes cause the same class of bug (“Don’t” → dont vs don-t).
The lesson: the sitemap is the crawl surface. Whatever generates it defines the truth, and your file-naming must match it byte for byte. Verify with a diff, not a spot check – extract every URL from the live sitemap and confirm a static file exists for each one.
Sitemap and robots.txt mistakes I keep finding on IDX sites
Prerendering is the big fix, but the supporting files sabotage IDX sites in their own special ways. Every one of these is from a real production site.
| Mistake | What it does |
| Sitemap lists URLs that serve empty shells | Worse than no sitemap: you’re inviting Google to crawl 66 pages of nothing, teaching it your site is low-value. |
| Sitemap: line in robots.txt points at the wrong domain | I found one pointing at the platform’s demo site domain (a template-copy leftover) and another pointing at an internal project name that isn’t even a registered domain. Google was being directed to someone else’s sitemap – or nowhere. |
| Canonical tag hardcoded to the wrong domain | Same template-copy failure mode. One site’s canonical pointed at a non-existent domain on every page, telling Google “the real version of this page is somewhere that doesn’t resolve.” |
| Site-name <h1> in the noscript fallback | A well-intentioned <noscript> bio block put <h1>Agent Name</h1> before the app root – so every prerendered page led with the same generic H1 ahead of its actual topic heading, diluting every page’s relevance signal. |
| Individual MLS listing URLs in the sitemap | Listing pages are transient – they expire when the property sells. Feeding thousands of soon-to-be-404 URLs to Google wastes crawl budget. Keep /listing/* out of the sitemap (and consider disallowing it in robots.txt). |
The pattern behind most of these: agent sites are built by copying a template or demo site, and hardcoded values – domains, site IDs, canonicals – survive the copy. The most extreme case I found was a site whose blog was serving the demo site’s content because a site-ID constant was never updated. Audit anything hardcoded in your HTML template against your actual production domain. And crucially: on a CSR site, the crawler-visible canonical and meta tags live in the static index.html, not in whatever JavaScript config your framework reads – on one repair I fixed the config file, redeployed, and the live canonical was still wrong because the static template had its own hardcoded copy. The template is what crawlers read; the config is what developers read.
What AI crawlers need that Googlebot doesn’t
Everything above gets you indexed by Google. But an increasing share of real estate discovery now happens inside AI assistants – buyers asking ChatGPT or Perplexity “who’s a good realtor in [city]” or “what’s the market like in [neighborhood].” Being citable there is a different, and in some ways stricter, bar.
1. No JavaScript rendering, period
GPTBot, ClaudeBot, PerplexityBot and their peers fetch raw HTML. There is no render queue, no second pass. Prerendering isn’t an optimization for these crawlers – it’s the entry ticket. This is why I test with an AI-crawler user agent rather than Googlebot’s: it holds the site to the stricter standard.
2. Let them in via robots.txt
Many robots.txt files predate these bots and only address User-agent: *. That technically covers them, but if you have any bot-specific rules, audit them. I explicitly welcome the AI crawlers I want citing my clients:
(Whether to allow AI training crawlers is a legitimate business decision – but for a local service business, being the answer an AI gives is nearly pure upside.)
3. llms.txt: a map for machines
llms.txt is an emerging convention: a plain-Markdown file at your site root that summarizes who you are and lists your key pages, so an AI agent can orient itself without crawling everything. For an agent site, mine look like:
# Jane Smith – Real Estate Agent
> Licensed real estate professional serving Springfield and surrounding areas.
## Contact
– Phone: (555) 555-0100
– License: 0123456
## Key Pages
– Homes for sale in Springfield: https://example.com/springfield
– Client reviews: https://example.com/reviews
– Real estate blog: https://example.com/blogs
## Service Areas
– Springfield, Maple Heights, Riverside
It costs ten minutes and gives every AI crawler a clean, unambiguous summary of exactly the facts you want repeated: name, license, service areas, and where the good content lives.
4. Structured data that answer engines can quote
JSON-LD structured data matters double for AI answers because it’s pre-parsed fact. The set I ship on every agent site:
- RealEstateAgent (site-wide): name, phone, license number, brokerage, service areas. This is the schema type that exists precisely for this business – use it, not generic LocalBusiness.
- BlogPosting on every post: headline, date, author, mainEntityOfPage. In one audit, the largest blog I checked – 200+ posts – was shipping none at all. Every post on every site should carry it, generated automatically by the same build script that prerenders the post.
- FAQPage on guide content: question-and-answer content is the closest match to how people query AI assistants, and marking it up makes it trivially extractable.
One rule: generate JSON-LD in the same build step that generates the HTML, from the same data. Hand-maintained structured data drifts out of sync with the visible content, and inconsistency is worse than absence. And don’t trust the build blindly – validate the output with a schema markup testing tool after each template change.
The checklist
- curl your homepage, three blog posts, and two neighborhood pages with a bot user agent. Real headlines and body text, or empty div?
- If empty: add build-time prerendering – one static HTML file per public route and per blog post.
- Add rewrite rules: static files first (wildcards for blogs), SPA catch-all last, with html and xml excluded from the catch-all.
- Diff every URL in your live sitemap against your static files. Fix slug mismatches exactly – including trailing hyphens and apostrophes.
- Verify robots.txt: correct Sitemap: domain, AI crawler user agents addressed, /listing/* and admin routes disallowed.
- Grep your static HTML template for hardcoded domains in canonical, og:url, and JSON-LD. Check for stray <h1>s in noscript blocks.
- Add llms.txt, RealEstateAgent schema site-wide, and BlogPosting schema on every post.
- Re-run step 1 after every deploy. Crawlability regresses silently; the test is one command.
None of this is exotic. It’s plumbing. But it’s plumbing that determines whether the content an agent works hard to produce is visible to the systems people actually use to find them – and on the evidence of my own audits, most IDX websites have never had it checked. It’s why I made build-time prerendering and structured data the default on the AI-assisted IDX websites I build: crawlability shouldn’t be a repair job.
Editor’s Note: This is a guest post by Craig Topham, the founder of AgentWinds, where he builds AI-assisted, crawlable-by-default websites for real estate agents and spends an unreasonable amount of time reading raw HTML the way crawlers do. Every failure described in this article came from a real production site he audited and repaired.
