Prerendered Content for Search and AI Bots

A product page can look complete to a shopper and still tell only part of the story to a crawler. Descriptions, specifications, and customer reviews may appear only after JavaScript runs, sometimes through a third-party service. If a search or AI crawler misses that content, it loses information that helps people discover and choose the product. Slow responses can also make a large catalog harder to crawl efficiently.

How Harper makes complete pages easier to read

Harper adds a page preparation and delivery layer behind the existing CDN. One cluster coordinates prerendering, while another stores and serves approved HTML snapshots across regions. The website stays in place. Permitted bots receive a ready-to-read version of the same public content, while ordinary visitors continue through the existing website path.

The preparation cluster follows the sitemap to discover URLs and schedules checks for changes. A daily pass can provide a baseline, with publishing events or submitted URLs triggering earlier refreshes. Headless-browser workers open pages, run their JavaScript, and wait for required content before capturing the result. For reviews loaded by a third party, the rendering rules must explicitly wait for those reviews and include them in the saved HTML.

New or changed pages are rendered and validated before replacing the approved copy. Checks can confirm that expected reviews, headings, product details, and links are present. Sitemap timestamps help prioritize work, but changes from a review provider may require a scheduled render or a separate update signal. Preparation runs ahead of requests so each bot visit does not need to rebuild the page.

At the CDN, configured bot detection and routing send eligible crawler requests to Harper. On Akamai, this can use edge routing logic such as EdgeWorkers alongside the chosen bot controls. The delivery cluster returns stored HTML close to the requester. Fabric manages regional replication, routing, and failover. Teams can add a delivery region in a few clicks by selecting the region and confirming deployment.

The same stored pages can also support origin resilience. If the origin fails, a configured CDN fallback can send eligible visitor requests to approved pages in Harper. Visitor fallback needs usable styles, images, navigation, and an acceptable content age, rather than just a stripped-down bot response. It can keep product discovery available and preserve purchase opportunities while the origin recovers. Checkout and other live functions still depend on their backing services.

Alternatively, relying on the website to render content

The usual starting point is the existing CDN and website. The CDN can cache the HTML it receives, but caching alone does not run the JavaScript that adds reviews or other content in the browser. Google can render JavaScript, with limitations. Other crawlers have different capabilities, so a page that works for one bot may remain incomplete for another.

Teams can move important content into server-rendered or statically generated HTML for all visitors. This is a strong long-term approach when the website can be changed. A separate prerendering service is another option when that change is difficult. It introduces its own rendering jobs, snapshot storage, refresh rules, and routing integration. Harper combines the coordination, stored pages, and regional serving layer while browser workers perform the rendering.

Where missing content and refreshes add work

A successful page load does not prove that every review or product detail arrived. Teams need to recognize incomplete renders, track changes beyond the sitemap, and refresh affected pages without overwhelming the origin or third-party providers. Large catalogs multiply that work. Separating preparation from delivery lets teams control browser workload independently while bots keep receiving approved pages.

Why Harper matters

• More of the page can inform discovery. Include public reviews and other JavaScript-loaded details so search and AI systems can read the information customers use to choose.
• Faster delivery across the catalog. Serve prepared HTML close to crawlers, reducing response time and repeated rendering work across popular and long-tail pages.
• Less bot-driven origin work. Reuse approved snapshots across requests and schedule preparation to control load on the website.
• More value from the same page store. Configure suitable snapshots for visitor fallback, helping preserve shopping journeys during origin disruption.
• Simpler global expansion. Add delivery regions in a few clicks, with Fabric managing replication and regional routing.

The commercial goal is more discoverable products and fewer lost visits. Readable, fast pages support that goal, but search engines and AI services still decide what to index, cite, or recommend. Measure discovery and revenue outcomes alongside page coverage and crawl performance.

Plan for complete, current pages

Define which public content must be present before a render is accepted. Preserve canonical URLs, status codes, indexing directives, and structured data. Keep the substance equivalent for people and bots, and honor crawler access policies. Separate locale and other public page variants, and exclude account-specific or private content from shared snapshots.

Set refresh schedules by content type, cap rendering concurrency, and choose what happens when a page is missing or too old. Do not replace a valid page with a failed render. Configure CDN cache separation and routing so bot responses cannot accidentally become the normal visitor experience. Test visitor fallback independently, including assets, live features, and recovery.

Related resources

Prerender for Bots explains Harper’s approach to preparing and delivering HTML. The public Static Prerender Template provides a starting point for scheduling, page storage, and headless-browser workers.

Why Your Brand Is Invisible to AI Search explores content accessibility for AI crawlers. Google’s rendering guidance recommends server-side or static rendering as long-term solutions and describes bot-specific rendering as a workaround.

Start with pages where content is being missed

Pilot product pages with third-party reviews, JavaScript-heavy category pages, and less-visited catalog pages. Compare the source HTML, rendered visitor page, and bot snapshot. Measure content completeness, refresh age, response time, crawl errors, origin load, and rendering cost. Track indexed coverage, referral traffic, and resulting conversions over time. Test an origin outage before enabling visitor fallback.

Bring your sitemap, priority crawlers, and examples of missing content to a Harper architecture workshop. Identify which pages to prepare first and where faster, more complete delivery can support discovery and revenue.

‍