Sitemap XML Format
urlset with url entries: loc, lastmod, changefreq, priority. Get the required loc field right first — it must be a fully-qualified, canonical URL on the same host as the sitemap itself. Interviewers often point out that Google treats changefreq and priority as weak hints and largely ignores them, so an accurate lastmod timestamp is the field that actually influences how often a page gets recrawled.
Sitemap Index
For sites with >50K URLs, generate multiple sitemap files + index file referencing them. The same limits apply to each child file: at most 50,000 URLs and 50MB uncompressed, which is the constraint that forces the split in the first place. A common follow-up is what happens past that scale — you create several index files and list each one in robots.txt, since the spec does not allow an index file to reference another index file.
SitemapEntry Table
SitemapEntry (
id,
url,
lastmod,
changefreq: always/hourly/daily/weekly/monthly/yearly/never,
priority DECIMAL 0.0-1.0,
sitemap_file_id
)Generation Process
Query SitemapEntry ordered by priority DESC → batch into files of max 50K URLs → generate XML → gzip → upload to S3 (sitemap1.xml.gz, sitemap2.xml.gz) → generate sitemap-index.xml. The ordering scan sorts the entire entry set — an O(n log n) step — so back it with an index on the columns you sort and filter by, and stream the write in 50K-row pages so memory stays bounded no matter how large the site grows.
Incremental Update
On content publish/update, upsert SitemapEntry + re-generate only affected sitemap file. This needs a stable mapping from a URL to its sitemap_file_id so you know which file to rewrite; keep that assignment sticky rather than re-sharding URLs across files on every change. Deletes matter too — on unpublish, remove the row and regenerate the file so stale URLs don’t linger in the index.
Priority Rules
- Homepage: 1.0 — the single most important entry point; reserve 1.0 for it alone so the relative ordering below it still carries meaning.
- Category pages: 0.8 — hubs that link out to many content pages; high enough to signal they’re worth crawling regularly but ranked below the homepage.
- Content pages: 0.7 — the bulk of the sitemap; keep these uniform so no single article outranks the sections that organize the site.
- Tag pages: 0.5 — thin, often near-duplicate index pages; keep them low, and consider dropping tags that point at only one or two items.
Exclusions
Dynamic pages excluded: search results, user-specific pages, admin URLs. The point is to spend crawl budget on canonical, indexable pages — listing search-result or faceted-filter URLs invites duplicate content and near-infinite crawl paths. Interviewers probe how you draw the line: a URL belongs in the sitemap only if it returns 200, is canonical to itself, and isn’t blocked by robots.txt or a noindex tag.
robots.txt Integration
Auto-append Sitemap: URL line to robots.txt. Use the absolute URL and point it at the index file rather than each child sitemap. Handle the edge case of an existing Sitemap: line — update it in place instead of appending a duplicate every time the job runs.
Search Engine Ping
GET request to Google/Bing ping URL after generation. Treat it as fire-and-forget: send it asynchronously and don’t fail generation if the ping errors, since crawlers still discover the sitemap through robots.txt. Note that Google deprecated its sitemap ping endpoint in 2023 in favor of robots.txt discovery, so the ping now matters mostly for Bing and IndexNow.
Schedule
Full regeneration weekly + incremental on every publish. The weekly full pass is a safety net that repairs drift from missed events or failed incremental writes; the per-publish incremental keeps lastmod fresh without rewriting every file. A strong answer scopes the incremental job to only the one sitemap file holding the changed URL, so a single edit never triggers a full rebuild.
See also: Atlassian Interview Guide
See also: Shopify Interview Guide
Keep sharpening your system design:
