AI Sitemap vs Flat llms.txt

Token Efficiency Benchmark Report (V2 — dual-strategy websites only, no correctness)

Model: gemini-3-flash-preview Generated: 2026-03-07 09:39:17 Questions: 70
Token Savings (Sitemap vs Flat)
81.0%
across websites with both strategies
Avg Latency
15.4s
Flat: 18.5s · Sitemap: 12.4s
Websites Tested
7
70 total questions

Total Tokens by Website

Token Savings % by Website (Sitemap vs Flat)

Latency Comparison (seconds)

Tool Calls Comparison (avg per question)

Per-Website Breakdown

vercel.com

MetricFlat llms.txtAI SitemapSavings
Total Tokens1,785,706217,12687.8%
Avg Context Tokens/Q79,0188,36289.4%
Avg Tool Calls/Q2.02.8-
Avg Latency (s)16.314.6-
CANNOT_FIND00-
Errors00-

sardine.ai

MetricFlat llms.txtAI SitemapSavings
Total Tokens1,293,045290,29777.5%
Avg Context Tokens/Q40,54710,11075.1%
Avg Tool Calls/Q2.73.2-
Avg Latency (s)14.812.9-
CANNOT_FIND00-
Errors00-

coinbase.com

MetricFlat llms.txtAI SitemapSavings
Total Tokens73,014223,883-206.6%
Avg Context Tokens/Q1,42115,916-1019.7%
Avg Tool Calls/Q4.02.1-
Avg Latency (s)16.110.9-
CANNOT_FIND20-
Errors30-

elevenlabs.io

MetricFlat llms.txtAI SitemapSavings
Total Tokens4,453,132260,84994.1%
Avg Context Tokens/Q150,95816,99388.7%
Avg Tool Calls/Q4.82.4-
Avg Latency (s)29.612.2-
CANNOT_FIND00-
Errors50-

upstash.com

MetricFlat llms.txtAI SitemapSavings
Total Tokens535,121320,47740.1%
Avg Context Tokens/Q19,74823,264-17.8%
Avg Tool Calls/Q6.72.2-
Avg Latency (s)23.214.1-
CANNOT_FIND00-
Errors30-

mintlify.com

MetricFlat llms.txtAI SitemapSavings
Total Tokens267,398172,69935.4%
Avg Context Tokens/Q12,85211,6959.0%
Avg Tool Calls/Q3.82.5-
Avg Latency (s)14.412.0-
CANNOT_FIND00-
Errors42-

langchain.com

MetricFlat llms.txtAI SitemapSavings
Total Tokens679,540238,37664.9%
Avg Context Tokens/Q29,92615,79347.2%
Avg Tool Calls/Q2.92.2-
Avg Latency (s)19.99.9-
CANNOT_FIND00-
Errors31-

Per-Question Detail

ID ▲▼ Question ▲▼ Strategy ▲▼ Answer ▲▼ Tokens ▲▼ Context ▲▼ Tools ▲▼ Found? ▲▼ Latency ▲▼
coinbase_01What is Coinbase AgentKit and what does it enable?flat10,5941,6754Y16.2s
coinbase_01What is Coinbase AgentKit and what does it enable?sitemap18,13613,6772Y9.3s
coinbase_02What types of wallets does the Coinbase Developer Platform oflat4,9181,5772Y10.6s
coinbase_02What types of wallets does the Coinbase Developer Platform ositemap18,15413,6772Y7.8s
coinbase_03What is the difference between Coinbase's custodial and selfflatERR-----
coinbase_03What is the difference between Coinbase's custodial and selfsitemap16,65811,7482Y13.2s
coinbase_04What is the Coinbase Paymaster and how does it work?flat8,6088734Y19.8s
coinbase_04What is the Coinbase Paymaster and how does it work?sitemap18,08413,6772Y7.9s
coinbase_05What is Coinbase Vault and how does it protect cryptocurrencflat11,62105N17.9s
coinbase_05What is Coinbase Vault and how does it protect cryptocurrencsitemap16,61611,7482Y11.0s
coinbase_06What is Coinbase One and what benefits does it offer?flat6,62904N14.5s
coinbase_06What is Coinbase One and what benefits does it offer?sitemap26,02320,4952Y10.9s
coinbase_07What testnets and faucet assets does the Coinbase Developer flat14,5463,2344Y17.6s
coinbase_07What testnets and faucet assets does the Coinbase Developer sitemap18,27413,6772Y8.3s
coinbase_08How does Coinbase secure private keys across its different wflatERR-----
coinbase_08How does Coinbase secure private keys across its different wsitemap52,71031,4373Y14.3s
coinbase_09What infrastructure does Coinbase provide for institutional flatERR-----
coinbase_09What infrastructure does Coinbase provide for institutional sitemap20,62815,3472Y13.2s
coinbase_10How can a developer build an AI agent that performs on-chainflat16,0982,5915Y16.0s
coinbase_10How can a developer build an AI agent that performs on-chainsitemap18,60013,6772Y13.3s
elevenlabs_01What is ElevenLabs Eleven v3 and how does it compare to prevflatERR-----
elevenlabs_01What is ElevenLabs Eleven v3 and how does it compare to prevsitemap40,91911,6834Y19.0s
elevenlabs_02What is the latency of ElevenLabs' Flash model for text-to-sflat547,373144,1343Y25.1s
elevenlabs_02What is the latency of ElevenLabs' Flash model for text-to-ssitemap32,54016,6233Y13.4s
elevenlabs_03What is ElevenLabs' conversational AI agent platform and whaflatERR-----
elevenlabs_03What is ElevenLabs' conversational AI agent platform and whasitemap27,05922,4842Y12.5s
elevenlabs_04How does ElevenLabs handle LLM integration and fallback in iflat1,290,168148,0327Y39.8s
elevenlabs_04How does ElevenLabs handle LLM integration and fallback in isitemap26,55622,4842Y11.6s
elevenlabs_05What SDKs does ElevenLabs offer for developers?flat548,648144,8713Y19.3s
elevenlabs_05What SDKs does ElevenLabs offer for developers?sitemap10,7967,3332Y8.3s
elevenlabs_06What integrations does ElevenLabs' conversational AI platforflat1,462,044145,1438Y42.6s
elevenlabs_06What integrations does ElevenLabs' conversational AI platforsitemap26,59622,4842Y8.9s
elevenlabs_07What character limits apply to different ElevenLabs TTS modeflat604,899172,6123Y21.1s
elevenlabs_07What character limits apply to different ElevenLabs TTS modesitemap32,28116,6233Y13.5s
elevenlabs_08How does ElevenLabs support HIPAA compliance for healthcare flatERR-----
elevenlabs_08How does ElevenLabs support HIPAA compliance for healthcare sitemap22,91517,7732Y10.5s
elevenlabs_09How would you build a multilingual customer service voice agflatERR-----
elevenlabs_09How would you build a multilingual customer service voice agsitemap27,34522,4842Y12.9s
elevenlabs_10What accessibility programs does ElevenLabs offer and what vflatERR-----
elevenlabs_10What accessibility programs does ElevenLabs offer and what vsitemap13,8429,9582Y11.5s
langchain_01What is the minimum Python version required to install LangCflat54,79421,6972Y7.2s
langchain_01What is the minimum Python version required to install LangCsitemap25,41420,7062Y6.9s
langchain_02How many monthly downloads does LangChain have and how many flatERR-----
langchain_02How many monthly downloads does LangChain have and how many sitemap25,57620,7062Y7.6s
langchain_03What is the price of LangChain's Pro tier?flatERR-----
langchain_03What is the price of LangChain's Pro tier?sitemapERR-----
langchain_04How many daily emails does C.H. Robinson process using LangCflat94,90424,8753Y48.9s
langchain_04How many daily emails does C.H. Robinson process using LangCsitemap7,5194,2642Y7.5s
langchain_05What is Klarna's AI Assistant's impact in terms of conversatflat131,82427,6884Y21.4s
langchain_05What is Klarna's AI Assistant's impact in terms of conversatsitemap7,4704,2642Y6.7s
langchain_06What is LangSmith Polly?flat58,99724,7612Y11.3s
langchain_06What is LangSmith Polly?sitemap70,75436,0733Y9.6s
langchain_07What external destinations does LangSmith support for bulk eflat63,80629,2682Y10.2s
langchain_07What external destinations does LangSmith support for bulk esitemap17,73913,4472Y7.5s
langchain_08Compare the enterprise outcomes of Podium and AppFolio when flatERR-----
langchain_08Compare the enterprise outcomes of Podium and AppFolio when sitemap8,4894,2642Y11.3s
langchain_09Explain how LangSmith's observability features work togetherflat157,49945,0534Y20.3s
langchain_09Explain how LangSmith's observability features work togethersitemap18,08813,4472Y8.8s
langchain_10Describe LangChain's multi-agent architecture patterns and hflat117,71636,1373Y19.8s
langchain_10Describe LangChain's multi-agent architecture patterns and hsitemap57,32724,9643Y22.9s
mintlify_01What is the central configuration file in Mintlify that contflat34,32822,3812Y11.6s
mintlify_01What is the central configuration file in Mintlify that contsitemap10,8686,5572Y11.1s
mintlify_02What OpenAPI and AsyncAPI specification versions does Mintliflat32,21810,4933Y9.7s
mintlify_02What OpenAPI and AsyncAPI specification versions does Mintlisitemap8,8145,4832Y6.5s
mintlify_03What are the llms.txt and llms-full.txt files that Mintlify flat16,6906,8882Y7.8s
mintlify_03What are the llms.txt and llms-full.txt files that Mintlify sitemap13,1569,3492Y7.4s
mintlify_04What are the limits for Mintlify Workflows (Beta), includingflatERR-----
mintlify_04What are the limits for Mintlify Workflows (Beta), includingsitemapERR-----
mintlify_05What company did Mintlify acquire, and what was that companyflatERR-----
mintlify_05What company did Mintlify acquire, and what was that companysitemap35,40518,7473Y12.0s
mintlify_06What is the cost per overage instance in Mintlify's pricing,flatERR-----
mintlify_06What is the cost per overage instance in Mintlify's pricing,sitemapERR-----
mintlify_07What OpenAPI extensions does Mintlify use to control API endflat16,1356,6142Y6.6s
mintlify_07What OpenAPI extensions does Mintlify use to control API endsitemap22,91211,2693Y11.6s
mintlify_08Explain how Mintlify's content negotiation works for AI agenflat61,14614,3067Y26.4s
mintlify_08Explain how Mintlify's content negotiation works for AI agensitemap13,9569,3492Y15.0s
mintlify_09Describe Mintlify's enterprise automation capabilities incluflat106,88116,4297Y24.2s
mintlify_09Describe Mintlify's enterprise automation capabilities inclusitemap30,81814,0613Y17.0s
mintlify_10How does Mintlify support its OSS Program and Startup PrograflatERR-----
mintlify_10How does Mintlify support its OSS Program and Startup Prograsitemap36,77018,7473Y15.5s
sardine_01What behavioral biometrics does Sardine use to detect fraud?flat143,94241,6003Y17.6s
sardine_01What behavioral biometrics does Sardine use to detect fraud?sitemap18,1876,9983Y14.0s
sardine_02What is the Sardine Feature Store?flat145,34141,5343Y16.5s
sardine_02What is the Sardine Feature Store?sitemap48,06516,5444Y16.1s
sardine_03What types of fraud does Sardine detect and prevent?flat194,12641,2914Y21.6s
sardine_03What types of fraud does Sardine detect and prevent?sitemap15,72710,7252Y9.4s
sardine_04What is Sardine's approach to KYC and how does it differ froflat94,72439,6692Y12.6s
sardine_04What is Sardine's approach to KYC and how does it differ frositemap8,8745,1112Y9.6s
sardine_05What is the Sardine Sponsor Bank Operating System?flat94,60339,6542Y11.0s
sardine_05What is the Sardine Sponsor Bank Operating System?sitemap50,27811,0545Y16.7s
sardine_06How does Sardine prevent SMS pumping attacks?flat95,64340,8302Y11.0s
sardine_06How does Sardine prevent SMS pumping attacks?sitemap7,2723,8392Y7.1s
sardine_07What compliance frameworks does Sardine support?flat142,61340,1083Y12.8s
sardine_07What compliance frameworks does Sardine support?sitemap8,7105,1112Y6.7s
sardine_08How does Sardine detect and prevent scams targeting elderly flat142,62840,0193Y16.6s
sardine_08How does Sardine detect and prevent scams targeting elderly sitemap65,53415,5615Y19.6s
sardine_09How does Sardine combine device intelligence with complianceflat143,76640,3263Y16.9s
sardine_09How does Sardine combine device intelligence with compliancesitemap19,6718,1613Y13.4s
sardine_10How does Sardine's approach to account takeover prevention dflat95,65940,4402Y11.8s
sardine_10How does Sardine's approach to account takeover prevention dsitemap47,97917,9994Y16.5s
upstash_01How does Upstash Redis differ from traditional Redis for serflat91,24830,5107Y26.8s
upstash_01How does Upstash Redis differ from traditional Redis for sersitemap36,76330,2192Y10.2s
upstash_02What is QStash and what are its core messaging features?flat14,1311,5526Y19.6s
upstash_02What is QStash and what are its core messaging features?sitemap15,83111,3142Y10.8s
upstash_03What is Upstash Vector and what index types does it support?flat129,68732,7687Y21.1s
upstash_03What is Upstash Vector and what index types does it support?sitemap33,73627,8752Y8.0s
upstash_04What embedding models does Upstash Vector support for automaflat142,69537,3829Y26.9s
upstash_04What embedding models does Upstash Vector support for automasitemap34,11627,8752Y9.3s
upstash_05What is QStash's pricing structure?flat9,7641,6354Y16.7s
upstash_05What is QStash's pricing structure?sitemap36,74819,3303Y19.2s
upstash_06How does QStash handle security and request verification?flat136,18132,6929Y30.3s
upstash_06How does QStash handle security and request verification?sitemap15,96211,3142Y14.3s
upstash_07What programming language SDKs does Upstash provide for RediflatERR-----
upstash_07What programming language SDKs does Upstash provide for Redisitemap38,41730,2192Y19.7s
upstash_08How would you build a RAG application using Upstash's serverflatERR-----
upstash_08How would you build a RAG application using Upstash's serversitemap34,64627,8752Y13.5s
upstash_09How does Upstash handle background job processing and task sflat11,4151,6945Y21.0s
upstash_09How does Upstash handle background job processing and task ssitemap49,59128,5973Y16.9s
upstash_10What is the full range of Upstash products and how do they wflatERR-----
upstash_10What is the full range of Upstash products and how do they wsitemap24,66718,0242Y18.9s
vercel_01What is the Vercel AI Gateway and what functionality does itflat90,85177,2771Y24.1s
vercel_01What is the Vercel AI Gateway and what functionality does itsitemap11,0117,0882Y10.7s
vercel_02What is Vercel Edge Config and what are its primary use caseflat176,33878,3072Y14.0s
vercel_02What is Vercel Edge Config and what are its primary use casesitemap8,7135,1762Y8.2s
vercel_03What programming language runtimes does Vercel Functions supflat269,97181,2303Y17.9s
vercel_03What programming language runtimes does Vercel Functions supsitemap17,9716,5633Y14.8s
vercel_04What is Vercel Blob and what access modes does it support?flat87,49177,2771Y6.9s
vercel_04What is Vercel Blob and what access modes does it support?sitemap8,1684,6882Y7.7s
vercel_05What is the Vercel AI SDK and which frameworks does it suppoflat176,64278,0522Y12.7s
vercel_05What is the Vercel AI SDK and which frameworks does it suppositemap10,5847,0882Y7.7s
vercel_06What is the Vercel Sandbox and what is it used for?flat177,78178,9622Y14.7s
vercel_06What is the Vercel Sandbox and what is it used for?sitemap10,6477,0882Y7.7s
vercel_07How do Vercel's middleware capabilities work and what can thflat272,29583,7453Y19.8s
vercel_07How do Vercel's middleware capabilities work and what can thsitemap8,7665,1762Y7.6s
vercel_08How can a developer build an AI voice agent on Vercel that sflat268,48679,6703Y20.7s
vercel_08How can a developer build an AI voice agent on Vercel that ssitemap11,2087,0882Y25.7s
vercel_09How would you set up a feature flag system on Vercel that inflat176,91878,3872Y17.0s
vercel_09How would you set up a feature flag system on Vercel that insitemap43,60211,2735Y20.6s
vercel_10What storage options does Vercel provide and how do they difflat88,93377,2771Y15.5s
vercel_10What storage options does Vercel provide and how do they difsitemap86,45622,3936Y35.1s
Generated by generate_report_v2.py — AI Sitemap vs Flat llms.txt Benchmark (V2)