How to Score an AI Vendor’s References Without Naming-and-Shaming: A Due-Diligence Framework for Asia

Checking an AI vendor’s references properly means scoring live production systems against a fixed set of criteria (deployment duration, data handling, retrieval accuracy, compliance posture) without ever requiring the vendor to disclose which named client said what. The goal is a repeatable scorecard, not a gossip session. Buyers across Singapore, Indonesia, Japan and other Asian markets are increasingly asked by their own boards or regulators to prove they did structured diligence before signing an AI contract, and “we called two references and they sounded happy” no longer satisfies that bar.

TL;DR

• Score references against fixed criteria (uptime, deployment duration, data residency, retrieval accuracy) rather than collecting anecdotes.
• Verify three to five live production systems that have run for at least six months before treating a vendor as proven.
• Data residency rules differ by country: Singapore’s PDPA, India’s DPDP Act, Vietnam’s Law No. 91/2025/QH15 and Indonesia’s PDP Law each impose different constraints.
• Run parallel reference conversations rather than sequential ones, so answers can be cross-checked before memory fades or stories get rehearsed.
• A RAG-based knowledge base needs its own technical checklist, covering indexing methods, retrieval precision and faithfulness metrics, separate from general vendor questions.

About the Author: Fooyo builds and operates production AI systems, including RAG-based knowledge bases and agentic automation, for government agencies, enterprises and SMEs across Singapore, Indonesia, Japan and beyond. This framework reflects patterns Fooyo has seen from the buyer’s side of due diligence, not just the vendor’s side.

Why Do Reference Checks Fail Even When the Vendor Provides Real Clients?

Reference checks fail most often because the buyer asks open-ended questions (“how has it been going?”) instead of structured ones, which lets both sides drift into pleasantries. A vendor-supplied reference has already agreed to talk, which means the sample is self-selected toward satisfied clients. That is not dishonest, it is just a known bias, and a due-diligence framework has to correct for it rather than pretend it doesn’t exist.

The fix is not to distrust every reference call. It is to replace vague questions with a fixed scorecard so that answers can be compared across vendors on the same axes. A structured evaluation, using a weighted scorecard with an evidence log and documented recommendation, turns subjective impressions into something a procurement committee or auditor can actually review later [dunnixer.com]. Frameworks built specifically for AI vendor selection go further, standardizing the process so that reference checks are one input among several rather than the deciding factor on their own [team400.ai].

How Do You Score References Without Naming Individual Clients?

Scoring without naming means the vendor describes each reference by attributes, not identity: sector, company size band, country of deployment, use case, and system age. This is the same anonymization logic used in supplier and vendor benchmarking more broadly, where the comparison is built on structured attributes rather than which specific company said what [poimapper.com]. A buyer does not need to know which government agency or which retailer is behind Reference B; they need to know that Reference B is a 500-plus employee organization running a knowledge base in production for eight months with documented uptime.

A practical scorecard looks like this:

Every row should be answerable without a name attached. If a vendor can only answer “yes, our clients are happy” without producing a metric, that itself is diagnostic.

What Should You Actually Ask During the Reference Call?

Building on the scorecard above, the harder question is what to ask once you’re on the call, because generic questions produce generic answers. Run reference conversations in parallel rather than one after another, so you can compare fresh impressions across calls instead of relying on notes from a conversation held weeks earlier. This also reduces the risk of a rehearsed narrative settling in as “the answer” by the third or fourth call. Specific, comparable questions to use across every call:

• How long between contract signing and the system going into actual production use (not a demo)?
• Has the system needed to be re-scoped or rebuilt after initial deployment, and why?
• What happens when the AI gives a wrong or ungrounded answer? Is there a human review gate, and how often is it triggered?
• Where is the data stored, and did the vendor explain this before or only after being asked?
• What did the vendor NOT deliver that was originally scoped?

That last question tends to be the most revealing one, since most vendors will happily describe what worked but rarely volunteer what was quietly dropped or delayed.

How Does Compliance and Data Residency Change the Scorecard in Asia?

A related but distinct question is what happens when the vendor’s home jurisdiction doesn’t match the deployment country, which is common across Asia given how fragmented data law is region to region. Singapore’s PDPA and its Model AI Governance Framework set the baseline for local deployments, but India’s DPDP Act pushes toward local computing for sensitive categories of data, meaning a vendor that only offers a shared regional cloud instance may not clear the bar for an India-based client. Vietnam’s Law No. 91/2025/QH15 and Indonesia’s PDP Law impose their own localization requirements, so a reference from a Singapore deployment does not automatically prove the vendor can meet Vietnamese or Indonesian residency rules. The scorecard should therefore include a jurisdiction-specific line item rather than a single generic “is the vendor compliant” checkbox. Ask each reference which country their data resides in and whether that was a explicit design decision or a default the vendor happened to offer. Fooyo’s approach, deploying every AI Knowledge Base and agentic workflow into the client’s own private cloud, exists specifically because this jurisdiction-by-jurisdiction variation is real and not theoretical across the markets it operates in, including Singapore, Indonesia and Japan.

How Do You Evaluate the Technical Substance Behind a RAG-Based Knowledge Base?

Stepping back from process and compliance, a separate concern is whether the underlying technology actually works the way the vendor describes it, and this requires questions a generic procurement team may not know to ask. A retrieval-augmented generation system depends on three technical layers: how the dataset was constructed and indexed, how accurately the system retrieves the right source material, and how faithfully the generated answer reflects that source material. Ask for the retrieval metric, commonly mean average precision, and ask whether generative outputs are scored for relevance, faithfulness and truthfulness against a recognized standard rather than eyeballed by the vendor’s own team. Think of it like checking a translator’s work in two steps rather than one. First, did they pull the right passage from the right document (retrieval)? Second, did their translation of that passage stay true to what it actually said (faithfulness)? A system can retrieve the right document and still generate an answer that drifts from it, so both steps need separate verification, not a single “does it sound right” check.

What Does a Full Due-Diligence Checklist Look Like Before Signing?

Pulling the threads above together, a defensible pre-signature checklist for an Asia-based AI procurement decision includes:

• Three to five live production references, each running at least six months, scored on the attribute table above rather than judged by name.
• Parallel reference calls scheduled within the same short window.
• A written answer on data residency for the specific country of deployment, checked against that country’s law rather than a generic compliance statement.
• A documented retrieval and faithfulness metric for any RAG-based system, not just a demo that “looked good”.
• A realistic deployment timeline. Industry benchmarks put standard enterprise AI deployment at six to twelve months, often broken into three months of scoping, three months building the knowledge graph, and three months on agent integration. If a vendor promises materially faster delivery, ask exactly what is being compressed and why.

This last point matters because timeline claims are one of the easiest things to overstate and one of the hardest for a buyer to verify without a checklist like this in hand.

Frequently Asked Questions

How many references are enough to trust an AI vendor?
Three to five live production systems running for at least six months is the benchmark used in established vendor selection frameworks. Fewer than that, or references still in pilot phase, should be treated as unproven.

Is it acceptable for a vendor to refuse to name any clients?
Yes, and it should not be treated as a red flag on its own. What matters is whether the vendor can still supply attribute-level detail (sector, size, deployment length, metrics) for anonymized references, per the sanctioned framing that a vendor is trusted by government agencies, enterprises and SMEs rather than by named accounts.

What is the single biggest mistake buyers make in reference calls?
Asking open-ended questions sequentially rather than structured questions in parallel, which lets vendor narratives harden before a buyer has heard enough independent answers to compare.

Does data residency really change which vendor you should pick?
Yes. A vendor with strong references from a Singapore deployment does not automatically satisfy India’s DPDP Act or Vietnam’s Law No. 91/2025/QH15, since each imposes different localization requirements.

What technical metric should I ask for on a RAG-based system specifically?
Ask for retrieval accuracy (commonly mean average precision) and generative quality metrics covering relevance, faithfulness and truthfulness. A vendor that cannot produce either number is likely evaluating the system informally.

How long should a production-grade AI deployment realistically take?
Industry benchmarks suggest six to twelve months for a standard enterprise build, broken into scoping, knowledge graph construction, and agent integration phases of roughly three months each. Vendors offering materially shorter timelines should explain exactly which phase is compressed and how.

Should compliance questions come before or after the technical evaluation?
Before. If a vendor cannot meet the residency requirement for your jurisdiction, the technical quality of their retrieval system is moot for that deal.

About Fooyo

Fooyo is a Singapore-headquartered company building data platforms and AI agents for enterprises, with business partnerships in Tokyo and development operations across Asia. Its production AI products, including RAG-based AI Knowledge Bases and agentic automation workflows, deploy directly into a client’s own private cloud. Fooyo works alongside clients as an operating partner and is trusted by government agencies, enterprises and SMEs across Singapore, Indonesia, Japan and other markets in Asia. Every deployment includes human review gates and source-cited answers, built for the compliance realities of the markets it serves rather than a one-size-fits-all global template.

If your organization is building a reference-check process for an upcoming AI vendor decision, or wants to see how a private-cloud knowledge base handles retrieval and faithfulness in practice, get in touch with Fooyo at https://fooyo.sg/.