Ten Questions That Expose AI Vendors Who Can't Deliver on Their Promises or Scale
The questions we'd want you to ask us before you sign an AI development contract
Buying AI development is unusually difficult to evaluate.
The work is technical enough that most buyers cannot assess the engineering directly. The market is still new enough that almost every AI company's website sounds similar.
"AI-powered."
"End-to-end AI solutions."
"Advanced AI agents."
"Enterprise-grade."
Everyone has impressive case studies. Everyone has logos. Everyone claims to have built something innovative.
So how do you tell who can actually deliver?
Ask better questions.
We build AI systems for a living, which makes this an unusual list for us to publish. These are the questions we would want a potential client to ask us, and the answers we believe a serious AI development partner should be prepared to provide.
If another vendor cannot answer them comfortably, that is useful information too.
1. What Have You Built That Is Still Running a Year Later?
Anyone can build a prototype.
The harder test is whether the system is still being used months after launch.
Ask:
What have you built that is still running today, and can I speak to that client?
You are looking for evidence that the system survived the transition from demo to production.
Ask what happened after launch.
Did users adopt it?
Did the vendor have to redesign parts of it?
What does ongoing support look like?
A production reference can tell you much more than a polished demo.
2. How Will We Measure Whether It Is Accurate?
"Good output" is not a measurement.
A serious answer should explain how the system will be evaluated, what test data will be used, which metrics matter, and what performance threshold needs to be reached.
For an AI agent, that could include accuracy, task completion, escalation rate, hallucination rate, or another metric appropriate to the workflow.
The exact metric will depend on the use case.
The principle does not:
You should know what success looks like before you start building.
Current AI vendor assessment frameworks recommend evaluating model behavior, accuracy, hallucination, reliability, and error modes before adoption rather than relying on a demonstration alone.
If the answer is simply, "We'll know when we see it," ask for something more measurable.
3. What Happens When the System Does Not Know?
This may be the most important question on the list.
AI systems will encounter situations they cannot handle.
The question is not whether that will happen.
It is what happens when it does.
Ask:
● What triggers a human handoff?
● What happens when two sources conflict?
● Can the system recognise uncertainty?
● What prevents it from taking an unsafe action?
● Can we review what happened afterward?
A production AI system needs a safe failure path.
Human oversight, escalation, logging, and the ability to intervene are important parts of current AI vendor due diligence.
An answer that sounds like "the model should be able to figure it out" is not a safety strategy.
4. What Happens to Our Data?
This question should come before you give a vendor access to your systems.
Ask exactly:
● What data will you collect or process?
● Where will it be stored?
● Who can access it?
● Will prompts, files, outputs, logs, or traces be retained?
● Will our data be used to train or improve any models?
● Which third-party models or subprocessors will receive it?
● How long will it be retained?
● What happens to it when the contract ends?
● Can we have it deleted?
Do not settle for a general statement such as "your data is secure."
Ask for the actual data flow.
The Best AI Vendor Should Be Comfortable With These Questions
If a vendor is uncomfortable with the list, that is useful information.
A serious AI development partner should be prepared to explain what it has built, how it measures performance, how the system handles uncertainty, what happens to your data, what it costs to operate, who owns the deliverables, who will build it, what happens when things go wrong, when AI is not the right answer, and how you take the system over.
At Gigaflop TechLab, we believe buyers should ask these questions of us too.
Even if it means occasionally losing a project.
"Don't ask an AI vendor what they can build. Ask them what happens when it goes wrong."
Siddharth Mishra, CEO, Gigaflop TechLab
We've put these ten questions into a one-page AI Vendor Evaluation Checklist, including the answers that should reassure you and the answers that should make you dig deeper.
No email required to read it.
If you want the deeper guide with scoring criteria, red flags, and a simple vendor comparison framework, you can choose to receive that separately.
What's Your Reaction?
