Identity verification vendors are usually evaluated on accuracy rates, supported document types, pricing and integration effort. Those are the right questions for choosing between two well-built platforms. They do not tell you whether a platform is well built. The signal that does is how the vendor’s engineering process handles a change — what gets checked automatically, what can block a release, and what happens when a check fails.
This is a checklist for that conversation, with the reasoning behind each question. It applies to aIDentix as much as to anyone else; the practices described are the ones used by DSS, the engineering team that builds it.
Why engineering process is a procurement question
Because identity verification failures are asymmetric. A false rejection loses a customer. A false acceptance admits someone who should not have been admitted, and you may not discover it for a year. Neither shows up in a demo, and both are downstream of ordinary software defects.
Accuracy benchmarks measure the model. Engineering process determines whether the code around the model — the routing, the thresholds, the state handling, the audit writing — does what it is supposed to. That code is where most production incidents originate.
1. What runs automatically on every change?
Ask: when a developer opens a pull request, what executes before anyone reviews it?
Why: checks that run automatically run every time. Checks that depend on someone remembering do not, particularly under deadline.
A concrete answer looks like the pipeline DSS configured for its own delivery, where opening a pull request triggers three things: static code analysis scanning for code smells, vulnerabilities and bugs; unit test execution against defined coverage thresholds; and compliance checks enforcing best practice and security policy. All before review, all on every change.
Warning sign: “our senior engineers review everything carefully.” True and irrelevant. Human review catches design problems; it does not reliably catch a dependency vulnerability at eleven at night.
2. Can a failed check actually block a release?
Ask: if the analysis flags a critical issue, what happens? Who can override it, and is the override recorded?
Why: this is the difference between a quality process and a quality dashboard. Many organisations run static analysis and ship regardless of what it says.
In the DSS setup, SonarQube quality gates are enforced rather than advisory: code failing the gate does not merge. A quality gate is a predefined set of conditions on security, reliability and maintainability, and enforcement means failure blocks the merge or the deployment rather than generating a notification.
Warning sign: a vendor who describes what their tooling detects without saying what it prevents. Detection without enforcement is a metric, not a control.
3. What is the test coverage threshold, and is it enforced in the pipeline?
Ask: not “do you have tests” — every vendor says yes. Ask what the coverage threshold is and whether falling below it fails the build.
Why: an unenforced target drifts downward under delivery pressure, always. An enforced threshold is a floor.
Follow-up worth asking: what is covered? Coverage concentrated on utility functions while the decision logic — thresholds, routing, failure handling — is thin is a common and misleading pattern.
4. How long does the pipeline take, and why does that matter to you?
Ask: how long from commit to deployable artefact?
Why: slow pipelines get bypassed. When validation takes hours, teams batch changes into larger releases, and larger releases fail in ways that are harder to diagnose and slower to roll back. Pipeline speed is a proxy for how often the vendor can safely ship a fix — including a fix you have asked for.
The DSS approach to this was architectural rather than a matter of trimming steps: build agents run as pods in Kubernetes with dependencies pre-installed, scaling dynamically so builds execute in parallel rather than queuing, and setup time per build is largely eliminated. Static analysis and pre-merge testing were also moved off the main deployment pipeline, which measurably improved its throughput.
Warning sign: a fast pipeline achieved by removing checks. Ask what runs, not just how long it takes.
5. What happens when a check fails at three in the morning?
Ask: is there an expedited path to production, and does it skip anything?
Why: almost every organisation has an emergency route. The question is whether it bypasses validation. That path is where unreviewed changes reach production, and it is where you should expect incidents to originate.
Good answer: the emergency path is faster because the pipeline is fast, not because it skips steps.
6. How do you know the change worked?
Ask: after deployment, what tells you whether the change behaved as expected in production?
Why: pre-merge checks validate against expectations. Production validates against reality. For verification platforms specifically, the metrics that matter are acceptance and rejection rate distributions by document type and issuing country — a regression that shifts rejection rates for one country’s ID format will not fail a unit test and will not be visible in aggregate accuracy.
7. What is the disclosure process for a defect that affected our data?
Ask: if a defect caused incorrect verification results, when and how would we hear about it?
Why: this is the question that separates vendors who have thought about their obligations from vendors who have not. For regulated buyers it is not optional — a verification defect may trigger reporting duties on your side, and you cannot meet a deadline you learn about late.
Warning sign: hesitation, or an answer that describes a status page. A status page covers outages. A defect that produced wrong answers while the system was fully available is a different event and needs a different process.
What the answers are worth
None of these questions is about identity verification. That is deliberate. Verification accuracy is what vendors optimise their sales material for; engineering discipline is what determines whether that accuracy survives contact with two years of feature development.
A vendor who answers all seven specifically, with named tooling and stated thresholds, is describing a process that exists. A vendor who answers in principle is describing an intention. The gap between those two is where production incidents live.
The practices described here — enforced quality gates, static analysis and coverage thresholds on every pull request, containerised parallel build agents — are the ones used by DSS in building aIDentix. We would rather you asked.
Frequently asked questions
How can businesses choose a reliable identity verification provider?
Beyond accuracy and pricing, ask how the vendor’s engineering process handles a change: what runs automatically on every pull request, whether a failed check can block a release, what the enforced test coverage threshold is, and what tells them after deployment that the change behaved as expected in production.
Why does a vendor’s engineering process matter when buying verification software?
Because verification failures are asymmetric — a false rejection loses a customer, a false acceptance may go undiscovered for a year — and both are downstream of ordinary software defects. Accuracy benchmarks measure the model; engineering discipline determines whether the code around it does what it should.
What are quality gates and why do they matter?
A quality gate is a predefined set of conditions on security, reliability and maintainability that code must meet. What matters is enforcement: an enforced gate blocks the merge or deployment when it fails, while an advisory one generates a notification that gets ignored. Detection without enforcement is a metric, not a control.
What is a warning sign when assessing a vendor’s engineering practices?
A vendor who describes what their tooling detects without saying what it prevents. Also: relying on careful human review instead of automated checks, since review catches design problems but will not reliably catch a dependency vulnerability at eleven at night. And an expedited release path that skips validation.
What should a vendor’s defect disclosure process cover?
Not just outages. A defect that produced incorrect verification results while the system was fully available is a different event from downtime and needs its own process. For regulated buyers this matters because a verification defect may trigger reporting obligations on your side, and you cannot meet a deadline you learn about late.