How should a dealership evaluate AI lead handling and follow-up?

Build an adversarial test set

Include incomplete inquiries, conflicting information, shared phones, opt-outs, existing appointments, sold vehicles, unavailable inventory, price questions, emergencies and abusive language.

Inspect every system action

Confirm how conversations, appointments, tasks, notes, source, status and attribution appear in the CRM. Test failure and duplicate-prevention behavior.

Set a supervised launch

Limit initial scope, sample interactions, define error thresholds, monitor complaints and opt-outs, and preserve an immediate disable and human takeover path.

Set pass criteria before the test

Decide in advance what counts as a pass: for example, no incorrect statement about price or availability, every opt-out honored, every unresolved question handed to a person, and every interaction visible in the CRM. Write the criteria down and score every test conversation against them. Without pre-set criteria, the evaluation drifts toward whichever demo looked most fluent, which is not the same as the one that behaves safely.

Who reviews the transcripts

Name a person, not a committee, who reads a sample of conversations every week for the first months and logs each error by type. A salesperson can judge tone and accuracy; a manager should judge whether escalation worked.

Give that reviewer authority to pause the system. Errors that nobody reads are still errors, and they show up first as customer complaints rather than as a dashboard number.

Measuring what the tool changed

Compare against the period or the leads handled before the tool, and against a group it does not touch where practical. Track appointments set, appointments that show, and repair orders or sales that follow, not messages sent.

Watch complaints, opt-outs and duplicate contacts as guardrails. A tool that raises activity but also raises complaints has not yet shown a benefit.

Common mistakes when evaluating AI follow-up

Testing only with clean, friendly inquiries hides the failures. Skipping consent and quiet-hour checks leaves compliance risk with the store.

Launching on every channel and every lead type at once makes problems hard to trace. Start narrow, keep a manual fallback, and expand only after the transcripts and the numbers support it.

A test conversation script

Write twenty test messages covering ordinary questions, a price challenge, an unavailable vehicle, a request to stop, a complaint and an off-topic query. Run them against each candidate and score responses for accuracy, tone and handoff. The script can be reused after updates to check that behavior has not changed.