A checklist for evaluating legal AI software before buying: 12 questions that prevent expensive mistakes

The legal AI software market has entered a phase of overabundance: a new tool appears every week and every vendor promises saved hours. For a law firm, the risk is no longer falling behind but buying badly: paying for a thin layer over a generic model, feeding client data into a non-compliant system, or discovering in production that the answers invent their sources. This checklist organises the evaluation into twelve questions that can be answered before signing.

Block 1: what happens to your clients' data

One: are the documents you upload used to train models? The correct answer is no, in writing and in the contract, not on the marketing page. Two: where is the data hosted and under which rules is it processed? EU hosting and a GDPR-compliant processing agreement are the minimum. Three: what happens to the data when the contract ends? Portability and verifiable deletion are part of the service, not a courtesy.

These three questions are not paranoia: professional rules already oblige lawyers to verify what comes out of these tools and to protect confidentiality, and the first court sanctions over AI use and confidentiality point exactly at that weak spot.

Block 2: what it produces and how you verify it

Four: does it cite real, linkable sources or generate plausible-looking references? Ask for a trial with your own cases and check the citations one by one; fake case law generated by AI and court sanctions show the cost of skipping that step. Five: where does its legal database come from? A system connected to updated official sources is not the same as a model trained on internet text. Six: how does the vendor document its error rates? If it cannot answer, final human control matters even more: the workflow for AI contract review with human control describes the right setup.

Block 3: the vendor's own regulatory compliance

Seven: how does the tool fit the European AI Act? Although the main burden falls on the provider, the firm needs to know whether the system falls into categories with reinforced obligations and what technical documentation exists. The AI Act calendar after the digital omnibus and its effects for firms summarises dates and duties. Eight: is there applicable Spanish supervision? The Spanish Agency for the Supervision of AI (AESIA) is the national reference, and the AI sandbox under Royal Decree 817/2023 shows which providers have passed through controlled environments and what public documentation exists about their testing.

Nine: which security certifications does it hold (ISO 27001, Spain's ENS or equivalents) and when was the last audit?

Block 4: integration, real cost and exit

Ten: does it integrate with your matter management system and email, or does it force duplicate work? Eleven: what is the real cost per matter or per user at your volume, including training time and the human review of outputs? Twelve: how is the contract terminated and in what format do you take your data and templates?

An afternoon of evaluation with these twelve questions saves months of trouble. The final rule is simple: buy the firm's AI the way you would hire an employee who will see every file: with references, with trials and with an agreed exit.

More practical analysis in our Legal AI section.