What to prepare before hiring an AI consultancy

What a study can show and what you must test

Brynjolfsson, Li and Raymond (2025) studied the introduction of generative assistance using data from 5,172 customer-support agents. Issues resolved per hour increased by 15% on average, with differences between workers. This is a finding from a specific business setting, not a forecast for your organisation.

Reference: Brynjolfsson, Li & Raymond, 2025

Human–AI collaboration also needs evaluation

Vaccaro, Almaatouq and Malone (2024) synthesised 106 experimental studies reporting 370 effect sizes. On average, human–AI combinations performed worse than the best alternative alone, with variation by task. Improving on a person working alone differs from outperforming the best alternative. This does not recommend removing human controls; it shows why workflow configuration matters.

Reference: Vaccaro, Almaatouq & Malone, 2024

1. Set an observable objective

Describe who performs the task, what information they use and what makes completion difficult. Choose a measurable outcome such as total drafting and validation time. Include quality criteria: faster output is unhelpful if it creates rework.

Avoid beginning with a desired percentage without a baseline. Establishing the current workload may be the first project phase.

2. Prepare representative work examples

Include routine and difficult cases: correct responses, insufficient information, contradictory documents and requests that require rejection or handover. Remove personal or confidential information that is unnecessary for evaluation.

Hypothetical example: for a service-information drafting assistant, prepare questions that the documentation can answer and others that it cannot. Asking for clarification may be the correct response to the latter. Do not reward invented details that make an answer look complete.

3. Map data and responsibilities

Record where each source lives, who authorises its use and who maintains it. Include systems the solution must consult and actions it would be permitted to perform.

• Reference documents and their maintenance owner.
• Access and integrations required for the pilot.
• Information that must not enter the system.
• The person reviewing results and handling incidents.
• An operational fallback if the service is unavailable.

4. Agree acceptance criteria before a demonstration

Request evaluation on examples beyond those selected for a demonstration. Include preparation, review and correction time. Define errors that prevent progression even when average results look favourable.

Compare current and assisted workflows on equivalent tasks. If the sample is small or cases differ substantially, state that limitation. A pilot informs a local decision; it does not automatically establish a general result.

5. Assess the complete proposal

Clarify scope, deliverables, dependencies and responsibilities. Ask how quality will be checked after model or documentation changes, what maintenance is included and which costs are excluded.

Expansion should depend on demonstrated usefulness, acceptable errors and operational capacity. Adjusting or dropping a use case can also be the right outcome. This preparation checklist is practical ALKATECH guidance; the cited studies provide context, not certification of the approach.

Frequently asked questions