Practical guide · Includes a copyable test pack and scorecard
An AI tool can produce an impressive answer and still be a poor purchase. The useful question is whether it completes a task you actually do, accurately enough that checking and correcting its work takes less time than doing the task yourself.
This guide gives you a repeatable way to test a text-based AI assistant before subscribing. Start with the fictional project notes below, then try a small set of representative, non-sensitive tasks from your own work. You will have expected answers to check, a record of mistakes, and a clearer basis for deciding whether to pay.
About this method: The test pack and scoring rules below were created for this guide. They are a practical screening exercise, not a validated benchmark or a claim that any particular product has passed. All sample names, project details, and calculations are illustrative.
1. Define one job before comparing tools
Write down the input, the result you need, and the mistakes that would make that result unusable. “Help with productivity” is too broad. “Turn meeting notes into an action list without inventing owners or deadlines” is specific enough to test.
| Write down | Example |
|---|---|
| Task | Convert project notes into an action list. |
| Input | A short meeting note containing dates, decisions, and unresolved questions. |
| Required output | A table with task, owner, deadline, and a supporting quote. |
| Unacceptable mistake | Inventing a deadline, assigning an unassigned task, or treating a proposal as approved. |
| Baseline | Time yourself doing the same task manually, including your final check. |
Use the same task definition for each candidate. Record the product, plan, model if shown, date, and whether web browsing or file access was enabled. Different settings can make otherwise similar comparisons misleading.
2. Copy this small test pack
Paste the following fictional notes into a new conversation. Keep the source identical across tools. For this initial exercise, ask the assistant to use only these notes.
Source notes — Cedar Studio planning meeting, 10 September 2026
1. The autumn campaign launches on 30 September 2026.
2. Mina will deliver the first landing-page draft by 18 September 2026.
3. Omar will review that draft by 22 September 2026.
4. The client has not approved the advertising budget.
5. Translation is required, but its owner and deadline have not been assigned.
6. The newsletter contains three links: the offer page, the FAQ page, and the contact page.
7. The FAQ page URL has not yet been supplied.
8. Last month there were 40 enquiries; 10 became paying customers.
9. A suggestion to launch on 25 September was discussed and rejected.
The notes deliberately include a rejected date, missing information, and a simple calculation. These details let you check whether a fluent answer is also faithful to its source. NIST identifies confidently generated false information, termed confabulation, as a generative AI risk. See NIST AI 600-1, section 2.2.
3. Run three tests with answers you can verify
Test A: Extract facts without filling gaps. Copy this prompt after the source notes:
“Using only the source notes, create a table with one row each for the landing-page draft, draft review, and translation. Use four columns: Task, Owner, Deadline, and Evidence. Evidence must be an exact quotation from the notes. Write ‘Not assigned’ wherever an owner or deadline is missing. Do not infer dates from the launch schedule.”
| Task | Expected owner | Expected deadline |
|---|---|---|
| Landing-page draft | Mina | 18 September 2026 |
| Draft review | Omar | 22 September 2026 |
| Translation | Not assigned | Not assigned |
Check the evidence column against the notes character by character where it matters. A plausible quotation that does not appear in the source is a failure, even if the surrounding answer sounds convincing.
Test B: Handle uncertainty and conflicting details. In another new conversation, paste the same notes and this prompt:
“Using only these notes, answer four questions: What is the agreed launch date? What is the approved advertising budget? What is the FAQ page URL? What percentage of last month’s enquiries became paying customers? For missing information, explicitly state that the notes do not supply it. Show the calculation for the percentage.”
The correct launch date is 30 September 2026. There is no approved budget amount, and the FAQ URL is not supplied. The conversion calculation is 10 ÷ 40 × 100 = 25%. An invented budget or URL is a material error. The rejected 25 September proposal must not replace the agreed date.
Test C: Follow output constraints. Start another new conversation with the notes and this prompt:
“Write exactly three bullet points and nothing else. Bullet 1 must state the agreed launch date. Bullet 2 must state the landing-page draft owner and deadline. Bullet 3 must state that translation has no assigned owner or deadline. Use no more than 60 words in total. Do not add recommendations.”
Check the three requested facts, the bullet count, and the word limit. A response can be factually correct but inconvenient for a workflow if it repeatedly ignores the format you need.
Run each test three times in fresh conversations. Save every answer, including failures; do not keep only the best response. Nine short runs can expose obvious inconsistency, but they do not establish a statistically reliable accuracy rate.
4. Add a task that resembles your real work
Passing a tiny sample only shows that the tool handled that sample. Next, try three non-sensitive examples from the job you defined: an ordinary case, an ambiguous case, and the largest or messiest input you regularly use. Prepare the expected facts before running the tool.
- Document work: include a scanned page or a difficult table if those are common in your files. Verify extracted names, numbers, and omitted rows.
- Writing: use a real length limit, audience, and list of facts the output must preserve. Judge the final usable draft rather than its first impression.
- Research: open every cited source. Check whether it supports the specific claim, not merely whether the link exists.
- Export: copy or download the result into the application you actually use. Check formatting and whether editing remains possible.
If your main task is document preparation, our PDFCraft guide provides another workflow to consider. Apply the same checks to recommendations here and elsewhere.
5. Measure time after checking and corrections
Start your AI timer before preparing the input. Stop only when the result is ready to use. Include upload preparation, prompt writing, generation, source checking, corrections, and export. If a run fails and you redo the work manually, include that time too.
Net time saved = manual completion time − total AI-assisted completion time.
| Illustrative measurement | Minutes |
|---|---|
| Manual task, including checking | 18 |
| Prepare input and prompt | 2 |
| Generate and refine output | 3 |
| Verify, correct, and export | 6 |
| Total AI-assisted time | 11 |
| Net saving per task | 7 |
At 30 comparable tasks per month, this hypothetical saving becomes 210 minutes, or 3.5 hours. A hypothetical €20 monthly subscription would cost about €5.71 per hour recovered, before other costs. These numbers are an example of the calculation, not a prediction of your savings or a product price.
Use your own measurements and include failed runs. If the tool makes a task faster but introduces mistakes you cannot reliably detect, a positive time saving is not enough to justify using it for that task.
6. Check the plan, data handling, and permissions
Before subscribing, read the provider’s current plan details and data policy. Record the usage limits, file-size limits, export options, renewal terms, and cancellation process. Check whether the tested model and features are available on the plan you intend to buy.
For data handling, find the provider’s statements about retention, deletion, and use of your inputs for training. Record the policy link and relevant account setting. If you cannot establish whether the service is suitable for your work data, keep the trial to fictional or non-sensitive material.
If the tool can access email, cloud files, or other applications, test with a limited sample and the narrowest permissions needed. OWASP describes how instructions embedded in external documents or web pages can redirect an AI assistant; it recommends controls including least privilege and human review for sensitive actions. Read OWASP’s prompt injection guidance.
A text-only evaluation does not certify that connected actions are safe. Test a proposed email as a draft and inspect it before enabling a workflow that sends messages.
7. Use this scorecard to make the decision
Score each dimension from 0 to 2 after reviewing all your runs. Keep the underlying notes: a total score alone can conceal an important failure.
| Dimension | 0 / 1 / 2 points |
|---|---|
| Accuracy | 0: material errors. 1: smaller factual corrections needed. 2: all checked facts correct. |
| Missing information | 0: invents answers. 1: inconsistent handling. 2: reliably identifies gaps. |
| Instructions | 0: often unusable format. 1: occasional reformatting. 2: consistently meets requirements. |
| Consistency | 0: recurring failures. 1: mixed results. 2: no observed failures across your runs. |
| Net time saving | 0: no saving. 1: small or inconsistent saving. 2: useful, repeatable saving. |
| Workflow fit | 0: cannot complete the workflow. 1: awkward workarounds. 2: usable input, output, and export. |
Suggested decision rule: A score of 10–12 out of 12 is a reason to consider a limited paid trial; 7–9 calls for more testing or a narrower use case; 0–6 suggests trying another approach. These are editorial rules of thumb, not industry standards.
Apply stop conditions before using the total: an invented fact that would materially affect your work, unresolved data-handling requirements, or an unauthorized external action should stop adoption for that use case regardless of the score. A failed tool may still be useful for a simpler, lower-risk task.
Keep a record you can reuse
Copy these fields into a note or spreadsheet for each candidate:
- Tool, plan, model if available, test date, and enabled features.
- Task definition and exact input.
- Prompt, run number, and saved output.
- Expected answer, observed mistakes, and severity.
- Manual time, total assisted time, and net saving.
- Scores, stop conditions, policy links, and final decision.
Retest when your task, the product, the model, or the plan changes. Your best choice is the tool that repeatedly produces usable work under your actual constraints. One successful demonstration is a starting point; a saved set of inputs, outputs, and corrections is evidence you can compare.
Sources and method notes
The exercises, sample notes, expected answers, scorecard, and time-saving example are original material prepared for this guide. They do not represent a hands-on review of a named product. The following primary sources support the discussion of AI risks and controls:
- NIST AI 600-1: Generative Artificial Intelligence Profile — confabulation and generative AI risk management.
- OWASP: LLM Prompt Injection Prevention Cheat Sheet — external-content risks, least privilege, and human review.
Featured photo: James McKinven / Unsplash. Illustrative workspace photograph; it does not show product test results.