A knowledge assistant earns its place when a colleague can use its answer to complete a real task. That is a more useful starting point than asking which model looks most impressive. The buying decision depends on whether the complete workflow improves: finding the right information, checking it and acting on it.
This article proposes a practical evaluation method. The example is illustrative, not a report of a CNPS customer deployment.
1. Choose one decision worth improving
Imagine a distributor whose service team answers installation questions from approved product manuals. “Answer every company question” is too broad for a first pilot. “Help service staff find the correct installation requirement for a named product version” gives the team a clear job, a defined audience and a way to judge the result.
Write down the current workflow. Who receives the question? Where do they look? Who checks a doubtful answer? How long does the entire task take? Keep several ordinary examples and several difficult ones. This baseline should include the effort of verification, because that effort remains part of the future workflow.
2. Give the assistant an accountable source collection
Treat the document collection as a maintained business asset. For each file, identify its owner, product version, effective date and intended audience. Resolve contradictory manuals before expecting an assistant to resolve them. Preserve meaningful headings, units, table context and links to the original material.
Decide who can approve an update and how withdrawn information leaves the system. A correct answer from last quarter may be wrong for the current product. Include a small update-and-removal exercise in the pilot so the team can observe how content changes reach users.
3. Test the answers you do not want to hear
Create a question set before tuning. Include ordinary factual questions, questions that combine documents, ambiguous requests, missing information and access-restricted material. Keep a separate holdout set for the final decision; repeatedly tuning against every question makes the result less informative.
| Test | What the reviewer checks |
|---|---|
| Factual answer | The answer matches the authoritative source and version |
| Citation | The cited passage supports the actual statement |
| Missing evidence | The assistant explains the gap and suggests a useful next step |
| Permissions | A user cannot retrieve material outside their assigned access |
| Ambiguity | The assistant requests the missing product or context |
| Update | A changed or withdrawn document is handled as agreed |
Record failures with the question, configuration and source version. Do not remove awkward examples simply because they lower the score.
4. Measure the whole job
Compare the time required to produce a usable, checked answer with the baseline. Also record correction effort, unresolved questions, response time under expected use and operating cost. A fluent first draft is only one step in the service task.
Set thresholds with the business owner before the test. Separate requirements that must pass, such as access boundaries, from improvements that can be traded against cost or speed. Review examples as well as aggregate numbers: a small number of serious errors can matter more than a convenient average.
5. Name the people behind the system
Assign ownership for documents, user access, technical maintenance and business acceptance. Map every service that handles documents or prompts. Hosting the application yourself does not, by itself, describe where every connected model or processing service runs; ask for the complete data path.
Plan a review when the model, document parser or retrieval configuration changes. Keep a route to a human colleague for unresolved questions. The assistant should fit the existing responsibility structure, with a clear owner for problems discovered after launch.
6. Turn the pilot into a buying decision
The final record should state the task, documents, configuration, measured results, known failures and operating owner. Choose among proceeding, improving a named weakness, narrowing the scope or stopping. Each outcome should be defensible from the evidence.
Use the knowledge pilot worksheet and Pick documents and one workflow for a UAE FastGPT first pilot to collect inputs. When you are ready, discuss a knowledge assistant evaluation with CNPS. Share the task, languages, user count, destination, timeline and non-sensitive data constraints so the next conversation can focus on a workable scope.
