Knowledge assistants grounded in your documents

This page is for IT leaders, service providers and business owners who want AI to answer questions from their own documents. Product manuals, internal policies and support knowledge bases can all become source material. What determines the value is not the model name, but whether the question set is real, permissions are clear and citations can be checked.

CNPS helps define the evaluation scope, prepare the evaluation set and agree acceptance, and can discuss implementation with open-source options such as FastGPT where appropriate. Public projects such as Dify, RAGFlow, Qwen and DeepSeek are research starting points, not CNPS customer deliveries. Implementation scope and software licensing are agreed separately.

Where this fits

Product manual and technical support Q&A

After-sales or support teams need accurate answers from manuals in several versions, with the source visible. Evaluate how version conflicts are handled and whether citations truly support the answer.

Internal policy and process lookup

Staff ask about expenses, approvals or compliance steps. Evaluate permission boundaries: each role should only retrieve the documents it is allowed to see, and outdated policies must be retired promptly.

First-pass triage of multilingual inquiries

Export teams receive inquiries in Chinese, English or Arabic and need a first answer from product materials before human review. Evaluate target-language performance and refusal behavior.

The expected output

A scoped knowledge assistant evaluation

What to share

Authorized sample documents, target languages, expected users, hosting requirements and a business owner.

What to evaluate

Build an evaluation set with expected answers.
Verify citations, model version and refusal behavior.
Test document permissions and update procedures.
Compare full running cost and human review time.

How the evaluation runs, step by step

Fix one task and a business owner
Choose one workflow, one set of authorized documents and one person accountable for answer quality. Identify outdated and conflicting versions and decide first which document is authoritative.
Build the evaluation set before tuning
Follow the resource method: everyday factual questions, cross-document comparisons, questions that should be refused for lack of evidence, and permission tests. Agree the set and thresholds before seeing results.
Compare candidates on the same set
Fix documents, prompts and model versions. Business reviewers compare correctness, citation support, refusals and latency, and record the time needed to correct each answer by hand.
Settle operations and cost before expanding
Document who updates content, approves access, reviews wrong answers and handles model changes, and map every external model call. Count hosting, inference, ingestion, support and human review before scaling.

Common mistakes

Replacing your own data test with public leaderboards
Rankings reflect general tasks, not your documents, terminology and language. Performance on the same evaluation set is the basis for selection.
Swapping hard questions after seeing results
Removing failures from the set distorts the evaluation. Keep failed results and separate tuning questions from held-out tests, so you know how the assistant will behave in production.
Equating open-source availability with free commercial use
Open-source licenses carry additional conditions, such as multi-tenant and branding restrictions. Check the license of the exact version and the hosting terms before launch.

What to record during the pilot

Write these down before the test starts, so the decision rests on numbers you collected rather than on impressions.

Correct answers on the evaluation set
The share of questions answered correctly according to the business reviewer, reported separately for everyday, cross-document and target-language questions.
Citation support rate
The share of citations that, when opened, actually support the answer. An answer with a decorative citation is counted as unsupported.
Correct refusals
How often the assistant declines questions that have no answer in the documents, and how often it invents one instead.
Permission test results
The number of role-specific questions where a user could retrieve a document outside their permission. The acceptable number is zero.
Review minutes and cost per answer
The average time a reviewer spends checking or correcting an answer, together with hosting, inference and ingestion cost per month.

Key terms on this page

These terms are used in the same sense on this page, in the linked resource and in our quotations.

Evaluation set: A fixed list of real questions with expected answers, sources and pass criteria, agreed before any tuning. It is the yardstick for every candidate.
Retrieval-augmented generation (RAG): The approach where the assistant retrieves passages from your documents and answers from them, rather than from general training data alone.
Citation: The reference the assistant shows for an answer. A citation counts only if the cited passage actually supports the statement when a reviewer opens it.
Refusal behavior: What the assistant does when the documents contain no answer. A correct refusal is a pass; an invented answer is a fail even if it sounds plausible.
Permission boundary: The rule that each user role can only retrieve the documents it is allowed to see, tested with role-specific questions.
Business owner: The named person accountable for answer quality, document updates and the decision to expand or stop.
Held-out test: Questions kept aside from tuning and run only at the end, so the result shows how the assistant behaves on questions it has not been adjusted for.
Total running cost: Hosting, inference, ingestion, licensing, support and human review time added together per month, not the model API price alone.

What to confirm

Implementation scope and software licensing need to be agreed. Open-source availability does not establish unrestricted resale rights.

Common questions

How many documents do we need to start?

Start with the authorized documents one workflow needs; a few dozen is often enough for a meaningful evaluation set. A small, authoritative scope is worth more than a large, uncleaned archive.

Will the evaluation send our documents to third parties?

It depends on the deployment and the model services chosen. Even a self-hosted application may send data to external model or speech services. We suggest evaluating with non-sensitive documents first and mapping every data flow.

Does CNPS handle implementation?

CNPS first helps define scope, the evaluation set and acceptance. Implementation capacity, commercial scope and software licensing are agreed separately for each requirement and are not included by default in the evaluation.

Can the assistant run entirely on our own servers?

Self-hosted deployment is possible with several open-source applications, but the language, embedding and speech models it calls may still be external services. Map every call before deciding what “on-premises” means for your data.

How do we keep answers current when documents change?

Name the person who updates content, define how retired documents are removed from retrieval and re-run part of the evaluation set after each significant update. An assistant without an update routine drifts out of date quietly.

Which languages can be evaluated?

Chinese, English and Arabic questions can all be included, but performance differs by question language and by document language. Build target-language questions into the evaluation set rather than assuming results transfer.

Continue exploring

Industry references are not presented as CNPS customer deliveries.