Skip to main content

Knowledge Retention

LLM-as-a-judge
Referenceless metric
Chatbot metric

The knowledge retention metric is a conversational metric that determines whether your LLM chatbot is able to retain factual information presented throughout a conversation.

info

This is great for a LLM powered questionnaire use case.

Required Arguments​

To use the KnowledgeRetentionMetric, you'll have to provide the following arguments when creating a ConversationalTestCase:

  • turns

You must provide the role and content for evaluation to happen. Read the How Is It Calculated section below to learn more.

Usage​

The KnowledgeRetentionMetric() can be used for end-to-end multi-turn evaluation:

from deepeval import evaluate
from deepeval.test_case import Turn, ConversationalTestCase
from deepeval.metrics import KnowledgeRetentionMetric

convo_test_case = ConversationalTestCase(
    turns=[Turn(role="...", content="..."), Turn(role="...", content="...")]
)
metric = KnowledgeRetentionMetric(threshold=0.5)

# To run metric as a standalone
# metric.measure(convo_test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[convo_test_case], metrics=[metric])

There are FIVE optional parameters when creating a KnowledgeRetentionMetric:

  • [Optional] threshold: a float representing the maximum passing threshold, defaulted to 0.5.
  • [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to 'gpt-4o'.
  • [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
  • [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 0. Defaulted to False.
  • [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.

As a standalone​

You can also run the KnowledgeRetentionMetric on a single test case as a standalone, one-off execution.

...

metric.measure(test_case)
print(metric.score, metric.reason)
caution

This is great for debugging or if you wish to build your own evaluation pipeline, but you will NOT get the benefits (testing reports, Confident AI platform) and all the optimizations (speed, caching, computation) the evaluate() function or deepeval test run offers.

How Is It Calculated?​

The KnowledgeRetentionMetric score is calculated according to the following equation:

Knowledge Retention=Number of Assistant Turns without Knowledge AttritionsTotal Number of Assistant Turns\text{Knowledge Retention} = \frac{\text{Number of Assistant Turns without Knowledge Attritions}}{\text{Total Number of Assistant Turns}}

The KnowledgeRetentionMetric first uses an LLM to extract knowledge supplied in "content" by the "user" role throughout turns, before using the same LLM to determine whether each corresponding "assistant" content indicates an inability to recall said knowledge.