DocsEvaluationHuman Annotation

Human Annotation for LLM apps

Collaborate with your team and add scores via the Langfuse UI. You can add scores to both traces and observations within a trace.

Annotate

Common use cases

  • Collaboration: Enable team collaboration by inviting other internal members to annotate a subset of traces and observations. This manual evaluation can enhance the overall accuracy and reliability of your results by incorporating diverse perspectives and expertise. See Annotation queues to manage and prioritize the annotation tasks effectively.
  • Annotation data consistency: Create score configurations for annotation workflows to ensure that all team members are using standardized scoring criteria. Hereby configure categorical, numerical or binary score types to capture different aspects of your data.
  • Evaluation of new product features: This feature can be useful for new use cases where no other scores have been allocated yet.
  • Benchmarking of other scores: Establish a human baseline score that can be used as a benchmark to compare and evaluate other scores. This can provide a clear standard of reference and enhance the objectivity of your performance evaluations.

Get started

Create score configurations

To use annotation, you need to create a score configuration. You can create multiple score configurations for different types of scores. Score configurations are immutable. However, you can archive configs if you no longer want to use them in annotation. Archived configs can be restored at any time.

To create a score configuration:

  • Navigate to Settings, locate the Score Configs table, and click Add new score config.
  • Specify the name of the score configuration and the type of score you want to create. You can choose between Categorical, Numeric, and Boolean score types.
  • Optionally, add a description to provide additional context for your team members.

Your configs are now available for annotation of traces and observations.

Create config

Data Labelling on LLM traces or observations

To annotate a trace or observation:

  • Navigate to the trace detail view.
  • Click on the Annotate button.
  • Select the scores you want to add.
  • Fill in the score values. The scores will be saved automatically. Annotation scores can be edited or deleted at any time.
  • Optionally, add comments to individual scores.

Annotate

View scores on trace or observation

Upon completing annotation click on the Scores tab to view a table of all the scores that have been added to the trace or observation.

Detail scores table

Annotation Queues

Annotation queues allow you to manage and prioritize your annotation tasks in a structured way. This feature is particularly useful for large-scale projects that benefit of human-in-the-loop evaluation at some scale. Queues streamline this process by allowing for specifying which traces/observations you’d like to annotate on which dimensions.

Where is this feature available?
  • Hobby
  • Pro
  • Team
  • Self Hosted
    (Enterprise)

Create annotation queues

Set up annotation queues to specify which traces/observations you’d like to annotate on which dimensions. Queues are fully mutable and editable even after annotation tasks have been created. You may also add/remove annotation tasks to/from queues at any time.

Populate annotation queues

Once you have created annotation queues, you can assign traces or observations to them. The easiest way to do this at scale is to navigate to the trace table view, optionally adjust the filters to narrow down the traces/observations you’d like to annotate, and then use the Actions > Add to queue button in the top right corner. You can also add single traces and observations to queues via the Annotate dropdown on the detail view.

Process annotation tasks

Navigate to the Annotate tab to view all the annotation queues that have been created. Select the queue you’d like to process. You will see all annotation tasks sequentially in the queue. Our queues are designed to be first-in-first-out (FIFO) and are fully concurrency safe. After adding scores on the defined dimensions, click on Complete + next to move to the next annotation task. If you’d like to view an overview of all annotation tasks of a given queue, you can click on the queue name in the queue table. This overview shows the status of each tasks.

GitHub Discussions

Was this page useful?

Questions? We're here to help

Subscribe to updates