Annotate a corpus with an agreement gate

Any 8 GB laptop for a small project. Multi-user deployment requires a secured server and database.

Component	Role	Price	Link
Label Studio 1.23.0	local annotation interface and export	Free / 免费Apache-2.0 Community Edition; hosting and administration are separatechecked 2026-08-02	https://github.com/HumanSignal/label-studio/releases/tag/1.23.0
krippendorff 0.8.2	nominal inter-annotator agreement calculation	Free / 免费MIT-licensed Python packagechecked 2026-08-02	https://pypi.org/project/krippendorff/0.8.2/

Label Studio classification interface

<View>
  <Text name="text" value="$text"/>
  <Choices name="label" toName="text" choice="single" showInLine="true">
    <Choice value="include"/>
    <Choice value="exclude"/>
    <Choice value="uncertain"/>
  </Choices>
  <TextArea name="rationale" toName="text" required="true"/>
</View>

Save as agreement.py

import pandas as pd, numpy as np, krippendorff
df=pd.read_csv('annotations.csv')
wide=df.pivot(index='annotator_id',columns='item_id',values='label')
levels={v:i for i,v in enumerate(sorted(df.label.dropna().unique()))}
values=wide.replace(levels).astype(float).to_numpy()
alpha=krippendorff.alpha(reliability_data=values,level_of_measurement='nominal')
print(f'items={wide.shape[1]} annotators={wide.shape[0]} alpha={alpha:.3f}')
wide.to_csv('agreement-matrix.csv')

Start locally and calculate agreement

uv venv --python 3.12 .venv
. .venv/bin/activate
uv pip install 'label-studio==1.23.0' 'krippendorff==0.8.2' pandas
label-studio start
# Export long-form annotations.csv with item_id, annotator_id and label, then run:
python agreement.py

Known pitfalls

• Agreement thresholds must be defined before viewing final results.
• High agreement can reflect an overly broad or trivial codebook.
• Adjudicated labels must be stored separately from original annotator decisions.
• Annotators must not see each other's labels during independent coding.