{"id":2210,"library":"pyannote-metrics","title":"pyannote-metrics","description":"pyannote.metrics is an open-source Python library, currently at version 4.0.0, designed for reproducible evaluation, diagnostic, and error analysis of speaker diarization systems. It provides a comprehensive set of evaluation metrics and a command-line interface, making it a critical tool for researchers in the field of speech processing. The library maintains a steady release cadence with regular updates and occasional major version changes that introduce breaking modifications.","status":"active","version":"4.0.0","language":"python","source_language":"en","source_url":"https://github.com/pyannote/pyannote-metrics","tags":["audio","speech","diarization","metrics","evaluation","speaker-diarization"],"install":[{"cmd":"pip install pyannote-metrics","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Core data structures for handling annotations and segments, fundamental for defining reference and hypothesis inputs to metrics.","package":"pyannote.core"},{"reason":"Provides reproducible experimental protocols for multimedia databases, often used in conjunction with metrics for standardized evaluation.","package":"pyannote.database","optional":true}],"imports":[{"note":"Primary import for computing the Diarization Error Rate.","symbol":"DiarizationErrorRate","correct":"from pyannote.metrics.diarization import DiarizationErrorRate"},{"note":"Required from 'pyannote.core' to define ground truth and hypothesis segments for evaluation.","symbol":"Annotation","correct":"from pyannote.core import Annotation"},{"note":"Required from 'pyannote.core' to define temporal segments within annotations.","symbol":"Segment","correct":"from pyannote.core import Segment"}],"quickstart":{"code":"from pyannote.core import Segment, Annotation\nfrom pyannote.metrics.diarization import DiarizationErrorRate\n\n# Define a reference (ground truth) annotation\nreference = Annotation(uri='file1')\nreference[Segment(0, 10)] = 'A'\nreference[Segment(12, 20)] = 'B'\nreference[Segment(24, 27)] = 'A'\nreference[Segment(30, 40)] = 'C'\n\n# Define a hypothesis (system output) annotation\nhypothesis = Annotation(uri='file1')\nhypothesis[Segment(2, 13)] = 'a'\nhypothesis[Segment(13, 14)] = 'd'\nhypothesis[Segment(14, 20)] = 'b'\nhypothesis[Segment(22, 38)] = 'c'\nhypothesis[Segment(38, 40)] = 'd'\n\n# Instantiate the Diarization Error Rate metric\nmetric = DiarizationErrorRate()\n\n# Compute the DER\nder_value = metric(reference, hypothesis)\nprint(f\"Diarization Error Rate: {der_value:.3f}\")","lang":"python","description":"This quickstart demonstrates how to compute the Diarization Error Rate (DER) using `pyannote.metrics`. It involves creating `Annotation` objects for both the reference and hypothesis, defining temporal `Segment`s with speaker labels, and then instantiating and calling the `DiarizationErrorRate` class."},"warnings":[{"fix":"Re-evaluate existing systems with the new metric behavior. Understand how overlapping speech is handled by your diarization system and pyannote.metrics to correctly interpret results.","message":"Version 3.3.0 introduced a breaking change by improving diarization purity and coverage to explicitly account for overlapping regions, which might alter previously obtained metric values for systems that handle overlap differently.","severity":"breaking","affected_versions":">=3.3.0"},{"fix":"Always use the evaluation tool and parameters (e.g., `collar`, `skip_overlap`) specified by the benchmark you are targeting, and explicitly report all settings. Avoid direct comparisons of scores obtained from different tools.","message":"Comparison of evaluation scores across different diarization evaluation tools (e.g., `pyannote.metrics` vs. `md-eval`) is not recommended due to varying design choices, default parameters (like collar size), and handling of speaker mapping and overlapping speech.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Explicitly define and report the `collar` setting in your experiments. Be aware that different collar values can change DER by several percentage points, making results incomparable if not standardized.","message":"The `collar` parameter, typically set to 0.25 (250 ms exclusion around boundaries), significantly impacts DER. Manual annotations often lack audio sample-level precision, making a collar common practice. However, strict benchmarks may use `collar=0.0`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Migrate to Python 3.10 or newer and convert non-RTTM annotation files to RTTM format for compatibility.","message":"Older versions (2.0.1) dropped support for Python 2.7 and all file formats except RTTM for evaluation. Ensure your environment uses Python 3.10+ and RTTM for input annotations.","severity":"breaking","affected_versions":"<2.0.1 (upgrade paths)"}],"env_vars":null,"search_vec":"'4.0.0':15 'analysi':23 'audio':74 'break':72 'cadenc':61 'chang':69 'command':39 'command-lin':38 'comprehens':31 'critic':45 'current':12 'design':16 'diagnost':20 'diariz':26,76,81 'error':22 'evalu':19,34,78 'field':51 'interfac':41 'introduc':71 'librari':11,56 'line':40 'maintain':57 'major':67 'make':42 'metric':3,35,77 'modif':73 'occasion':66 'open':8 'open-sourc':7 'process':54 'provid':29 'pyannot':2 'pyannote-metr':1 'pyannote.metrics':4 'python':10 'regular':63 'releas':60 'reproduc':18 'research':48 'set':32 'sourc':9 'speaker':25,80 'speaker-diar':79 'speech':53,75 'steadi':59 'system':27 'tool':46 'updat':64 'version':14,68","created_at":"2026-04-09T18:48:13.204501+00:00","updated_at":"2026-04-16T18:19:03.520022+00:00","problems":[{"fix":"Ensure the library is correctly installed using pip: `pip install pyannote.metrics`. If using a virtual environment, activate it before installation. Also, ensure `pyannote.core` is installed, as it's a dependency: `pip install pyannote.core`.","cause":"The `pyannote.metrics` library or one of its core dependencies like `pyannote.core` is not installed, or the Python environment where the code is run does not have access to the installed package.","error":"ModuleNotFoundError: No module named 'pyannote.metrics'"},{"fix":"Before computing metrics, ensure that both the reference and hypothesis `Annotation` objects contain actual segments. You may need to add checks for empty annotations (e.g., `if len(reference) > 0 and len(hypothesis) > 0:`) and handle such cases gracefully, perhaps by assigning a default value or skipping the metric computation.","cause":"This error often occurs when calculating metrics like `SegmentationPurity` or `SegmentationCoverage` on empty reference or hypothesis annotations, leading to division by zero in the metric calculation.","error":"ZeroDivisionError: float division by zero"},{"fix":"Verify that the `Annotation` objects (reference and hypothesis) used for metric computation are not empty. Implement checks to ensure annotations contain segments before calling metric functions to avoid operating on zero-size arrays.","cause":"Similar to `ZeroDivisionError`, this `ValueError` arises when metrics (like `SegmentationCoverage` or `SegmentationPurityCoverageFMeasure`) attempt to perform reduction operations (like finding a maximum) on arrays that are empty, which can happen with empty reference or hypothesis annotations.","error":"ValueError: zero-size array to reduction operation maximum which has no identity"},{"fix":"Upgrade all `pyannote` libraries to compatible versions, ideally the latest stable releases, to ensure consistent API usage across the ecosystem. For example, `pip install --upgrade pyannote.database pyannote.pipeline`.","cause":"This error typically indicates an incompatibility between versions of `pyannote.database` and `pyannote.pipeline` (or other components of the pyannote ecosystem). An older version of `pyannote.database`'s `get_protocol` function might not accept the `progress` argument, which was introduced or removed in different versions.","error":"TypeError: get_protocol() got an unexpected keyword argument 'progress'"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"4.1","cli_name":"pyannote-metrics","cli_version":"Traceback (most recent call last):","type":"library","homepage":null,"github":null,"docs":null,"changelog":null,"pypi":"https://pypi.org/project/pyannote-metrics/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["ai-ml"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-28","next_check":"2026-07-28","install_tag":null}}