Robots Atlas>ROBOTS ATLAS
Evaluation

WER

ActiveUpdated: 16 October 2026Published
Key innovation
Reducing speech transcription quality to a single, reproducible number based on word-level edit distance, accounting for substitutions, deletions and insertions.
Category
Evaluation
Abstraction level
Benchmark
Operation level
Evaluation (runtime)Inference
Use cases
Evaluating ASR (speech-to-text) systemsComparing speech recognition models (e.g. Whisper, NVIDIA Parakeet)Monitoring transcription-quality regressionsASR benchmarks (e.g. LibriSpeech)Assessing quality across languages and acoustic conditions

How it works

1) The hypothesis transcript (ASR output) and the reference transcript are normalized (e.g. lowercasing, punctuation removal). 2) The two word sequences are aligned by computing the minimum word-level edit (Levenshtein) distance. 3) Three error types are counted: substitutions (S โ€” a word replaced by another), deletions (D โ€” a reference word omitted) and insertions (I โ€” a word added by the system). 4) The error sum is divided by the number of reference words N: WER = (S + D + I) / N. The metric is based on Levenshtein distance and is normally reported after the same normalization across all compared systems.

Problem solved

An objective, reproducible and cross-model way of measuring speech transcription accuracy is needed. Subjective 'by ear' judgement cannot reliably compare systems or track progress. WER reduces ASR quality to a single number that accounts for three error types โ€” substitutions, deletions and insertions โ€” against a reference transcript.

Components

Text normalizationPreprocessing

Normalizes hypothesis and reference (lowercasing, punctuation removal, number formatting) before alignment.

Official

Levenshtein alignmentSequence alignment

Dynamic programming computing the minimum word-level edit distance between hypothesis and reference.

Error tallying (S, D, I)Aggregation

Classifies edit operations into substitutions, deletions and insertions.

Normalization by NScaling

Divides the error sum by the reference word count N, giving the result (usually in %).

Implementation

Reference implementations
jiwer
Python
Hugging Face evaluate โ€” wer
Python
torchmetrics WordErrorRate
Python
NIST SCTK / sclite
C
Official
Implementation pitfalls
Inconsistent normalizationCritical

Differing lowercasing / punctuation / number rules make WER non-comparable across systems and papers.

Fix:Use a single, explicitly documented normalizer for all compared systems.
WER can exceed 100%Medium

With many insertions the error sum can exceed N; WER is not bounded to [0, 1].

Fix:Treat WER as an error rate, not a probability; also report absolute S/D/I counts.
Equal error weightingMedium

S, D, I are weighted equally regardless of impact on meaning; WER does not capture semantics or word importance.

Fix:Complement WER with semantic or weighted metrics when meaning matters.
Word-boundary sensitivityHigh

WER is ill-defined for languages without clear word boundaries (Chinese, Japanese).

Fix:Use CER (Character Error Rate) for such languages.

Evolution

1966
Levenshtein distance
Inflection point

Vladimir Levenshtein formalizes edit distance (substitutions, deletions, insertions) โ€” the mathematical basis of WER.

2015
LibriSpeech benchmark

Panayotov et al. (ICASSP 2015) release a large open corpus, standardizing WER reporting (test-clean / test-other).

2022
Whisper and zero-shot WER

OpenAI Whisper popularizes zero-shot WER reporting across many languages and acoustic conditions.

Hyperparameters (configurable axes)

Normalization schemeCritical

Lowercasing, punctuation, number and abbreviation rules; drastically affects the score and comparability.

lowercase + usuniฤ™cie interpunkcji
Whisper EnglishTextNormalizer
GranularityHigh

Level at which errors are counted: words (WER) or characters (CER).

word (WER)
character (CER)
Tokenization / word boundariesHigh

Definition of word boundaries; problematic for languages without spaces (Chinese, Japanese) โ€” hence CER.

Computational complexity

Time complexity: O(mยทn). Space complexity: O(mยทn).

Compute bottleneck

Edit-distance alignment

The quadratic DP alignment dominates cost; negligible for short utterances, noticeable for very long sequences.

Parallelism

Parallelism level
Partially parallel

Each utterance is scored independently, so a corpus scales embarrassingly in parallel; a single DP alignment is sequential.

Scope
Inference

Hardware requirements

Primary

A pure string operation of negligible cost; computed on CPU, independent of accelerators.