Text Normalization API

hyperion.text_norm provides reusable normalization for speech transcripts, ASR targets, and text-based evaluation. Normalization policy is task-specific: record the normalizer configuration with any metric result that depends on it.

Core normalizers

class hyperion.text_norm.BasicTextNormalizer(remove_punctuation: bool = True, remove_symbols: bool = True, remove_diacritics: bool = False, split_chars: bool = False)[source]

A configurable text normalization class for Unicode text.

This class supports: - Unicode normalization (NFKC/NFKD) - Removal of punctuation, symbols, and diacritics - Bracket/parenthesis content stripping - Optional grapheme-based character splitting

remove_punctuation

Whether to remove punctuation characters.

Type:

bool

remove_symbols

Whether to remove symbol characters.

Type:

bool

remove_diacritics

Whether to remove diacritical marks.

Type:

bool

split_chars

Whether to split characters into grapheme clusters.

Type:

bool

remove_map

Internal map of Unicode categories to remove.

Type:

str

__init__(remove_punctuation: bool = True, remove_symbols: bool = True, remove_diacritics: bool = False, split_chars: bool = False) → None[source]

Initialize the BasicTextNormalizer.

Parameters:

  • remove_punctuation (bool) – If True, remove punctuation.

  • remove_symbols (bool) – If True, remove symbols.

  • remove_diacritics (bool) – If True, remove diacritics.

  • split_chars (bool) – If True, split text into Unicode grapheme clusters.

_normalize_unicode_text(text: str, keep: str) → str[source]

Normalize and clean a Unicode string.

Applies NFKC (or NFKD when removing diacritics), and removes specified categories like marks (M), symbols (S), and punctuation (P).

Parameters:

  • text (str) – Input text to normalize.

  • keep (str) – Characters to keep unchanged, even if they match removal rules.

Returns:

Normalized and cleaned text.

Return type:

str

__call__(text: str) → str[source]

Apply normalization pipeline to a string.

This includes: - Lowercasing - Replacing unicode apostrophes - Removing bracketed/parenthesized content - Unicode category-based cleaning - Whitespace normalization - Optional character splitting

Parameters:

text (str) – Input string.

Returns:

Normalized string.

Return type:

str

class hyperion.text_norm.EnglishTextNormalizer(standardize_numbers=False, standardize_numbers_rev=True, remove_fillers=True, remove_punctuation: bool = True, remove_symbols: bool = True, remove_diacritics: bool = False, split_chars: bool = False)[source]

This is a modified version of the Whisper text normalizer designed to enhance compatibility across various ASRs.

Key features:

  1. Idempotency: output is unchanged with repeated application.

  2. The original Whisper-tailored number normalization is replaced with one that is compatible with

    other ASR systems, mapping numerals into spelled-out numbers. See EnglishReverseNumberNormalizer for details and limitations.

  3. Filler words are removed by default, similar to the original normalizer: [‘hmm’, ‘uh’, ‘ah’, ‘eh’].

    This is for compatibility with ASRs trained to ignore these.

  4. Added normalization for some common words: okay -> ok, everyday -> every day etc.

__init__(standardize_numbers=False, standardize_numbers_rev=True, remove_fillers=True, remove_punctuation: bool = True, remove_symbols: bool = True, remove_diacritics: bool = False, split_chars: bool = False) → None[source]

Initialize the BasicTextNormalizer.

Parameters:

  • remove_punctuation (bool) – If True, remove punctuation.

  • remove_symbols (bool) – If True, remove symbols.

  • remove_diacritics (bool) – If True, remove diacritics.

  • split_chars (bool) – If True, split text into Unicode grapheme clusters.

_normalize_unicode_text(text: str, keep: str) → str

Normalize and clean a Unicode string.

Applies NFKC (or NFKD when removing diacritics), and removes specified categories like marks (M), symbols (S), and punctuation (P).

Parameters:

  • text (str) – Input text to normalize.

  • keep (str) – Characters to keep unchanged, even if they match removal rules.

Returns:

Normalized and cleaned text.

Return type:

str

BasicTextNormalizer handles Unicode, punctuation/symbol removal, diacritics, bracketed text, whitespace, and optional grapheme splitting. EnglishTextNormalizer adds English-oriented behavior. Apply the same normalizer to references and hypotheses before computing text metrics.

Numbers and spelling

class hyperion.text_norm.english_number_normalizers.EnglishNumberNormalizer[source]

A robust English number normalization utility that converts spelled-out numbers and numeric expressions into standard Arabic numeral forms.

Features:

  • Converts written numbers (e.g., “twenty five”) to digits (“25”)

  • Handles suffixes like ordinals (“first”, “101st”) and decades (“1960s”)

  • Converts currency expressions (e.g., “$20 million” to “20000000 dollars”)

  • Supports expressions like “one oh one” to “101”

  • Deals with multipliers (hundred, thousand, million, etc.)

  • Supports common filler modifiers like “double” and “triple”

  • Preserves and normalizes special symbols (%, ¢, $, £, €)

  • Configurable for integrating into ASR post-processing or NLP pipelines

zeros

Set of zero-word variants like ‘o’ and ‘zero’.

Type:

set

ones

Mapping of one-to-nineteen words to their numeric values.

Type:

dict

ones_plural

Plural forms of one-to-nineteen words.

Type:

dict

ones_ordinal

Ordinal forms (e.g., ‘first’, ‘second’, etc.).

Type:

dict

ones_suffixed

Combined plural and ordinal forms.

Type:

dict

tens

Tens words (e.g., ‘twenty’, ‘thirty’) mapped to values.

Type:

dict

tens_plural

Plural forms of tens.

Type:

dict

tens_ordinal

Ordinal forms of tens (e.g., ‘twentieth’).

Type:

dict

tens_suffixed

Combined plural and ordinal tens.

Type:

dict

multipliers

Magnitude multipliers (e.g., ‘million’, ‘billion’).

Type:

dict

multipliers_plural

Plural forms of multipliers.

Type:

dict

multipliers_ordinal

Ordinal forms of multipliers.

Type:

dict

multipliers_suffixed

Combined plural and ordinal multipliers.

Type:

dict

preceding_prefixers

Words that act as numeric prefixes (e.g., ‘minus’).

Type:

dict

following_prefixers

Words that follow numbers (e.g., ‘dollars’, ‘cents’).

Type:

dict

prefixes

All recognized prefix symbols.

Type:

set

suffixers

Postfix modifiers such as ‘percent’.

Type:

dict

specials

Special control words like ‘and’, ‘point’, ‘double’.

Type:

set

decimals

Words allowed in decimal numbers.

Type:

set

words

All valid words handled by the normalizer.

Type:

set

literal_words

Words like ‘one’ or ‘ones’ that may be left unconverted.

Type:

set

__init__() → None[source]
_init_digits() → None[source]

Initializes mappings for zero, one-to-nineteen, plural, and ordinal forms.

_init_tens() → None[source]

Initializes mappings for tens (twenty, thirty, etc.), including plural and ordinal forms.

_init_multipliers() → None[source]

Initializes mappings for large number multipliers like thousand, million, etc., including plural and ordinal forms.

_init_prefix_suffix() → None[source]

Initializes mappings for prefix symbols (positive, negative) and suffix terms (currency, percent).

_init_specials() → None[source]

Initializes special control words such as ‘and’, ‘double’, ‘point’.

_build_vocab() → None[source]

Builds the full set of recognized words from all category dictionaries.

process_words(words: List[str]) → Iterator[str][source]

Process a list of tokenized words and convert recognized number patterns (e.g., ‘twenty five’, ‘one oh one’, ‘ten thousand dollars’) into numeric strings.

Parameters:

words (List[str]) – A tokenized list of words from a text or transcript.

Yields:

Iterator[str] – Converted or unchanged word tokens, as normalized string values.

preprocess(s: str) → str[source]

Preprocess a string to normalize patterns that could affect numeric interpretation.

This method performs the following:

  • Replaces patterns like “<number> and a half” with “<number> point five” if the preceding word is a recognized number or multiplier.

  • Inserts spaces between number-letter boundaries (e.g., “20th” → “20 th”) to improve tokenization.

  • Removes spaces between numbers and suffixes (e.g., “20 th” → “20th”).

Parameters:

s (str) – The raw input string.

Returns:

A preprocessed string suitable for further normalization.

Return type:

str

postprocess(s: str) → str[source]

Postprocess the normalized string to refine currency and singular word forms.

This method applies the following transformations:

  • Converts patterns like “$2 and ¢7” into “$2.07” (combining integer and cent parts).

  • Converts “$0.xx” into “¢xx” when appropriate (extracting cents).

  • Replaces standalone “1” or “1s” with “one” or “ones” for improved readability.

Parameters:

s (str) – The normalized string to be postprocessed.

Returns:

A cleaned-up string with currency and linguistic adjustments.

Return type:

str

__call__(s: str) → str[source]

Normalize a string containing spelled-out English numbers into numeric form.

This is the main entry point for using the normalizer. It applies:

  1. Preprocessing (e.g., handling “and a half”, spacing around suffixes)

  2. Word-level normalization via process_words

  3. Postprocessing (e.g., combining “$2 and ¢7” into “$2.07”, replacing “1” with “one”)

Parameters:

s (str) – The raw input string to normalize.

Returns:

The fully normalized string with numbers in standard Arabic format.

Return type:

str

class hyperion.text_norm.english_number_normalizers.EnglishReverseNumberNormalizer[source]

A reverse number normalizer that approximates the inverse of EnglishNumberNormalizer.

Converts Arabic numerals (e.g., ‘365’) back into spelled-out English forms (e.g., ‘three hundred sixty five’). This is useful for comparing Whisper’s output to ASRs that cannot produce numerals directly.

Motivation:

Whisper outputs rich numeric expressions like “$20” or “50%”, which many ASRs cannot generate. This class converts Whisper’s output back into spoken-word forms to ensure fair comparison.

Examples

  • “365” -> “three hundred sixty five”

  • “$20” -> “twenty dollars”

  • “50%” -> “fifty percent”

  • “12th” -> “twelfth”

  • “12s” -> “twelves”

  • “90th” -> “ninetieth”

  • “90s” -> “nineties”

  • “70 000” -> “seventy thousand” (special case)

Caveats:

  • Only supports numbers in the range 0–1000

  • Does not handle signs like ‘+’ or ‘-’

  • Some ambiguity is inherent (e.g., “100” → “one hundred” vs. “a hundred”)

int_to_ones

Mapping from integer values (1–19) to their spelled-out word forms.

Type:

dict[int, str]

int_to_tens

Mapping from tens values (20, 30, …, 90) to their word forms.

Type:

dict[int, str]

str_to_ones_suffixed

Mapping from numeric strings with suffixes (e.g., ‘12th’, ’12s’) to their spoken equivalents.

Type:

dict[str, str]

str_to_tens_suffixed

Mapping from suffixed tens (e.g., ’90s’, ‘90th’) to spelled-out versions like ‘nineties’, ‘ninetieth’.

Type:

dict[str, str]

__init__() → None[source]
__call__(s: str) → str[source]

Converts numeric expressions in a string back to their approximate spelled-out equivalents.

Rewrites:

  • Currency symbols (e.g., “$20” → “twenty dollars”)

  • Percent signs (e.g., “50%” → “fifty percent”)

  • Ordinal suffixes (e.g., “12th” → “twelfth”)

  • Plurals (e.g., “20s” → “twenties”)

Parameters:

s (str) – A string containing numeric expressions.

Returns:

The normalized string with numerals converted back to words.

Return type:

str

_build_vocab() → None

Builds the full set of recognized words from all category dictionaries.

_init_digits() → None

Initializes mappings for zero, one-to-nineteen, plural, and ordinal forms.

_init_multipliers() → None

Initializes mappings for large number multipliers like thousand, million, etc., including plural and ordinal forms.

_init_prefix_suffix() → None

Initializes mappings for prefix symbols (positive, negative) and suffix terms (currency, percent).

_init_specials() → None

Initializes special control words such as ‘and’, ‘double’, ‘point’.

_init_tens() → None

Initializes mappings for tens (twenty, thirty, etc.), including plural and ordinal forms.

postprocess(s: str) → str

Postprocess the normalized string to refine currency and singular word forms.

This method applies the following transformations:

  • Converts patterns like “$2 and ¢7” into “$2.07” (combining integer and cent parts).

  • Converts “$0.xx” into “¢xx” when appropriate (extracting cents).

  • Replaces standalone “1” or “1s” with “one” or “ones” for improved readability.

Parameters:

s (str) – The normalized string to be postprocessed.

Returns:

A cleaned-up string with currency and linguistic adjustments.

Return type:

str

preprocess(s: str) → str

Preprocess a string to normalize patterns that could affect numeric interpretation.

This method performs the following:

  • Replaces patterns like “<number> and a half” with “<number> point five” if the preceding word is a recognized number or multiplier.

  • Inserts spaces between number-letter boundaries (e.g., “20th” → “20 th”) to improve tokenization.

  • Removes spaces between numbers and suffixes (e.g., “20 th” → “20th”).

Parameters:

s (str) – The raw input string.

Returns:

A preprocessed string suitable for further normalization.

Return type:

str

process_words(words: List[str]) → Iterator[str]

Process a list of tokenized words and convert recognized number patterns (e.g., ‘twenty five’, ‘one oh one’, ‘ten thousand dollars’) into numeric strings.

Parameters:

words (List[str]) – A tokenized list of words from a text or transcript.

Yields:

Iterator[str] – Converted or unchanged word tokens, as normalized string values.

class hyperion.text_norm.spelling_normalizer.SpellingNormalizer(mapping_name: str)[source]

A rule-based normalizer that replaces words in a string using a predefined JSON mapping.

This class loads a word-to-word mapping from a JSON file and applies substitutions on input strings, replacing each word with its mapped value (if any).

Example

If the mapping file contains {“colour”: “color”}, then:

SpellingNormalizer(“british_to_american”)(“colour is nice”) → “color is nice”

mapping

A dictionary loaded from the specified JSON file, mapping input words to their normalized forms.

Type:

dict

__init__(mapping_name: str)[source]
__call__(s: str) → str[source]

Apply the loaded spelling normalization to an input string.

Each word in the string is replaced by its corresponding value in the mapping if a match is found; otherwise, the original word is preserved.

Parameters:

s (str) – The input string to normalize.

Returns:

The normalized string.

Return type:

str

The number and spelling helpers are English-specific. Do not silently apply them to another language; use a language-appropriate normalization policy and document it with the evaluation protocol.

See also