Data Preparation API

hyperion.data_prep converts supported corpora into Hyperion recording, segment, class, and dataset-manifest tables. Each corpus preparer is registered under a dataset name and exposed through hyperion-prepare-data.

Base extension point

class hyperion.data_prep.data_prep.DataPrep(corpus_dir: str | Path, output_dir: str | Path, use_kaldi_ids: bool = False, target_sample_freq: int | None = None, num_threads: int = 10)[source]

Base class for data preparation. Handles parallel audio processing and metadata generation.

corpus_dir

Path to input data directory.

Type:

Path

output_dir

Path to output data directory.

Type:

Path

use_kaldi_ids

Whether to prefix segment IDs with speaker IDs (Kaldi style).

Type:

bool

target_sample_freq

Target audio sampling frequency.

Type:

int

num_threads

Number of threads for parallel processing.

Type:

int

registry = {'arts_phase1_eval3_british': <class 'hyperion.data_prep.arts_phase1_eval3_british.ARTSPhase1Eval3BritishDataPrep'>, 'asvspoof2015': <class 'hyperion.data_prep.asvspoof2015.ASVSpoof2015DataPrep'>, 'asvspoof2017': <class 'hyperion.data_prep.asvspoof2017.ASVSpoof2017DataPrep'>, 'asvspoof2019': <class 'hyperion.data_prep.asvspoof2019.ASVSpoof2019DataPrep'>, 'asvspoof2021': <class 'hyperion.data_prep.asvspoof2021.ASVSpoof2021DataPrep'>, 'asvspoof2024': <class 'hyperion.data_prep.asvspoof2024.ASVSpoof2024DataPrep'>, 'commonvoice': <class 'hyperion.data_prep.mozilla_commonvoice.CommonVoiceDataPrep'>, 'fake_codec': <class 'hyperion.data_prep.fake_codec.FakeCodecDataPrep'>, 'gigaspeech': <class 'hyperion.data_prep.gigaspeech.GigaSpeechDataPrep'>, 'hf_dataset': <class 'hyperion.data_prep.hf_dataset.HFDatasetDataPrep'>, 'iarpa_mx6_debug': <class 'hyperion.data_prep.iarpa_mx6_debug.IARPAMixer6DebugDataPrep'>, 'janus_multimedia': <class 'hyperion.data_prep.janus_multimedia.JanusMultimediaDataPrep'>, 'ldc2024e41': <class 'hyperion.data_prep.ldc2024e41.LDC2024E41DataPrep'>, 'ldc2025e05': <class 'hyperion.data_prep.ldc2025e05.LDC2025E05DataPrep'>, 'ldc2025e08': <class 'hyperion.data_prep.ldc2025e08.LDC2025E08DataPrep'>, 'libriheavy': <class 'hyperion.data_prep.libriheavy.LibriHeavyDataPrep'>, 'librilight': <class 'hyperion.data_prep.librilight.LibriLightDataPrep'>, 'librispeech': <class 'hyperion.data_prep.librispeech.LibriSpeechDataPrep'>, 'libritts': <class 'hyperion.data_prep.libritts.LibriTTSDataPrep'>, 'libritts-r': <class 'hyperion.data_prep.libritts_r.LibriTTS_R_DataPrep'>, 'mls': <class 'hyperion.data_prep.mls.MLSDataPrep'>, 'musan': <class 'hyperion.data_prep.musan.MusanDataPrep'>, 'ravdess': <class 'hyperion.data_prep.ravdess.RAVDESSPrep'>, 'rirs': <class 'hyperion.data_prep.rirs.RIRSDataPrep'>, 'sre16': <class 'hyperion.data_prep.sre16.SRE16DataPrep'>, 'sre18': <class 'hyperion.data_prep.sre18.SRE18DataPrep'>, 'sre19_av': <class 'hyperion.data_prep.sre19_av.SRE19AVDataPrep'>, 'sre19_cts': <class 'hyperion.data_prep.sre19_cts.SRE19CTSDataPrep'>, 'sre21': <class 'hyperion.data_prep.sre21.SRE21DataPrep'>, 'sre24': <class 'hyperion.data_prep.sre24.SRE24DataPrep'>, 'sre_cts_superset': <class 'hyperion.data_prep.sre_cts_superset.SRECTSSupersetDataPrep'>, 'vctk': <class 'hyperion.data_prep.vctk.VCTKDataPrep'>, 'voxceleb1': <class 'hyperion.data_prep.voxceleb1.VoxCeleb1DataPrep'>, 'voxceleb2': <class 'hyperion.data_prep.voxceleb2.VoxCeleb2DataPrep'>, 'voxsrc22': <class 'hyperion.data_prep.voxsrc22.VoxSRC22DataPrep'>}
__init__(corpus_dir: str | Path, output_dir: str | Path, use_kaldi_ids: bool = False, target_sample_freq: int | None = None, num_threads: int = 10)[source]
static dataset_name() → str[source]

Returns a unique identifier for the dataset.

static _get_recording_duration(recordings: RecordingSet, i: int, n: int) → Tuple[List[float], List[float]][source]

Helper function to calculate duration and sample rate for audio chunks.

get_recording_duration(recording_set: RecordingSet)[source]

Computes and appends duration and sampling frequency for each recording.

Parameters:

recording_set (dict) – A dictionary expected to be updated with ‘duration’ and ‘sample_freq’.

static _age_to_arts_age_group(values: _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]) → Series[source]

Map numeric ages to ARTS age buckets used across this recipe.

Parameters:

values – Input ages, typically numeric or castable to numeric.

Returns:

Pandas categorical series with ARTS age-group labels.

static _language_to_alpha3(values: Series | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]) → Series[source]

Convert language names to ISO 639-3 alpha-3 codes.

Parameters:

values – Input language names, typically strings.

Returns:

Pandas series with ISO 639-3 alpha-3 language codes.

static add_class_args(parser: ArgumentParser) → None[source]

Adds command-line arguments to the parser for configuring DataPrep.

Parameters:

parser (ArgumentParser) – Argument parser to which arguments will be added.

All preparers share corpus_dir and output_dir inputs and can optionally set a target sample frequency. Corpus-specific subclasses add only the inputs needed to locate that corpus’s audio and annotations.

Use a registered preparer

Discover names and their version-specific requirements from the CLI:

hyperion-prepare-data --help
hyperion-prepare-data voxceleb1 --help

The output should be inspected as CSV manifests before training: recording and segment ids, storage paths, speaker labels, durations, and sample rates must align. See Prepare Dataset Metadata and VAD for the operational workflow.

Add a new preparer

A new preparer subclasses DataPrep, implements a unique dataset_name, and implements its corpus-specific preparation flow. Subclass registration is automatic. Keep corpus parsing in that module and write standard CSV manifests instead of embedding corpus-specific behavior in training commands.

See also