Utility Layer

Overview

hyperion.utils defines the package’s identifier and table contracts: dataset manifests, enrollment mappings, trial keys, and score containers. These objects keep rows and columns aligned across preparation, extraction, scoring, and evaluation.

CSV/TSV tables are the maintained interchange format for manifests and indexes. Kaldi-style list helpers remain available for compatibility, but new workflows should use the table classes and CSV files.

Alignment rules

InfoTable-derived manifests use an id column as their stable key. Verification tables use enrollment model ids for rows and test segment ids for columns. TrialScores must use the same ordered model/segment sets as the TrialKey or TrialNdx it is evaluated against.

See Hyperion Data Model for the complete relationship between manifests, enrollment maps, trials, and scores.

Dataset and table abstractions

See Working With Info Tables for a practical tutorial on manifest usage and how the different InfoTable child classes relate to each other. See Working With HyperDataset for a dataset-level tutorial covering how those manifests are bundled and manipulated together.

class hyperion.utils.InfoTable(df: DataFrame | T)

Base class for storing structured metadata in a tabular format.

This class wraps a pandas DataFrame and adds helper methods for working with audio-visual dataset metadata such as recordings, segments, and features. Maintains a consistent interface for operations like filtering, merging, and indexing.

df

The internal DataFrame storing the metadata.

Type:

pd.DataFrame

__init__(df: DataFrame | T) → None

Initialize an InfoTable from a DataFrame or another InfoTable.

Parameters:

df (Union[pd.DataFrame, InfoTable]) – Input data.

static is_valid_df(df: DataFrame) → bool

Check if the DataFrame is valid for InfoTable.

Parameters:

df (pd.DataFrame) – DataFrame to check.

Returns:

True if valid, False otherwise.

Return type:

bool

fix_dtypes() → None

Ensure the ‘id’ column is of string type.

convert_col_to_str(column: str) → None

Ensure a specific column is of string type.

Parameters:

column (str) – Column name to convert.

copy() → T

Return a deep copy of the InfoTable.

Returns:

A deep copy.

Return type:

InfoTable

clone() → T

Alias for copy().

Returns:

A deep copy.

Return type:

InfoTable

property columns: Index

Get the DataFrame columns.

Returns:

Column names.

Return type:

pd.Index

property loc: _InfoTableIndexer

Access a group of rows and columns by label(s).

Returns:

Indexer that wraps .loc.

Return type:

_InfoTableIndexer

property iloc: _InfoTableIndexer

Access a group of rows and columns by integer position(s).

Returns:

Indexer that wraps .iloc.

Return type:

_InfoTableIndexer

property at: _InfoTableAtIndexer

Access a single value for a row/column label pair.

Returns:

Indexer that wraps .at.

Return type:

_InfoTableAtIndexer

property iat: _InfoTableAtIndexer

Access a single value for a row/column integer position pair.

Returns:

Indexer that wraps .iat.

Return type:

_InfoTableAtIndexer

__getitem__(key: Any) → T | Series

Get item from the internal DataFrame.

Parameters:

key – Key used for indexing.

Returns:

Sub-table or Series.

Return type:

Union[InfoTable, pd.Series]

__setitem__(key: Any, value: Any) → None

Set item in the internal DataFrame.

Parameters:

• key – Column label or index.

• value – Value to assign.

__contains__(key: Any) → bool

Check whether key is in the DataFrame columns.

Parameters:

key – Key to check.

Returns:

True if key is in columns.

Return type:

bool

property index: Index

Get the index of the DataFrame.

Returns:

DataFrame index.

Return type:

pd.Index

property eval: Callable

Return the DataFrame.eval method.

Returns:

The eval method for evaluating expressions.

Return type:

Callable

property iterrows: Callable

Return the DataFrame.iterrows generator.

Returns:

Yields (index, Series) pairs.

Return type:

Callable

dropna(*args: Any, **kwargs: Any) → T | None

Return a new InfoTable with missing values dropped.

Parameters:

• *args – Passed to pandas.DataFrame.dropna.

• **kwargs – Passed to pandas.DataFrame.dropna.

Returns:

A new instance with rows (or columns) with NA removed.

Return type:

InfoTable

query(expr: str, **kwargs: Any) → T

Filters rows using a boolean expression string.

Parameters:

• expr (str) – A string expression to evaluate, using column names as variables.

• **kwargs – Passed through to pandas.DataFrame.query().

Returns:

A new InfoTable with filtered rows.

Return type:

InfoTable

xs(key: Any, axis: int = 0, level: int | str = None, drop_level: bool = True) → T | Series

Returns a cross-section (row or column) from the DataFrame.

Parameters:

• key – Label or tuple of labels to select.

• axis (int) – Axis to retrieve from (0 for index, 1 for columns).

• level – Level in MultiIndex to use.

• drop_level (bool) – Whether to drop the level(s) from the result.

Returns:

If the result is a DataFrame, wraps it as InfoTable.

Return type:

InfoTable or Series

head(n: int = 5) → T

Return the first n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

tail(n: int = 5) → T

Return the last n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

sample(n: int = 1, random_state: int | RandomState | Generator | BitGenerator | None = None) → T

Return a random sample of rows.

Parameters:

• n (int) – Number of rows.

• random_state – Seed for reproducibility.

Returns:

Sampled InfoTable.

Return type:

InfoTable

drop(labels: str | int | List[Any] | ndarray | Index | Series | None = None, axis: int | str = 0, index: str | int | List[Any] | ndarray | Index | Series | None = None, columns: str | int | List[Any] | ndarray | Index | Series | None = None, level: int | str | None = None, inplace: bool = False, errors: str = 'raise') → T | None

Drop specified labels.

Parameters:

• labels – Index or column labels.

• axis (int) – Whether to drop rows (0) or columns (1).

• index – Alias for labels along the index (rows).

• columns – Column labels to drop.

• level – Level in MultiIndex from which to drop.

• inplace (bool) – Modify in place.

• errors (str) – If ‘ignore’, suppress errors for nonexistent labels.

Returns:

Modified InfoTable or None.

Return type:

Optional[InfoTable]

save(file_path: str | Path, sep: str | None = None) → None

Save the InfoTable to a file.

Parameters:

• file_path (str or Path) – Path to save the file.

• sep (Optional[str]) – Column separator (default inferred from extension).

classmethod from_lists(ids: List[str], column_names: List[str], column_data: List[List[Any]]) → T

Create InfoTable from lists of IDs and corresponding column data.

Parameters:

• ids (List[str]) – List of IDs.

• column_names (List[str]) – List of column names.

• column_data (List[List[Any]]) – Column values.

Returns:

Constructed table.

Return type:

InfoTable

classmethod from_dict(df_dict: Dict[str, List[Any]]) → T

Create InfoTable from a dictionary.

Parameters:

df_dict (Dict[str, List[Any]]) – Column data including ‘id’.

Returns:

Constructed table.

Return type:

InfoTable

classmethod load(file_path: str | Path, sep: str | None = None, name: str = 'class_id') → T

Load InfoTable from file.

Parameters:

• file_path (str or Path) – Path to the input file.

• sep (Optional[str]) – Column separator.

• name (str) – Name of the second column (used for Kaldi format).

Returns:

Loaded table.

Return type:

InfoTable

sort(column: str = 'id', ascending: bool = True, inplace: bool = True) → T | None

Sort the InfoTable by a specific column.

Parameters:

• column (str) – Column name to sort by.

• ascending (bool) – Sort in ascending order.

• inplace (bool) – Sort in place or return a new object.

Returns:

Sorted InfoTable or None if inplace.

Return type:

Optional[InfoTable]

split(idx: int, num_parts: int, group_by: str | None = None) → T

Split the InfoTable into parts and return the selected part.

Parameters:

• idx (int) – Part to return (1-based).

• num_parts (int) – Total number of parts.

• group_by (Optional[str]) – Column to group by when splitting.

Returns:

The selected part of the split.

Return type:

InfoTable

classmethod cat(tables: List[T]) → T

Concatenate multiple InfoTables.

Parameters:

tables (List[InfoTable]) – List of InfoTable objects to concatenate.

Returns:

Concatenated InfoTable.

Return type:

InfoTable

Raises:

AssertionError – If resulting DataFrame has duplicate IDs.

filter(predicate: Callable[[DataFrame], Series | ndarray] | str | None = None, items: List[Any] | None = None, iindex: ndarray | None = None, columns: List[str] | None = None, by: str = 'id', keep: bool = True, raise_if_missing: bool = True) → T

Filter the InfoTable based on a predicate, item list, index list, or column subset.

Parameters:

• predicate (Callable, optional) – Function that returns a boolean mask, e.g.: lambda df: df[“duration”] > 1.0.

• items (List[Any], optional) – Items to include/exclude from the ‘by’ column like df.loc[items, by], used only if predicate is None

• iindex (np.ndarray, optional) – Integer indices to include/exclude like df.iloc[iindex], used if predicate and items are None

• columns (List[str], optional) – Columns to retain or remove.

• by (str) – Column name to use with ‘items’.

• keep (bool) – Whether to keep or exclude matched rows/columns.

• raise_if_missing (bool) – Raise error if items are missing.

Returns:

Filtered InfoTable.

Return type:

InfoTable

Raises:

Exception – If items are not found and raise_if_missing is True.

histogram(column: str, bins: int | None = None, density: bool = True, kind: str = 'bar', color: str = 'C0', output_file: str | Path | None = None, dropna: bool = True) → None

Plot a histogram of a column, handling string or numeric data.

Parameters:

• column (str) – Column name to plot. Title will include the column name.

• bins (Optional[int]) – Number of bins for numeric columns. Ignored for strings.

• density (bool) – If True, plot density/relative frequency; otherwise counts.

• kind (str) – Histogram style, either “bar” or “line”.

• color (str) – Matplotlib color spec for the plot.

• output_file (Optional[PathLike]) – If provided, save the figure; otherwise show it.

• dropna (bool) – If True, ignore NA values before plotting.

scatter2d(x_column: str, y_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 2D scattergram for two numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

scatter3d(x_column: str, y_column: str, z_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 3D scattergram for three numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• z_column (str) – Column name for the z-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

__eq__(other: Any) → bool

Equal operator

__ne__(other: Any) → bool

Non-equal operator

__cmp__(other: Any) → int

Comparison operator

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffle the rows of the InfoTable.

Parameters:

• seed (int) – Seed for random number generator.

• rng (np.random.Generator, optional) – Numpy random generator.

Returns:

Shuffled indices.

Return type:

np.ndarray

set_index(keys: str | List[str], inplace: bool = True) → T | None

Set the DataFrame index using one or more columns.

Parameters:

• keys (str or list) – Column(s) to use as index.

• inplace (bool) – Whether to modify in place.

Returns:

Modified InfoTable if not inplace.

Return type:

Optional[InfoTable]

reset_index() → None

Reset the DataFrame index to the ‘id’ column.

Returns:

None

get_loc(keys: str | List[str] | ndarray) → int | ndarray | List[int]

Get integer location(s) for the given key(s).

Parameters:

keys (str, list, or np.ndarray) – Index label(s).

Returns:

Location(s) in the index.

Return type:

int, np.ndarray, or List[int]

get_col_idx(keys: str | List[str]) → int | ndarray

Get the integer index position(s) of the specified column(s).

Parameters:

keys (str or list) – Column name(s).

Returns:

Position(s) of the column(s).

Return type:

int or np.ndarray

__getattr__(name: str) → Any

Provide attribute-style access to DataFrame columns and attributes.

Falls back to self.df[name] if name is a column. Falls back to getattr(self.df, name) if it’s a DataFrame method or attribute. Special methods (e.g., __setstate__) raise AttributeError immediately to allow correct behavior during pickling/unpickling.

Parameters:

name (str) – The attribute being accessed.

Returns:

The corresponding column, method, or attribute.

Return type:

Any

Raises:

AttributeError – If the attribute is not found.

add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Add new columns from another InfoTable or DataFrame.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – The table to merge columns from.

• column_names (str or list, optional) – Columns to include from the right table.

• on (str or list) – Key(s) from the current table.

• right_on (str or list, optional) – Key(s) from the right table.

• replace_overlapping (bool) – Replace overlapping columns if True.

• ignore_overlapping (bool) – If True, skip adding columns that already exist.

• remove_missing (bool) – Use inner join (drop unmatched rows) if True.

legacy_add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Legacy implementation of add_columns that uses a full pandas merge.

This keeps the original behavior for compatibility/testing.

replace_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None) → None

Replace column values with those from another table.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – Table to source values from.

• column_names (str or list, optional) – Columns to replace. If None, all.

harmonize_columns_by_majority_vote(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the majority value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the voting group (e.g., speaker).

• target_columns (Union[str, List[str]]) – Column or columns to harmonize via majority vote.

Raises:

KeyError – If the voter or target columns are missing.

harmonize_columns_by_average(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the average value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the group used for averaging.

• target_columns (Union[str, List[str]]) – Numeric column(s) to harmonize via averaging.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If any target column is not numeric.

harmonize_age_given_decade(voter_columns: str | List[str], target_column: str, decade_column: str = 'age_decade') → None

Force all rows sharing voter_columns to take the averaged age in target_column, constrained by the decade labels in decade_column.

If the averaged age falls outside the decade bounds, the age is set to the upper boundary for that decade.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric age column to harmonize.

• decade_column (str) – Column indicating the age decade labels.

Raises:

• KeyError – If the voter, target, or decade columns are missing.

• TypeError – If the target column is not numeric.

harmonize_column_by_majority_cluster(voter_columns: str | List[str], target_column: str, suspect_column: str = 'suspect', std_threshold: float | None = None, max_iter: int = 20) → None

Harmonize a numeric column using the dominant cluster within each voter group, while flagging rows from the smaller cluster as suspect.

If std_threshold is provided, clustering is only performed when the group’s standard deviation exceeds the threshold.

The standard deviation of the dominant cluster is stored in a column named f"{target_column}_std" for the non-suspect rows.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric column to harmonize.

• suspect_column (str) – Column used to flag suspect rows.

• std_threshold (Optional[float]) – If set, only split groups above this std.

• max_iter (int) – Maximum iterations for the 1D two-means refinement.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If the target column is not numeric.

classmethod merge(left_table: T | DataFrame, right_table: T | DataFrame, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Tuple[str, str] = ('_x', '_y'), copy: bool | None = None, indicator: str | bool = False, validate: str | None = None) → T

Merge two InfoTables or DataFrames into a new InfoTable.

Parameters:

• left_table (Union[InfoTable, pd.DataFrame]) – Left-hand table.

• right_table (Union[InfoTable, pd.DataFrame]) – Right-hand table.

• how (str) – Merge method (e.g., ‘inner’, ‘outer’, ‘left’, ‘right’).

• on (Union[str, List[str], None]) – Column(s) to join on.

• left_on (Union[str, List[str], None]) – Column(s) from the left table to join on.

• right_on (Union[str, List[str], None]) – Column(s) from the right table to join on.

• left_index (bool) – Use index from the left table as join key.

• right_index (bool) – Use index from the right table as join key.

• sort (bool) – Sort the result by the join keys.

• suffixes (Tuple[str, str]) – Suffixes to apply to overlapping column names.

• copy (Optional[bool]) – If False, avoid copying data where possible.

• indicator (Union[str, bool]) – Adds a column to the output DataFrame called ‘_merge’.

• validate (Optional[str]) – Check if merge is of specified type.

Returns:

A new merged InfoTable.

Return type:

InfoTable

replace(to_replace: Any = None, value: Any = None, inplace: bool = False, **kwargs: Any) → T | None

Replace values in the DataFrame.

Parameters:

• to_replace – What to replace.

• value – Value to replace with.

• inplace (bool) – Whether to modify the table in-place.

• **kwargs – Additional keyword args passed to pd.DataFrame.replace().

Returns:

New InfoTable if not inplace, else None.

Return type:

Optional[InfoTable]

class hyperion.utils.HyperDataset(segments: SegmentSet | str | Path, classes: Dict[str, ClassInfo | str | Path] | None = None, recordings: RecordingSet | str | Path | None = None, images: ImageSet | str | Path | None = None, videos: VideoSet | str | Path | None = None, features: Dict[str, FeatureSet | str | Path] | None = None, vads: Dict[str, VADSet | str | Path] | None = None, diarizations: Dict[str, DiarizationSet | str | Path] | None = None, enrollments: Dict[str, EnrollmentMap | str | Path] | None = None, trials: Dict[str, TrialKey | TrialNdx | SparseTrialKey | str | Path] | None = None, sparse_trials: bool = False, table_sep: str | None = None, trials_sep: str | None = None)

Container that groups segments with their related resources.

The dataset keeps references to tables (segments, recordings, features, etc.) either as in-memory objects or filesystem paths. Paths are loaded lazily when an accessor is called so large datasets can be defined without reading everything into memory upfront.

segments

SegmentSet object or path; this is the only required table.

classes

Mapping from class name to ClassInfo object or path.

recordings

RecordingSet object or path aligned with segment recording.

images

ImageSet object or path aligned with segment image.

videos

VideoSet object or path aligned with segment video.

features

Mapping from feature name to FeatureSet object or path.

vads

Mapping from VAD name to VADSet object or path.

diarizations

Mapping from diarization name to DiarizationSet object or path.

enrollments

Mapping from enrollment name to EnrollmentMap object or path.

trials

Mapping from trial name to TrialKey/TrialNdx/SparseTrialKey object or path.

sparse_trials

If True, load trials using SparseTrialKey to save memory.

table_sep

Default column separator when reading/writing tables.

trials_sep

Separator for trial manifests; falls back to table_sep when None.

Examples

>>> import pandas as pd
>>> from hyperion.utils import HyperDataset, RecordingSet, SegmentSet
>>> segments = SegmentSet(
...     pd.DataFrame(
...         {
...             "id": ["utt1", "utt2", "utt3"],
...             "recording": ["rec1", "rec1", "rec2"],
...             "speaker": ["spk1", "spk1", "spk2"],
...             "duration": [1.2, 0.9, 2.4],
...         }
...     )
... )
>>> recordings = RecordingSet(
...     pd.DataFrame(
...         {
...             "id": ["rec1", "rec2"],
...             "storage_path": ["/data/audio/rec1.wav", "/data/audio/rec2.wav"],
...         }
...     )
... )
>>> dataset = HyperDataset(segments=segments, recordings=recordings)
>>> dataset.add_classes_from_segments("speaker")
>>> len(dataset), sorted(dataset.classes_keys())
(3, ['speaker'])
>>> dataset.save("tmp/my_dataset", force_save_all=True)
>>> ds2 = HyperDataset.load("tmp/my_dataset", lazy=True)
>>> len(ds2.segments())
3

__init__(segments: SegmentSet | str | Path, classes: Dict[str, ClassInfo | str | Path] | None = None, recordings: RecordingSet | str | Path | None = None, images: ImageSet | str | Path | None = None, videos: VideoSet | str | Path | None = None, features: Dict[str, FeatureSet | str | Path] | None = None, vads: Dict[str, VADSet | str | Path] | None = None, diarizations: Dict[str, DiarizationSet | str | Path] | None = None, enrollments: Dict[str, EnrollmentMap | str | Path] | None = None, trials: Dict[str, TrialKey | TrialNdx | SparseTrialKey | str | Path] | None = None, sparse_trials: bool = False, table_sep: str | None = None, trials_sep: str | None = None) → None

Initialize the dataset wrapper and optionally register auxiliary tables.

Parameters:

• segments – SegmentSet instance or path to the segments table; required anchor for the dataset.

• classes – Optional mapping from class name to ClassInfo object or path containing class labels.

• recordings – Optional RecordingSet object or path aligned with the recording column.

• images – Optional ImageSet object or path aligned with the image column.

• videos – Optional VideoSet object or path aligned with the video column.

• features – Optional mapping from feature name to FeatureSet object or path.

• vads – Optional mapping from VAD name to VADSet object or path.

• diarizations – Optional mapping from diarization name to DiarizationSet object or path.

• enrollments – Optional mapping from enrollment name to EnrollmentMap object or path.

• trials – Optional mapping from trial name to TrialKey/TrialNdx/SparseTrialKey object or path.

• sparse_trials – If True, load trial files using SparseTrialKey for memory efficiency.

• table_sep – Column separator used when reading or writing tabular manifests.

• trials_sep – Optional separator for trial manifests; defaults to table_sep when None.

fix_segments_dtypes() → None

Ensure any class columns in the segments table are stored as strings.

_fix_segments_dtypes(segments: SegmentSet) → None

Convert class columns in a SegmentSet to string dtype.

Parameters:

segments – SegmentSet whose columns will be adjusted in-place.

describe() → Dict[str, int | float | str]

Summarize dataset counts and duration, logging a human-readable message.

Returns:

Counts per component plus a msg field.

Return type:

Dict[str, Union[int, float, str]]

get_dataset_files() → List[Path]

Collect all manifest file paths referenced by the dataset.

Returns:

Paths for segments, recordings/videos, and auxiliary manifests.

Return type:

List[Path]

_delete_files(dataset_dir: str | Path) → None

Delete files queued for removal if they are not part of the saved dataset.

Parameters:

dataset_dir – Target directory where dataset manifests are stored.

_parse_dict_args(data: Dict[str, ClassInfo | RecordingSet | ImageSet | VideoSet | FeatureSet | VADSet | DiarizationSet | EnrollmentMap | TrialKey | TrialNdx | SparseTrialKey | str | Path] | None, types: type | Tuple[type, ...]) → Tuple[Dict[str, ClassInfo | RecordingSet | ImageSet | VideoSet | FeatureSet | VADSet | DiarizationSet | EnrollmentMap | TrialKey | TrialNdx | SparseTrialKey | None] | None, Dict[str, Path | None] | None]

Split a mapping into separate object and path dictionaries.

Parameters:

• data – Mapping whose values are either instances of types or paths.

• types – Class or tuple of classes expected for in-memory objects.

Returns:

Objects and paths keyed by name.

Return type:

Tuple[Optional[Dict[str, object]], Optional[Dict[str, Path]]]

clone() → HyperDataset

Return a deep copy of the dataset.

segments(keep_loaded: bool = True) → SegmentSet

Access the segments table, loading from disk if needed.

Parameters:

keep_loaded – If True, cache the loaded SegmentSet on the instance.

Returns:

Segment metadata for the dataset.

Return type:

SegmentSet

__len__() → int

Number of segments in the dataset.

property has_recordings: bool

Whether a recordings manifest is available (loaded or path).

property has_images: bool

Whether an images manifest is available (loaded or path).

property has_videos: bool

Whether a videos manifest is available (loaded or path).

recordings(keep_loaded: bool = True) → RecordingSet

Access the recordings table, loading from disk if needed.

Parameters:

keep_loaded – If True, cache the loaded RecordingSet.

Returns:

Recording metadata aligned with segments.

Return type:

RecordingSet

images(keep_loaded: bool = True) → ImageSet

Access the images table, loading from disk if needed.

Parameters:

keep_loaded – If True, cache the loaded ImageSet.

Returns:

Image metadata aligned with segments.

Return type:

ImageSet

videos(keep_loaded: bool = True) → VideoSet

Access the videos table, loading from disk if needed.

Parameters:

keep_loaded – If True, cache the loaded VideoSet.

Returns:

Video metadata aligned with segments.

Return type:

VideoSet

features_keys() → Iterable[str]

Return names of feature sets present in the dataset.

features_value(key: str, keep_loaded: bool = True) → FeatureSet

Access a feature manifest by name.

Parameters:

• key – Name of the feature set.

• keep_loaded – If True, cache the loaded FeatureSet.

Returns:

Feature manifest referenced by key.

Return type:

FeatureSet

vads_keys() → Iterable[str]

Return names of VAD sets present in the dataset.

vads_value(key: str, keep_loaded: bool = True) → VADSet

Access a VAD manifest by name.

Parameters:

• key – Name of the VAD set.

• keep_loaded – If True, cache the loaded VADSet.

Returns:

VAD manifest referenced by key.

Return type:

VADSet

diarizations_keys() → Iterable[str]

Return names of diarization sets present in the dataset.

diarizations_value(key: str, keep_loaded: bool = True) → DiarizationSet

Access a diarization manifest by name.

Parameters:

• key – Name of the diarization set.

• keep_loaded – If True, cache the loaded DiarizationSet.

Returns:

Diarization manifest referenced by key.

Return type:

DiarizationSet

classes_keys() → Iterable[str]

Return names of class info tables present in the dataset.

classes_value(key: str, keep_loaded: bool = True) → ClassInfo

Access a class info manifest by name.

Parameters:

• key – Name of the class info table.

• keep_loaded – If True, cache the loaded ClassInfo.

Returns:

Class metadata referenced by key.

Return type:

ClassInfo

enrollments_value(key: str, keep_loaded: bool = True) → EnrollmentMap

Access an enrollment map by name.

Parameters:

• key – Name of the enrollment map.

• keep_loaded – If True, cache the loaded EnrollmentMap.

Returns:

Enrollment manifest referenced by key.

Return type:

EnrollmentMap

trials_value(key: str, keep_loaded: bool = True) → TrialKey | TrialNdx | SparseTrialKey

Access a trials object by name, loading lazily from disk.

Parameters:

• key – Name of the trials entry.

• keep_loaded – If True, cache the loaded trials structure.

Returns:

Trials data referenced by key.

Return type:

Union[TrialKey, TrialNdx, SparseTrialKey]

features(keep_loaded: bool = True) → Iterator[Tuple[str, FeatureSet]]

Iterate over all feature sets, loading lazily if necessary.

Parameters:

keep_loaded – If True, cache each loaded FeatureSet.

Yields:

Tuple[str, FeatureSet] – Feature name and manifest.

vads(keep_loaded: bool = True) → Iterator[Tuple[str, VADSet]]

Iterate over all VAD sets, loading lazily if necessary.

Parameters:

keep_loaded – If True, cache each loaded VADSet.

Yields:

Tuple[str, VADSet] – VAD name and manifest.

diarizations(keep_loaded: bool = True) → Iterator[Tuple[str, DiarizationSet]]

Iterate over all diarization sets, loading lazily if necessary.

Parameters:

keep_loaded – If True, cache each loaded DiarizationSet.

Yields:

Tuple[str, DiarizationSet] – Diarization name and manifest.

classes(keep_loaded: bool = True) → Iterator[Tuple[str, ClassInfo]]

Iterate over all class info tables, loading lazily if necessary.

Parameters:

keep_loaded – If True, cache each loaded ClassInfo.

Yields:

Tuple[str, ClassInfo] – Class name and table.

enrollments(keep_loaded: bool = True) → Iterator[Tuple[str, EnrollmentMap]]

Iterate over all enrollment maps, loading lazily if necessary.

Parameters:

keep_loaded – If True, cache each loaded EnrollmentMap.

Yields:

Tuple[str, EnrollmentMap] – Enrollment name and map.

trials(keep_loaded: bool = True) → Iterator[Tuple[str, TrialKey | TrialNdx | SparseTrialKey]]

Iterate over all trials, loading lazily if necessary.

Parameters:

keep_loaded – If True, cache each loaded trials object.

Yields:

Tuple[str, Union[TrialKey, TrialNdx, SparseTrialKey]] – Trial name and data.

static resolve_dataset_path(dataset_path: str | Path) → Tuple[Path, Path]

Normalize a dataset path to directory and YAML manifest.

Parameters:

dataset_path – Path to a dataset directory or dataset YAML file.

Returns:

Dataset directory and dataset YAML file path.

Return type:

Tuple[Path, Path]

static resolve_file_path(dataset_dir: str | Path, file_path: str | Path) → Path

Resolve a manifest path relative to the dataset directory.

Parameters:

• dataset_dir – Base directory for the dataset.

• file_path – Absolute or relative manifest path.

Returns:

Resolved file path.

Return type:

Path

save(dataset_path: str | Path, update_paths: bool = True, table_sep: str | None = None, force_save_all: bool = False) → None

Persist the dataset manifests to disk.

Parameters:

• dataset_path – Directory to hold manifests or path to a dataset YAML file.

• update_paths – Whether to update internal file paths after saving.

• table_sep – Separator to use when writing tabular files (overrides instance default).

• force_save_all – If True, save every table; otherwise only save loaded/changed files. Trials use trials_sep when provided.

Returns:

None

save_changed(dataset_path: str | Path, update_paths: bool = True, table_sep: str | None = None, trials_sep: str | None = None) → None

Save only manifests that changed or are missing in the target directory.

Parameters:

• dataset_path – Directory to hold manifests or path to a dataset YAML file.

• update_paths – Whether to update internal file paths after saving.

• table_sep – Separator to use when writing tabular files (overrides instance default).

• trials_sep – Separator to use when writing trial files (overrides instance default).

Returns:

None

save_all(dataset_path: str | Path, update_paths: bool = True, table_sep: str | None = None, trials_sep: str | None = None) → None

Save every manifest to disk, regardless of change tracking.

Parameters:

• dataset_path – Directory to hold manifests or path to a dataset YAML file.

• update_paths – Whether to update internal file paths after saving.

• table_sep – Separator to use when writing tabular files (overrides instance default).

• trials_sep – Separator to use when writing trial files (overrides instance default).

Returns:

None

update_from_disk() → None

Eagerly load every registered manifest into memory.

classmethod load(dataset_path: str | Path, lazy: bool = True, sparse_trials: bool = False) → HyperDataset

Instantiate a dataset from a manifest directory or YAML file.

Parameters:

• dataset_path – Directory containing manifests or a dataset YAML file.

• lazy – If True, defer loading manifests until accessed.

• sparse_trials – If True, load trial files as SparseTrialKey.

Returns:

Dataset pointing to the referenced manifests.

Return type:

HyperDataset

set_segments(segments: str | Path | SegmentSet) → None

Replace the segments table reference.

Parameters:

segments – SegmentSet instance or path to a segments manifest.

Returns:

None

set_recordings(recordings: str | Path | RecordingSet, update_seg_durs: bool = False) → None

Attach a recordings manifest to the dataset.

Parameters:

• recordings – RecordingSet instance or path to a recordings manifest.

• update_seg_durs – If True, populate segment durations from recordings.

Returns:

None

set_images(images: str | Path | ImageSet) → None

Attach an images manifest to the dataset.

Parameters:

images – ImageSet instance or path to an images manifest.

Returns:

None

set_videos(videos: str | Path | VideoSet, update_seg_durs: bool = False) → None

Attach a videos manifest to the dataset.

Parameters:

• videos – VideoSet instance or path to a videos manifest.

• update_seg_durs – If True, populate segment durations from videos.

Returns:

None

add_features(features_name: str, features: str | Path | FeatureSet) → None

Register a feature manifest under a given name.

Parameters:

• features_name – Identifier for the feature set.

• features – FeatureSet instance or path to a features manifest.

Returns:

None

add_vads(vads_name: str, vads: str | Path | VADSet) → None

Register a VAD manifest under a given name.

Parameters:

• vads_name – Identifier for the VAD set.

• vads – VADSet instance or path to a VAD manifest.

Returns:

None

add_diarizations(diarizations_name: str, diarizations: str | Path | DiarizationSet) → None

Register a diarization manifest under a given name.

Parameters:

• diarizations_name – Identifier for the diarization set.

• diarizations – DiarizationSet instance or path to a diarization manifest.

Returns:

None

add_classes(classes_name: str, classes: str | Path | ClassInfo) → None

Register a class info table under a given name.

Parameters:

• classes_name – Identifier for the class table.

• classes – ClassInfo instance or path to a class manifest.

Returns:

None

add_enrollments(enrollments_name: str, enrollments: str | Path | EnrollmentMap) → None

Register an enrollment map under a given name.

Parameters:

• enrollments_name – Identifier for the enrollment map.

• enrollments – EnrollmentMap instance or path to an enrollment manifest.

Returns:

None

add_trials(trials_name: str, trials: str | Path | TrialKey | TrialNdx | SparseTrialKey) → None

Register a trials object under a given name.

Parameters:

• trials_name – Identifier for the trials entry.

• trials – TrialKey, TrialNdx, SparseTrialKey instance or path to a trials file.

Returns:

None

remove_recordings() → None

Detach recordings and mark backing file for deletion if present.

remove_images() → None

Detach images and mark backing file for deletion if present.

remove_videos() → None

Detach videos and mark backing file for deletion if present.

remove_features(features_name: str | None = None) → None

Remove one feature set or all feature sets.

Parameters:

features_name – Identifier of the feature set to remove. If None, remove every feature set.

Returns:

None

remove_vads(vads_name: str | None = None) → None

Remove one VAD set or all VAD sets.

Parameters:

vads_name – Identifier of the VAD set to remove. If None, remove every VAD set.

Returns:

None

remove_diarizations(diarizations_name: str | None = None) → None

Remove one diarization set or all diarization sets.

Parameters:

diarizations_name – Identifier of the diarization set to remove. If None, remove every diarization set.

Returns:

None

remove_classes(classes_name: str | None = None) → None

Remove one class info table or all class info tables.

Parameters:

classes_name – Identifier of the class info table to remove. If None, remove every class info table.

Returns:

None

remove_enrollments(enrollments_name: str | None = None) → None

Remove one enrollment map or all enrollment maps.

Parameters:

enrollments_name – Identifier of the enrollment map to remove. If None, remove every enrollment map.

Returns:

None

remove_trials(trials_name: str | None = None) → None

Remove one trials entry or all trials entries.

Parameters:

trials_name – Identifier of the trials entry to remove. If None, remove every trials entry.

Returns:

None

add_cols_to_segments(right_table: InfoTable | DataFrame | str | Path, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, remove_missing: bool = False, create_class_info: bool = False) → None

Join additional columns into the segments table.

Parameters:

• right_table – InfoTable/DataFrame or path, or a string key referring to a registered manifest.

• column_names – Columns to add; defaults to all columns.

• on – Column(s) in segments used for the join.

• right_on – Column(s) in the right table used for the join.

• remove_missing – If True, drop segments with missing join keys.

• create_class_info – If True, build ClassInfo tables for newly added columns.

Returns:

None

clean(rebuild_class_idx: bool = False) → None

Drop orphaned entries across manifests based on current segments.

Parameters:

rebuild_class_idx – If True, rebuild integer class indices after filtering.

Returns:

None

_split_into_trials_and_cohort(segments: SegmentSet, num_tar_trials: int, num_trial_speakers: int, seed: int) → Tuple[TrialKey, EnrollmentMap, SegmentSet]

Create a trials list and cohort split from a subset of segments.

Parameters:

• segments – SegmentSet to sample from.

• num_tar_trials – Number of target trials to generate.

• num_trial_speakers – Number of speakers to include in trials.

• seed – Random seed for reproducibility.

Returns:

Trials, enrollments, and cohort segments.

Return type:

Tuple[TrialKey, EnrollmentMap, SegmentSet]

split_into_trials_and_cohort(num_1k_tar_trials: int, num_trial_speakers: int, intra_gender: bool = True, trials_name: str = 'trials_qmf', seed: int = 1123) → Tuple[HyperDataset, HyperDataset]

Split dataset into a trial subset and a cohort subset for QMF training.

Parameters:

• num_1k_tar_trials – Target trials expressed in thousands (e.g., 10 -> 10k trials).

• num_trial_speakers – Number of speakers to use for trials.

• intra_gender – If True, build trials separately within each gender.

• trials_name – Name used to store trials in the returned dataset.

• seed – Random seed for reproducibility.

Returns:

Dataset with trials/enrollments and dataset with cohort only.

Return type:

Tuple[HyperDataset, HyperDataset]

remove_short_segments(min_length: float, length_name: str = 'duration') → None

Remove segments shorter than a given length.

Parameters:

• min_length – Minimum allowed duration.

• length_name – Column to compare against min_length.

Returns:

None

remove_classes_few_segments(class_name: str, min_segs: int, rebuild_idx: bool = False) → None

Drop classes with fewer than min_segs segments.

Parameters:

• class_name – Column name representing the class label.

• min_segs – Minimum number of segments required to keep a class.

• rebuild_idx – If True, rebuild class indices after filtering.

Returns:

None

remove_classes_few_toomany_segments(class_name: str, min_segs: int, max_segs: int | None, rebuild_idx: bool = False) → None

Drop classes with too few or too many segments.

Parameters:

• class_name – Column name representing the class label.

• min_segs – Minimum number of segments required to keep a class.

• max_segs – Maximum number of segments allowed to keep a class; None to ignore.

• rebuild_idx – If True, rebuild class indices after filtering.

Returns:

None

remove_class_ids(class_name: str, class_ids: List[str], remove_na: bool, rebuild_idx: bool = False) → None

Remove specific class ids (and optionally NaNs) from the dataset.

Parameters:

• class_name – Column name representing the class label.

• class_ids – List of class identifiers to drop.

• remove_na – If True, drop rows with missing class labels.

• rebuild_idx – If True, rebuild class indices after filtering.

Returns:

None

filter_by_segments(segments: SegmentSet | List[str], rebuild_class_idx: bool = False, keep: bool = True) → None

Filter dataset by a list of segment ids or a SegmentSet.

Parameters:

• segments – Segment ids or SegmentSet to define the filter.

• rebuild_class_idx – If True, rebuild class indices after filtering.

• keep – If True, keep the provided ids; otherwise drop them.

Returns:

None

filter_by_segments_predicate(predicate: str, rebuild_class_idx: bool = False, keep: bool = True) → None

Filter dataset by an expression evaluated on the segments table.

Parameters:

• predicate – Query string passed to SegmentSet.filter.

• rebuild_class_idx – If True, rebuild class indices after filtering.

• keep – If True, keep rows matching predicate; otherwise drop them.

Returns:

None

filter_by_classes(class_name: str, classes: ClassInfo | List[str], remove_na: bool, rebuild_idx: bool = False, keep: bool = True) → None

Filter dataset by class membership.

Parameters:

• class_name – Column name representing the class label.

• classes – ClassInfo object or list of class ids to keep/drop.

• remove_na – If True, drop rows with missing class labels.

• rebuild_idx – If True, rebuild class indices after filtering.

• keep – If True, retain matching classes; otherwise drop them.

Returns:

None

filter_by_classes_and_enrollments(class_name: str, classes: ClassInfo | List[str], enrollment_name: str, enrollments: EnrollmentMap, remove_na: bool, rebuild_idx: bool = False, keep: bool = True) → None

Filter dataset by class membership and enrollment ids, updating trials too.

Parameters:

• class_name – Column name representing the class label.

• classes – ClassInfo object or list of class ids to keep/drop.

• enrollment_name – Enrollment map key to filter.

• enrollments – Enrollment map providing ids to retain/drop.

• remove_na – If True, drop rows with missing class labels.

• rebuild_idx – If True, rebuild class indices after filtering.

• keep – If True, retain matching classes/enrollments; otherwise drop them.

Returns:

None

rebuild_class_idx(class_name: str) → None

Recompute integer class indices for a given class info table.

Parameters:

class_name – Name of the class info table.

Returns:

None

_segments_split(val_prob: float, rng: Generator) → Tuple[SegmentSet, SegmentSet]

Randomly split segments into train/validation folds.

Parameters:

• val_prob – Fraction of segments to place in validation.

• rng – Random generator to use for permutation.

Returns:

Training and validation segments.

Return type:

Tuple[SegmentSet, SegmentSet]

_segments_split_joint_classes(val_prob: float, joint_classes: List[str], min_train_samples: int, rng: Generator) → Tuple[SegmentSet, SegmentSet]

Split ensuring each joint class combination appears in both splits.

Parameters:

• val_prob – Fraction of samples per class to place in validation.

• joint_classes – Columns defining joint class membership.

• min_train_samples – Minimum training samples per joint class.

• rng – Random generator to use for permutation.

Returns:

Training and validation segments.

Return type:

Tuple[SegmentSet, SegmentSet]

_segments_split_disjoint_classes(val_prob: float, disjoint_classes: List[str], rng: Generator) → Tuple[SegmentSet, SegmentSet]

Split ensuring disjoint sets of classes between train and validation.

Parameters:

• val_prob – Fraction of segments to place in validation.

• disjoint_classes – Columns defining mutually exclusive classes.

• rng – Random generator to use for permutation.

Returns:

Training and validation segments.

Return type:

Tuple[SegmentSet, SegmentSet]

_segments_split_joint_and_disjoint_classes(val_prob: float, joint_classes: List[str], disjoint_clases: List[str], min_train_samples: int, rng: Generator) → Tuple[SegmentSet, SegmentSet]

Placeholder for joint/disjoint class split logic.

split_train_val(val_prob: float, joint_classes: List[str] | None = None, disjoint_classes: List[str] | None = None, min_train_samples: int = 1, seed: int = 11235813) → Tuple[HyperDataset, HyperDataset]

Create train/validation dataset splits with optional class constraints.

Parameters:

• val_prob – Fraction of segments to place in validation.

• joint_classes – Columns that must appear in both splits.

• disjoint_classes – Columns that must not overlap between splits.

• min_train_samples – Minimum samples per joint class when joint_classes is used.

• seed – Random seed for reproducibility.

Returns:

Train and validation datasets.

Return type:

Tuple[HyperDataset, HyperDataset]

_segments_split_folds(num_folds: int, rng: Generator) → Tuple[List[SegmentSet], List[SegmentSet]]

Randomly split segments into num_folds folds.

Parameters:

• num_folds – Number of folds to create.

• rng – Random generator to use for permutation.

Returns:

Training and test folds.

Return type:

Tuple[List[SegmentSet], List[SegmentSet]]

_segments_split_folds_joint_classes(num_folds: int, joint_classes: List[str], rng: Generator) → Tuple[List[SegmentSet], List[SegmentSet]]

Create folds while keeping each joint class combination in every fold.

Parameters:

• num_folds – Number of folds to create.

• joint_classes – Columns defining joint class membership.

• rng – Random generator to use for permutation.

Returns:

Training and test folds.

Return type:

Tuple[List[SegmentSet], List[SegmentSet]]

_segments_split_folds_disjoint_classes(num_folds: float, disjoint_classes: List[str], rng: Generator) → Tuple[List[SegmentSet], List[SegmentSet]]

Create folds such that class groups are disjoint across folds.

Parameters:

• num_folds – Number of folds to create.

• disjoint_classes – Columns defining mutually exclusive classes.

• rng – Random generator to use for permutation.

Returns:

Training and test folds.

Return type:

Tuple[List[SegmentSet], List[SegmentSet]]

_segments_split_folds_joint_and_disjoint_classes(num_folds: int, joint_classes: List[str], disjoint_classes: List[str], rng: Generator) → Tuple[List[SegmentSet], List[SegmentSet]]

Create folds balancing both joint and disjoint class constraints.

split_folds(num_folds: int, joint_classes: List[str] | None = None, disjoint_classes: List[str] | None = None, seed: int = 11235813) → Tuple[List[HyperDataset], List[HyperDataset]]

Create cross-validation folds with optional class constraints.

Parameters:

• num_folds – Number of folds to create.

• joint_classes – Columns that must appear across all folds.

• disjoint_classes – Columns that must be disjoint across folds.

• seed – Random seed for reproducibility.

Returns:

Training and test datasets per fold.

Return type:

Tuple[List[HyperDataset], List[HyperDataset]]

classmethod merge(datasets: List[HyperDataset]) → HyperDataset

Concatenate multiple HyperDataset objects into one.

Parameters:

datasets – Iterable of HyperDataset instances to merge.

Returns:

New dataset containing concatenated manifests where possible.

Return type:

HyperDataset

add_classes_from_segments(class_names: str | List[str] | ndarray) → None

Build ClassInfo objects from columns already present in segments.

Parameters:

class_names – Column name or list/array of column names to convert to ClassInfo.

Returns:

None

classmethod from_recordings(recordings: RecordingSet | str | Path) → HyperDataset

Create a dataset from recordings when no segmentation exists.

Parameters:

recordings – RecordingSet object or path to a RecordingSet manifest.

Returns:

Dataset whose segments mirror the recordings table.

Return type:

HyperDataset

classmethod from_segments(segments: SegmentSet | str | Path, recordings: RecordingSet | str | Path | None = None, class_names: List[str] | None = None) → HyperDataset

Create a dataset from a SegmentSet with optional recordings and classes.

Parameters:

• segments – SegmentSet object or path to a segments manifest.

• recordings – Optional RecordingSet object or path.

• class_names – Optional class columns to convert into ClassInfo tables.

Returns:

Dataset built from the provided manifests.

Return type:

HyperDataset

classmethod from_lhotse(cuts: lhotse.CutSet | str | Path | None = None, recordings: lhotse.RecordingSet | str | Path | None = None, supervisions: lhotse.SupervisionSet | str | Path | None = None) → HyperDataset

Create a dataset from Lhotse cuts or from recordings + supervisions.

Parameters:

• cuts – Lhotse CutSet object or path to a CutSet manifest.

• recordings – Optional Lhotse RecordingSet object or path.

• supervisions – Optional Lhotse SupervisionSet object or path.

Returns:

Dataset derived from the Lhotse manifests.

Return type:

HyperDataset

classmethod from_kaldi(kaldi_data_dir: str | Path) → HyperDataset

Create a dataset from a Kaldi-style data directory.

Parameters:

kaldi_data_dir – Path to a Kaldi data directory.

Returns:

Dataset populated from Kaldi manifests.

Return type:

HyperDataset

append_seg_suffix(seg_suffix: str) → None

Append a suffix to all segment ids (and aligned manifest ids).

cat_segments(group_by: str | List[str], max_duration: float | None = None, inplace: bool = False) → HyperDataset

Concatenate segments within groups and rebuild recordings with a sox pipe.

Parameters:

• group_by – Column name or list of columns to define concatenation groups.

• max_duration – Maximum duration in seconds for each concatenated segment. When exceeded, a new concatenated segment is started.

• inplace – If True, modify the dataset in place; otherwise return a clone.

Returns:

Dataset with concatenated segments and recordings.

Return type:

HyperDataset

sample_random_subsegments(subsegments_per_segment: int = 1, min_duration: float = 0.0, max_duration: float | None = None, seg_suffix: str | None = None, random_start: bool = True, seed: int = 11235813, rng: Generator | None = None, inplace: bool = True) → HyperDataset

Sample random subsegments for each segment and optionally apply to the dataset.

Parameters:

• subsegments_per_segment – Number of subsegments to draw from each original segment.

• min_duration – Minimum duration of sampled subsegments.

• max_duration – Maximum duration of sampled subsegments; None for full length.

• seg_suffix – Optional suffix to append to new segment ids.

• random_start – If True, choose random start within the segment.

• seed – Seed for the internal random generator if rng is not provided.

• rng – Optional numpy Generator to control randomness.

• inplace – If True, modify current dataset; otherwise return a cloned dataset.

Returns:

Dataset containing the sampled subsegments.

Return type:

HyperDataset

Domain-specific table sets built on top of InfoTable:

class hyperion.utils.SegmentSet(df: DataFrame | SegmentSet)

Store metadata for speech segments.

The table uses id as segment identifier and may include columns such as recording, start, duration, image, and video.

Examples

>>> import pandas as pd
>>> from hyperion.utils.segment_set import SegmentSet
>>> df = pd.DataFrame({"id": ["seg1"], "recording": ["rec1"], "duration": [1.2]})
>>> segs = SegmentSet(df)
>>> segs.has_time_marks
True
>>> segs.recording(["seg1"]).tolist()
['rec1']
>>> marks = segs.recording_time_marks(["seg1"])
>>> list(marks.columns)
['recording', 'start', 'duration']

__init__(df: DataFrame | SegmentSet) → None

Initialize a segment set and normalize basic timing columns.

Parameters:

df (pd.DataFrame or SegmentSet) – Input segment table.

property has_time_marks: bool

Whether recording/time-mark columns are present.

Returns:

True when recording, start, and duration exist.

Return type:

bool

property has_recording_ids: bool

Whether a recording column is present.

Returns:

True if recording exists.

Return type:

bool

property has_recording: bool

Alias for has_recording_ids().

Returns:

True if recording exists.

Return type:

bool

recording(ids: ndarray | List[str] | None = None) → Series | ndarray | List[str]

Get recording IDs for segments.

Parameters:

• ids (Union[np.ndarray, List[str], None]) – Segment IDs to query. If

• None

• series. (return the full recording)

Returns:

Recording IDs. Falls back to segment id when recording is missing.

Return type:

Union[pd.Series, np.ndarray, List[str]]

image(ids: ndarray | List[str] | None = None) → Series | ndarray | List[str]

Get image IDs associated with segments.

Parameters:

• ids (Union[np.ndarray, List[str], None]) – Segment IDs to query. If

• None

• series. (return the full image)

Returns:

Image IDs. Falls back to segment id when image is missing.

Return type:

Union[pd.Series, np.ndarray, List[str]]

video(ids: ndarray | List[str] | None = None) → Series | ndarray | List[str]

Get video IDs associated with segments.

Parameters:

• ids (Union[np.ndarray, List[str], None]) – Segment IDs to query. If

• None

• series. (return the full video)

Returns:

Video IDs.

Return type:

Union[pd.Series, np.ndarray, List[str]]

recording_ids(ids: ndarray | List[str] | None = None) → Series | ndarray | List[str]

Alias for recording().

Parameters:

ids (Union[np.ndarray, List[str], None]) – Segment IDs to query.

Returns:

Recording IDs.

Return type:

Union[pd.Series, np.ndarray, List[str]]

recording_time_marks(ids: ndarray | List[str]) → DataFrame

Return recording name, start time, and duration for selected segments.

Parameters:

ids (Union[np.ndarray, List[str]]) – Segment IDs to query.

Returns:

Columns [recording_or_id, start, duration].

Return type:

pd.DataFrame

sample_random_subsegments(subsegments_per_segment: int = 1, min_duration: float = 0.0, max_duration: float | None = None, seg_suffix: str | None = None, random_start: bool = True, seed: int = 11235813, rng: Generator | None = None) → SegmentSet

Sample random subsegments from each segment.

Parameters:

• subsegments_per_segment (int) – Number of subsegments to sample per row.

• min_duration (float) – Minimum sampled duration.

• max_duration (Optional[float]) – Maximum sampled duration. If None,

• bound. (each segment's original duration is used as upper)

• seg_suffix (Optional[str]) – Optional suffix for generated segment IDs.

• random_start (bool) – If True, sample random start offsets; otherwise

• 0.0. (use)

• seed (int) – RNG seed used when rng is None.

• rng (Optional[np.random.Generator]) – Optional external RNG.

Returns:

New table containing sampled subsegments.

Return type:

SegmentSet

__cmp__(other: Any) → int

Comparison operator

__contains__(key: Any) → bool

Check whether key is in the DataFrame columns.

Parameters:

key – Key to check.

Returns:

True if key is in columns.

Return type:

bool

__eq__(other: Any) → bool

Equal operator

__getattr__(name: str) → Any

Provide attribute-style access to DataFrame columns and attributes.

Falls back to self.df[name] if name is a column. Falls back to getattr(self.df, name) if it’s a DataFrame method or attribute. Special methods (e.g., __setstate__) raise AttributeError immediately to allow correct behavior during pickling/unpickling.

Parameters:

name (str) – The attribute being accessed.

Returns:

The corresponding column, method, or attribute.

Return type:

Any

Raises:

AttributeError – If the attribute is not found.

__getitem__(key: Any) → T | Series

Get item from the internal DataFrame.

Parameters:

key – Key used for indexing.

Returns:

Sub-table or Series.

Return type:

Union[InfoTable, pd.Series]

__ne__(other: Any) → bool

Non-equal operator

__setitem__(key: Any, value: Any) → None

Set item in the internal DataFrame.

Parameters:

• key – Column label or index.

• value – Value to assign.

add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Add new columns from another InfoTable or DataFrame.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – The table to merge columns from.

• column_names (str or list, optional) – Columns to include from the right table.

• on (str or list) – Key(s) from the current table.

• right_on (str or list, optional) – Key(s) from the right table.

• replace_overlapping (bool) – Replace overlapping columns if True.

• ignore_overlapping (bool) – If True, skip adding columns that already exist.

• remove_missing (bool) – Use inner join (drop unmatched rows) if True.

property at: _InfoTableAtIndexer

Access a single value for a row/column label pair.

Returns:

Indexer that wraps .at.

Return type:

_InfoTableAtIndexer

classmethod cat(tables: List[T]) → T

Concatenate multiple InfoTables.

Parameters:

tables (List[InfoTable]) – List of InfoTable objects to concatenate.

Returns:

Concatenated InfoTable.

Return type:

InfoTable

Raises:

AssertionError – If resulting DataFrame has duplicate IDs.

clone() → T

Alias for copy().

Returns:

A deep copy.

Return type:

InfoTable

property columns: Index

Get the DataFrame columns.

Returns:

Column names.

Return type:

pd.Index

convert_col_to_str(column: str) → None

Ensure a specific column is of string type.

Parameters:

column (str) – Column name to convert.

copy() → T

Return a deep copy of the InfoTable.

Returns:

A deep copy.

Return type:

InfoTable

drop(labels: str | int | List[Any] | ndarray | Index | Series | None = None, axis: int | str = 0, index: str | int | List[Any] | ndarray | Index | Series | None = None, columns: str | int | List[Any] | ndarray | Index | Series | None = None, level: int | str | None = None, inplace: bool = False, errors: str = 'raise') → T | None

Drop specified labels.

Parameters:

• labels – Index or column labels.

• axis (int) – Whether to drop rows (0) or columns (1).

• index – Alias for labels along the index (rows).

• columns – Column labels to drop.

• level – Level in MultiIndex from which to drop.

• inplace (bool) – Modify in place.

• errors (str) – If ‘ignore’, suppress errors for nonexistent labels.

Returns:

Modified InfoTable or None.

Return type:

Optional[InfoTable]

dropna(*args: Any, **kwargs: Any) → T | None

Return a new InfoTable with missing values dropped.

Parameters:

• *args – Passed to pandas.DataFrame.dropna.

• **kwargs – Passed to pandas.DataFrame.dropna.

Returns:

A new instance with rows (or columns) with NA removed.

Return type:

InfoTable

property eval: Callable

Return the DataFrame.eval method.

Returns:

The eval method for evaluating expressions.

Return type:

Callable

filter(predicate: Callable[[DataFrame], Series | ndarray] | str | None = None, items: List[Any] | None = None, iindex: ndarray | None = None, columns: List[str] | None = None, by: str = 'id', keep: bool = True, raise_if_missing: bool = True) → T

Filter the InfoTable based on a predicate, item list, index list, or column subset.

Parameters:

• predicate (Callable, optional) – Function that returns a boolean mask, e.g.: lambda df: df[“duration”] > 1.0.

• items (List[Any], optional) – Items to include/exclude from the ‘by’ column like df.loc[items, by], used only if predicate is None

• iindex (np.ndarray, optional) – Integer indices to include/exclude like df.iloc[iindex], used if predicate and items are None

• columns (List[str], optional) – Columns to retain or remove.

• by (str) – Column name to use with ‘items’.

• keep (bool) – Whether to keep or exclude matched rows/columns.

• raise_if_missing (bool) – Raise error if items are missing.

Returns:

Filtered InfoTable.

Return type:

InfoTable

Raises:

Exception – If items are not found and raise_if_missing is True.

fix_dtypes() → None

Ensure the ‘id’ column is of string type.

classmethod from_dict(df_dict: Dict[str, List[Any]]) → T

Create InfoTable from a dictionary.

Parameters:

df_dict (Dict[str, List[Any]]) – Column data including ‘id’.

Returns:

Constructed table.

Return type:

InfoTable

classmethod from_lists(ids: List[str], column_names: List[str], column_data: List[List[Any]]) → T

Create InfoTable from lists of IDs and corresponding column data.

Parameters:

• ids (List[str]) – List of IDs.

• column_names (List[str]) – List of column names.

• column_data (List[List[Any]]) – Column values.

Returns:

Constructed table.

Return type:

InfoTable

get_col_idx(keys: str | List[str]) → int | ndarray

Get the integer index position(s) of the specified column(s).

Parameters:

keys (str or list) – Column name(s).

Returns:

Position(s) of the column(s).

Return type:

int or np.ndarray

get_loc(keys: str | List[str] | ndarray) → int | ndarray | List[int]

Get integer location(s) for the given key(s).

Parameters:

keys (str, list, or np.ndarray) – Index label(s).

Returns:

Location(s) in the index.

Return type:

int, np.ndarray, or List[int]

harmonize_age_given_decade(voter_columns: str | List[str], target_column: str, decade_column: str = 'age_decade') → None

Force all rows sharing voter_columns to take the averaged age in target_column, constrained by the decade labels in decade_column.

If the averaged age falls outside the decade bounds, the age is set to the upper boundary for that decade.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric age column to harmonize.

• decade_column (str) – Column indicating the age decade labels.

Raises:

• KeyError – If the voter, target, or decade columns are missing.

• TypeError – If the target column is not numeric.

harmonize_column_by_majority_cluster(voter_columns: str | List[str], target_column: str, suspect_column: str = 'suspect', std_threshold: float | None = None, max_iter: int = 20) → None

Harmonize a numeric column using the dominant cluster within each voter group, while flagging rows from the smaller cluster as suspect.

If std_threshold is provided, clustering is only performed when the group’s standard deviation exceeds the threshold.

The standard deviation of the dominant cluster is stored in a column named f"{target_column}_std" for the non-suspect rows.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric column to harmonize.

• suspect_column (str) – Column used to flag suspect rows.

• std_threshold (Optional[float]) – If set, only split groups above this std.

• max_iter (int) – Maximum iterations for the 1D two-means refinement.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If the target column is not numeric.

harmonize_columns_by_average(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the average value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the group used for averaging.

• target_columns (Union[str, List[str]]) – Numeric column(s) to harmonize via averaging.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If any target column is not numeric.

harmonize_columns_by_majority_vote(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the majority value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the voting group (e.g., speaker).

• target_columns (Union[str, List[str]]) – Column or columns to harmonize via majority vote.

Raises:

KeyError – If the voter or target columns are missing.

head(n: int = 5) → T

Return the first n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

histogram(column: str, bins: int | None = None, density: bool = True, kind: str = 'bar', color: str = 'C0', output_file: str | Path | None = None, dropna: bool = True) → None

Plot a histogram of a column, handling string or numeric data.

Parameters:

• column (str) – Column name to plot. Title will include the column name.

• bins (Optional[int]) – Number of bins for numeric columns. Ignored for strings.

• density (bool) – If True, plot density/relative frequency; otherwise counts.

• kind (str) – Histogram style, either “bar” or “line”.

• color (str) – Matplotlib color spec for the plot.

• output_file (Optional[PathLike]) – If provided, save the figure; otherwise show it.

• dropna (bool) – If True, ignore NA values before plotting.

property iat: _InfoTableAtIndexer

Access a single value for a row/column integer position pair.

Returns:

Indexer that wraps .iat.

Return type:

_InfoTableAtIndexer

property iloc: _InfoTableIndexer

Access a group of rows and columns by integer position(s).

Returns:

Indexer that wraps .iloc.

Return type:

_InfoTableIndexer

property index: Index

Get the index of the DataFrame.

Returns:

DataFrame index.

Return type:

pd.Index

static is_valid_df(df: DataFrame) → bool

Check if the DataFrame is valid for InfoTable.

Parameters:

df (pd.DataFrame) – DataFrame to check.

Returns:

True if valid, False otherwise.

Return type:

bool

property iterrows: Callable

Return the DataFrame.iterrows generator.

Returns:

Yields (index, Series) pairs.

Return type:

Callable

legacy_add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Legacy implementation of add_columns that uses a full pandas merge.

This keeps the original behavior for compatibility/testing.

classmethod load(file_path: str | Path, sep: str | None = None, name: str = 'class_id') → T

Load InfoTable from file.

Parameters:

• file_path (str or Path) – Path to the input file.

• sep (Optional[str]) – Column separator.

• name (str) – Name of the second column (used for Kaldi format).

Returns:

Loaded table.

Return type:

InfoTable

property loc: _InfoTableIndexer

Access a group of rows and columns by label(s).

Returns:

Indexer that wraps .loc.

Return type:

_InfoTableIndexer

classmethod merge(left_table: T | DataFrame, right_table: T | DataFrame, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Tuple[str, str] = ('_x', '_y'), copy: bool | None = None, indicator: str | bool = False, validate: str | None = None) → T

Merge two InfoTables or DataFrames into a new InfoTable.

Parameters:

• left_table (Union[InfoTable, pd.DataFrame]) – Left-hand table.

• right_table (Union[InfoTable, pd.DataFrame]) – Right-hand table.

• how (str) – Merge method (e.g., ‘inner’, ‘outer’, ‘left’, ‘right’).

• on (Union[str, List[str], None]) – Column(s) to join on.

• left_on (Union[str, List[str], None]) – Column(s) from the left table to join on.

• right_on (Union[str, List[str], None]) – Column(s) from the right table to join on.

• left_index (bool) – Use index from the left table as join key.

• right_index (bool) – Use index from the right table as join key.

• sort (bool) – Sort the result by the join keys.

• suffixes (Tuple[str, str]) – Suffixes to apply to overlapping column names.

• copy (Optional[bool]) – If False, avoid copying data where possible.

• indicator (Union[str, bool]) – Adds a column to the output DataFrame called ‘_merge’.

• validate (Optional[str]) – Check if merge is of specified type.

Returns:

A new merged InfoTable.

Return type:

InfoTable

query(expr: str, **kwargs: Any) → T

Filters rows using a boolean expression string.

Parameters:

• expr (str) – A string expression to evaluate, using column names as variables.

• **kwargs – Passed through to pandas.DataFrame.query().

Returns:

A new InfoTable with filtered rows.

Return type:

InfoTable

replace(to_replace: Any = None, value: Any = None, inplace: bool = False, **kwargs: Any) → T | None

Replace values in the DataFrame.

Parameters:

• to_replace – What to replace.

• value – Value to replace with.

• inplace (bool) – Whether to modify the table in-place.

• **kwargs – Additional keyword args passed to pd.DataFrame.replace().

Returns:

New InfoTable if not inplace, else None.

Return type:

Optional[InfoTable]

replace_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None) → None

Replace column values with those from another table.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – Table to source values from.

• column_names (str or list, optional) – Columns to replace. If None, all.

reset_index() → None

Reset the DataFrame index to the ‘id’ column.

Returns:

None

sample(n: int = 1, random_state: int | RandomState | Generator | BitGenerator | None = None) → T

Return a random sample of rows.

Parameters:

• n (int) – Number of rows.

• random_state – Seed for reproducibility.

Returns:

Sampled InfoTable.

Return type:

InfoTable

save(file_path: str | Path, sep: str | None = None) → None

Save the InfoTable to a file.

Parameters:

• file_path (str or Path) – Path to save the file.

• sep (Optional[str]) – Column separator (default inferred from extension).

scatter2d(x_column: str, y_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 2D scattergram for two numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

scatter3d(x_column: str, y_column: str, z_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 3D scattergram for three numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• z_column (str) – Column name for the z-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

set_index(keys: str | List[str], inplace: bool = True) → T | None

Set the DataFrame index using one or more columns.

Parameters:

• keys (str or list) – Column(s) to use as index.

• inplace (bool) – Whether to modify in place.

Returns:

Modified InfoTable if not inplace.

Return type:

Optional[InfoTable]

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffle the rows of the InfoTable.

Parameters:

• seed (int) – Seed for random number generator.

• rng (np.random.Generator, optional) – Numpy random generator.

Returns:

Shuffled indices.

Return type:

np.ndarray

sort(column: str = 'id', ascending: bool = True, inplace: bool = True) → T | None

Sort the InfoTable by a specific column.

Parameters:

• column (str) – Column name to sort by.

• ascending (bool) – Sort in ascending order.

• inplace (bool) – Sort in place or return a new object.

Returns:

Sorted InfoTable or None if inplace.

Return type:

Optional[InfoTable]

split(idx: int, num_parts: int, group_by: str | None = None) → T

Split the InfoTable into parts and return the selected part.

Parameters:

• idx (int) – Part to return (1-based).

• num_parts (int) – Total number of parts.

• group_by (Optional[str]) – Column to group by when splitting.

Returns:

The selected part of the split.

Return type:

InfoTable

tail(n: int = 5) → T

Return the last n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

xs(key: Any, axis: int = 0, level: int | str = None, drop_level: bool = True) → T | Series

Returns a cross-section (row or column) from the DataFrame.

Parameters:

• key – Label or tuple of labels to select.

• axis (int) – Axis to retrieve from (0 for index, 1 for columns).

• level – Level in MultiIndex to use.

• drop_level (bool) – Whether to drop the level(s) from the result.

Returns:

If the result is a DataFrame, wraps it as InfoTable.

Return type:

InfoTable or Series

class hyperion.utils.RecordingSet(df: DataFrame | T)

InfoTable specialization for audio-recording manifests.

The table must contain id and storage_path columns.

Examples

>>> import pandas as pd
>>> from hyperion.utils.recording_set import RecordingSet
>>> df = pd.DataFrame({"id": ["utt1"], "storage_path": ["/audio/utt1.wav"]})
>>> recs = RecordingSet(df)
>>> recs.is_valid_df(recs.df)
True
>>> recs2 = recs.filter(items=["utt1"])
>>> len(recs2)
1

__init__(df: DataFrame | T) → None

Initialize a recording set.

Parameters:

df (pd.DataFrame or RecordingSet) – Input metadata table.

static is_valid_df(df: DataFrame) → bool

Check if the DataFrame is valid for InfoTable.

Parameters:

df (pd.DataFrame) – DataFrame to check.

Returns:

True if valid, False otherwise.

Return type:

bool

save(file_path: str | Path, sep: str | None = None) → None

Save the recording manifest to disk.

Parameters:

• file_path (PathLike) – Output file path.

• sep (Optional[str]) – Delimiter for non-.scp files.

classmethod load(file_path: str | Path, sep: str | None = None) → T

Load a recording manifest from disk.

Parameters:

• file_path (PathLike) – Input file path.

• sep (Optional[str]) – Delimiter for non-.scp files.

Returns:

Loaded recording set.

Return type:

RecordingSet

static _get_durations_old(recordings: RecordingSet, i: int, n: int) → Tuple[List[int], List[float]]

Legacy duration extraction helper based on sequential audio reads.

Parameters:

• recordings (RecordingSet) – Source recordings table.

• i (int) – 1-based partition index.

• n (int) – Number of partitions.

Returns:

Sample rates and durations.

Return type:

Tuple[List[int], List[float]]

static _get_durations(recordings: RecordingSet, i: int, n: int, progress: Any | None = None, report_every: int = 1000) → Tuple[List[str], List[int], List[float]]

Duration extraction helper with file-header and fallback decoding logic.

Parameters:

• recordings (RecordingSet) – Source recordings table.

• i (int) – 1-based partition index.

• n (int) – Number of partitions.

• progress (Optional[Any]) – Shared counter proxy with a value field.

• report_every (int) – Counter update interval.

Returns:

Recording IDs, sample rates, and durations.

Return type:

Tuple[List[str], List[int], List[float]]

get_durations(num_threads: int = 16, report_every: int = 5000) → None

Estimate recording duration and sample rate with a process pool.

This version periodically reports progress and writes duration and sample_freq columns back into the table.

Parameters:

• num_threads (int) – Number of worker processes.

• report_every (int) – Progress update interval in processed recordings.

__cmp__(other: Any) → int

Comparison operator

__contains__(key: Any) → bool

Check whether key is in the DataFrame columns.

Parameters:

key – Key to check.

Returns:

True if key is in columns.

Return type:

bool

__eq__(other: Any) → bool

Equal operator

__getattr__(name: str) → Any

Provide attribute-style access to DataFrame columns and attributes.

Falls back to self.df[name] if name is a column. Falls back to getattr(self.df, name) if it’s a DataFrame method or attribute. Special methods (e.g., __setstate__) raise AttributeError immediately to allow correct behavior during pickling/unpickling.

Parameters:

name (str) – The attribute being accessed.

Returns:

The corresponding column, method, or attribute.

Return type:

Any

Raises:

AttributeError – If the attribute is not found.

__getitem__(key: Any) → T | Series

Get item from the internal DataFrame.

Parameters:

key – Key used for indexing.

Returns:

Sub-table or Series.

Return type:

Union[InfoTable, pd.Series]

__ne__(other: Any) → bool

Non-equal operator

__setitem__(key: Any, value: Any) → None

Set item in the internal DataFrame.

Parameters:

• key – Column label or index.

• value – Value to assign.

add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Add new columns from another InfoTable or DataFrame.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – The table to merge columns from.

• column_names (str or list, optional) – Columns to include from the right table.

• on (str or list) – Key(s) from the current table.

• right_on (str or list, optional) – Key(s) from the right table.

• replace_overlapping (bool) – Replace overlapping columns if True.

• ignore_overlapping (bool) – If True, skip adding columns that already exist.

• remove_missing (bool) – Use inner join (drop unmatched rows) if True.

property at: _InfoTableAtIndexer

Access a single value for a row/column label pair.

Returns:

Indexer that wraps .at.

Return type:

_InfoTableAtIndexer

classmethod cat(tables: List[T]) → T

Concatenate multiple InfoTables.

Parameters:

tables (List[InfoTable]) – List of InfoTable objects to concatenate.

Returns:

Concatenated InfoTable.

Return type:

InfoTable

Raises:

AssertionError – If resulting DataFrame has duplicate IDs.

clone() → T

Alias for copy().

Returns:

A deep copy.

Return type:

InfoTable

property columns: Index

Get the DataFrame columns.

Returns:

Column names.

Return type:

pd.Index

convert_col_to_str(column: str) → None

Ensure a specific column is of string type.

Parameters:

column (str) – Column name to convert.

copy() → T

Return a deep copy of the InfoTable.

Returns:

A deep copy.

Return type:

InfoTable

drop(labels: str | int | List[Any] | ndarray | Index | Series | None = None, axis: int | str = 0, index: str | int | List[Any] | ndarray | Index | Series | None = None, columns: str | int | List[Any] | ndarray | Index | Series | None = None, level: int | str | None = None, inplace: bool = False, errors: str = 'raise') → T | None

Drop specified labels.

Parameters:

• labels – Index or column labels.

• axis (int) – Whether to drop rows (0) or columns (1).

• index – Alias for labels along the index (rows).

• columns – Column labels to drop.

• level – Level in MultiIndex from which to drop.

• inplace (bool) – Modify in place.

• errors (str) – If ‘ignore’, suppress errors for nonexistent labels.

Returns:

Modified InfoTable or None.

Return type:

Optional[InfoTable]

dropna(*args: Any, **kwargs: Any) → T | None

Return a new InfoTable with missing values dropped.

Parameters:

• *args – Passed to pandas.DataFrame.dropna.

• **kwargs – Passed to pandas.DataFrame.dropna.

Returns:

A new instance with rows (or columns) with NA removed.

Return type:

InfoTable

property eval: Callable

Return the DataFrame.eval method.

Returns:

The eval method for evaluating expressions.

Return type:

Callable

filter(predicate: Callable[[DataFrame], Series | ndarray] | str | None = None, items: List[Any] | None = None, iindex: ndarray | None = None, columns: List[str] | None = None, by: str = 'id', keep: bool = True, raise_if_missing: bool = True) → T

Filter the InfoTable based on a predicate, item list, index list, or column subset.

Parameters:

• predicate (Callable, optional) – Function that returns a boolean mask, e.g.: lambda df: df[“duration”] > 1.0.

• items (List[Any], optional) – Items to include/exclude from the ‘by’ column like df.loc[items, by], used only if predicate is None

• iindex (np.ndarray, optional) – Integer indices to include/exclude like df.iloc[iindex], used if predicate and items are None

• columns (List[str], optional) – Columns to retain or remove.

• by (str) – Column name to use with ‘items’.

• keep (bool) – Whether to keep or exclude matched rows/columns.

• raise_if_missing (bool) – Raise error if items are missing.

Returns:

Filtered InfoTable.

Return type:

InfoTable

Raises:

Exception – If items are not found and raise_if_missing is True.

fix_dtypes() → None

Ensure the ‘id’ column is of string type.

classmethod from_dict(df_dict: Dict[str, List[Any]]) → T

Create InfoTable from a dictionary.

Parameters:

df_dict (Dict[str, List[Any]]) – Column data including ‘id’.

Returns:

Constructed table.

Return type:

InfoTable

classmethod from_lists(ids: List[str], column_names: List[str], column_data: List[List[Any]]) → T

Create InfoTable from lists of IDs and corresponding column data.

Parameters:

• ids (List[str]) – List of IDs.

• column_names (List[str]) – List of column names.

• column_data (List[List[Any]]) – Column values.

Returns:

Constructed table.

Return type:

InfoTable

get_col_idx(keys: str | List[str]) → int | ndarray

Get the integer index position(s) of the specified column(s).

Parameters:

keys (str or list) – Column name(s).

Returns:

Position(s) of the column(s).

Return type:

int or np.ndarray

get_loc(keys: str | List[str] | ndarray) → int | ndarray | List[int]

Get integer location(s) for the given key(s).

Parameters:

keys (str, list, or np.ndarray) – Index label(s).

Returns:

Location(s) in the index.

Return type:

int, np.ndarray, or List[int]

harmonize_age_given_decade(voter_columns: str | List[str], target_column: str, decade_column: str = 'age_decade') → None

Force all rows sharing voter_columns to take the averaged age in target_column, constrained by the decade labels in decade_column.

If the averaged age falls outside the decade bounds, the age is set to the upper boundary for that decade.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric age column to harmonize.

• decade_column (str) – Column indicating the age decade labels.

Raises:

• KeyError – If the voter, target, or decade columns are missing.

• TypeError – If the target column is not numeric.

harmonize_column_by_majority_cluster(voter_columns: str | List[str], target_column: str, suspect_column: str = 'suspect', std_threshold: float | None = None, max_iter: int = 20) → None

Harmonize a numeric column using the dominant cluster within each voter group, while flagging rows from the smaller cluster as suspect.

If std_threshold is provided, clustering is only performed when the group’s standard deviation exceeds the threshold.

The standard deviation of the dominant cluster is stored in a column named f"{target_column}_std" for the non-suspect rows.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric column to harmonize.

• suspect_column (str) – Column used to flag suspect rows.

• std_threshold (Optional[float]) – If set, only split groups above this std.

• max_iter (int) – Maximum iterations for the 1D two-means refinement.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If the target column is not numeric.

harmonize_columns_by_average(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the average value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the group used for averaging.

• target_columns (Union[str, List[str]]) – Numeric column(s) to harmonize via averaging.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If any target column is not numeric.

harmonize_columns_by_majority_vote(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the majority value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the voting group (e.g., speaker).

• target_columns (Union[str, List[str]]) – Column or columns to harmonize via majority vote.

Raises:

KeyError – If the voter or target columns are missing.

head(n: int = 5) → T

Return the first n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

histogram(column: str, bins: int | None = None, density: bool = True, kind: str = 'bar', color: str = 'C0', output_file: str | Path | None = None, dropna: bool = True) → None

Plot a histogram of a column, handling string or numeric data.

Parameters:

• column (str) – Column name to plot. Title will include the column name.

• bins (Optional[int]) – Number of bins for numeric columns. Ignored for strings.

• density (bool) – If True, plot density/relative frequency; otherwise counts.

• kind (str) – Histogram style, either “bar” or “line”.

• color (str) – Matplotlib color spec for the plot.

• output_file (Optional[PathLike]) – If provided, save the figure; otherwise show it.

• dropna (bool) – If True, ignore NA values before plotting.

property iat: _InfoTableAtIndexer

Access a single value for a row/column integer position pair.

Returns:

Indexer that wraps .iat.

Return type:

_InfoTableAtIndexer

property iloc: _InfoTableIndexer

Access a group of rows and columns by integer position(s).

Returns:

Indexer that wraps .iloc.

Return type:

_InfoTableIndexer

property index: Index

Get the index of the DataFrame.

Returns:

DataFrame index.

Return type:

pd.Index

property iterrows: Callable

Return the DataFrame.iterrows generator.

Returns:

Yields (index, Series) pairs.

Return type:

Callable

legacy_add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Legacy implementation of add_columns that uses a full pandas merge.

This keeps the original behavior for compatibility/testing.

property loc: _InfoTableIndexer

Access a group of rows and columns by label(s).

Returns:

Indexer that wraps .loc.

Return type:

_InfoTableIndexer

classmethod merge(left_table: T | DataFrame, right_table: T | DataFrame, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Tuple[str, str] = ('_x', '_y'), copy: bool | None = None, indicator: str | bool = False, validate: str | None = None) → T

Merge two InfoTables or DataFrames into a new InfoTable.

Parameters:

• left_table (Union[InfoTable, pd.DataFrame]) – Left-hand table.

• right_table (Union[InfoTable, pd.DataFrame]) – Right-hand table.

• how (str) – Merge method (e.g., ‘inner’, ‘outer’, ‘left’, ‘right’).

• on (Union[str, List[str], None]) – Column(s) to join on.

• left_on (Union[str, List[str], None]) – Column(s) from the left table to join on.

• right_on (Union[str, List[str], None]) – Column(s) from the right table to join on.

• left_index (bool) – Use index from the left table as join key.

• right_index (bool) – Use index from the right table as join key.

• sort (bool) – Sort the result by the join keys.

• suffixes (Tuple[str, str]) – Suffixes to apply to overlapping column names.

• copy (Optional[bool]) – If False, avoid copying data where possible.

• indicator (Union[str, bool]) – Adds a column to the output DataFrame called ‘_merge’.

• validate (Optional[str]) – Check if merge is of specified type.

Returns:

A new merged InfoTable.

Return type:

InfoTable

query(expr: str, **kwargs: Any) → T

Filters rows using a boolean expression string.

Parameters:

• expr (str) – A string expression to evaluate, using column names as variables.

• **kwargs – Passed through to pandas.DataFrame.query().

Returns:

A new InfoTable with filtered rows.

Return type:

InfoTable

replace(to_replace: Any = None, value: Any = None, inplace: bool = False, **kwargs: Any) → T | None

Replace values in the DataFrame.

Parameters:

• to_replace – What to replace.

• value – Value to replace with.

• inplace (bool) – Whether to modify the table in-place.

• **kwargs – Additional keyword args passed to pd.DataFrame.replace().

Returns:

New InfoTable if not inplace, else None.

Return type:

Optional[InfoTable]

replace_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None) → None

Replace column values with those from another table.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – Table to source values from.

• column_names (str or list, optional) – Columns to replace. If None, all.

reset_index() → None

Reset the DataFrame index to the ‘id’ column.

Returns:

None

sample(n: int = 1, random_state: int | RandomState | Generator | BitGenerator | None = None) → T

Return a random sample of rows.

Parameters:

• n (int) – Number of rows.

• random_state – Seed for reproducibility.

Returns:

Sampled InfoTable.

Return type:

InfoTable

scatter2d(x_column: str, y_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 2D scattergram for two numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

scatter3d(x_column: str, y_column: str, z_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 3D scattergram for three numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• z_column (str) – Column name for the z-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

set_index(keys: str | List[str], inplace: bool = True) → T | None

Set the DataFrame index using one or more columns.

Parameters:

• keys (str or list) – Column(s) to use as index.

• inplace (bool) – Whether to modify in place.

Returns:

Modified InfoTable if not inplace.

Return type:

Optional[InfoTable]

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffle the rows of the InfoTable.

Parameters:

• seed (int) – Seed for random number generator.

• rng (np.random.Generator, optional) – Numpy random generator.

Returns:

Shuffled indices.

Return type:

np.ndarray

sort(column: str = 'id', ascending: bool = True, inplace: bool = True) → T | None

Sort the InfoTable by a specific column.

Parameters:

• column (str) – Column name to sort by.

• ascending (bool) – Sort in ascending order.

• inplace (bool) – Sort in place or return a new object.

Returns:

Sorted InfoTable or None if inplace.

Return type:

Optional[InfoTable]

split(idx: int, num_parts: int, group_by: str | None = None) → T

Split the InfoTable into parts and return the selected part.

Parameters:

• idx (int) – Part to return (1-based).

• num_parts (int) – Total number of parts.

• group_by (Optional[str]) – Column to group by when splitting.

Returns:

The selected part of the split.

Return type:

InfoTable

tail(n: int = 5) → T

Return the last n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

xs(key: Any, axis: int = 0, level: int | str = None, drop_level: bool = True) → T | Series

Returns a cross-section (row or column) from the DataFrame.

Parameters:

• key – Label or tuple of labels to select.

• axis (int) – Axis to retrieve from (0 for index, 1 for columns).

• level – Level in MultiIndex to use.

• drop_level (bool) – Whether to drop the level(s) from the result.

Returns:

If the result is a DataFrame, wraps it as InfoTable.

Return type:

InfoTable or Series

class hyperion.utils.FeatureSet(df: DataFrame | T)

InfoTable specialization for feature manifests.

The table must contain id and storage_path columns.

Examples

>>> import pandas as pd
>>> from hyperion.utils.feature_set import FeatureSet
>>> df = pd.DataFrame({"id": ["utt1"], "storage_path": ["feats.ark:123"]})
>>> feats = FeatureSet(df)
>>> feats.add_prefix_to_storage_path("/mnt/data")
>>> feats.df.loc["utt1", "storage_path"]
'/mnt/data/feats.ark:123'
>>> feats2 = feats.filter(items=["utt1"])
>>> len(feats2)
1

__init__(df: DataFrame | T) → None

Initialize a feature set.

Parameters:

df (pd.DataFrame or FeatureSet) – Input metadata table.

add_prefix_to_storage_path(prefix: str | Path) → None

Prepend a directory prefix to storage_path values.

Parameters:

prefix (PathLike) – Prefix path to join with each storage path.

save(file_path: str | Path, sep: str | None = None) → None

Save the feature set to disk.

Parameters:

• file_path (PathLike) – Output file path.

• sep (Optional[str]) – Delimiter for non-.scp files.

classmethod load(file_path: str | Path, sep: str | None = None) → T

Load a feature set from disk.

Parameters:

• file_path (PathLike) – Input file path.

• sep (Optional[str]) – Delimiter for non-.scp files.

Returns:

Loaded feature set.

Return type:

FeatureSet

__cmp__(other: Any) → int

Comparison operator

__contains__(key: Any) → bool

Check whether key is in the DataFrame columns.

Parameters:

key – Key to check.

Returns:

True if key is in columns.

Return type:

bool

__eq__(other: Any) → bool

Equal operator

__getattr__(name: str) → Any

Provide attribute-style access to DataFrame columns and attributes.

Falls back to self.df[name] if name is a column. Falls back to getattr(self.df, name) if it’s a DataFrame method or attribute. Special methods (e.g., __setstate__) raise AttributeError immediately to allow correct behavior during pickling/unpickling.

Parameters:

name (str) – The attribute being accessed.

Returns:

The corresponding column, method, or attribute.

Return type:

Any

Raises:

AttributeError – If the attribute is not found.

__getitem__(key: Any) → T | Series

Get item from the internal DataFrame.

Parameters:

key – Key used for indexing.

Returns:

Sub-table or Series.

Return type:

Union[InfoTable, pd.Series]

__ne__(other: Any) → bool

Non-equal operator

__setitem__(key: Any, value: Any) → None

Set item in the internal DataFrame.

Parameters:

• key – Column label or index.

• value – Value to assign.

add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Add new columns from another InfoTable or DataFrame.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – The table to merge columns from.

• column_names (str or list, optional) – Columns to include from the right table.

• on (str or list) – Key(s) from the current table.

• right_on (str or list, optional) – Key(s) from the right table.

• replace_overlapping (bool) – Replace overlapping columns if True.

• ignore_overlapping (bool) – If True, skip adding columns that already exist.

• remove_missing (bool) – Use inner join (drop unmatched rows) if True.

property at: _InfoTableAtIndexer

Access a single value for a row/column label pair.

Returns:

Indexer that wraps .at.

Return type:

_InfoTableAtIndexer

classmethod cat(tables: List[T]) → T

Concatenate multiple InfoTables.

Parameters:

tables (List[InfoTable]) – List of InfoTable objects to concatenate.

Returns:

Concatenated InfoTable.

Return type:

InfoTable

Raises:

AssertionError – If resulting DataFrame has duplicate IDs.

clone() → T

Alias for copy().

Returns:

A deep copy.

Return type:

InfoTable

property columns: Index

Get the DataFrame columns.

Returns:

Column names.

Return type:

pd.Index

convert_col_to_str(column: str) → None

Ensure a specific column is of string type.

Parameters:

column (str) – Column name to convert.

copy() → T

Return a deep copy of the InfoTable.

Returns:

A deep copy.

Return type:

InfoTable

drop(labels: str | int | List[Any] | ndarray | Index | Series | None = None, axis: int | str = 0, index: str | int | List[Any] | ndarray | Index | Series | None = None, columns: str | int | List[Any] | ndarray | Index | Series | None = None, level: int | str | None = None, inplace: bool = False, errors: str = 'raise') → T | None

Drop specified labels.

Parameters:

• labels – Index or column labels.

• axis (int) – Whether to drop rows (0) or columns (1).

• index – Alias for labels along the index (rows).

• columns – Column labels to drop.

• level – Level in MultiIndex from which to drop.

• inplace (bool) – Modify in place.

• errors (str) – If ‘ignore’, suppress errors for nonexistent labels.

Returns:

Modified InfoTable or None.

Return type:

Optional[InfoTable]

dropna(*args: Any, **kwargs: Any) → T | None

Return a new InfoTable with missing values dropped.

Parameters:

• *args – Passed to pandas.DataFrame.dropna.

• **kwargs – Passed to pandas.DataFrame.dropna.

Returns:

A new instance with rows (or columns) with NA removed.

Return type:

InfoTable

property eval: Callable

Return the DataFrame.eval method.

Returns:

The eval method for evaluating expressions.

Return type:

Callable

filter(predicate: Callable[[DataFrame], Series | ndarray] | str | None = None, items: List[Any] | None = None, iindex: ndarray | None = None, columns: List[str] | None = None, by: str = 'id', keep: bool = True, raise_if_missing: bool = True) → T

Filter the InfoTable based on a predicate, item list, index list, or column subset.

Parameters:

• predicate (Callable, optional) – Function that returns a boolean mask, e.g.: lambda df: df[“duration”] > 1.0.

• items (List[Any], optional) – Items to include/exclude from the ‘by’ column like df.loc[items, by], used only if predicate is None

• iindex (np.ndarray, optional) – Integer indices to include/exclude like df.iloc[iindex], used if predicate and items are None

• columns (List[str], optional) – Columns to retain or remove.

• by (str) – Column name to use with ‘items’.

• keep (bool) – Whether to keep or exclude matched rows/columns.

• raise_if_missing (bool) – Raise error if items are missing.

Returns:

Filtered InfoTable.

Return type:

InfoTable

Raises:

Exception – If items are not found and raise_if_missing is True.

fix_dtypes() → None

Ensure the ‘id’ column is of string type.

classmethod from_dict(df_dict: Dict[str, List[Any]]) → T

Create InfoTable from a dictionary.

Parameters:

df_dict (Dict[str, List[Any]]) – Column data including ‘id’.

Returns:

Constructed table.

Return type:

InfoTable

classmethod from_lists(ids: List[str], column_names: List[str], column_data: List[List[Any]]) → T

Create InfoTable from lists of IDs and corresponding column data.

Parameters:

• ids (List[str]) – List of IDs.

• column_names (List[str]) – List of column names.

• column_data (List[List[Any]]) – Column values.

Returns:

Constructed table.

Return type:

InfoTable

get_col_idx(keys: str | List[str]) → int | ndarray

Get the integer index position(s) of the specified column(s).

Parameters:

keys (str or list) – Column name(s).

Returns:

Position(s) of the column(s).

Return type:

int or np.ndarray

get_loc(keys: str | List[str] | ndarray) → int | ndarray | List[int]

Get integer location(s) for the given key(s).

Parameters:

keys (str, list, or np.ndarray) – Index label(s).

Returns:

Location(s) in the index.

Return type:

int, np.ndarray, or List[int]

harmonize_age_given_decade(voter_columns: str | List[str], target_column: str, decade_column: str = 'age_decade') → None

Force all rows sharing voter_columns to take the averaged age in target_column, constrained by the decade labels in decade_column.

If the averaged age falls outside the decade bounds, the age is set to the upper boundary for that decade.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric age column to harmonize.

• decade_column (str) – Column indicating the age decade labels.

Raises:

• KeyError – If the voter, target, or decade columns are missing.

• TypeError – If the target column is not numeric.

harmonize_column_by_majority_cluster(voter_columns: str | List[str], target_column: str, suspect_column: str = 'suspect', std_threshold: float | None = None, max_iter: int = 20) → None

Harmonize a numeric column using the dominant cluster within each voter group, while flagging rows from the smaller cluster as suspect.

If std_threshold is provided, clustering is only performed when the group’s standard deviation exceeds the threshold.

The standard deviation of the dominant cluster is stored in a column named f"{target_column}_std" for the non-suspect rows.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric column to harmonize.

• suspect_column (str) – Column used to flag suspect rows.

• std_threshold (Optional[float]) – If set, only split groups above this std.

• max_iter (int) – Maximum iterations for the 1D two-means refinement.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If the target column is not numeric.

harmonize_columns_by_average(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the average value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the group used for averaging.

• target_columns (Union[str, List[str]]) – Numeric column(s) to harmonize via averaging.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If any target column is not numeric.

harmonize_columns_by_majority_vote(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the majority value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the voting group (e.g., speaker).

• target_columns (Union[str, List[str]]) – Column or columns to harmonize via majority vote.

Raises:

KeyError – If the voter or target columns are missing.

head(n: int = 5) → T

Return the first n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

histogram(column: str, bins: int | None = None, density: bool = True, kind: str = 'bar', color: str = 'C0', output_file: str | Path | None = None, dropna: bool = True) → None

Plot a histogram of a column, handling string or numeric data.

Parameters:

• column (str) – Column name to plot. Title will include the column name.

• bins (Optional[int]) – Number of bins for numeric columns. Ignored for strings.

• density (bool) – If True, plot density/relative frequency; otherwise counts.

• kind (str) – Histogram style, either “bar” or “line”.

• color (str) – Matplotlib color spec for the plot.

• output_file (Optional[PathLike]) – If provided, save the figure; otherwise show it.

• dropna (bool) – If True, ignore NA values before plotting.

property iat: _InfoTableAtIndexer

Access a single value for a row/column integer position pair.

Returns:

Indexer that wraps .iat.

Return type:

_InfoTableAtIndexer

property iloc: _InfoTableIndexer

Access a group of rows and columns by integer position(s).

Returns:

Indexer that wraps .iloc.

Return type:

_InfoTableIndexer

property index: Index

Get the index of the DataFrame.

Returns:

DataFrame index.

Return type:

pd.Index

static is_valid_df(df: DataFrame) → bool

Check if the DataFrame is valid for InfoTable.

Parameters:

df (pd.DataFrame) – DataFrame to check.

Returns:

True if valid, False otherwise.

Return type:

bool

property iterrows: Callable

Return the DataFrame.iterrows generator.

Returns:

Yields (index, Series) pairs.

Return type:

Callable

legacy_add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Legacy implementation of add_columns that uses a full pandas merge.

This keeps the original behavior for compatibility/testing.

property loc: _InfoTableIndexer

Access a group of rows and columns by label(s).

Returns:

Indexer that wraps .loc.

Return type:

_InfoTableIndexer

classmethod merge(left_table: T | DataFrame, right_table: T | DataFrame, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Tuple[str, str] = ('_x', '_y'), copy: bool | None = None, indicator: str | bool = False, validate: str | None = None) → T

Merge two InfoTables or DataFrames into a new InfoTable.

Parameters:

• left_table (Union[InfoTable, pd.DataFrame]) – Left-hand table.

• right_table (Union[InfoTable, pd.DataFrame]) – Right-hand table.

• how (str) – Merge method (e.g., ‘inner’, ‘outer’, ‘left’, ‘right’).

• on (Union[str, List[str], None]) – Column(s) to join on.

• left_on (Union[str, List[str], None]) – Column(s) from the left table to join on.

• right_on (Union[str, List[str], None]) – Column(s) from the right table to join on.

• left_index (bool) – Use index from the left table as join key.

• right_index (bool) – Use index from the right table as join key.

• sort (bool) – Sort the result by the join keys.

• suffixes (Tuple[str, str]) – Suffixes to apply to overlapping column names.

• copy (Optional[bool]) – If False, avoid copying data where possible.

• indicator (Union[str, bool]) – Adds a column to the output DataFrame called ‘_merge’.

• validate (Optional[str]) – Check if merge is of specified type.

Returns:

A new merged InfoTable.

Return type:

InfoTable

query(expr: str, **kwargs: Any) → T

Filters rows using a boolean expression string.

Parameters:

• expr (str) – A string expression to evaluate, using column names as variables.

• **kwargs – Passed through to pandas.DataFrame.query().

Returns:

A new InfoTable with filtered rows.

Return type:

InfoTable

replace(to_replace: Any = None, value: Any = None, inplace: bool = False, **kwargs: Any) → T | None

Replace values in the DataFrame.

Parameters:

• to_replace – What to replace.

• value – Value to replace with.

• inplace (bool) – Whether to modify the table in-place.

• **kwargs – Additional keyword args passed to pd.DataFrame.replace().

Returns:

New InfoTable if not inplace, else None.

Return type:

Optional[InfoTable]

replace_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None) → None

Replace column values with those from another table.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – Table to source values from.

• column_names (str or list, optional) – Columns to replace. If None, all.

reset_index() → None

Reset the DataFrame index to the ‘id’ column.

Returns:

None

sample(n: int = 1, random_state: int | RandomState | Generator | BitGenerator | None = None) → T

Return a random sample of rows.

Parameters:

• n (int) – Number of rows.

• random_state – Seed for reproducibility.

Returns:

Sampled InfoTable.

Return type:

InfoTable

scatter2d(x_column: str, y_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 2D scattergram for two numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

scatter3d(x_column: str, y_column: str, z_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 3D scattergram for three numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• z_column (str) – Column name for the z-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

set_index(keys: str | List[str], inplace: bool = True) → T | None

Set the DataFrame index using one or more columns.

Parameters:

• keys (str or list) – Column(s) to use as index.

• inplace (bool) – Whether to modify in place.

Returns:

Modified InfoTable if not inplace.

Return type:

Optional[InfoTable]

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffle the rows of the InfoTable.

Parameters:

• seed (int) – Seed for random number generator.

• rng (np.random.Generator, optional) – Numpy random generator.

Returns:

Shuffled indices.

Return type:

np.ndarray

sort(column: str = 'id', ascending: bool = True, inplace: bool = True) → T | None

Sort the InfoTable by a specific column.

Parameters:

• column (str) – Column name to sort by.

• ascending (bool) – Sort in ascending order.

• inplace (bool) – Sort in place or return a new object.

Returns:

Sorted InfoTable or None if inplace.

Return type:

Optional[InfoTable]

split(idx: int, num_parts: int, group_by: str | None = None) → T

Split the InfoTable into parts and return the selected part.

Parameters:

• idx (int) – Part to return (1-based).

• num_parts (int) – Total number of parts.

• group_by (Optional[str]) – Column to group by when splitting.

Returns:

The selected part of the split.

Return type:

InfoTable

tail(n: int = 5) → T

Return the last n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

xs(key: Any, axis: int = 0, level: int | str = None, drop_level: bool = True) → T | Series

Returns a cross-section (row or column) from the DataFrame.

Parameters:

• key – Label or tuple of labels to select.

• axis (int) – Axis to retrieve from (0 for index, 1 for columns).

• level – Level in MultiIndex to use.

• drop_level (bool) – Whether to drop the level(s) from the result.

Returns:

If the result is a DataFrame, wraps it as InfoTable.

Return type:

InfoTable or Series

class hyperion.utils.VADSet(df: DataFrame | T)

FeatureSet specialization for voice-activity-detection manifests.

Examples

>>> import pandas as pd
>>> from hyperion.utils.vad_set import VADSet
>>> df = pd.DataFrame({"id": ["utt1"], "storage_path": ["vad.ark:10"]})
>>> vad = VADSet(df)
>>> list(vad.columns)
['id', 'storage_path']
>>> vad.add_prefix_to_storage_path("/mnt/vad")
>>> vad.df.loc["utt1", "storage_path"]
'/mnt/vad/vad.ark:10'

__init__(df: DataFrame | T) → None

Initialize a VAD set.

Parameters:

df (pd.DataFrame or VADSet) – Input VAD metadata table.

__cmp__(other: Any) → int

Comparison operator

__contains__(key: Any) → bool

Check whether key is in the DataFrame columns.

Parameters:

key – Key to check.

Returns:

True if key is in columns.

Return type:

bool

__eq__(other: Any) → bool

Equal operator

__getattr__(name: str) → Any

Provide attribute-style access to DataFrame columns and attributes.

Falls back to self.df[name] if name is a column. Falls back to getattr(self.df, name) if it’s a DataFrame method or attribute. Special methods (e.g., __setstate__) raise AttributeError immediately to allow correct behavior during pickling/unpickling.

Parameters:

name (str) – The attribute being accessed.

Returns:

The corresponding column, method, or attribute.

Return type:

Any

Raises:

AttributeError – If the attribute is not found.

__getitem__(key: Any) → T | Series

Get item from the internal DataFrame.

Parameters:

key – Key used for indexing.

Returns:

Sub-table or Series.

Return type:

Union[InfoTable, pd.Series]

__ne__(other: Any) → bool

Non-equal operator

__setitem__(key: Any, value: Any) → None

Set item in the internal DataFrame.

Parameters:

• key – Column label or index.

• value – Value to assign.

add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Add new columns from another InfoTable or DataFrame.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – The table to merge columns from.

• column_names (str or list, optional) – Columns to include from the right table.

• on (str or list) – Key(s) from the current table.

• right_on (str or list, optional) – Key(s) from the right table.

• replace_overlapping (bool) – Replace overlapping columns if True.

• ignore_overlapping (bool) – If True, skip adding columns that already exist.

• remove_missing (bool) – Use inner join (drop unmatched rows) if True.

add_prefix_to_storage_path(prefix: str | Path) → None

Prepend a directory prefix to storage_path values.

Parameters:

prefix (PathLike) – Prefix path to join with each storage path.

property at: _InfoTableAtIndexer

Access a single value for a row/column label pair.

Returns:

Indexer that wraps .at.

Return type:

_InfoTableAtIndexer

classmethod cat(tables: List[T]) → T

Concatenate multiple InfoTables.

Parameters:

tables (List[InfoTable]) – List of InfoTable objects to concatenate.

Returns:

Concatenated InfoTable.

Return type:

InfoTable

Raises:

AssertionError – If resulting DataFrame has duplicate IDs.

clone() → T

Alias for copy().

Returns:

A deep copy.

Return type:

InfoTable

property columns: Index

Get the DataFrame columns.

Returns:

Column names.

Return type:

pd.Index

convert_col_to_str(column: str) → None

Ensure a specific column is of string type.

Parameters:

column (str) – Column name to convert.

copy() → T

Return a deep copy of the InfoTable.

Returns:

A deep copy.

Return type:

InfoTable

drop(labels: str | int | List[Any] | ndarray | Index | Series | None = None, axis: int | str = 0, index: str | int | List[Any] | ndarray | Index | Series | None = None, columns: str | int | List[Any] | ndarray | Index | Series | None = None, level: int | str | None = None, inplace: bool = False, errors: str = 'raise') → T | None

Drop specified labels.

Parameters:

• labels – Index or column labels.

• axis (int) – Whether to drop rows (0) or columns (1).

• index – Alias for labels along the index (rows).

• columns – Column labels to drop.

• level – Level in MultiIndex from which to drop.

• inplace (bool) – Modify in place.

• errors (str) – If ‘ignore’, suppress errors for nonexistent labels.

Returns:

Modified InfoTable or None.

Return type:

Optional[InfoTable]

dropna(*args: Any, **kwargs: Any) → T | None

Return a new InfoTable with missing values dropped.

Parameters:

• *args – Passed to pandas.DataFrame.dropna.

• **kwargs – Passed to pandas.DataFrame.dropna.

Returns:

A new instance with rows (or columns) with NA removed.

Return type:

InfoTable

property eval: Callable

Return the DataFrame.eval method.

Returns:

The eval method for evaluating expressions.

Return type:

Callable

filter(predicate: Callable[[DataFrame], Series | ndarray] | str | None = None, items: List[Any] | None = None, iindex: ndarray | None = None, columns: List[str] | None = None, by: str = 'id', keep: bool = True, raise_if_missing: bool = True) → T

Filter the InfoTable based on a predicate, item list, index list, or column subset.

Parameters:

• predicate (Callable, optional) – Function that returns a boolean mask, e.g.: lambda df: df[“duration”] > 1.0.

• items (List[Any], optional) – Items to include/exclude from the ‘by’ column like df.loc[items, by], used only if predicate is None

• iindex (np.ndarray, optional) – Integer indices to include/exclude like df.iloc[iindex], used if predicate and items are None

• columns (List[str], optional) – Columns to retain or remove.

• by (str) – Column name to use with ‘items’.

• keep (bool) – Whether to keep or exclude matched rows/columns.

• raise_if_missing (bool) – Raise error if items are missing.

Returns:

Filtered InfoTable.

Return type:

InfoTable

Raises:

Exception – If items are not found and raise_if_missing is True.

fix_dtypes() → None

Ensure the ‘id’ column is of string type.

classmethod from_dict(df_dict: Dict[str, List[Any]]) → T

Create InfoTable from a dictionary.

Parameters:

df_dict (Dict[str, List[Any]]) – Column data including ‘id’.

Returns:

Constructed table.

Return type:

InfoTable

classmethod from_lists(ids: List[str], column_names: List[str], column_data: List[List[Any]]) → T

Create InfoTable from lists of IDs and corresponding column data.

Parameters:

• ids (List[str]) – List of IDs.

• column_names (List[str]) – List of column names.

• column_data (List[List[Any]]) – Column values.

Returns:

Constructed table.

Return type:

InfoTable

get_col_idx(keys: str | List[str]) → int | ndarray

Get the integer index position(s) of the specified column(s).

Parameters:

keys (str or list) – Column name(s).

Returns:

Position(s) of the column(s).

Return type:

int or np.ndarray

get_loc(keys: str | List[str] | ndarray) → int | ndarray | List[int]

Get integer location(s) for the given key(s).

Parameters:

keys (str, list, or np.ndarray) – Index label(s).

Returns:

Location(s) in the index.

Return type:

int, np.ndarray, or List[int]

harmonize_age_given_decade(voter_columns: str | List[str], target_column: str, decade_column: str = 'age_decade') → None

Force all rows sharing voter_columns to take the averaged age in target_column, constrained by the decade labels in decade_column.

If the averaged age falls outside the decade bounds, the age is set to the upper boundary for that decade.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric age column to harmonize.

• decade_column (str) – Column indicating the age decade labels.

Raises:

• KeyError – If the voter, target, or decade columns are missing.

• TypeError – If the target column is not numeric.

harmonize_column_by_majority_cluster(voter_columns: str | List[str], target_column: str, suspect_column: str = 'suspect', std_threshold: float | None = None, max_iter: int = 20) → None

Harmonize a numeric column using the dominant cluster within each voter group, while flagging rows from the smaller cluster as suspect.

If std_threshold is provided, clustering is only performed when the group’s standard deviation exceeds the threshold.

The standard deviation of the dominant cluster is stored in a column named f"{target_column}_std" for the non-suspect rows.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric column to harmonize.

• suspect_column (str) – Column used to flag suspect rows.

• std_threshold (Optional[float]) – If set, only split groups above this std.

• max_iter (int) – Maximum iterations for the 1D two-means refinement.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If the target column is not numeric.

harmonize_columns_by_average(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the average value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the group used for averaging.

• target_columns (Union[str, List[str]]) – Numeric column(s) to harmonize via averaging.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If any target column is not numeric.

harmonize_columns_by_majority_vote(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the majority value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the voting group (e.g., speaker).

• target_columns (Union[str, List[str]]) – Column or columns to harmonize via majority vote.

Raises:

KeyError – If the voter or target columns are missing.

head(n: int = 5) → T

Return the first n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

histogram(column: str, bins: int | None = None, density: bool = True, kind: str = 'bar', color: str = 'C0', output_file: str | Path | None = None, dropna: bool = True) → None

Plot a histogram of a column, handling string or numeric data.

Parameters:

• column (str) – Column name to plot. Title will include the column name.

• bins (Optional[int]) – Number of bins for numeric columns. Ignored for strings.

• density (bool) – If True, plot density/relative frequency; otherwise counts.

• kind (str) – Histogram style, either “bar” or “line”.

• color (str) – Matplotlib color spec for the plot.

• output_file (Optional[PathLike]) – If provided, save the figure; otherwise show it.

• dropna (bool) – If True, ignore NA values before plotting.

property iat: _InfoTableAtIndexer

Access a single value for a row/column integer position pair.

Returns:

Indexer that wraps .iat.

Return type:

_InfoTableAtIndexer

property iloc: _InfoTableIndexer

Access a group of rows and columns by integer position(s).

Returns:

Indexer that wraps .iloc.

Return type:

_InfoTableIndexer

property index: Index

Get the index of the DataFrame.

Returns:

DataFrame index.

Return type:

pd.Index

static is_valid_df(df: DataFrame) → bool

Check if the DataFrame is valid for InfoTable.

Parameters:

df (pd.DataFrame) – DataFrame to check.

Returns:

True if valid, False otherwise.

Return type:

bool

property iterrows: Callable

Return the DataFrame.iterrows generator.

Returns:

Yields (index, Series) pairs.

Return type:

Callable

legacy_add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Legacy implementation of add_columns that uses a full pandas merge.

This keeps the original behavior for compatibility/testing.

classmethod load(file_path: str | Path, sep: str | None = None) → T

Load a feature set from disk.

Parameters:

• file_path (PathLike) – Input file path.

• sep (Optional[str]) – Delimiter for non-.scp files.

Returns:

Loaded feature set.

Return type:

FeatureSet

property loc: _InfoTableIndexer

Access a group of rows and columns by label(s).

Returns:

Indexer that wraps .loc.

Return type:

_InfoTableIndexer

classmethod merge(left_table: T | DataFrame, right_table: T | DataFrame, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Tuple[str, str] = ('_x', '_y'), copy: bool | None = None, indicator: str | bool = False, validate: str | None = None) → T

Merge two InfoTables or DataFrames into a new InfoTable.

Parameters:

• left_table (Union[InfoTable, pd.DataFrame]) – Left-hand table.

• right_table (Union[InfoTable, pd.DataFrame]) – Right-hand table.

• how (str) – Merge method (e.g., ‘inner’, ‘outer’, ‘left’, ‘right’).

• on (Union[str, List[str], None]) – Column(s) to join on.

• left_on (Union[str, List[str], None]) – Column(s) from the left table to join on.

• right_on (Union[str, List[str], None]) – Column(s) from the right table to join on.

• left_index (bool) – Use index from the left table as join key.

• right_index (bool) – Use index from the right table as join key.

• sort (bool) – Sort the result by the join keys.

• suffixes (Tuple[str, str]) – Suffixes to apply to overlapping column names.

• copy (Optional[bool]) – If False, avoid copying data where possible.

• indicator (Union[str, bool]) – Adds a column to the output DataFrame called ‘_merge’.

• validate (Optional[str]) – Check if merge is of specified type.

Returns:

A new merged InfoTable.

Return type:

InfoTable

query(expr: str, **kwargs: Any) → T

Filters rows using a boolean expression string.

Parameters:

• expr (str) – A string expression to evaluate, using column names as variables.

• **kwargs – Passed through to pandas.DataFrame.query().

Returns:

A new InfoTable with filtered rows.

Return type:

InfoTable

replace(to_replace: Any = None, value: Any = None, inplace: bool = False, **kwargs: Any) → T | None

Replace values in the DataFrame.

Parameters:

• to_replace – What to replace.

• value – Value to replace with.

• inplace (bool) – Whether to modify the table in-place.

• **kwargs – Additional keyword args passed to pd.DataFrame.replace().

Returns:

New InfoTable if not inplace, else None.

Return type:

Optional[InfoTable]

replace_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None) → None

Replace column values with those from another table.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – Table to source values from.

• column_names (str or list, optional) – Columns to replace. If None, all.

reset_index() → None

Reset the DataFrame index to the ‘id’ column.

Returns:

None

sample(n: int = 1, random_state: int | RandomState | Generator | BitGenerator | None = None) → T

Return a random sample of rows.

Parameters:

• n (int) – Number of rows.

• random_state – Seed for reproducibility.

Returns:

Sampled InfoTable.

Return type:

InfoTable

save(file_path: str | Path, sep: str | None = None) → None

Save the feature set to disk.

Parameters:

• file_path (PathLike) – Output file path.

• sep (Optional[str]) – Delimiter for non-.scp files.

scatter2d(x_column: str, y_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 2D scattergram for two numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

scatter3d(x_column: str, y_column: str, z_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 3D scattergram for three numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• z_column (str) – Column name for the z-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

set_index(keys: str | List[str], inplace: bool = True) → T | None

Set the DataFrame index using one or more columns.

Parameters:

• keys (str or list) – Column(s) to use as index.

• inplace (bool) – Whether to modify in place.

Returns:

Modified InfoTable if not inplace.

Return type:

Optional[InfoTable]

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffle the rows of the InfoTable.

Parameters:

• seed (int) – Seed for random number generator.

• rng (np.random.Generator, optional) – Numpy random generator.

Returns:

Shuffled indices.

Return type:

np.ndarray

sort(column: str = 'id', ascending: bool = True, inplace: bool = True) → T | None

Sort the InfoTable by a specific column.

Parameters:

• column (str) – Column name to sort by.

• ascending (bool) – Sort in ascending order.

• inplace (bool) – Sort in place or return a new object.

Returns:

Sorted InfoTable or None if inplace.

Return type:

Optional[InfoTable]

split(idx: int, num_parts: int, group_by: str | None = None) → T

Split the InfoTable into parts and return the selected part.

Parameters:

• idx (int) – Part to return (1-based).

• num_parts (int) – Total number of parts.

• group_by (Optional[str]) – Column to group by when splitting.

Returns:

The selected part of the split.

Return type:

InfoTable

tail(n: int = 5) → T

Return the last n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

xs(key: Any, axis: int = 0, level: int | str = None, drop_level: bool = True) → T | Series

Returns a cross-section (row or column) from the DataFrame.

Parameters:

• key – Label or tuple of labels to select.

• axis (int) – Axis to retrieve from (0 for index, 1 for columns).

• level – Level in MultiIndex to use.

• drop_level (bool) – Whether to drop the level(s) from the result.

Returns:

If the result is a DataFrame, wraps it as InfoTable.

Return type:

InfoTable or Series

class hyperion.utils.ImageSet(df: DataFrame | T)

InfoTable specialization for image manifests.

The table must contain id and storage_path columns.

Examples

>>> import pandas as pd
>>> from hyperion.utils.image_set import ImageSet
>>> df = pd.DataFrame({"id": ["img1"], "storage_path": ["a/b/c.jpg"]})
>>> images = ImageSet(df)
>>> images.df.loc["img1", "storage_path"]
'a/b/c.jpg'
>>> images2 = images.filter(items=["img1"])
>>> list(images2.index)
['img1']

__init__(df: DataFrame | T) → None

Initialize an image set.

Parameters:

df (pd.DataFrame or ImageSet) – Input metadata table.

__cmp__(other: Any) → int

Comparison operator

__contains__(key: Any) → bool

Check whether key is in the DataFrame columns.

Parameters:

key – Key to check.

Returns:

True if key is in columns.

Return type:

bool

__eq__(other: Any) → bool

Equal operator

__getattr__(name: str) → Any

Provide attribute-style access to DataFrame columns and attributes.

Falls back to self.df[name] if name is a column. Falls back to getattr(self.df, name) if it’s a DataFrame method or attribute. Special methods (e.g., __setstate__) raise AttributeError immediately to allow correct behavior during pickling/unpickling.

Parameters:

name (str) – The attribute being accessed.

Returns:

The corresponding column, method, or attribute.

Return type:

Any

Raises:

AttributeError – If the attribute is not found.

__getitem__(key: Any) → T | Series

Get item from the internal DataFrame.

Parameters:

key – Key used for indexing.

Returns:

Sub-table or Series.

Return type:

Union[InfoTable, pd.Series]

__ne__(other: Any) → bool

Non-equal operator

__setitem__(key: Any, value: Any) → None

Set item in the internal DataFrame.

Parameters:

• key – Column label or index.

• value – Value to assign.

add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Add new columns from another InfoTable or DataFrame.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – The table to merge columns from.

• column_names (str or list, optional) – Columns to include from the right table.

• on (str or list) – Key(s) from the current table.

• right_on (str or list, optional) – Key(s) from the right table.

• replace_overlapping (bool) – Replace overlapping columns if True.

• ignore_overlapping (bool) – If True, skip adding columns that already exist.

• remove_missing (bool) – Use inner join (drop unmatched rows) if True.

property at: _InfoTableAtIndexer

Access a single value for a row/column label pair.

Returns:

Indexer that wraps .at.

Return type:

_InfoTableAtIndexer

classmethod cat(tables: List[T]) → T

Concatenate multiple InfoTables.

Parameters:

tables (List[InfoTable]) – List of InfoTable objects to concatenate.

Returns:

Concatenated InfoTable.

Return type:

InfoTable

Raises:

AssertionError – If resulting DataFrame has duplicate IDs.

clone() → T

Alias for copy().

Returns:

A deep copy.

Return type:

InfoTable

property columns: Index

Get the DataFrame columns.

Returns:

Column names.

Return type:

pd.Index

convert_col_to_str(column: str) → None

Ensure a specific column is of string type.

Parameters:

column (str) – Column name to convert.

copy() → T

Return a deep copy of the InfoTable.

Returns:

A deep copy.

Return type:

InfoTable

drop(labels: str | int | List[Any] | ndarray | Index | Series | None = None, axis: int | str = 0, index: str | int | List[Any] | ndarray | Index | Series | None = None, columns: str | int | List[Any] | ndarray | Index | Series | None = None, level: int | str | None = None, inplace: bool = False, errors: str = 'raise') → T | None

Drop specified labels.

Parameters:

• labels – Index or column labels.

• axis (int) – Whether to drop rows (0) or columns (1).

• index – Alias for labels along the index (rows).

• columns – Column labels to drop.

• level – Level in MultiIndex from which to drop.

• inplace (bool) – Modify in place.

• errors (str) – If ‘ignore’, suppress errors for nonexistent labels.

Returns:

Modified InfoTable or None.

Return type:

Optional[InfoTable]

dropna(*args: Any, **kwargs: Any) → T | None

Return a new InfoTable with missing values dropped.

Parameters:

• *args – Passed to pandas.DataFrame.dropna.

• **kwargs – Passed to pandas.DataFrame.dropna.

Returns:

A new instance with rows (or columns) with NA removed.

Return type:

InfoTable

property eval: Callable

Return the DataFrame.eval method.

Returns:

The eval method for evaluating expressions.

Return type:

Callable

filter(predicate: Callable[[DataFrame], Series | ndarray] | str | None = None, items: List[Any] | None = None, iindex: ndarray | None = None, columns: List[str] | None = None, by: str = 'id', keep: bool = True, raise_if_missing: bool = True) → T

Filter the InfoTable based on a predicate, item list, index list, or column subset.

Parameters:

• predicate (Callable, optional) – Function that returns a boolean mask, e.g.: lambda df: df[“duration”] > 1.0.

• items (List[Any], optional) – Items to include/exclude from the ‘by’ column like df.loc[items, by], used only if predicate is None

• iindex (np.ndarray, optional) – Integer indices to include/exclude like df.iloc[iindex], used if predicate and items are None

• columns (List[str], optional) – Columns to retain or remove.

• by (str) – Column name to use with ‘items’.

• keep (bool) – Whether to keep or exclude matched rows/columns.

• raise_if_missing (bool) – Raise error if items are missing.

Returns:

Filtered InfoTable.

Return type:

InfoTable

Raises:

Exception – If items are not found and raise_if_missing is True.

fix_dtypes() → None

Ensure the ‘id’ column is of string type.

classmethod from_dict(df_dict: Dict[str, List[Any]]) → T

Create InfoTable from a dictionary.

Parameters:

df_dict (Dict[str, List[Any]]) – Column data including ‘id’.

Returns:

Constructed table.

Return type:

InfoTable

classmethod from_lists(ids: List[str], column_names: List[str], column_data: List[List[Any]]) → T

Create InfoTable from lists of IDs and corresponding column data.

Parameters:

• ids (List[str]) – List of IDs.

• column_names (List[str]) – List of column names.

• column_data (List[List[Any]]) – Column values.

Returns:

Constructed table.

Return type:

InfoTable

get_col_idx(keys: str | List[str]) → int | ndarray

Get the integer index position(s) of the specified column(s).

Parameters:

keys (str or list) – Column name(s).

Returns:

Position(s) of the column(s).

Return type:

int or np.ndarray

get_loc(keys: str | List[str] | ndarray) → int | ndarray | List[int]

Get integer location(s) for the given key(s).

Parameters:

keys (str, list, or np.ndarray) – Index label(s).

Returns:

Location(s) in the index.

Return type:

int, np.ndarray, or List[int]

harmonize_age_given_decade(voter_columns: str | List[str], target_column: str, decade_column: str = 'age_decade') → None

Force all rows sharing voter_columns to take the averaged age in target_column, constrained by the decade labels in decade_column.

If the averaged age falls outside the decade bounds, the age is set to the upper boundary for that decade.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric age column to harmonize.

• decade_column (str) – Column indicating the age decade labels.

Raises:

• KeyError – If the voter, target, or decade columns are missing.

• TypeError – If the target column is not numeric.

harmonize_column_by_majority_cluster(voter_columns: str | List[str], target_column: str, suspect_column: str = 'suspect', std_threshold: float | None = None, max_iter: int = 20) → None

Harmonize a numeric column using the dominant cluster within each voter group, while flagging rows from the smaller cluster as suspect.

If std_threshold is provided, clustering is only performed when the group’s standard deviation exceeds the threshold.

The standard deviation of the dominant cluster is stored in a column named f"{target_column}_std" for the non-suspect rows.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric column to harmonize.

• suspect_column (str) – Column used to flag suspect rows.

• std_threshold (Optional[float]) – If set, only split groups above this std.

• max_iter (int) – Maximum iterations for the 1D two-means refinement.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If the target column is not numeric.

harmonize_columns_by_average(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the average value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the group used for averaging.

• target_columns (Union[str, List[str]]) – Numeric column(s) to harmonize via averaging.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If any target column is not numeric.

harmonize_columns_by_majority_vote(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the majority value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the voting group (e.g., speaker).

• target_columns (Union[str, List[str]]) – Column or columns to harmonize via majority vote.

Raises:

KeyError – If the voter or target columns are missing.

head(n: int = 5) → T

Return the first n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

histogram(column: str, bins: int | None = None, density: bool = True, kind: str = 'bar', color: str = 'C0', output_file: str | Path | None = None, dropna: bool = True) → None

Plot a histogram of a column, handling string or numeric data.

Parameters:

• column (str) – Column name to plot. Title will include the column name.

• bins (Optional[int]) – Number of bins for numeric columns. Ignored for strings.

• density (bool) – If True, plot density/relative frequency; otherwise counts.

• kind (str) – Histogram style, either “bar” or “line”.

• color (str) – Matplotlib color spec for the plot.

• output_file (Optional[PathLike]) – If provided, save the figure; otherwise show it.

• dropna (bool) – If True, ignore NA values before plotting.

property iat: _InfoTableAtIndexer

Access a single value for a row/column integer position pair.

Returns:

Indexer that wraps .iat.

Return type:

_InfoTableAtIndexer

property iloc: _InfoTableIndexer

Access a group of rows and columns by integer position(s).

Returns:

Indexer that wraps .iloc.

Return type:

_InfoTableIndexer

property index: Index

Get the index of the DataFrame.

Returns:

DataFrame index.

Return type:

pd.Index

static is_valid_df(df: DataFrame) → bool

Check if the DataFrame is valid for InfoTable.

Parameters:

df (pd.DataFrame) – DataFrame to check.

Returns:

True if valid, False otherwise.

Return type:

bool

property iterrows: Callable

Return the DataFrame.iterrows generator.

Returns:

Yields (index, Series) pairs.

Return type:

Callable

legacy_add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Legacy implementation of add_columns that uses a full pandas merge.

This keeps the original behavior for compatibility/testing.

classmethod load(file_path: str | Path, sep: str | None = None, name: str = 'class_id') → T

Load InfoTable from file.

Parameters:

• file_path (str or Path) – Path to the input file.

• sep (Optional[str]) – Column separator.

• name (str) – Name of the second column (used for Kaldi format).

Returns:

Loaded table.

Return type:

InfoTable

property loc: _InfoTableIndexer

Access a group of rows and columns by label(s).

Returns:

Indexer that wraps .loc.

Return type:

_InfoTableIndexer

classmethod merge(left_table: T | DataFrame, right_table: T | DataFrame, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Tuple[str, str] = ('_x', '_y'), copy: bool | None = None, indicator: str | bool = False, validate: str | None = None) → T

Merge two InfoTables or DataFrames into a new InfoTable.

Parameters:

• left_table (Union[InfoTable, pd.DataFrame]) – Left-hand table.

• right_table (Union[InfoTable, pd.DataFrame]) – Right-hand table.

• how (str) – Merge method (e.g., ‘inner’, ‘outer’, ‘left’, ‘right’).

• on (Union[str, List[str], None]) – Column(s) to join on.

• left_on (Union[str, List[str], None]) – Column(s) from the left table to join on.

• right_on (Union[str, List[str], None]) – Column(s) from the right table to join on.

• left_index (bool) – Use index from the left table as join key.

• right_index (bool) – Use index from the right table as join key.

• sort (bool) – Sort the result by the join keys.

• suffixes (Tuple[str, str]) – Suffixes to apply to overlapping column names.

• copy (Optional[bool]) – If False, avoid copying data where possible.

• indicator (Union[str, bool]) – Adds a column to the output DataFrame called ‘_merge’.

• validate (Optional[str]) – Check if merge is of specified type.

Returns:

A new merged InfoTable.

Return type:

InfoTable

query(expr: str, **kwargs: Any) → T

Filters rows using a boolean expression string.

Parameters:

• expr (str) – A string expression to evaluate, using column names as variables.

• **kwargs – Passed through to pandas.DataFrame.query().

Returns:

A new InfoTable with filtered rows.

Return type:

InfoTable

replace(to_replace: Any = None, value: Any = None, inplace: bool = False, **kwargs: Any) → T | None

Replace values in the DataFrame.

Parameters:

• to_replace – What to replace.

• value – Value to replace with.

• inplace (bool) – Whether to modify the table in-place.

• **kwargs – Additional keyword args passed to pd.DataFrame.replace().

Returns:

New InfoTable if not inplace, else None.

Return type:

Optional[InfoTable]

replace_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None) → None

Replace column values with those from another table.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – Table to source values from.

• column_names (str or list, optional) – Columns to replace. If None, all.

reset_index() → None

Reset the DataFrame index to the ‘id’ column.

Returns:

None

sample(n: int = 1, random_state: int | RandomState | Generator | BitGenerator | None = None) → T

Return a random sample of rows.

Parameters:

• n (int) – Number of rows.

• random_state – Seed for reproducibility.

Returns:

Sampled InfoTable.

Return type:

InfoTable

save(file_path: str | Path, sep: str | None = None) → None

Save the InfoTable to a file.

Parameters:

• file_path (str or Path) – Path to save the file.

• sep (Optional[str]) – Column separator (default inferred from extension).

scatter2d(x_column: str, y_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 2D scattergram for two numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

scatter3d(x_column: str, y_column: str, z_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 3D scattergram for three numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• z_column (str) – Column name for the z-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

set_index(keys: str | List[str], inplace: bool = True) → T | None

Set the DataFrame index using one or more columns.

Parameters:

• keys (str or list) – Column(s) to use as index.

• inplace (bool) – Whether to modify in place.

Returns:

Modified InfoTable if not inplace.

Return type:

Optional[InfoTable]

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffle the rows of the InfoTable.

Parameters:

• seed (int) – Seed for random number generator.

• rng (np.random.Generator, optional) – Numpy random generator.

Returns:

Shuffled indices.

Return type:

np.ndarray

sort(column: str = 'id', ascending: bool = True, inplace: bool = True) → T | None

Sort the InfoTable by a specific column.

Parameters:

• column (str) – Column name to sort by.

• ascending (bool) – Sort in ascending order.

• inplace (bool) – Sort in place or return a new object.

Returns:

Sorted InfoTable or None if inplace.

Return type:

Optional[InfoTable]

split(idx: int, num_parts: int, group_by: str | None = None) → T

Split the InfoTable into parts and return the selected part.

Parameters:

• idx (int) – Part to return (1-based).

• num_parts (int) – Total number of parts.

• group_by (Optional[str]) – Column to group by when splitting.

Returns:

The selected part of the split.

Return type:

InfoTable

tail(n: int = 5) → T

Return the last n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

xs(key: Any, axis: int = 0, level: int | str = None, drop_level: bool = True) → T | Series

Returns a cross-section (row or column) from the DataFrame.

Parameters:

• key – Label or tuple of labels to select.

• axis (int) – Axis to retrieve from (0 for index, 1 for columns).

• level – Level in MultiIndex to use.

• drop_level (bool) – Whether to drop the level(s) from the result.

Returns:

If the result is a DataFrame, wraps it as InfoTable.

Return type:

InfoTable or Series

class hyperion.utils.VideoSet(df: DataFrame | T)

InfoTable specialization for audiovisual recording manifests.

The table must contain id and storage_path columns.

Examples

>>> import pandas as pd
>>> from hyperion.utils.video_set import VideoSet
>>> df = pd.DataFrame({"id": ["vid1"], "storage_path": ["/video/vid1.mp4"]})
>>> videos = VideoSet(df)
>>> videos.df.loc["vid1", "storage_path"]
'/video/vid1.mp4'
>>> videos2 = videos.filter(items=["vid1"])
>>> len(videos2)
1

__init__(df: DataFrame | T) → None

Initialize a video set.

Parameters:

df (pd.DataFrame or VideoSet) – Input metadata table.

static _get_metadata(videos: VideoSet, i: int, n: int) → Tuple[List[int], List[float], List[float], List[float]]

Collect audio/video metadata for one data partition.

Parameters:

• videos (VideoSet) – Source table.

• i (int) – 1-based partition index.

• n (int) – Number of partitions.

Returns:

Sample rates, audio durations, FPS values, and video durations.

Return type:

Tuple[List[int], List[float], List[float], List[float]]

get_metadata(num_threads: int = 16) → None

Populate duration and frame-rate metadata using a thread pool.

Parameters:

num_threads (int) – Maximum number of worker threads.

__cmp__(other: Any) → int

Comparison operator

__contains__(key: Any) → bool

Check whether key is in the DataFrame columns.

Parameters:

key – Key to check.

Returns:

True if key is in columns.

Return type:

bool

__eq__(other: Any) → bool

Equal operator

__getattr__(name: str) → Any

Provide attribute-style access to DataFrame columns and attributes.

Falls back to self.df[name] if name is a column. Falls back to getattr(self.df, name) if it’s a DataFrame method or attribute. Special methods (e.g., __setstate__) raise AttributeError immediately to allow correct behavior during pickling/unpickling.

Parameters:

name (str) – The attribute being accessed.

Returns:

The corresponding column, method, or attribute.

Return type:

Any

Raises:

AttributeError – If the attribute is not found.

__getitem__(key: Any) → T | Series

Get item from the internal DataFrame.

Parameters:

key – Key used for indexing.

Returns:

Sub-table or Series.

Return type:

Union[InfoTable, pd.Series]

__ne__(other: Any) → bool

Non-equal operator

__setitem__(key: Any, value: Any) → None

Set item in the internal DataFrame.

Parameters:

• key – Column label or index.

• value – Value to assign.

add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Add new columns from another InfoTable or DataFrame.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – The table to merge columns from.

• column_names (str or list, optional) – Columns to include from the right table.

• on (str or list) – Key(s) from the current table.

• right_on (str or list, optional) – Key(s) from the right table.

• replace_overlapping (bool) – Replace overlapping columns if True.

• ignore_overlapping (bool) – If True, skip adding columns that already exist.

• remove_missing (bool) – Use inner join (drop unmatched rows) if True.

property at: _InfoTableAtIndexer

Access a single value for a row/column label pair.

Returns:

Indexer that wraps .at.

Return type:

_InfoTableAtIndexer

classmethod cat(tables: List[T]) → T

Concatenate multiple InfoTables.

Parameters:

tables (List[InfoTable]) – List of InfoTable objects to concatenate.

Returns:

Concatenated InfoTable.

Return type:

InfoTable

Raises:

AssertionError – If resulting DataFrame has duplicate IDs.

clone() → T

Alias for copy().

Returns:

A deep copy.

Return type:

InfoTable

property columns: Index

Get the DataFrame columns.

Returns:

Column names.

Return type:

pd.Index

convert_col_to_str(column: str) → None

Ensure a specific column is of string type.

Parameters:

column (str) – Column name to convert.

copy() → T

Return a deep copy of the InfoTable.

Returns:

A deep copy.

Return type:

InfoTable

drop(labels: str | int | List[Any] | ndarray | Index | Series | None = None, axis: int | str = 0, index: str | int | List[Any] | ndarray | Index | Series | None = None, columns: str | int | List[Any] | ndarray | Index | Series | None = None, level: int | str | None = None, inplace: bool = False, errors: str = 'raise') → T | None

Drop specified labels.

Parameters:

• labels – Index or column labels.

• axis (int) – Whether to drop rows (0) or columns (1).

• index – Alias for labels along the index (rows).

• columns – Column labels to drop.

• level – Level in MultiIndex from which to drop.

• inplace (bool) – Modify in place.

• errors (str) – If ‘ignore’, suppress errors for nonexistent labels.

Returns:

Modified InfoTable or None.

Return type:

Optional[InfoTable]

dropna(*args: Any, **kwargs: Any) → T | None

Return a new InfoTable with missing values dropped.

Parameters:

• *args – Passed to pandas.DataFrame.dropna.

• **kwargs – Passed to pandas.DataFrame.dropna.

Returns:

A new instance with rows (or columns) with NA removed.

Return type:

InfoTable

property eval: Callable

Return the DataFrame.eval method.

Returns:

The eval method for evaluating expressions.

Return type:

Callable

filter(predicate: Callable[[DataFrame], Series | ndarray] | str | None = None, items: List[Any] | None = None, iindex: ndarray | None = None, columns: List[str] | None = None, by: str = 'id', keep: bool = True, raise_if_missing: bool = True) → T

Filter the InfoTable based on a predicate, item list, index list, or column subset.

Parameters:

• predicate (Callable, optional) – Function that returns a boolean mask, e.g.: lambda df: df[“duration”] > 1.0.

• items (List[Any], optional) – Items to include/exclude from the ‘by’ column like df.loc[items, by], used only if predicate is None

• iindex (np.ndarray, optional) – Integer indices to include/exclude like df.iloc[iindex], used if predicate and items are None

• columns (List[str], optional) – Columns to retain or remove.

• by (str) – Column name to use with ‘items’.

• keep (bool) – Whether to keep or exclude matched rows/columns.

• raise_if_missing (bool) – Raise error if items are missing.

Returns:

Filtered InfoTable.

Return type:

InfoTable

Raises:

Exception – If items are not found and raise_if_missing is True.

fix_dtypes() → None

Ensure the ‘id’ column is of string type.

classmethod from_dict(df_dict: Dict[str, List[Any]]) → T

Create InfoTable from a dictionary.

Parameters:

df_dict (Dict[str, List[Any]]) – Column data including ‘id’.

Returns:

Constructed table.

Return type:

InfoTable

classmethod from_lists(ids: List[str], column_names: List[str], column_data: List[List[Any]]) → T

Create InfoTable from lists of IDs and corresponding column data.

Parameters:

• ids (List[str]) – List of IDs.

• column_names (List[str]) – List of column names.

• column_data (List[List[Any]]) – Column values.

Returns:

Constructed table.

Return type:

InfoTable

get_col_idx(keys: str | List[str]) → int | ndarray

Get the integer index position(s) of the specified column(s).

Parameters:

keys (str or list) – Column name(s).

Returns:

Position(s) of the column(s).

Return type:

int or np.ndarray

get_loc(keys: str | List[str] | ndarray) → int | ndarray | List[int]

Get integer location(s) for the given key(s).

Parameters:

keys (str, list, or np.ndarray) – Index label(s).

Returns:

Location(s) in the index.

Return type:

int, np.ndarray, or List[int]

harmonize_age_given_decade(voter_columns: str | List[str], target_column: str, decade_column: str = 'age_decade') → None

Force all rows sharing voter_columns to take the averaged age in target_column, constrained by the decade labels in decade_column.

If the averaged age falls outside the decade bounds, the age is set to the upper boundary for that decade.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric age column to harmonize.

• decade_column (str) – Column indicating the age decade labels.

Raises:

• KeyError – If the voter, target, or decade columns are missing.

• TypeError – If the target column is not numeric.

harmonize_column_by_majority_cluster(voter_columns: str | List[str], target_column: str, suspect_column: str = 'suspect', std_threshold: float | None = None, max_iter: int = 20) → None

Harmonize a numeric column using the dominant cluster within each voter group, while flagging rows from the smaller cluster as suspect.

If std_threshold is provided, clustering is only performed when the group’s standard deviation exceeds the threshold.

The standard deviation of the dominant cluster is stored in a column named f"{target_column}_std" for the non-suspect rows.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric column to harmonize.

• suspect_column (str) – Column used to flag suspect rows.

• std_threshold (Optional[float]) – If set, only split groups above this std.

• max_iter (int) – Maximum iterations for the 1D two-means refinement.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If the target column is not numeric.

harmonize_columns_by_average(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the average value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the group used for averaging.

• target_columns (Union[str, List[str]]) – Numeric column(s) to harmonize via averaging.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If any target column is not numeric.

harmonize_columns_by_majority_vote(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the majority value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the voting group (e.g., speaker).

• target_columns (Union[str, List[str]]) – Column or columns to harmonize via majority vote.

Raises:

KeyError – If the voter or target columns are missing.

head(n: int = 5) → T

Return the first n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

histogram(column: str, bins: int | None = None, density: bool = True, kind: str = 'bar', color: str = 'C0', output_file: str | Path | None = None, dropna: bool = True) → None

Plot a histogram of a column, handling string or numeric data.

Parameters:

• column (str) – Column name to plot. Title will include the column name.

• bins (Optional[int]) – Number of bins for numeric columns. Ignored for strings.

• density (bool) – If True, plot density/relative frequency; otherwise counts.

• kind (str) – Histogram style, either “bar” or “line”.

• color (str) – Matplotlib color spec for the plot.

• output_file (Optional[PathLike]) – If provided, save the figure; otherwise show it.

• dropna (bool) – If True, ignore NA values before plotting.

property iat: _InfoTableAtIndexer

Access a single value for a row/column integer position pair.

Returns:

Indexer that wraps .iat.

Return type:

_InfoTableAtIndexer

property iloc: _InfoTableIndexer

Access a group of rows and columns by integer position(s).

Returns:

Indexer that wraps .iloc.

Return type:

_InfoTableIndexer

property index: Index

Get the index of the DataFrame.

Returns:

DataFrame index.

Return type:

pd.Index

static is_valid_df(df: DataFrame) → bool

Check if the DataFrame is valid for InfoTable.

Parameters:

df (pd.DataFrame) – DataFrame to check.

Returns:

True if valid, False otherwise.

Return type:

bool

property iterrows: Callable

Return the DataFrame.iterrows generator.

Returns:

Yields (index, Series) pairs.

Return type:

Callable

legacy_add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Legacy implementation of add_columns that uses a full pandas merge.

This keeps the original behavior for compatibility/testing.

classmethod load(file_path: str | Path, sep: str | None = None, name: str = 'class_id') → T

Load InfoTable from file.

Parameters:

• file_path (str or Path) – Path to the input file.

• sep (Optional[str]) – Column separator.

• name (str) – Name of the second column (used for Kaldi format).

Returns:

Loaded table.

Return type:

InfoTable

property loc: _InfoTableIndexer

Access a group of rows and columns by label(s).

Returns:

Indexer that wraps .loc.

Return type:

_InfoTableIndexer

classmethod merge(left_table: T | DataFrame, right_table: T | DataFrame, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Tuple[str, str] = ('_x', '_y'), copy: bool | None = None, indicator: str | bool = False, validate: str | None = None) → T

Merge two InfoTables or DataFrames into a new InfoTable.

Parameters:

• left_table (Union[InfoTable, pd.DataFrame]) – Left-hand table.

• right_table (Union[InfoTable, pd.DataFrame]) – Right-hand table.

• how (str) – Merge method (e.g., ‘inner’, ‘outer’, ‘left’, ‘right’).

• on (Union[str, List[str], None]) – Column(s) to join on.

• left_on (Union[str, List[str], None]) – Column(s) from the left table to join on.

• right_on (Union[str, List[str], None]) – Column(s) from the right table to join on.

• left_index (bool) – Use index from the left table as join key.

• right_index (bool) – Use index from the right table as join key.

• sort (bool) – Sort the result by the join keys.

• suffixes (Tuple[str, str]) – Suffixes to apply to overlapping column names.

• copy (Optional[bool]) – If False, avoid copying data where possible.

• indicator (Union[str, bool]) – Adds a column to the output DataFrame called ‘_merge’.

• validate (Optional[str]) – Check if merge is of specified type.

Returns:

A new merged InfoTable.

Return type:

InfoTable

query(expr: str, **kwargs: Any) → T

Filters rows using a boolean expression string.

Parameters:

• expr (str) – A string expression to evaluate, using column names as variables.

• **kwargs – Passed through to pandas.DataFrame.query().

Returns:

A new InfoTable with filtered rows.

Return type:

InfoTable

replace(to_replace: Any = None, value: Any = None, inplace: bool = False, **kwargs: Any) → T | None

Replace values in the DataFrame.

Parameters:

• to_replace – What to replace.

• value – Value to replace with.

• inplace (bool) – Whether to modify the table in-place.

• **kwargs – Additional keyword args passed to pd.DataFrame.replace().

Returns:

New InfoTable if not inplace, else None.

Return type:

Optional[InfoTable]

replace_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None) → None

Replace column values with those from another table.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – Table to source values from.

• column_names (str or list, optional) – Columns to replace. If None, all.

reset_index() → None

Reset the DataFrame index to the ‘id’ column.

Returns:

None

sample(n: int = 1, random_state: int | RandomState | Generator | BitGenerator | None = None) → T

Return a random sample of rows.

Parameters:

• n (int) – Number of rows.

• random_state – Seed for reproducibility.

Returns:

Sampled InfoTable.

Return type:

InfoTable

save(file_path: str | Path, sep: str | None = None) → None

Save the InfoTable to a file.

Parameters:

• file_path (str or Path) – Path to save the file.

• sep (Optional[str]) – Column separator (default inferred from extension).

scatter2d(x_column: str, y_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 2D scattergram for two numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

scatter3d(x_column: str, y_column: str, z_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 3D scattergram for three numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• z_column (str) – Column name for the z-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

set_index(keys: str | List[str], inplace: bool = True) → T | None

Set the DataFrame index using one or more columns.

Parameters:

• keys (str or list) – Column(s) to use as index.

• inplace (bool) – Whether to modify in place.

Returns:

Modified InfoTable if not inplace.

Return type:

Optional[InfoTable]

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffle the rows of the InfoTable.

Parameters:

• seed (int) – Seed for random number generator.

• rng (np.random.Generator, optional) – Numpy random generator.

Returns:

Shuffled indices.

Return type:

np.ndarray

sort(column: str = 'id', ascending: bool = True, inplace: bool = True) → T | None

Sort the InfoTable by a specific column.

Parameters:

• column (str) – Column name to sort by.

• ascending (bool) – Sort in ascending order.

• inplace (bool) – Sort in place or return a new object.

Returns:

Sorted InfoTable or None if inplace.

Return type:

Optional[InfoTable]

split(idx: int, num_parts: int, group_by: str | None = None) → T

Split the InfoTable into parts and return the selected part.

Parameters:

• idx (int) – Part to return (1-based).

• num_parts (int) – Total number of parts.

• group_by (Optional[str]) – Column to group by when splitting.

Returns:

The selected part of the split.

Return type:

InfoTable

tail(n: int = 5) → T

Return the last n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

xs(key: Any, axis: int = 0, level: int | str = None, drop_level: bool = True) → T | Series

Returns a cross-section (row or column) from the DataFrame.

Parameters:

• key – Label or tuple of labels to select.

• axis (int) – Axis to retrieve from (0 for index, 1 for columns).

• level – Level in MultiIndex to use.

• drop_level (bool) – Whether to drop the level(s) from the result.

Returns:

If the result is a DataFrame, wraps it as InfoTable.

Return type:

InfoTable or Series

class hyperion.utils.DiarizationSet(df: DataFrame | T)

InfoTable specialization for diarization-related manifests.

The table must contain id (from InfoTable) and storage_path.

Examples

>>> import pandas as pd
>>> from hyperion.utils.diarization_set import DiarizationSet
>>> df = pd.DataFrame({"id": ["utt1"], "storage_path": ["seg1.rttm"]})
>>> diar = DiarizationSet(df)
>>> diar.add_prefix_to_storage_path("/data/")
>>> diar.df.loc["utt1", "storage_path"]
'/data/seg1.rttm'
>>> diar2 = diar.copy()
>>> len(diar2)
1

__init__(df: DataFrame | T) → None

Initialize a diarization set.

Parameters:

df (pd.DataFrame or DiarizationSet) – Input metadata table.

add_prefix_to_storage_path(prefix: str | Path) → None

Prefix values in the storage_path column.

Parameters:

prefix (PathLike) – Prefix prepended to each value in storage_path.

__cmp__(other: Any) → int

Comparison operator

__contains__(key: Any) → bool

Check whether key is in the DataFrame columns.

Parameters:

key – Key to check.

Returns:

True if key is in columns.

Return type:

bool

__eq__(other: Any) → bool

Equal operator

__getattr__(name: str) → Any

Provide attribute-style access to DataFrame columns and attributes.

Falls back to self.df[name] if name is a column. Falls back to getattr(self.df, name) if it’s a DataFrame method or attribute. Special methods (e.g., __setstate__) raise AttributeError immediately to allow correct behavior during pickling/unpickling.

Parameters:

name (str) – The attribute being accessed.

Returns:

The corresponding column, method, or attribute.

Return type:

Any

Raises:

AttributeError – If the attribute is not found.

__getitem__(key: Any) → T | Series

Get item from the internal DataFrame.

Parameters:

key – Key used for indexing.

Returns:

Sub-table or Series.

Return type:

Union[InfoTable, pd.Series]

__ne__(other: Any) → bool

Non-equal operator

__setitem__(key: Any, value: Any) → None

Set item in the internal DataFrame.

Parameters:

• key – Column label or index.

• value – Value to assign.

add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Add new columns from another InfoTable or DataFrame.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – The table to merge columns from.

• column_names (str or list, optional) – Columns to include from the right table.

• on (str or list) – Key(s) from the current table.

• right_on (str or list, optional) – Key(s) from the right table.

• replace_overlapping (bool) – Replace overlapping columns if True.

• ignore_overlapping (bool) – If True, skip adding columns that already exist.

• remove_missing (bool) – Use inner join (drop unmatched rows) if True.

property at: _InfoTableAtIndexer

Access a single value for a row/column label pair.

Returns:

Indexer that wraps .at.

Return type:

_InfoTableAtIndexer

classmethod cat(tables: List[T]) → T

Concatenate multiple InfoTables.

Parameters:

tables (List[InfoTable]) – List of InfoTable objects to concatenate.

Returns:

Concatenated InfoTable.

Return type:

InfoTable

Raises:

AssertionError – If resulting DataFrame has duplicate IDs.

clone() → T

Alias for copy().

Returns:

A deep copy.

Return type:

InfoTable

property columns: Index

Get the DataFrame columns.

Returns:

Column names.

Return type:

pd.Index

convert_col_to_str(column: str) → None

Ensure a specific column is of string type.

Parameters:

column (str) – Column name to convert.

copy() → T

Return a deep copy of the InfoTable.

Returns:

A deep copy.

Return type:

InfoTable

drop(labels: str | int | List[Any] | ndarray | Index | Series | None = None, axis: int | str = 0, index: str | int | List[Any] | ndarray | Index | Series | None = None, columns: str | int | List[Any] | ndarray | Index | Series | None = None, level: int | str | None = None, inplace: bool = False, errors: str = 'raise') → T | None

Drop specified labels.

Parameters:

• labels – Index or column labels.

• axis (int) – Whether to drop rows (0) or columns (1).

• index – Alias for labels along the index (rows).

• columns – Column labels to drop.

• level – Level in MultiIndex from which to drop.

• inplace (bool) – Modify in place.

• errors (str) – If ‘ignore’, suppress errors for nonexistent labels.

Returns:

Modified InfoTable or None.

Return type:

Optional[InfoTable]

dropna(*args: Any, **kwargs: Any) → T | None

Return a new InfoTable with missing values dropped.

Parameters:

• *args – Passed to pandas.DataFrame.dropna.

• **kwargs – Passed to pandas.DataFrame.dropna.

Returns:

A new instance with rows (or columns) with NA removed.

Return type:

InfoTable

property eval: Callable

Return the DataFrame.eval method.

Returns:

The eval method for evaluating expressions.

Return type:

Callable

filter(predicate: Callable[[DataFrame], Series | ndarray] | str | None = None, items: List[Any] | None = None, iindex: ndarray | None = None, columns: List[str] | None = None, by: str = 'id', keep: bool = True, raise_if_missing: bool = True) → T

Filter the InfoTable based on a predicate, item list, index list, or column subset.

Parameters:

• predicate (Callable, optional) – Function that returns a boolean mask, e.g.: lambda df: df[“duration”] > 1.0.

• items (List[Any], optional) – Items to include/exclude from the ‘by’ column like df.loc[items, by], used only if predicate is None

• iindex (np.ndarray, optional) – Integer indices to include/exclude like df.iloc[iindex], used if predicate and items are None

• columns (List[str], optional) – Columns to retain or remove.

• by (str) – Column name to use with ‘items’.

• keep (bool) – Whether to keep or exclude matched rows/columns.

• raise_if_missing (bool) – Raise error if items are missing.

Returns:

Filtered InfoTable.

Return type:

InfoTable

Raises:

Exception – If items are not found and raise_if_missing is True.

fix_dtypes() → None

Ensure the ‘id’ column is of string type.

classmethod from_dict(df_dict: Dict[str, List[Any]]) → T

Create InfoTable from a dictionary.

Parameters:

df_dict (Dict[str, List[Any]]) – Column data including ‘id’.

Returns:

Constructed table.

Return type:

InfoTable

classmethod from_lists(ids: List[str], column_names: List[str], column_data: List[List[Any]]) → T

Create InfoTable from lists of IDs and corresponding column data.

Parameters:

• ids (List[str]) – List of IDs.

• column_names (List[str]) – List of column names.

• column_data (List[List[Any]]) – Column values.

Returns:

Constructed table.

Return type:

InfoTable

get_col_idx(keys: str | List[str]) → int | ndarray

Get the integer index position(s) of the specified column(s).

Parameters:

keys (str or list) – Column name(s).

Returns:

Position(s) of the column(s).

Return type:

int or np.ndarray

get_loc(keys: str | List[str] | ndarray) → int | ndarray | List[int]

Get integer location(s) for the given key(s).

Parameters:

keys (str, list, or np.ndarray) – Index label(s).

Returns:

Location(s) in the index.

Return type:

int, np.ndarray, or List[int]

harmonize_age_given_decade(voter_columns: str | List[str], target_column: str, decade_column: str = 'age_decade') → None

Force all rows sharing voter_columns to take the averaged age in target_column, constrained by the decade labels in decade_column.

If the averaged age falls outside the decade bounds, the age is set to the upper boundary for that decade.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric age column to harmonize.

• decade_column (str) – Column indicating the age decade labels.

Raises:

• KeyError – If the voter, target, or decade columns are missing.

• TypeError – If the target column is not numeric.

harmonize_column_by_majority_cluster(voter_columns: str | List[str], target_column: str, suspect_column: str = 'suspect', std_threshold: float | None = None, max_iter: int = 20) → None

Harmonize a numeric column using the dominant cluster within each voter group, while flagging rows from the smaller cluster as suspect.

If std_threshold is provided, clustering is only performed when the group’s standard deviation exceeds the threshold.

The standard deviation of the dominant cluster is stored in a column named f"{target_column}_std" for the non-suspect rows.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric column to harmonize.

• suspect_column (str) – Column used to flag suspect rows.

• std_threshold (Optional[float]) – If set, only split groups above this std.

• max_iter (int) – Maximum iterations for the 1D two-means refinement.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If the target column is not numeric.

harmonize_columns_by_average(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the average value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the group used for averaging.

• target_columns (Union[str, List[str]]) – Numeric column(s) to harmonize via averaging.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If any target column is not numeric.

harmonize_columns_by_majority_vote(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the majority value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the voting group (e.g., speaker).

• target_columns (Union[str, List[str]]) – Column or columns to harmonize via majority vote.

Raises:

KeyError – If the voter or target columns are missing.

head(n: int = 5) → T

Return the first n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

histogram(column: str, bins: int | None = None, density: bool = True, kind: str = 'bar', color: str = 'C0', output_file: str | Path | None = None, dropna: bool = True) → None

Plot a histogram of a column, handling string or numeric data.

Parameters:

• column (str) – Column name to plot. Title will include the column name.

• bins (Optional[int]) – Number of bins for numeric columns. Ignored for strings.

• density (bool) – If True, plot density/relative frequency; otherwise counts.

• kind (str) – Histogram style, either “bar” or “line”.

• color (str) – Matplotlib color spec for the plot.

• output_file (Optional[PathLike]) – If provided, save the figure; otherwise show it.

• dropna (bool) – If True, ignore NA values before plotting.

property iat: _InfoTableAtIndexer

Access a single value for a row/column integer position pair.

Returns:

Indexer that wraps .iat.

Return type:

_InfoTableAtIndexer

property iloc: _InfoTableIndexer

Access a group of rows and columns by integer position(s).

Returns:

Indexer that wraps .iloc.

Return type:

_InfoTableIndexer

property index: Index

Get the index of the DataFrame.

Returns:

DataFrame index.

Return type:

pd.Index

static is_valid_df(df: DataFrame) → bool

Check if the DataFrame is valid for InfoTable.

Parameters:

df (pd.DataFrame) – DataFrame to check.

Returns:

True if valid, False otherwise.

Return type:

bool

property iterrows: Callable

Return the DataFrame.iterrows generator.

Returns:

Yields (index, Series) pairs.

Return type:

Callable

legacy_add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Legacy implementation of add_columns that uses a full pandas merge.

This keeps the original behavior for compatibility/testing.

classmethod load(file_path: str | Path, sep: str | None = None, name: str = 'class_id') → T

Load InfoTable from file.

Parameters:

• file_path (str or Path) – Path to the input file.

• sep (Optional[str]) – Column separator.

• name (str) – Name of the second column (used for Kaldi format).

Returns:

Loaded table.

Return type:

InfoTable

property loc: _InfoTableIndexer

Access a group of rows and columns by label(s).

Returns:

Indexer that wraps .loc.

Return type:

_InfoTableIndexer

classmethod merge(left_table: T | DataFrame, right_table: T | DataFrame, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Tuple[str, str] = ('_x', '_y'), copy: bool | None = None, indicator: str | bool = False, validate: str | None = None) → T

Merge two InfoTables or DataFrames into a new InfoTable.

Parameters:

• left_table (Union[InfoTable, pd.DataFrame]) – Left-hand table.

• right_table (Union[InfoTable, pd.DataFrame]) – Right-hand table.

• how (str) – Merge method (e.g., ‘inner’, ‘outer’, ‘left’, ‘right’).

• on (Union[str, List[str], None]) – Column(s) to join on.

• left_on (Union[str, List[str], None]) – Column(s) from the left table to join on.

• right_on (Union[str, List[str], None]) – Column(s) from the right table to join on.

• left_index (bool) – Use index from the left table as join key.

• right_index (bool) – Use index from the right table as join key.

• sort (bool) – Sort the result by the join keys.

• suffixes (Tuple[str, str]) – Suffixes to apply to overlapping column names.

• copy (Optional[bool]) – If False, avoid copying data where possible.

• indicator (Union[str, bool]) – Adds a column to the output DataFrame called ‘_merge’.

• validate (Optional[str]) – Check if merge is of specified type.

Returns:

A new merged InfoTable.

Return type:

InfoTable

query(expr: str, **kwargs: Any) → T

Filters rows using a boolean expression string.

Parameters:

• expr (str) – A string expression to evaluate, using column names as variables.

• **kwargs – Passed through to pandas.DataFrame.query().

Returns:

A new InfoTable with filtered rows.

Return type:

InfoTable

replace(to_replace: Any = None, value: Any = None, inplace: bool = False, **kwargs: Any) → T | None

Replace values in the DataFrame.

Parameters:

• to_replace – What to replace.

• value – Value to replace with.

• inplace (bool) – Whether to modify the table in-place.

• **kwargs – Additional keyword args passed to pd.DataFrame.replace().

Returns:

New InfoTable if not inplace, else None.

Return type:

Optional[InfoTable]

replace_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None) → None

Replace column values with those from another table.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – Table to source values from.

• column_names (str or list, optional) – Columns to replace. If None, all.

reset_index() → None

Reset the DataFrame index to the ‘id’ column.

Returns:

None

sample(n: int = 1, random_state: int | RandomState | Generator | BitGenerator | None = None) → T

Return a random sample of rows.

Parameters:

• n (int) – Number of rows.

• random_state – Seed for reproducibility.

Returns:

Sampled InfoTable.

Return type:

InfoTable

save(file_path: str | Path, sep: str | None = None) → None

Save the InfoTable to a file.

Parameters:

• file_path (str or Path) – Path to save the file.

• sep (Optional[str]) – Column separator (default inferred from extension).

scatter2d(x_column: str, y_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 2D scattergram for two numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

scatter3d(x_column: str, y_column: str, z_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 3D scattergram for three numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• z_column (str) – Column name for the z-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

set_index(keys: str | List[str], inplace: bool = True) → T | None

Set the DataFrame index using one or more columns.

Parameters:

• keys (str or list) – Column(s) to use as index.

• inplace (bool) – Whether to modify in place.

Returns:

Modified InfoTable if not inplace.

Return type:

Optional[InfoTable]

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffle the rows of the InfoTable.

Parameters:

• seed (int) – Seed for random number generator.

• rng (np.random.Generator, optional) – Numpy random generator.

Returns:

Shuffled indices.

Return type:

np.ndarray

sort(column: str = 'id', ascending: bool = True, inplace: bool = True) → T | None

Sort the InfoTable by a specific column.

Parameters:

• column (str) – Column name to sort by.

• ascending (bool) – Sort in ascending order.

• inplace (bool) – Sort in place or return a new object.

Returns:

Sorted InfoTable or None if inplace.

Return type:

Optional[InfoTable]

split(idx: int, num_parts: int, group_by: str | None = None) → T

Split the InfoTable into parts and return the selected part.

Parameters:

• idx (int) – Part to return (1-based).

• num_parts (int) – Total number of parts.

• group_by (Optional[str]) – Column to group by when splitting.

Returns:

The selected part of the split.

Return type:

InfoTable

tail(n: int = 5) → T

Return the last n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

xs(key: Any, axis: int = 0, level: int | str = None, drop_level: bool = True) → T | Series

Returns a cross-section (row or column) from the DataFrame.

Parameters:

• key – Label or tuple of labels to select.

• axis (int) – Axis to retrieve from (0 for index, 1 for columns).

• level – Level in MultiIndex to use.

• drop_level (bool) – Whether to drop the level(s) from the result.

Returns:

If the result is a DataFrame, wraps it as InfoTable.

Return type:

InfoTable or Series

Trial/key/score structures

See Working With Trial Tables for a practical guide to TrialNdx, TrialKey, TrialScores, and their sparse variants.

class hyperion.utils.TrialNdx(model_set: List[str] | ndarray | None = None, seg_set: List[str] | ndarray | None = None, trial_mask: ndarray | None = None)

Contains the trial index to run speaker recognition trials.

BOSARIS-compatible trial index.

model_set

List of model names.

seg_set

List of test segment names.

trial_mask

Boolean matrix with the trials to execute to True (num_models x num_segments).

Examples

>>> import numpy as np
>>> from hyperion.utils.trial_ndx import TrialNdx
>>> ndx = TrialNdx(
...     model_set=["m1", "m2"],
...     seg_set=["s1", "s2", "s3"],
...     trial_mask=np.array([[1, 0, 1], [0, 1, 1]], dtype=bool),
... )
>>> ndx.num_models, ndx.num_tests
(2, 3)
>>> ndx_part = ndx.split(1, 2, 1, 1)
>>> ndx_part.trial_mask.shape
(1, 3)

__init__(model_set: List[str] | ndarray | None = None, seg_set: List[str] | ndarray | None = None, trial_mask: ndarray | None = None) → None

property num_models: int

property num_tests: int

copy() → TrialNdx

Makes a copy of the object

sort() → None

Sorts the object by model and test segment names.

save(file_path: str | Path, sep: str | None = None) → None

Saves object to txt/h5 file.

Parameters:

file_path – File to write the list.

save_h5(file_path: str | Path) → None

Saves object to h5 file.

Parameters:

file_path – File to write the list.

save_txt(file_path: str | Path) → None

Saves object to txt file.

Parameters:

file_path – File to write the list.

save_table(file_path: str | Path, sep: str | None = None) → None

Saves object to a pandas table file.

Parameters:

file_path – File to write the list.

classmethod load(file_path: str | Path, sep: str | None = None) → TrialNdx

Loads object from txt/h5 file

Parameters:

file_path – File to read the list.

Returns:

TrialNdx object.

classmethod load_h5(file_path: str | Path) → TrialNdx

Loads object from h5 file

Parameters:

file_path – File to read the list.

Returns:

TrialNdx object.

classmethod load_txt(file_path: str | Path) → TrialNdx

Loads object from txt file

Parameters:

file_path – File to read the list.

Returns:

TrialNdx object.

classmethod load_table(file_path: str | Path, sep: str | None = None) → TrialNdx

Loads object from pandas table file

Parameters:

file_path – File to read the list.

Returns:

TrialNdx object.

classmethod merge(ndx_list: List[TrialNdx]) → TrialNdx

Merges several index objects.

Parameters:

ndx_list – List of TrialNdx objects.

Returns:

Merged TrialNdx object.

static parse_eval_set(ndx: TrialNdx, enroll: object, test: object | None = None, eval_set: str = 'enroll-test') → Tuple[TrialNdx, object]

Prepares the data structures required for evaluation.

Parameters:

• ndx – TrialNdx object containing the trials for the main evaluation.

• enroll – Utt2Info where key are file_ids and second column are model names

• test – Utt2Info of where key are test segments names. Needed in the cases enroll-coh and coh-coh.

• eval_set – Type of evaluation enroll-test: main evaluation of enrollment vs test segments. enroll-coh: enrollment vs cohort segments. coh-test: cohort vs test segments. coh-coh: cohort vs cohort segments.

Returns:

TrialNdx object enroll: SCPList

Return type:

ndx

filter(model_set: ndarray | List[str], seg_set: ndarray | List[str], keep: bool = True, raise_missing: bool = True) → TrialNdx

Removes elements from TrialNdx object.

Parameters:

• model_set – List of models to keep or remove.

• seg_set – List of test segments to keep or remove.

• keep – If True, we keep the elements in model_set/seg_set, if False, we remove the elements in model_set/seg_set.

Returns:

Filtered TrialNdx object.

filter_by_model(model_set: ndarray | List[str], keep: bool = True, raise_missing: bool = True) → TrialNdx

Removes elements from TrialNdx object.

Parameters:

• model_set – List of models to keep or remove.

• keep – If True, we keep the elements in model_set/seg_set, if False, we remove the elements in model_set/seg_set.

Returns:

Filtered TrialNdx object.

split(model_idx: int, num_model_parts: int, seg_idx: int, num_seg_parts: int) → TrialNdx

Splits the TrialNdx into num_model_parts x num_seg_parts and returns part

(model_idx, seg_idx).

Parameters:

• model_idx – Model index of the part to return from 1 to num_model_parts.

• num_model_parts – Number of parts to split the model list.

• seg_idx – Segment index of the part to return from 1 to num_model_parts.

• num_seg_parts – Number of parts to split the test segment list.

Returns:

Subpart of the TrialNdx

validate() → None

Validates the attributes of the TrialNdx object.

apply_segmentation_to_test(segment_list: object) → TrialNdx

Splits test segment into multiple sub-segments Useful to create ndx for spk diarization or tracking.

Parameters:

segment_list – ExtSegmentList object with mapping of file_id to ext_segment_id

Returns:

New TrialNdx object with segment_ids in test instead of file_id.

__eq__(other: object) → bool

Equal operator

__ne__(other: object) → bool

Non-equal operator

__cmp__(other: object) → int

Comparison operator

test() → None

class hyperion.utils.TrialKey(model_set: List[str] | ndarray | None = None, seg_set: List[str] | ndarray | None = None, tar: ndarray | None = None, non: ndarray | None = None, spoof: ndarray | None = None, model_cond: ndarray | None = None, seg_cond: ndarray | None = None, trial_cond: ndarray | None = None, model_cond_name: List[str] | ndarray | None = None, seg_cond_name: List[str] | ndarray | None = None, trial_cond_name: List[str] | ndarray | None = None)

Contains the trial key for speaker recognition trials.

Bosaris compatible Key.

model_set

List of model names.

seg_set

List of test segment names.

tar

Boolean matrix with target trials to True (num_models x num_segments).

non

Boolean matrix with non-target trials to True (num_models x num_segments).

spoof

Boolean matrix with spoof trials to True (num_models x num_segments)

model_cond

Conditions related to the model.

seg_cond

Conditions related to the test segment.

trial_cond

Conditions related to the combination of model and test segment.

model_cond_name

String list with the names of the model conditions.

seg_cond_name

String list with the names of the segment conditions.

trial_cond_name

String list with the names of the trial conditions.

Examples

>>> import numpy as np
>>> from hyperion.utils.trial_key import TrialKey
>>> key = TrialKey(
...     model_set=["m1", "m2"],
...     seg_set=["s1", "s2"],
...     tar=np.array([[1, 0], [0, 1]], dtype=bool),
...     non=np.array([[0, 1], [1, 0]], dtype=bool),
... )
>>> ndx = key.to_ndx()
>>> ndx.trial_mask.shape
(2, 2)
>>> key_small = key.filter(["m1"], ["s1", "s2"], keep=True)
>>> key_small.tar.shape
(1, 2)

__init__(model_set: List[str] | ndarray | None = None, seg_set: List[str] | ndarray | None = None, tar: ndarray | None = None, non: ndarray | None = None, spoof: ndarray | None = None, model_cond: ndarray | None = None, seg_cond: ndarray | None = None, trial_cond: ndarray | None = None, model_cond_name: List[str] | ndarray | None = None, seg_cond_name: List[str] | ndarray | None = None, trial_cond_name: List[str] | ndarray | None = None) → None

property num_models: int

property num_tests: int

copy() → TrialKey

Makes a copy of the object

sort() → None

Sorts the object by model and test segment names.

save(file_path: str | Path, sep: str | None = None) → None

Saves object to txt/h5 file.

Parameters:

file_path – File to write the list.

save_h5(file_path: str | Path) → None

Saves object to h5 file.

Parameters:

file_path – File to write the list.

save_txt(file_path: str | Path) → None

Saves object to txt file.

Parameters:

file_path – File to write the list.

save_table(file_path: str | Path, sep: str | None = None) → None

Saves object to txt file.

Parameters:

file_path – File to write the list.

classmethod load(file_path: str | Path, sep: str | None = None) → TrialKey

Loads object from txt/h5 file

Parameters:

file_path – File to read the list.

Returns:

TrialKey object.

classmethod load_h5(file_path: str | Path) → TrialKey

Loads object from h5 file

Parameters:

file_path – File to read the list.

Returns:

TrialKey object.

classmethod load_txt(file_path: str | Path) → TrialKey

Loads object from txt file

Parameters:

file_path – File to read the list.

Returns:

TrialKey object.

classmethod load_table(file_path: str | Path, sep: str | None = None) → TrialKey

Loads object from pandas table file

Parameters:

file_path – File to read the list.

Returns:

TrialKey object.

classmethod merge(key_list: List[TrialKey]) → TrialKey

Merges several key objects.

Parameters:

key_list – List of TrialKey objects.

Returns:

Merged TrialKey object.

filter(model_set: ndarray | List[str], seg_set: ndarray | List[str], keep: bool = True, raise_missing: bool = True) → TrialKey

Removes elements from TrialKey object.

Parameters:

• model_set – List of models to keep or remove.

• seg_set – List of test segments to keep or remove.

• keep – If True, we keep the elements in model_set/seg_set, if False, we remove the elements in model_set/seg_set.

Returns:

Filtered TrialKey object.

filter_by_model(model_set: ndarray | List[str], keep: bool = True, raise_missing: bool = True) → TrialKey

Removes elements from TrialKey object.

Parameters:

• model_set – List of models to keep or remove.

• keep – If True, we keep the elements in model_set/seg_set, if False, we remove the elements in model_set/seg_set.

Returns:

Filtered TrialKey object.

split(model_idx: int, num_model_parts: int, seg_idx: int, num_seg_parts: int) → TrialKey

Splits the TrialKey into num_model_parts x num_seg_parts and returns part

(model_idx, seg_idx).

Parameters:

• model_idx – Model index of the part to return from 1 to num_model_parts.

• num_model_parts – Number of parts to split the model list.

• seg_idx – Segment index of the part to return from 1 to num_model_parts.

• num_seg_parts – Number of parts to split the test segment list.

Returns:

Subpart of the TrialKey

to_ndx() → TrialNdx

Converts TrialKey object into TrialNdx object.

Returns:

TrialNdx object.

validate() → None

Validates the attributes of the TrialKey object.

__eq__(other: object) → bool

Equal operator

__ne__(other: object) → bool

Non-equal operator

__cmp__(other: object) → int

Comparison operator

test() → None

class hyperion.utils.TrialScores(model_set: List[str] | ndarray | None = None, seg_set: List[str] | ndarray | None = None, scores: ndarray | None = None, score_mask: ndarray | None = None, q_measures: Dict[str, ndarray] | None = None)

Container for speaker recognition trial scores, compatible with BOSARIS toolkit.

model_set

Array of model IDs.

Type:

np.ndarray

seg_set

Array of segment IDs.

Type:

np.ndarray

scores

Score matrix (num_models x num_segments).

Type:

np.ndarray

score_mask

Boolean matrix indicating which scores are valid.

Type:

np.ndarray

q_measures

Optional dictionary of quality measures.

Type:

Optional[Dict[str, np.ndarray]]

Examples

>>> import numpy as np
>>> from hyperion.utils.trial_key import TrialKey
>>> from hyperion.utils.trial_scores import TrialScores
>>> key = TrialKey(
...     model_set=["m1", "m2"],
...     seg_set=["s1", "s2"],
...     tar=np.array([[1, 0], [0, 1]], dtype=bool),
...     non=np.array([[0, 1], [1, 0]], dtype=bool),
... )
>>> scores = TrialScores(
...     model_set=["m1", "m2"],
...     seg_set=["s1", "s2"],
...     scores=np.array([[2.1, -0.4], [-1.2, 1.7]], dtype=np.float32),
...     score_mask=np.ones((2, 2), dtype=bool),
... )
>>> tar, non = scores.get_tar_non(key)
>>> tar.shape, non.shape
((2,), (2,))

__init__(model_set: List[str] | ndarray | None = None, seg_set: List[str] | ndarray | None = None, scores: ndarray | None = None, score_mask: ndarray | None = None, q_measures: Dict[str, ndarray] | None = None) → None

property num_models: int

property num_tests: int

copy() → TrialScores

Makes a copy of the object

sort() → None

Sorts the object by model and test segment names.

save(file_path: str | Path, sep: str | None = None) → None

Saves the object to a file (HDF5, TXT, or CSV/TSV)

Parameters:

file_path – File to write the list.

save_h5(file_path: str | Path) → None

Saves object to h5 file.

Parameters:

file_path – File to write the list.

save_txt(file_path: str | Path) → None

Saves the object to a plain text file (space-separated)

Parameters:

file_path – File to write the list.

save_table(file_path: str | Path, sep: str | None = None) → None

Saves the object to a CSV/TSV table using Pandas. :param file_path: File to write the list.

classmethod load(file_path: str | Path, sep: str | None = None) → TrialScores

Loads a TrialScores object from file. :param file_path: File to read the list.

Returns:

TrialScores object.

classmethod load_h5(file_path: str | Path) → TrialScores

Loads object from h5 file

Parameters:

file_path – File to read the list.

Returns:

TrialScores object.

classmethod load_txt(file_path: str | Path) → TrialScores

Loads object from txt file

Parameters:

file_path – File to read the list.

Returns:

TrialScores object.

classmethod load_table(file_path: str | Path, sep: str | None = None) → TrialScores

Loads object from pandas table file

Parameters:

file_path – File to read the list.

Returns:

TrialScores object.

classmethod merge(scr_list: List[TrialScores]) → TrialScores

Merges several score objects.

Parameters:

scr_list – List of TrialScores objects.

Returns:

Merged TrialScores object.

filter(model_set: ndarray | List[str], seg_set: ndarray | List[str], keep: bool = True, raise_missing: bool = True) → TrialScores

Removes elements from TrialScores object.

Parameters:

• model_set – List of models to keep or remove.

• seg_set – List of test segments to keep or remove.

• keep – If True, we keep the elements in model_set/seg_set, if False, we remove the elements in model_set/seg_set.

• raise_missing – Raises exception if there are elements in model_set or seg_set that are not in the object.

Returns:

Filtered TrialScores object.

split(model_idx: int, num_model_parts: int, seg_idx: int, num_seg_parts: int) → TrialScores

Splits the TrialScores into num_model_parts x num_seg_parts and returns part

(model_idx, seg_idx).

Parameters:

• model_idx – Model index of the part to return from 1 to num_model_parts.

• num_model_parts – Number of parts to split the model list.

• seg_idx – Segment index of the part to return from 1 to num_model_parts.

• num_seg_parts – Number of parts to split the test segment list.

Returns:

Subpart of the TrialScores

validate() → None

Validates the attributes of the TrialScores object.

align_with_ndx(ndx: TrialNdx | TrialKey, raise_missing: bool = True) → TrialScores

Aligns scores, model_set, and seg_set with a TrialNdx or TrialKey object.

Parameters:

• ndx (TrialNdx or TrialKey) – Index object indicating which trials to align with.

• raise_missing (bool) – Whether to raise an error if some trials are missing.

Returns:

Aligned TrialScores object.

Return type:

TrialScores

get_tar_non(key: TrialKey) → Tuple[ndarray, ndarray]

Returns target and non-target scores using a TrialKey.

Parameters:

key (TrialKey) – TrialKey with target/non-target trial masks.

Returns:

Target scores, Non-target scores.

Return type:

Tuple[np.ndarray, np.ndarray]

get_tar_non_spoof(key: TrialKey) → Tuple[ndarray, ndarray, ndarray]

Returns target, non-target, and spoofing scores using a TrialKey.

Parameters:

key (TrialKey) – TrialKey with target, non-target, and optionally spoof trial masks.

Returns:

Target scores, Non-target scores, Spoof scores.

Return type:

Tuple[np.ndarray, np.ndarray, np.ndarray]

get_tar_non_q_measures(key: TrialKey, q_names: List[str] | None = None, return_dict: bool = False) → Tuple[Dict[str, ndarray] | ndarray, Dict[str, ndarray] | ndarray]

Returns quality measures for target and non-target trials.

Parameters:

• key (TrialKey) – TrialKey object.

• q_names (list of str, optional) – Names of quality measures to extract. All are used if None.

• return_dict (bool) – If True, returns dictionaries; if False, returns stacked arrays.

Returns:

(target quality measures, non-target quality measures)

Return type:

Tuple

get_class_sim(key: TrialKey, model_classes: List[str] | ndarray | None = None, seg_classes: List[str] | ndarray | None = None) → Tuple[ndarray, ndarray, ndarray]

Returns the class similarity scores for the trials in key.

Parameters:

key – SparseTrialKey object.

Returns:

Numpy array with the class similarity scores. M(i,j) average similarity between class i and class j.

set_missing_to_value(ndx: TrialNdx | TrialKey, val: float) → TrialScores

Sets scores missing in score_mask but present in ndx to a specific value.

Parameters:

• ndx (TrialNdx or TrialKey) – Index of trials.

• val (float) – Value to assign to missing scores.

Returns:

The modified TrialScores object.

Return type:

TrialScores

transform(f: Callable[[ndarray], ndarray]) → None

Applies a transformation function to the scores at valid (True) score_mask positions.

Parameters:

f (callable) – A function to apply to score values.

__eq__(other: object) → bool

Equal operator

__ne__(other: object) → bool

Non-equal operator

__cmp__(other: object) → int

Comparison operator

test() → None

class hyperion.utils.SparseTrialNdx(model_set: List[str] | ndarray | None, seg_set: List[str] | ndarray | None, trial_mask: spmatrix)

Contains sparse trial indices for speaker recognition trials.

Bosaris compatible Ndx.

model_set

List of model names.

seg_set

List of test segment names.

trial_mask

Sparse boolean matrix with the trials to execute to True (num_models x num_segments).

Examples

>>> import numpy as np
>>> import scipy.sparse as sparse
>>> from hyperion.utils.sparse_trial_ndx import SparseTrialNdx
>>> ndx = SparseTrialNdx(
...     model_set=["m1", "m2"],
...     seg_set=["s1", "s2", "s3"],
...     trial_mask=sparse.csr_matrix(np.array([[1, 0, 1], [0, 1, 1]], dtype=bool)),
... )
>>> ndx.num_models, ndx.num_tests
(2, 3)
>>> ndx_part = ndx.split(1, 2, 1, 1)
>>> ndx_part.trial_mask.shape
(1, 3)

__init__(model_set: List[str] | ndarray | None, seg_set: List[str] | ndarray | None, trial_mask: spmatrix) → None

static _full_trial_mask(num_models: int, num_tests: int) → csr_matrix

Creates an all-True sparse mask without allocating a dense matrix.

sort() → None

Sorts the object by model and test segment names.

save_h5(file_path: str | Path) → None

Saves object to h5 file.

Parameters:

file_path – File to write the list.

save_txt(file_path: str | Path) → None

Saves object to txt file.

Parameters:

file_path – File to write the list.

save_table(file_path: str | Path, sep: str | None = None) → None

Saves object to pandas table file.

Parameters:

file_path – File to write the list.

classmethod load_h5(file_path: str | Path) → SparseTrialNdx

Loads object from h5 file

Parameters:

file_path – File to read the list.

Returns:

TrialNdx object.

classmethod load_txt(file_path: str | Path) → SparseTrialNdx

Loads object from txt file

Parameters:

file_path – File to read the list.

Returns:

SparseTrialNdx object.

classmethod load_table(file_path: str | Path, sep: str | None = None) → SparseTrialNdx

Loads object from pandas table file.

Parameters:

file_path – File to read the list.

Returns:

SparseTrialNdx object.

classmethod merge(ndx_list: List[SparseTrialNdx]) → SparseTrialNdx

Merges several index objects.

Parameters:

ndx_list – List of SparseTrialNdx objects.

Returns:

Merged SparseTrialNdx object.

static parse_eval_set(ndx: SparseTrialNdx, enroll: object, test: object | None = None, eval_set: str = 'enroll-test') → Tuple[SparseTrialNdx, object]

Prepares sparse data structures required for evaluation.

filter(model_set: ndarray | List[str], seg_set: ndarray | List[str], keep: bool = True, raise_missing: bool = True) → SparseTrialNdx

Removes elements from SparseTrialNdx object.

Parameters:

• model_set – List of models to keep or remove.

• seg_set – List of test segments to keep or remove.

• keep – If True, keeps elements in model_set/seg_set.

• raise_missing – Raises error if requested models or segments are missing.

Returns:

Filtered SparseTrialNdx object.

filter_by_model(model_set: ndarray | List[str], keep: bool = True, raise_missing: bool = True) → SparseTrialNdx

Removes model entries from SparseTrialNdx object.

Parameters:

• model_set – List of models to keep or remove.

• keep – If True, keeps elements in model_set.

• raise_missing – Raises error if requested models are missing.

Returns:

Filtered SparseTrialNdx object.

split(model_idx: int, num_model_parts: int, seg_idx: int, num_seg_parts: int) → SparseTrialNdx

Splits the object and returns one subpart.

validate() → None

Validates the attributes of the SparseTrialNdx object.

apply_segmentation_to_test(segment_list: object) → SparseTrialNdx

Splits test segments into multiple sub-segments.

Parameters:

segment_list – ExtSegmentList object with mapping of file_id to ext_segment_id.

Returns:

New SparseTrialNdx with segment_ids in test instead of file_id.

classmethod from_trial_ndx(ndx: TrialNdx) → SparseTrialNdx

Builds a SparseTrialNdx from a dense TrialNdx.

to_trial_ndx() → TrialNdx

Converts SparseTrialNdx to dense TrialNdx.

__eq__(other: object) → bool

Equal operator.

__ne__(other: object) → bool

Non-equal operator.

__cmp__(other: object) → int

Comparison operator.

copy() → TrialNdx

Makes a copy of the object

classmethod load(file_path: str | Path, sep: str | None = None) → TrialNdx

Loads object from txt/h5 file

Parameters:

file_path – File to read the list.

Returns:

TrialNdx object.

property num_models: int

property num_tests: int

save(file_path: str | Path, sep: str | None = None) → None

Saves object to txt/h5 file.

Parameters:

file_path – File to write the list.

test() → None

class hyperion.utils.SparseTrialKey(model_set: List[str] | ndarray | None = None, seg_set: List[str] | ndarray | None = None, tar: spmatrix | None = None, non: spmatrix | None = None, spoof: spmatrix | None = None, model_cond: ndarray | None = None, seg_cond: ndarray | None = None, trial_cond: ndarray | None = None, model_cond_name: List[str] | ndarray | None = None, seg_cond_name: List[str] | ndarray | None = None, trial_cond_name: List[str] | ndarray | None = None)

Contains the trial key for speaker recognition trials.

Bosaris compatible Key.

model_set

List of model names.

seg_set

List of test segment names.

tar

Boolean matrix with target trials to True (num_models x num_segments).

non

Boolean matrix with non-target trials to True (num_models x num_segments).

spoof

Boolean matrix with spoof trials to True (num_models x num_segments).

model_cond

Conditions related to the model.

seg_cond

Conditions related to the test segment.

trial_cond

Conditions related to the combination of model and test segment.

model_cond_name

String list with the names of the model conditions.

seg_cond_name

String list with the names of the segment conditions.

trial_cond_name

String list with the names of the trial conditions.

Examples

>>> import numpy as np
>>> import scipy.sparse as sparse
>>> from hyperion.utils.sparse_trial_key import SparseTrialKey
>>> tar = sparse.csr_matrix(np.array([[1, 0], [0, 1]], dtype=bool))
>>> non = sparse.csr_matrix(np.array([[0, 1], [1, 0]], dtype=bool))
>>> key = SparseTrialKey(model_set=["m1", "m2"], seg_set=["s1", "s2"], tar=tar, non=non)
>>> ndx = key.to_ndx()
>>> ndx.trial_mask.shape
(2, 2)
>>> from hyperion.utils.trial_key import TrialKey
>>> dense_key = TrialKey(
...     model_set=["m1", "m2"],
...     seg_set=["s1", "s2"],
...     tar=np.array([[1, 0], [0, 1]], dtype=bool),
...     non=np.array([[0, 1], [1, 0]], dtype=bool),
... )
>>> sparse_key = SparseTrialKey.from_trial_key(dense_key)
>>> sparse_key.tar.nnz
2

__init__(model_set: List[str] | ndarray | None = None, seg_set: List[str] | ndarray | None = None, tar: spmatrix | None = None, non: spmatrix | None = None, spoof: spmatrix | None = None, model_cond: ndarray | None = None, seg_cond: ndarray | None = None, trial_cond: ndarray | None = None, model_cond_name: List[str] | ndarray | None = None, seg_cond_name: List[str] | ndarray | None = None, trial_cond_name: List[str] | ndarray | None = None) → None

sort() → None

Sorts the object by model and test segment names.

save_h5(file_path: str | Path) → None

Saves object to h5 file.

Parameters:

file_path – File to write the list.

save_txt(file_path: str | Path) → None

Saves object to txt file.

Parameters:

file_path – File to write the list.

save_table(file_path: str | Path, sep: str | None = None) → None

Saves object to txt file.

Parameters:

file_path – File to write the list.

classmethod load_h5(file_path: str | Path) → SparseTrialKey

Loads object from h5 file

Parameters:

file_path – File to read the list.

Returns:

TrialKey object.

classmethod load_txt(file_path: str | Path) → SparseTrialKey

Loads object from txt file

Parameters:

file_path – File to read the list.

Returns:

TrialKey object.

classmethod load_table(file_path: str | Path, sep: str | None = None) → SparseTrialKey

Loads object from txt file

Parameters:

file_path – File to read the list.

Returns:

SparseTrialKey object.

classmethod merge(key_list: List[SparseTrialKey]) → SparseTrialKey

Merges several key objects.

Parameters:

key_list – List of TrialKey objects.

Returns:

Merged TrialKey object.

filter(model_set: ndarray | List[str], seg_set: ndarray | List[str], keep: bool = True, raise_missing: bool = True) → SparseTrialKey

Removes elements from SparseTrialKey object.

filter_by_model(model_set: ndarray | List[str], keep: bool = True, raise_missing: bool = True) → SparseTrialKey

Removes model entries from SparseTrialKey object.

split(model_idx: int, num_model_parts: int, seg_idx: int, num_seg_parts: int) → SparseTrialKey

Splits the SparseTrialKey and returns one subpart.

to_ndx() → TrialNdx

Converts TrialKey object into TrialNdx object.

Returns:

TrialNdx object.

validate() → None

Validates the attributes of the TrialKey object.

classmethod from_trial_key(key: TrialKey) → SparseTrialKey

__eq__(other: object) → bool

Equal operator

__cmp__(other: object) → int

Comparison operator

__ne__(other: object) → bool

Non-equal operator

copy() → TrialKey

Makes a copy of the object

classmethod load(file_path: str | Path, sep: str | None = None) → TrialKey

Loads object from txt/h5 file

Parameters:

file_path – File to read the list.

Returns:

TrialKey object.

property num_models: int

property num_tests: int

save(file_path: str | Path, sep: str | None = None) → None

Saves object to txt/h5 file.

Parameters:

file_path – File to write the list.

test() → None

class hyperion.utils.SparseTrialScores(model_set: List[str] | ndarray | None = None, seg_set: List[str] | ndarray | None = None, scores: spmatrix | None = None, score_mask: spmatrix | None = None)

Contains the scores for the speaker recognition trials.

Bosaris compatible Scores.

model_set

List of model names.

seg_set

List of test segment names.

scores

Matrix with the scores (num_models x num_segments).

score_mask

Boolean matrix with the trials with valid scores to True (num_models x num_segments).

Examples

>>> import numpy as np
>>> from hyperion.utils.trial_scores import TrialScores
>>> from hyperion.utils.sparse_trial_scores import SparseTrialScores
>>> dense = TrialScores(
...     model_set=["m1", "m2"],
...     seg_set=["s1", "s2"],
...     scores=np.array([[1.2, 0.0], [0.0, -0.8]], dtype=np.float32),
...     score_mask=np.array([[1, 0], [0, 1]], dtype=bool),
... )
>>> sparse_scores = SparseTrialScores.from_trial_scores(dense)
>>> sparse_scores.score_mask.nnz
2
>>> dense_back = sparse_scores.to_trial_scores()
>>> dense_back.scores.shape
(2, 2)

__init__(model_set: List[str] | ndarray | None = None, seg_set: List[str] | ndarray | None = None, scores: spmatrix | None = None, score_mask: spmatrix | None = None) → None

static _extract_scores_from_mask(scores: spmatrix, mask: spmatrix) → ndarray

Extracts scores selected by a sparse mask as a 1-D NumPy array.

sort() → None

Sorts the object by model and test segment names.

save_h5(file_path: str | Path) → None

Saves object to h5 file.

Parameters:

file_path – File to write the list.

save_txt(file_path: str | Path) → None

Saves object to txt file.

Parameters:

file_path – File to write the list.

save_table(file_path: str | Path, sep: str | None = None) → None

Saves object to a pandas table file.

Parameters:

file_path – File to write the list.

classmethod load_h5(file_path: str | Path) → SparseTrialScores

Loads object from h5 file

Parameters:

file_path – File to read the list.

Returns:

TrialScores object.

classmethod load_txt(file_path: str | Path) → SparseTrialScores

Loads object from h5 file

Parameters:

file_path – File to read the list.

Returns:

SparseTrialScores object.

classmethod load_table(file_path: str | Path, sep: str | None = None) → SparseTrialScores

Loads object from pandas table file

Parameters:

file_path – File to read the list.

Returns:

TrialScores object.

classmethod merge(scr_list: List[SparseTrialScores]) → SparseTrialScores

Merges several SparseTrialScores objects.

Parameters:

scr_list – List of SparseTrialScores objects.

Returns:

Merged SparseTrialScores object.

split(model_idx: int, num_model_parts: int, seg_idx: int, num_seg_parts: int) → SparseTrialScores

Splits the TrialScores into num_model_parts x num_seg_parts and returns part

(model_idx, seg_idx).

Parameters:

• model_idx – Model index of the part to return from 1 to num_model_parts.

• num_model_parts – Number of parts to split the model list.

• seg_idx – Segment index of the part to return from 1 to num_model_parts.

• num_seg_parts – Number of parts to split the test segment list.

Returns:

Subpart of the TrialScores

validate() → None

Validates the attributes of the TrialKey object.

filter(model_set: ndarray | List[str], seg_set: ndarray | List[str], keep: bool = True, raise_missing: bool = True) → SparseTrialScores

Removes elements from TrialScores object.

Parameters:

• model_set – List of models to keep or remove.

• seg_set – List of test segments to keep or remove.

• keep – If True, we keep the elements in model_set/seg_set, if False, we remove the elements in model_set/seg_set.

• raise_missing – Raises exception if there are elements in model_set or seg_set that are not in the object.

Returns:

Filtered TrialScores object.

align_with_ndx(ndx: TrialNdx | TrialKey | SparseTrialKey, raise_missing: bool = True) → SparseTrialScores

Aligns scores, model_set and seg_set with TrialNdx or TrialKey.

Parameters:

• ndx – TrialNdx or TrialKey object.

• raise_missing – Raises exception if there are trials in ndx that are not in the score object.

Returns:

Aligned TrialScores object.

get_tar_non(key: TrialKey | SparseTrialKey) → Tuple[ndarray, ndarray]

Returns target and non target scores.

Parameters:

key – TrialKey object.

Returns:

Numpy array with target scores. Numpy array with non-target scores.

get_tar_non_spoof(key: TrialKey | SparseTrialKey) → Tuple[ndarray, ndarray, ndarray]

Returns target, non-target and spoof scores.

Parameters:

key – TrialKey or SparseTrialKey object.

Returns:

Numpy array with target scores. Numpy array with non-target scores. Numpy array with spoof scores.

get_valid_scores(ndx: TrialNdx | TrialKey | SparseTrialKey | None = None) → ndarray

get_class_sim(key: SparseTrialKey, model_classes: List[str] | ndarray | None = None, seg_classes: List[str] | ndarray | None = None) → Tuple[ndarray, ndarray, ndarray]

Returns the class similarity scores for the trials in key.

Parameters:

key – SparseTrialKey object.

Returns:

Numpy array with the class similarity scores. M(i,j) average similarity between class i and class j.

set_valid_scores(scores: ndarray | List[float], ndx: TrialNdx | TrialKey | SparseTrialKey | None = None) → None

classmethod from_trial_scores(scr: TrialScores) → SparseTrialScores

to_trial_scores() → TrialScores

set_missing_to_value(ndx: TrialNdx | TrialKey | SparseTrialKey, val: float) → SparseTrialScores

Aligns the scores with a TrialNdx and sets the trials with missing scores to the same value.

Parameters:

• ndx – TrialNdx or TrialKey object.

• val – Value for the missing scores.

Returns:

Aligned SparseTrialScores object.

__eq__(other: object) → bool

Equal operator

__cmp__(other: object) → int

Comparison operator

__ne__(other: object) → bool

Non-equal operator

copy() → TrialScores

Makes a copy of the object

get_tar_non_q_measures(key: TrialKey, q_names: List[str] | None = None, return_dict: bool = False) → Tuple[Dict[str, ndarray] | ndarray, Dict[str, ndarray] | ndarray]

Returns quality measures for target and non-target trials.

Parameters:

• key (TrialKey) – TrialKey object.

• q_names (list of str, optional) – Names of quality measures to extract. All are used if None.

• return_dict (bool) – If True, returns dictionaries; if False, returns stacked arrays.

Returns:

(target quality measures, non-target quality measures)

Return type:

Tuple

classmethod load(file_path: str | Path, sep: str | None = None) → TrialScores

Loads a TrialScores object from file. :param file_path: File to read the list.

Returns:

TrialScores object.

property num_models: int

property num_tests: int

save(file_path: str | Path, sep: str | None = None) → None

Saves the object to a file (HDF5, TXT, or CSV/TSV)

Parameters:

file_path – File to write the list.

test() → None

transform(f: Callable[[ndarray], ndarray]) → None

Applies a transformation function to the scores at valid (True) score_mask positions.

Parameters:

f (callable) – A function to apply to score values.

Enrollment and class metadata

class hyperion.utils.EnrollmentMap(df: DataFrame | T)

Mapping between enrollment model IDs and segment IDs.

Required columns are id (model identifier) and segmentid.

Examples

>>> import pandas as pd
>>> from hyperion.utils.enrollment_map import EnrollmentMap
>>> df = pd.DataFrame({"id": ["m1", "m1", "m2"], "segmentid": ["s1", "s2", "s3"]})
>>> emap = EnrollmentMap(df)
>>> uniq, inv = emap.model_idx()
>>> uniq.tolist()
['m1', 'm2']
>>> emap_part = emap.split(1, 2)
>>> isinstance(emap_part, EnrollmentMap)
True
>>> merged = EnrollmentMap.cat([emap_part, emap.split(2, 2)])
>>> len(merged) == len(emap)
True

__init__(df: DataFrame | T) → None

Initialize the enrollment map.

Parameters:

df (pd.DataFrame or EnrollmentMap) – Input mapping table.

split(idx: int, num_parts: int) → EnrollmentMap

Split the map into num_parts and return partition idx.

Parameters:

• idx (int) – 1-based partition index to return.

• num_parts (int) – Total number of partitions.

Returns:

Requested partition.

Return type:

EnrollmentMap

save(file_path: str | Path, sep: str | None = None, nist_compatible: bool = True) → None

Save the enrollment map to disk.

Parameters:

• file_path (PathLike) – Output path.

• sep (Optional[str]) – Optional delimiter override for non-.scp files.

• nist_compatible (bool) – If True, save id as modelid.

classmethod load(file_path: str | Path, sep: str | None = None) → T

Load an EnrollmentMap from file.

Parameters:

• file_path (PathLike) – File to read.

• sep (Optional[str]) – Delimiter for text/CSV/TSV formats.

Returns:

Loaded enrollment map.

Return type:

EnrollmentMap

classmethod cat(tables: List[T]) → T

Concatenate several enrollment maps.

Parameters:

tables (List[EnrollmentMap]) – Input tables.

Returns:

Concatenated table.

Return type:

EnrollmentMap

model_idx(modelids: List[str] | ndarray | None = None) → Tuple[ndarray, ndarray] | ndarray

Return mapping from segments to model indices.

Parameters:

• modelids (Optional[Union[List[str], np.ndarray]]) – Ordered model IDs

• None (used to assign integer indices. If)

• with (IDs are inferred)

• np.unique.

Returns:

If modelids is None, returns (unique_modelids, segment_to_model_index). Otherwise returns only segment_to_model_index and assigns -1 to rows whose model ID does not appear in modelids.

Return type:

Union[Tuple[np.ndarray, np.ndarray], np.ndarray]

get_unique_modelid_df() → DataFrame

Return a DataFrame with sorted unique model IDs.

Returns:

Unique rows over the model-id columns.

Return type:

pd.DataFrame

__cmp__(other: Any) → int

Comparison operator

__contains__(key: Any) → bool

Check whether key is in the DataFrame columns.

Parameters:

key – Key to check.

Returns:

True if key is in columns.

Return type:

bool

__eq__(other: Any) → bool

Equal operator

__getattr__(name: str) → Any

Provide attribute-style access to DataFrame columns and attributes.

Falls back to self.df[name] if name is a column. Falls back to getattr(self.df, name) if it’s a DataFrame method or attribute. Special methods (e.g., __setstate__) raise AttributeError immediately to allow correct behavior during pickling/unpickling.

Parameters:

name (str) – The attribute being accessed.

Returns:

The corresponding column, method, or attribute.

Return type:

Any

Raises:

AttributeError – If the attribute is not found.

__getitem__(key: Any) → T | Series

Get item from the internal DataFrame.

Parameters:

key – Key used for indexing.

Returns:

Sub-table or Series.

Return type:

Union[InfoTable, pd.Series]

__ne__(other: Any) → bool

Non-equal operator

__setitem__(key: Any, value: Any) → None

Set item in the internal DataFrame.

Parameters:

• key – Column label or index.

• value – Value to assign.

add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Add new columns from another InfoTable or DataFrame.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – The table to merge columns from.

• column_names (str or list, optional) – Columns to include from the right table.

• on (str or list) – Key(s) from the current table.

• right_on (str or list, optional) – Key(s) from the right table.

• replace_overlapping (bool) – Replace overlapping columns if True.

• ignore_overlapping (bool) – If True, skip adding columns that already exist.

• remove_missing (bool) – Use inner join (drop unmatched rows) if True.

property at: _InfoTableAtIndexer

Access a single value for a row/column label pair.

Returns:

Indexer that wraps .at.

Return type:

_InfoTableAtIndexer

clone() → T

Alias for copy().

Returns:

A deep copy.

Return type:

InfoTable

property columns: Index

Get the DataFrame columns.

Returns:

Column names.

Return type:

pd.Index

convert_col_to_str(column: str) → None

Ensure a specific column is of string type.

Parameters:

column (str) – Column name to convert.

copy() → T

Return a deep copy of the InfoTable.

Returns:

A deep copy.

Return type:

InfoTable

drop(labels: str | int | List[Any] | ndarray | Index | Series | None = None, axis: int | str = 0, index: str | int | List[Any] | ndarray | Index | Series | None = None, columns: str | int | List[Any] | ndarray | Index | Series | None = None, level: int | str | None = None, inplace: bool = False, errors: str = 'raise') → T | None

Drop specified labels.

Parameters:

• labels – Index or column labels.

• axis (int) – Whether to drop rows (0) or columns (1).

• index – Alias for labels along the index (rows).

• columns – Column labels to drop.

• level – Level in MultiIndex from which to drop.

• inplace (bool) – Modify in place.

• errors (str) – If ‘ignore’, suppress errors for nonexistent labels.

Returns:

Modified InfoTable or None.

Return type:

Optional[InfoTable]

dropna(*args: Any, **kwargs: Any) → T | None

Return a new InfoTable with missing values dropped.

Parameters:

• *args – Passed to pandas.DataFrame.dropna.

• **kwargs – Passed to pandas.DataFrame.dropna.

Returns:

A new instance with rows (or columns) with NA removed.

Return type:

InfoTable

property eval: Callable

Return the DataFrame.eval method.

Returns:

The eval method for evaluating expressions.

Return type:

Callable

filter(predicate: Callable[[DataFrame], Series | ndarray] | str | None = None, items: List[Any] | None = None, iindex: ndarray | None = None, columns: List[str] | None = None, by: str = 'id', keep: bool = True, raise_if_missing: bool = True) → T

Filter the InfoTable based on a predicate, item list, index list, or column subset.

Parameters:

• predicate (Callable, optional) – Function that returns a boolean mask, e.g.: lambda df: df[“duration”] > 1.0.

• items (List[Any], optional) – Items to include/exclude from the ‘by’ column like df.loc[items, by], used only if predicate is None

• iindex (np.ndarray, optional) – Integer indices to include/exclude like df.iloc[iindex], used if predicate and items are None

• columns (List[str], optional) – Columns to retain or remove.

• by (str) – Column name to use with ‘items’.

• keep (bool) – Whether to keep or exclude matched rows/columns.

• raise_if_missing (bool) – Raise error if items are missing.

Returns:

Filtered InfoTable.

Return type:

InfoTable

Raises:

Exception – If items are not found and raise_if_missing is True.

fix_dtypes() → None

Ensure the ‘id’ column is of string type.

classmethod from_dict(df_dict: Dict[str, List[Any]]) → T

Create InfoTable from a dictionary.

Parameters:

df_dict (Dict[str, List[Any]]) – Column data including ‘id’.

Returns:

Constructed table.

Return type:

InfoTable

classmethod from_lists(ids: List[str], column_names: List[str], column_data: List[List[Any]]) → T

Create InfoTable from lists of IDs and corresponding column data.

Parameters:

• ids (List[str]) – List of IDs.

• column_names (List[str]) – List of column names.

• column_data (List[List[Any]]) – Column values.

Returns:

Constructed table.

Return type:

InfoTable

get_col_idx(keys: str | List[str]) → int | ndarray

Get the integer index position(s) of the specified column(s).

Parameters:

keys (str or list) – Column name(s).

Returns:

Position(s) of the column(s).

Return type:

int or np.ndarray

get_loc(keys: str | List[str] | ndarray) → int | ndarray | List[int]

Get integer location(s) for the given key(s).

Parameters:

keys (str, list, or np.ndarray) – Index label(s).

Returns:

Location(s) in the index.

Return type:

int, np.ndarray, or List[int]

harmonize_age_given_decade(voter_columns: str | List[str], target_column: str, decade_column: str = 'age_decade') → None

Force all rows sharing voter_columns to take the averaged age in target_column, constrained by the decade labels in decade_column.

If the averaged age falls outside the decade bounds, the age is set to the upper boundary for that decade.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric age column to harmonize.

• decade_column (str) – Column indicating the age decade labels.

Raises:

• KeyError – If the voter, target, or decade columns are missing.

• TypeError – If the target column is not numeric.

harmonize_column_by_majority_cluster(voter_columns: str | List[str], target_column: str, suspect_column: str = 'suspect', std_threshold: float | None = None, max_iter: int = 20) → None

Harmonize a numeric column using the dominant cluster within each voter group, while flagging rows from the smaller cluster as suspect.

If std_threshold is provided, clustering is only performed when the group’s standard deviation exceeds the threshold.

The standard deviation of the dominant cluster is stored in a column named f"{target_column}_std" for the non-suspect rows.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric column to harmonize.

• suspect_column (str) – Column used to flag suspect rows.

• std_threshold (Optional[float]) – If set, only split groups above this std.

• max_iter (int) – Maximum iterations for the 1D two-means refinement.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If the target column is not numeric.

harmonize_columns_by_average(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the average value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the group used for averaging.

• target_columns (Union[str, List[str]]) – Numeric column(s) to harmonize via averaging.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If any target column is not numeric.

harmonize_columns_by_majority_vote(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the majority value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the voting group (e.g., speaker).

• target_columns (Union[str, List[str]]) – Column or columns to harmonize via majority vote.

Raises:

KeyError – If the voter or target columns are missing.

head(n: int = 5) → T

Return the first n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

histogram(column: str, bins: int | None = None, density: bool = True, kind: str = 'bar', color: str = 'C0', output_file: str | Path | None = None, dropna: bool = True) → None

Plot a histogram of a column, handling string or numeric data.

Parameters:

• column (str) – Column name to plot. Title will include the column name.

• bins (Optional[int]) – Number of bins for numeric columns. Ignored for strings.

• density (bool) – If True, plot density/relative frequency; otherwise counts.

• kind (str) – Histogram style, either “bar” or “line”.

• color (str) – Matplotlib color spec for the plot.

• output_file (Optional[PathLike]) – If provided, save the figure; otherwise show it.

• dropna (bool) – If True, ignore NA values before plotting.

property iat: _InfoTableAtIndexer

Access a single value for a row/column integer position pair.

Returns:

Indexer that wraps .iat.

Return type:

_InfoTableAtIndexer

property iloc: _InfoTableIndexer

Access a group of rows and columns by integer position(s).

Returns:

Indexer that wraps .iloc.

Return type:

_InfoTableIndexer

property index: Index

Get the index of the DataFrame.

Returns:

DataFrame index.

Return type:

pd.Index

static is_valid_df(df: DataFrame) → bool

Check if the DataFrame is valid for InfoTable.

Parameters:

df (pd.DataFrame) – DataFrame to check.

Returns:

True if valid, False otherwise.

Return type:

bool

property iterrows: Callable

Return the DataFrame.iterrows generator.

Returns:

Yields (index, Series) pairs.

Return type:

Callable

legacy_add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Legacy implementation of add_columns that uses a full pandas merge.

This keeps the original behavior for compatibility/testing.

property loc: _InfoTableIndexer

Access a group of rows and columns by label(s).

Returns:

Indexer that wraps .loc.

Return type:

_InfoTableIndexer

classmethod merge(left_table: T | DataFrame, right_table: T | DataFrame, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Tuple[str, str] = ('_x', '_y'), copy: bool | None = None, indicator: str | bool = False, validate: str | None = None) → T

Merge two InfoTables or DataFrames into a new InfoTable.

Parameters:

• left_table (Union[InfoTable, pd.DataFrame]) – Left-hand table.

• right_table (Union[InfoTable, pd.DataFrame]) – Right-hand table.

• how (str) – Merge method (e.g., ‘inner’, ‘outer’, ‘left’, ‘right’).

• on (Union[str, List[str], None]) – Column(s) to join on.

• left_on (Union[str, List[str], None]) – Column(s) from the left table to join on.

• right_on (Union[str, List[str], None]) – Column(s) from the right table to join on.

• left_index (bool) – Use index from the left table as join key.

• right_index (bool) – Use index from the right table as join key.

• sort (bool) – Sort the result by the join keys.

• suffixes (Tuple[str, str]) – Suffixes to apply to overlapping column names.

• copy (Optional[bool]) – If False, avoid copying data where possible.

• indicator (Union[str, bool]) – Adds a column to the output DataFrame called ‘_merge’.

• validate (Optional[str]) – Check if merge is of specified type.

Returns:

A new merged InfoTable.

Return type:

InfoTable

query(expr: str, **kwargs: Any) → T

Filters rows using a boolean expression string.

Parameters:

• expr (str) – A string expression to evaluate, using column names as variables.

• **kwargs – Passed through to pandas.DataFrame.query().

Returns:

A new InfoTable with filtered rows.

Return type:

InfoTable

replace(to_replace: Any = None, value: Any = None, inplace: bool = False, **kwargs: Any) → T | None

Replace values in the DataFrame.

Parameters:

• to_replace – What to replace.

• value – Value to replace with.

• inplace (bool) – Whether to modify the table in-place.

• **kwargs – Additional keyword args passed to pd.DataFrame.replace().

Returns:

New InfoTable if not inplace, else None.

Return type:

Optional[InfoTable]

replace_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None) → None

Replace column values with those from another table.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – Table to source values from.

• column_names (str or list, optional) – Columns to replace. If None, all.

reset_index() → None

Reset the DataFrame index to the ‘id’ column.

Returns:

None

sample(n: int = 1, random_state: int | RandomState | Generator | BitGenerator | None = None) → T

Return a random sample of rows.

Parameters:

• n (int) – Number of rows.

• random_state – Seed for reproducibility.

Returns:

Sampled InfoTable.

Return type:

InfoTable

scatter2d(x_column: str, y_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 2D scattergram for two numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

scatter3d(x_column: str, y_column: str, z_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 3D scattergram for three numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• z_column (str) – Column name for the z-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

set_index(keys: str | List[str], inplace: bool = True) → T | None

Set the DataFrame index using one or more columns.

Parameters:

• keys (str or list) – Column(s) to use as index.

• inplace (bool) – Whether to modify in place.

Returns:

Modified InfoTable if not inplace.

Return type:

Optional[InfoTable]

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffle the rows of the InfoTable.

Parameters:

• seed (int) – Seed for random number generator.

• rng (np.random.Generator, optional) – Numpy random generator.

Returns:

Shuffled indices.

Return type:

np.ndarray

sort(column: str = 'id', ascending: bool = True, inplace: bool = True) → T | None

Sort the InfoTable by a specific column.

Parameters:

• column (str) – Column name to sort by.

• ascending (bool) – Sort in ascending order.

• inplace (bool) – Sort in place or return a new object.

Returns:

Sorted InfoTable or None if inplace.

Return type:

Optional[InfoTable]

tail(n: int = 5) → T

Return the last n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

xs(key: Any, axis: int = 0, level: int | str = None, drop_level: bool = True) → T | Series

Returns a cross-section (row or column) from the DataFrame.

Parameters:

• key – Label or tuple of labels to select.

• axis (int) – Axis to retrieve from (0 for index, 1 for columns).

• level – Level in MultiIndex to use.

• drop_level (bool) – Whether to drop the level(s) from the result.

Returns:

If the result is a DataFrame, wraps it as InfoTable.

Return type:

InfoTable or Series

class hyperion.utils.ClassInfo(df: DataFrame | T)

A subclass of InfoTable for managing classification metadata.

Ensures that each entry has a unique class index and maintains class weights.

df

Underlying DataFrame containing: - ‘id’ (str): Unique identifier for each class entry. - ‘class_idx’ (int): Unique index assigned to each class. - ‘weights’ (float): Class weights normalized to sum to 1.

Type:

pd.DataFrame

Examples

>>> import pandas as pd
>>> from hyperion.utils.class_info import ClassInfo
>>> df = pd.DataFrame({"id": ["spk1", "spk2", "spk3"]})
>>> ci = ClassInfo(df)
>>> ci.num_classes
3
>>> ci.weights(["spk1", "spk2"]).tolist()
[0.3333333333333333, 0.3333333333333333]
>>> ci.set_zero_weight(["spk1"])
>>> ci.weights("spk1")
0.0
>>> ci2 = ci.filter(items=["spk2", "spk3"], rebuild_idx=True)
>>> ci2.num_classes
2

__init__(df: DataFrame | T) → None

Initialize ClassInfo with automatic class index and normalized weights.

Parameters:

df (pd.DataFrame or ClassInfo) – Input data.

add_class_idx(sort_by_id: bool = False) → None

Assign a unique integer class index to each row.

set_uniform_weights() → None

Set uniform weights across all classes.

set_weights(weights: Series | ndarray) → None

Set class weights and normalize them.

Parameters:

weights (pd.Series or np.ndarray) – Raw weights.

renorm_weights() → None

Renormalize existing weights to ensure they sum to 1.

exp_weights(x: int | float) → None

Raise weights to the power of x and re-normalize.

Parameters:

x (int or float) – Exponent to apply to weights.

set_zero_weight(ids: List[str] | ndarray) → None

Set weights of selected IDs to zero and renormalize the rest.

Parameters:

ids (list or np.ndarray) – List of IDs to zero out.

weights(ids: str | List[str]) → float | Series

Get the weight(s) for given ID(s).

Parameters:

ids (str or list) – Single ID or list of IDs.

Returns:

Corresponding weight(s).

Return type:

float or pd.Series

property num_classes: int

Number of distinct classes in the table.

Returns:

Maximum class index + 1.

Return type:

int

sort_by_idx(ascending: bool = True) → None

Sort entries by class index.

Parameters:

ascending (bool) – Whether to sort in ascending order.

classmethod load(file_path: str | Path, sep: str | None = None) → T

Load ClassInfo from file.

Parameters:

• file_path (str or Path) – Path to the input file.

• sep (Optional[str]) – Column separator.

Returns:

Loaded ClassInfo instance.

Return type:

ClassInfo

classmethod cat(tables: List[T]) → T

Concatenate multiple ClassInfo tables.

Parameters:

tables (List[ClassInfo]) – List of ClassInfo objects.

Returns:

Concatenated and validated ClassInfo.

Return type:

ClassInfo

filter(predicate: Callable[[DataFrame], Series | ndarray] | None = None, items: List[Any] | None = None, iindex: ndarray | None = None, columns: List[str] | None = None, by: str = 'id', keep: bool = True, rebuild_idx: bool = False) → T

Filter rows from ClassInfo with optional index rebuilding.

Parameters:

• predicate (Callable, optional) – Boolean function to filter rows.

• items (list, optional) – Items to filter by.

• iindex (np.ndarray, optional) – Row indices.

• columns (list, optional) – Columns to keep.

• by (str) – Column to apply item filter on.

• keep (bool) – Whether to keep or exclude matching rows.

• rebuild_idx (bool) – Reassign class_idx after filtering.

Returns:

Filtered and optionally reindexed ClassInfo.

Return type:

ClassInfo

__cmp__(other: Any) → int

Comparison operator

__contains__(key: Any) → bool

Check whether key is in the DataFrame columns.

Parameters:

key – Key to check.

Returns:

True if key is in columns.

Return type:

bool

__eq__(other: Any) → bool

Equal operator

__getattr__(name: str) → Any

Provide attribute-style access to DataFrame columns and attributes.

Falls back to self.df[name] if name is a column. Falls back to getattr(self.df, name) if it’s a DataFrame method or attribute. Special methods (e.g., __setstate__) raise AttributeError immediately to allow correct behavior during pickling/unpickling.

Parameters:

name (str) – The attribute being accessed.

Returns:

The corresponding column, method, or attribute.

Return type:

Any

Raises:

AttributeError – If the attribute is not found.

__getitem__(key: Any) → T | Series

Get item from the internal DataFrame.

Parameters:

key – Key used for indexing.

Returns:

Sub-table or Series.

Return type:

Union[InfoTable, pd.Series]

__ne__(other: Any) → bool

Non-equal operator

__setitem__(key: Any, value: Any) → None

Set item in the internal DataFrame.

Parameters:

• key – Column label or index.

• value – Value to assign.

add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Add new columns from another InfoTable or DataFrame.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – The table to merge columns from.

• column_names (str or list, optional) – Columns to include from the right table.

• on (str or list) – Key(s) from the current table.

• right_on (str or list, optional) – Key(s) from the right table.

• replace_overlapping (bool) – Replace overlapping columns if True.

• ignore_overlapping (bool) – If True, skip adding columns that already exist.

• remove_missing (bool) – Use inner join (drop unmatched rows) if True.

property at: _InfoTableAtIndexer

Access a single value for a row/column label pair.

Returns:

Indexer that wraps .at.

Return type:

_InfoTableAtIndexer

clone() → T

Alias for copy().

Returns:

A deep copy.

Return type:

InfoTable

property columns: Index

Get the DataFrame columns.

Returns:

Column names.

Return type:

pd.Index

convert_col_to_str(column: str) → None

Ensure a specific column is of string type.

Parameters:

column (str) – Column name to convert.

copy() → T

Return a deep copy of the InfoTable.

Returns:

A deep copy.

Return type:

InfoTable

drop(labels: str | int | List[Any] | ndarray | Index | Series | None = None, axis: int | str = 0, index: str | int | List[Any] | ndarray | Index | Series | None = None, columns: str | int | List[Any] | ndarray | Index | Series | None = None, level: int | str | None = None, inplace: bool = False, errors: str = 'raise') → T | None

Drop specified labels.

Parameters:

• labels – Index or column labels.

• axis (int) – Whether to drop rows (0) or columns (1).

• index – Alias for labels along the index (rows).

• columns – Column labels to drop.

• level – Level in MultiIndex from which to drop.

• inplace (bool) – Modify in place.

• errors (str) – If ‘ignore’, suppress errors for nonexistent labels.

Returns:

Modified InfoTable or None.

Return type:

Optional[InfoTable]

dropna(*args: Any, **kwargs: Any) → T | None

Return a new InfoTable with missing values dropped.

Parameters:

• *args – Passed to pandas.DataFrame.dropna.

• **kwargs – Passed to pandas.DataFrame.dropna.

Returns:

A new instance with rows (or columns) with NA removed.

Return type:

InfoTable

property eval: Callable

Return the DataFrame.eval method.

Returns:

The eval method for evaluating expressions.

Return type:

Callable

fix_dtypes() → None

Ensure the ‘id’ column is of string type.

classmethod from_dict(df_dict: Dict[str, List[Any]]) → T

Create InfoTable from a dictionary.

Parameters:

df_dict (Dict[str, List[Any]]) – Column data including ‘id’.

Returns:

Constructed table.

Return type:

InfoTable

classmethod from_lists(ids: List[str], column_names: List[str], column_data: List[List[Any]]) → T

Create InfoTable from lists of IDs and corresponding column data.

Parameters:

• ids (List[str]) – List of IDs.

• column_names (List[str]) – List of column names.

• column_data (List[List[Any]]) – Column values.

Returns:

Constructed table.

Return type:

InfoTable

get_col_idx(keys: str | List[str]) → int | ndarray

Get the integer index position(s) of the specified column(s).

Parameters:

keys (str or list) – Column name(s).

Returns:

Position(s) of the column(s).

Return type:

int or np.ndarray

get_loc(keys: str | List[str] | ndarray) → int | ndarray | List[int]

Get integer location(s) for the given key(s).

Parameters:

keys (str, list, or np.ndarray) – Index label(s).

Returns:

Location(s) in the index.

Return type:

int, np.ndarray, or List[int]

harmonize_age_given_decade(voter_columns: str | List[str], target_column: str, decade_column: str = 'age_decade') → None

Force all rows sharing voter_columns to take the averaged age in target_column, constrained by the decade labels in decade_column.

If the averaged age falls outside the decade bounds, the age is set to the upper boundary for that decade.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric age column to harmonize.

• decade_column (str) – Column indicating the age decade labels.

Raises:

• KeyError – If the voter, target, or decade columns are missing.

• TypeError – If the target column is not numeric.

harmonize_column_by_majority_cluster(voter_columns: str | List[str], target_column: str, suspect_column: str = 'suspect', std_threshold: float | None = None, max_iter: int = 20) → None

Harmonize a numeric column using the dominant cluster within each voter group, while flagging rows from the smaller cluster as suspect.

If std_threshold is provided, clustering is only performed when the group’s standard deviation exceeds the threshold.

The standard deviation of the dominant cluster is stored in a column named f"{target_column}_std" for the non-suspect rows.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) defining the group used for averaging.

• target_column (str) – Numeric column to harmonize.

• suspect_column (str) – Column used to flag suspect rows.

• std_threshold (Optional[float]) – If set, only split groups above this std.

• max_iter (int) – Maximum iterations for the 1D two-means refinement.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If the target column is not numeric.

harmonize_columns_by_average(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the average value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the group used for averaging.

• target_columns (Union[str, List[str]]) – Numeric column(s) to harmonize via averaging.

Raises:

• KeyError – If the voter or target columns are missing.

• TypeError – If any target column is not numeric.

harmonize_columns_by_majority_vote(voter_columns: str | List[str], target_columns: str | List[str]) → None

Force all rows sharing voter_columns to take the majority value on target_columns.

Parameters:

• voter_columns (Union[str, List[str]]) – Column(s) that define the voting group (e.g., speaker).

• target_columns (Union[str, List[str]]) – Column or columns to harmonize via majority vote.

Raises:

KeyError – If the voter or target columns are missing.

head(n: int = 5) → T

Return the first n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

histogram(column: str, bins: int | None = None, density: bool = True, kind: str = 'bar', color: str = 'C0', output_file: str | Path | None = None, dropna: bool = True) → None

Plot a histogram of a column, handling string or numeric data.

Parameters:

• column (str) – Column name to plot. Title will include the column name.

• bins (Optional[int]) – Number of bins for numeric columns. Ignored for strings.

• density (bool) – If True, plot density/relative frequency; otherwise counts.

• kind (str) – Histogram style, either “bar” or “line”.

• color (str) – Matplotlib color spec for the plot.

• output_file (Optional[PathLike]) – If provided, save the figure; otherwise show it.

• dropna (bool) – If True, ignore NA values before plotting.

property iat: _InfoTableAtIndexer

Access a single value for a row/column integer position pair.

Returns:

Indexer that wraps .iat.

Return type:

_InfoTableAtIndexer

property iloc: _InfoTableIndexer

Access a group of rows and columns by integer position(s).

Returns:

Indexer that wraps .iloc.

Return type:

_InfoTableIndexer

property index: Index

Get the index of the DataFrame.

Returns:

DataFrame index.

Return type:

pd.Index

static is_valid_df(df: DataFrame) → bool

Check if the DataFrame is valid for InfoTable.

Parameters:

df (pd.DataFrame) – DataFrame to check.

Returns:

True if valid, False otherwise.

Return type:

bool

property iterrows: Callable

Return the DataFrame.iterrows generator.

Returns:

Yields (index, Series) pairs.

Return type:

Callable

legacy_add_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None, on: str | List[str] | ndarray = 'id', right_on: None | str | List[str] | ndarray = None, replace_overlapping: bool = False, ignore_overlapping: bool = False, remove_missing: bool = False) → None

Legacy implementation of add_columns that uses a full pandas merge.

This keeps the original behavior for compatibility/testing.

property loc: _InfoTableIndexer

Access a group of rows and columns by label(s).

Returns:

Indexer that wraps .loc.

Return type:

_InfoTableIndexer

classmethod merge(left_table: T | DataFrame, right_table: T | DataFrame, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Tuple[str, str] = ('_x', '_y'), copy: bool | None = None, indicator: str | bool = False, validate: str | None = None) → T

Merge two InfoTables or DataFrames into a new InfoTable.

Parameters:

• left_table (Union[InfoTable, pd.DataFrame]) – Left-hand table.

• right_table (Union[InfoTable, pd.DataFrame]) – Right-hand table.

• how (str) – Merge method (e.g., ‘inner’, ‘outer’, ‘left’, ‘right’).

• on (Union[str, List[str], None]) – Column(s) to join on.

• left_on (Union[str, List[str], None]) – Column(s) from the left table to join on.

• right_on (Union[str, List[str], None]) – Column(s) from the right table to join on.

• left_index (bool) – Use index from the left table as join key.

• right_index (bool) – Use index from the right table as join key.

• sort (bool) – Sort the result by the join keys.

• suffixes (Tuple[str, str]) – Suffixes to apply to overlapping column names.

• copy (Optional[bool]) – If False, avoid copying data where possible.

• indicator (Union[str, bool]) – Adds a column to the output DataFrame called ‘_merge’.

• validate (Optional[str]) – Check if merge is of specified type.

Returns:

A new merged InfoTable.

Return type:

InfoTable

query(expr: str, **kwargs: Any) → T

Filters rows using a boolean expression string.

Parameters:

• expr (str) – A string expression to evaluate, using column names as variables.

• **kwargs – Passed through to pandas.DataFrame.query().

Returns:

A new InfoTable with filtered rows.

Return type:

InfoTable

replace(to_replace: Any = None, value: Any = None, inplace: bool = False, **kwargs: Any) → T | None

Replace values in the DataFrame.

Parameters:

• to_replace – What to replace.

• value – Value to replace with.

• inplace (bool) – Whether to modify the table in-place.

• **kwargs – Additional keyword args passed to pd.DataFrame.replace().

Returns:

New InfoTable if not inplace, else None.

Return type:

Optional[InfoTable]

replace_columns(right_table: T | DataFrame, column_names: None | str | List[str] | ndarray = None) → None

Replace column values with those from another table.

Parameters:

• right_table (Union[InfoTable, pd.DataFrame]) – Table to source values from.

• column_names (str or list, optional) – Columns to replace. If None, all.

reset_index() → None

Reset the DataFrame index to the ‘id’ column.

Returns:

None

sample(n: int = 1, random_state: int | RandomState | Generator | BitGenerator | None = None) → T

Return a random sample of rows.

Parameters:

• n (int) – Number of rows.

• random_state – Seed for reproducibility.

Returns:

Sampled InfoTable.

Return type:

InfoTable

save(file_path: str | Path, sep: str | None = None) → None

Save the InfoTable to a file.

Parameters:

• file_path (str or Path) – Path to save the file.

• sep (Optional[str]) – Column separator (default inferred from extension).

scatter2d(x_column: str, y_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 2D scattergram for two numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

scatter3d(x_column: str, y_column: str, z_column: str, color: str = 'C0', marker: str = 'o', sample_frac: float = 1.0, output_file: str | Path | None = None, dropna: bool = True) → None

Plot a 3D scattergram for three numeric columns.

Parameters:

• x_column (str) – Column name for the x-axis.

• y_column (str) – Column name for the y-axis.

• z_column (str) – Column name for the z-axis.

• color (str) – Matplotlib color for points.

• marker (str) – Matplotlib marker style.

• sample_frac (float) – Fraction (0,1] of points to plot, sampled randomly.

• output_file (Optional[PathLike]) – If provided, save figure; otherwise show it.

• dropna (bool) – Drop NA rows before plotting.

set_index(keys: str | List[str], inplace: bool = True) → T | None

Set the DataFrame index using one or more columns.

Parameters:

• keys (str or list) – Column(s) to use as index.

• inplace (bool) – Whether to modify in place.

Returns:

Modified InfoTable if not inplace.

Return type:

Optional[InfoTable]

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffle the rows of the InfoTable.

Parameters:

• seed (int) – Seed for random number generator.

• rng (np.random.Generator, optional) – Numpy random generator.

Returns:

Shuffled indices.

Return type:

np.ndarray

sort(column: str = 'id', ascending: bool = True, inplace: bool = True) → T | None

Sort the InfoTable by a specific column.

Parameters:

• column (str) – Column name to sort by.

• ascending (bool) – Sort in ascending order.

• inplace (bool) – Sort in place or return a new object.

Returns:

Sorted InfoTable or None if inplace.

Return type:

Optional[InfoTable]

split(idx: int, num_parts: int, group_by: str | None = None) → T

Split the InfoTable into parts and return the selected part.

Parameters:

• idx (int) – Part to return (1-based).

• num_parts (int) – Total number of parts.

• group_by (Optional[str]) – Column to group by when splitting.

Returns:

The selected part of the split.

Return type:

InfoTable

tail(n: int = 5) → T

Return the last n rows.

Parameters:

n (int) – Number of rows.

Returns:

A new InfoTable.

Return type:

InfoTable

xs(key: Any, axis: int = 0, level: int | str = None, drop_level: bool = True) → T | Series

Returns a cross-section (row or column) from the DataFrame.

Parameters:

• key – Label or tuple of labels to select.

• axis (int) – Axis to retrieve from (0 for index, 1 for columns).

• level – Level in MultiIndex to use.

• drop_level (bool) – Whether to drop the level(s) from the result.

Returns:

If the result is a DataFrame, wraps it as InfoTable.

Return type:

InfoTable or Series

Compatibility helper structures

These types support legacy Kaldi/BOSARIS-style data interchange. Prefer RecordingSet, FeatureSet, VADSet, and CSV-indexed archives in new package code.

class hyperion.utils.SCPList(key: List[str] | ndarray, file_path: List[str] | ndarray, offset: Sequence[int] | ndarray | None = None, range_spec: Sequence[Sequence[int]] | ndarray | None = None)

Class to manipulate script lists.

key

segment key name.

file_path

path to the file on hard drive, wav, ark or hdf5 file.

offset

Byte in Ark file where the data is located.

range_spec

range of frames (rows) to read.

key_to_index

Dictionary that returns the position of a key in the list.

__init__(key: List[str] | ndarray, file_path: List[str] | ndarray, offset: Sequence[int] | ndarray | None = None, range_spec: Sequence[Sequence[int]] | ndarray | None = None) → None

validate() → None

Validates the attributes of the SCPList object.

copy() → SCPList

Makes a copy of the object.

__len__() → int

Returns the number of elements in the list.

len() → int

Returns the number of elements in the list.

_create_dict() → None

Creates dictionary that returns the position of a segment in the list.

get_index(key: Any) → int

Returns the position of key in the list.

__contains__(key: Any) → bool

Returns True if the list contains the key

__getitem__(key: str | int | integer) → Tuple[Any, Any, Any | None, ndarray | None] | Tuple[Any, Any | None, ndarray | None]

Access list data by key or integer index.

For a string key, returns (file_path, offset, range_spec). For an integer index, returns (key, file_path, offset, range_spec).

Parameters:

key – String key or integer index.

Returns:

Data associated with key as described above.

add_prefix_to_filepath(prefix: str) → None

Adds a prefix to the file path

sort() → None

Sorts the list by key

save(file_path: str | Path, sep: str = ' ', offset_sep: str = ':') → None

Saves script list to text file.

Parameters:

• file_path – File to write the list.

• sep – Separator between the key and file_path in the text file.

• offset_sep – Separator between file_path and offset.

static parse_script(script: Sequence[str], offset_sep: str) → Tuple[List[str], List[int] | None, ndarray | None]

Parses the parts of the second field of the scp text file.

Parameters:

• script – Second column of scp file.

• offset_sep – Separator between file_path and offset.

Returns:

file_path, offset and range_spec.

classmethod load(file_path: str | Path, sep: str = ' ', offset_sep: str = ':', is_wav: bool = False) → SCPList

Loads script list from text file.

Parameters:

• file_path – File to read the list.

• sep – Separator between the key and file_path in the text file.

• offset_sep – Separator between file_path and offset.

Returns:

SCPList object.

split(idx: int, num_parts: int, group_by_key: bool = True) → SCPList

Splits SCPList into num_parts and return part idx.

Parameters:

• idx – Part to return from 1 to num_parts.

• num_parts – Number of parts to split the list.

• group_by_key – If True, all the lines with the same key go to the same part.

Returns:

Sub SCPList

classmethod merge(scp_lists: Sequence[SCPList]) → SCPList

Merges several SCPList.

Parameters:

scp_lists – List of SCPLists

Returns:

SCPList object concatenation the scp_lists.

filter(filter_key: List[str] | ndarray, keep: bool = True) → SCPList

Removes elements from SCPList object by key

Parameters:

• filter_key – List with the keys of the elements to keep or remove.

• keep – If True, we keep the elements in filter_key; if False, we remove the elements in filter_key;

Returns:

SCPList object.

filter_paths(filter_key: List[str] | ndarray, keep: bool = True) → SCPList

Removes elements of SCPList by file_path

Parameters:

• filter_key – List with the file_path of the elements to keep or remove.

• keep – If True, we keep the elements in filter_key; if False, we remove the elements in filter_key;

Returns:

SCPList object.

filter_index(index: Sequence[int] | ndarray, keep: bool = True) → SCPList

Removes elements of SCPList by index

Parameters:

• filter_key – List with the index of the elements to keep or remove.

• keep – If True, we keep the elements in filter_key; if False, we remove the elements in filter_key;

Returns:

SCPList object.

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffles the elements of the list.

Parameters:

• seed – Seed for random number generator.

• rng – numpy random number generator object.

Returns:

Index used to shuffle the list.

__eq__(other: object) → bool

Equal operator

__ne__(other: object) → bool

Non-equal operator

__cmp__(other: object) → int

Comparison operator

class hyperion.utils.Utt2Info(utt_info: DataFrame)

Class to manipulate utt2spk, utt2lang, etc. files.

key

Utterance keys.

info

Info values associated to each key.

key_to_index

Dictionary that returns the row position of each key.

__init__(utt_info: DataFrame) → None

validate() → None

Validates the attributes of the Utt2Info object.

classmethod create(key: Sequence[str] | ndarray, info: Sequence[object] | ndarray) → Utt2Info

property num_info_fields: int

property key: ndarray

property info: ndarray

copy() → Utt2Info

Makes a copy of the object.

__len__() → int

Returns the number of elements in the list.

len() → int

Returns the number of elements in the list.

_create_dict() → None

Creates dictionary that returns the position of a segment in the list.

get_index(key: str) → int

Returns the position of key in the list.

__contains__(key: str) → bool

Returns True if the list contains key.

__getitem__(key: str) → object | ndarray

__getitem__(key: int | integer) → Tuple[str, object] | Tuple[str, ndarray]

Return entry by key or index.

Parameters:

key – Utterance key or row index.

Returns:

If key is a string, returns the info value(s) for that key. If key is an integer, returns (key, info) for that row.

sort(field: int | str = 0) → None

Sort rows by key (field=0) or by a selected info field.

save(file_path: str | Path, sep: str = ' ') → None

Save utt2info table to text file.

Parameters:

• file_path – Destination file path.

• sep – Field separator.

classmethod load(file_path: str | Path, sep: str = ' ', dtype: Dict[int, object] | None = None) → Utt2Info

Load an utt2info table from a text file.

Parameters:

• file_path – File to read.

• sep – Field separator.

• dtype – Optional dictionary with pandas dtypes by column index.

Returns:

Loaded Utt2Info object.

split(idx: int, num_parts: int, group_by_field: int | str = 0) → Utt2Info

Split table into num_parts and return part idx.

Parameters:

• idx – Part to return from 1 to num_parts.

• num_parts – Number of parts to split the list.

• group_by_field – If non-zero, rows with the same value in this field are kept in the same split.

Returns:

Sub Utt2Info object.

classmethod merge(info_lists: Sequence[Utt2Info]) → Utt2Info

Merge several Utt2Info tables.

Parameters:

info_lists – List of Utt2Info objects.

Returns:

Concatenated Utt2Info object.

filter(filter_key: Sequence[str] | ndarray, keep: bool = True) → Utt2Info

Filter rows by utterance key.

Parameters:

• filter_key – Keys to keep or remove.

• keep – If True, keep keys in filter_key. If False, remove keys in filter_key.

Returns:

Filtered Utt2Info object.

filter_info(filter_key: Sequence[object] | ndarray, field: int | str = 1, keep: bool = True) → Utt2Info

Filter rows by value in an info field.

Parameters:

• filter_key – Info values to keep or remove.

• field – Column index or name to filter on.

• keep – If True, keep values in filter_key. If False, remove values in filter_key.

Returns:

Filtered Utt2Info object.

filter_index(index: Sequence[int] | ndarray, keep: bool = True) → Utt2Info

Filter rows by positional index.

Parameters:

• index – Integer indices to keep or remove.

• keep – If True, keep index. If False, remove index.

Returns:

Filtered Utt2Info object.

shuffle(seed: int = 1024, rng: Generator | None = None) → ndarray

Shuffles the elements of the list.

Parameters:

• seed – Seed for random number generator.

• rng – numpy random number generator object.

Returns:

Index used to shuffle the list.

__eq__(other: object) → bool

Return True when both tables are equal.

__ne__(other: object) → bool

Return True when tables are different.

__cmp__(other: object) → int

Compatibility comparison method.

class hyperion.utils.SegmentList(segments: DataFrame, index_by_file: bool = True)

Class to manipulate segment files

segments

Pandas dataframe.

_index_by_file

if True the df is index by file name, if False by segment id.

iter_idx

index of the current element for the iterator.

uniq_file_id

unique file names.

__init__(segments: DataFrame, index_by_file: bool = True) → None

classmethod create(segment_id: Sequence[str] | ndarray, file_id: Sequence[str] | ndarray, tbeg: Sequence[float] | ndarray, tend: Sequence[float] | ndarray, index_by_file: bool = True) → SegmentList

validate() → None

Validates the attributes of the SegmentList object.

property index_by_file: bool

property file_id: ndarray

property segment_id: ndarray

property tbeg: ndarray

property tend: ndarray

copy() → SegmentList

Makes a copy of the object.

segments_ids_from_file(file_id: Any) → ndarray

Returns segments_ids corresponding to a given file_id

__len__() → int

Returns the number of segments in the list.

__contains__(key: Any) → bool

Returns True if the segments contains the key

getitem_by_key(key: str) → SegmentList | Series

Access segments by file or segment identifier.

Parameters:

key – Segment or file key.

Returns:

A SegmentList for a file or a row from the segment table.

getitem_by_index(index: int) → SegmentList | Series

Access segments by integer position.

Parameters:

index – Segment or file position.

Returns:

A SegmentList for a file or a row from the segment table.

__getitem__(key: str | int | integer) → SegmentList | Series

Access segments by file/segment key or integer position.

Parameters:

key – Segment or file key, or integer position.

Returns:

A SegmentList for a file or a row from the segment table.

save(file_path: str | Path, sep: str = ' ') → None

Saves segments to text file.

Parameters:

• file_path – File to write the list.

• sep – Separator between the fields

classmethod load(file_path: str | Path, sep: str = ' ', index_by_file: bool = True) → SegmentList

Loads script list from text file.

Parameters:

• file_path – File to read the list.

• sep – Separator between the key and file_path in the text file.

Returns:

SegmentList object.

filter(filter_key: Sequence[Any] | ndarray, keep: bool = True) → SegmentList

split(idx: int, num_parts: int) → SegmentList

classmethod merge(segment_lists: Sequence[SegmentList], index_by_file: bool = True) → SegmentList

to_bin_vad(key: Any, frame_shift: float = 10, num_frames: int | None = None) → ndarray

Converts segments to binary VAD

Parameters:

• key – Segment or file key

• frame_shift – frame_shift in milliseconds

• num_frames – number of frames of file corresponding to key, if None it takes the maximum tend for file

Returns:

if index_by_file is True if returns VAD joining all segments of one file else if returns VAD for one given segment

__eq__(other: object) → bool

Equal operator

__ne__(other: object) → bool

Non-equal operator

__cmp__(other: object) → int

Comparison operator

class hyperion.utils.RTTM(segments: DataFrame, index_by_file: bool = True)

Class to manipulate rttm files

df

Pandas dataframe.

_index_by_file

if True the df is indexed by file name, if False by segment id.

iter_idx

index of the current element for the iterator.

unique_file_key

unique file names.

__init__(segments: DataFrame, index_by_file: bool = True) → None

classmethod create(segment_type: Sequence[Any], file_id: Sequence[Any], chnl: Sequence[Any] | None = None, tbeg: Sequence[Any] | None = None, tdur: Sequence[Any] | None = None, ortho: Sequence[Any] | None = None, stype: Sequence[Any] | None = None, name: Sequence[Any] | None = None, conf: Sequence[Any] | None = None, slat: Sequence[Any] | None = None, index_by_file: bool = True) → RTTM

classmethod create_spkdiar(file_id: Sequence[Any], tbeg: Sequence[Any], tdur: Sequence[Any], spk_id: Sequence[Any], conf: Sequence[Any] | None = None, chnl: Sequence[Any] | None = None, index_by_file: bool = True, prepend_file_id: bool = False) → RTTM

classmethod create_spkdiar_single_file(file_id: str, tbeg: Sequence[Any], tdur: Sequence[Any], spk_id: Sequence[Any], conf: Sequence[Any] | None = None, chnl: Sequence[Any] | None = None, index_by_file: bool = True, prepend_file_id: bool = False) → RTTM

classmethod create_spkdiar_from_segments(segments: DataFrame, spk_id: Sequence[Any], conf: Sequence[Any] | None = None, chnl: Sequence[Any] | None = None, index_by_file: bool = True, prepend_file_id: bool = False) → RTTM

classmethod create_spkdiar_from_ext_segments(ext_segments: Any, chnl: Sequence[Any] | None = None, index_by_file: bool = True, prepend_file_id: bool = False) → RTTM

validate() → None

Validates the attributes of the RTTM object.

property index_by_file: bool

property file_id: ndarray

property tbeg: ndarray

property tdur: ndarray

property name: ndarray

copy() → RTTM

Makes a copy of the object.

property num_files: int

property total_num_spks: int

property num_spks_per_file: Dict[Any, int]

property avg_num_spks_per_file: float

__len__() → int

Returns the number of segments in the list.

__contains__(key: Any) → bool

Returns True if the segments contains the key

__getitem__(key: str | int) → RTTM | Series

Access RTTM segments by file key or integer position.

Parameters:

key – File key or integer position.

Returns:

An RTTM object for a file or a row from the segment table.

save(file_path: str | Path, sep: str = ' ') → None

Saves segments to text file.

Parameters:

• file_path – File to write the list.

• sep – Separator between the fields

classmethod load(file_path: str | Path, sep: str = ' ', index_by_file: bool = True) → RTTM

Loads script list from text file.

Parameters:

• file_path – File to read the list.

• sep – Separator between the key and file_path in the text file.

Returns:

SegmentList object.

filter(filter_key: Sequence[Any], keep: bool = True) → RTTM

split(idx: int, num_parts: int) → RTTM

classmethod merge(rttm_list: Sequence[RTTM], index_by_file: bool = True) → RTTM

merge_adjacent_segments(t_margin: float = 0) → None

__eq__(other: object) → bool

Equal operator

__ne__(other: object) → bool

Non-equal operator

__cmp__(other: object) → int

Comparison operator

get_segment_names_from_timestamps(file_id: Any, timestamps: Sequence[Sequence[float]], segment_type: str = 'SPEAKER', min_seg_dur: float = 0.1) → Tuple[ndarray, List[Any], List[float]]

get_files_with_names_diff_to_file(file_id: Any, segment_type: str = 'SPEAKER') → ndarray

prepend_file_id_to_name(segment_type: str = 'SPEAKER') → None

get_segments_from_file(file_id: Any) → DataFrame

get_uniq_names_for_file(file_id: Any | None = None) → ndarray

get_bin_frame_mask_for_spk(file_id: Any, name: Any, frame_length: float = 0.025, frame_shift: float = 0.01, snip_edges: bool = False, signal_length: float | None = None, max_frames: int | None = None) → ndarray

Returns binary mask of a given speaker to select feature frames

Parameters:

• file_id – file identifier

• name – speaker id

• frame_length – frame-length used to compute the VAD

• frame_shift – frame-shift used to compute the VAD

• snip_edges – if True, computing VAD used snip-edges option

• signal_length – total duration of the signal, if None it takes it from the last timestamp

• max_frames – expected number of frames, if None it computes automatically

Returns:

Binary VAD np.array

get_bin_sample_mask_for_spk(file_id: Any, name: Any, fs: float, signal_length: float | None = None, max_samples: int | None = None) → ndarray

Returns binary mask of a given speaker to select waveform samples

Parameters:

• file_id – file identifier

• name – speaker id

• fs – sampling frequency

• signal_length – total duration of the signal, if None it takes it from the last timestamp

• max_frames – expected number of frames, if None it computes automatically

Returns:

Binary mask np.array

compute_stats(nbins_dur: int | None = None) → Tuple[Series, Tuple[ndarray, ndarray], int]

to_segment_list() → SegmentList

sort() → None

tbeg_is_sorted() → bool

class hyperion.utils.KaldiMatrix(data: ndarray)

Class to read/write uncompressed kaldi matrices/vectors.

When compressed matrix is found in file, it calls KaldiCompressedMatrix class automatically to uncompress.

data

numpy array with the matrix/vector values.

__init__(data: ndarray) → None

to_ndarray() → ndarray

Returns:

numpy array containing the matrix/vector

property num_rows: int

property num_cols: int

classmethod read(f: IO[Any], binary: bool, row_offset: int = 0, num_rows: int = 0, sequential_mode: bool = True) → KaldiMatrix

Reads kaldi matrix/vector from file.

Parameters:

• f – Python file object

• binary – True if we read from binary file and False if we read from text file.

• row_offset – Reads matrix starting from a given row instead of row 0.

• num_rows – Number of rows to read; if 0, read all rows.

• sequential_mode – True if we are reading the ark file sequentially and False if we are using random access.

Returns:

KaldiMatrix object.

write(f: IO[Any], binary: bool) → None

Writes matrix/vector to ark file.

Parameters:

• f – Python file object.

• binary – True if we write in binary file and False if we write to text file.

static read_shape(f: IO[Any], binary: bool, sequential_mode: bool = True) → Tuple[int, ...]

Reads the shape of the current matrix/vector in the ark file.

Parameters:

• f – Python file object

• binary – True if we read from binary file and False if we read from text file.

• sequential_mode – True if we are reading the ark file sequentially and False if we are using random access. In sequential_mode=True it moves the file pointer to the next matrix.

Returns:

Tuple object with shape.

class hyperion.utils.KaldiCompressedMatrix(data: bytes | None = None)

Class to read/write compressed kaldi matrices.

When compressed matrix is found in file, it calls KaldiCompressedMatrix class automatically to uncompress.

data

numpy byte array with the compressed coded matrix.

data_format

{1, 2, 3, 4}

min_value

Minimum value in the matrix.

data_range

max_value - min_value

num_rows

Number of rows in the matrix

num_columns

Number of columns in the matrix

__init__(data: bytes | None = None) → None

get_data_attrs() → Tuple[ndarray, Dict[str, Any]]

Returns:

Coded matrix values in 2D format. Dictionary object with data attributes: data_format, min_value, data_range, percentiles.

classmethod build_from_data_attrs(data: ndarray, attrs: Dict[str, Any]) → KaldiCompressedMatrix

Builds object from coded values and attributes

Parameters:

• data – Coded matrix values in 2D format.

• attrs – Dictionary object with data attributes: data_format, min_value, data_range, percentiles.

Returns:

KaldiCompressedMatrix object.

_unpack_header() → None

Unpacks attributes from header

_pack_header() → bytes

Creates header from the object attributes

scale(alpha: int | float) → None

Multiplies matrix by alpha

_compute_global_header(mat: ndarray, method: str) → bytes

Computes the header

Parameters:

• mat – numpy array with the uncompressed matrix.

• method – Compression method.

Returns:

Byte array with header.

static _get_read_info(header: bytes, row_offset: int = 0, num_rows: int = 0) → Tuple[bytes, int, int, int, int, int]

Gets info needed to read the matrix from file

static _data_size(header: bytes) → int

Returns:

Number of bytes of the coded matrix.

classmethod compress(mat: ndarray | KaldiMatrix, method: str = 'auto') → KaldiCompressedMatrix

Creates compressed matrix from uncompressed numpy matrix :param mat: numpy array with the uncompressed matrix. :param method: Compression method.

Returns:

KaldiCompressedMatrix object.

_compute_column_header(v: ndarray) → bytes

Creates the column headers for the speech-feat compression.

Parameters:

v – numpy array with the column to compress.

Returns:

Byte array with the header of the column containing the 0, 25, 75 and 100 percentile values.

_compress_column(v: ndarray) → Tuple[bytes, bytes]

Compress column for the speech-feat compression.

Parameters:

v – numpy array with the column to compress.

Returns:

Byte array with the header of the column containing the 0, 25, 75 and 100 percentile values. Byte array with the coded column.

_uncompress_column(col_header: bytes, col_data: bytes) → ndarray

Compress column for the speech-feat compression.

Parameters:

• col_header – Byte array with the header of the column containing the 0, 25, 75 and 100 percentile values.

• col_data – Byte array with the coded column.

Returns:

numpy array with the uncompressed column

static _float_to_char(v: ndarray, p0: float, p25: float, p75: float, p100: float) → ndarray

Codes the column from float to bytes using the given percentiles

static _char_to_float(v: bytes, p0: float, p25: float, p75: float, p100: float) → ndarray

Decodes the column from bytes to float using the given percentiles

to_ndarray() → ndarray

Decompresses matrix to a NumPy array. :returns: numpy array with uncompressed matrix.

to_matrix() → KaldiMatrix

Decompresses matrix to a KaldiMatrix object. :returns: KaldiMatrix with uncompressed matrix.

classmethod read(f: IO[Any], binary: bool, row_offset: int = 0, num_rows: int = 0, sequential_mode: bool = True) → KaldiCompressedMatrix

Reads a Kaldi compressed matrix/vector from a file.

Parameters:

• f – Python file object

• binary – True if we read from binary file and False if we read from text file.

• row_offset – Reads matrix starting from a given row instead of row 0.

• num_rows – Number of rows to read; if 0, read all rows.

• sequential_mode – True if we are reading the ark file sequentially and False if we are using random access.

Returns:

KaldiCompressedMatrix object.

write(f: IO[Any], binary: bool) → None

Writes matrix/vector to ark file.

Parameters:

• f – Python file object.

• binary – True if we write in binary file and False if we write to text file.

static read_shape(f: IO[Any], binary: bool, sequential_mode: bool = True) → Tuple[int, ...]

Reads the shape of the current matrix/vector in the ark file.

Parameters:

• f – Python file object

• binary – True if we read from binary file and False if we read from text file.

• sequential_mode – True if we are reading the ark file sequentially and False if we are using random access. In sequential_mode=True it moves the file pointer to the next matrix.

Returns:

Tuple object with shape.

Miscellaneous utilities

hyperion.utils.misc contains low-level convenience helpers used internally by package subsystems. It is not a supported extension namespace; import a named utility only when another documented public API requires it. The supported table, trial, dataset, and Kaldi-style contracts are documented on Foundation API Contracts.