PyTorch Training Support

This reference covers the reusable pieces that sit between a task model and a trainer: classification output layers, metrics, loggers, and learning-rate or weight-decay schedules. Configure these through the existing trainer and factory interfaces; avoid duplicating their update loops in a model.

Classification output layers

These layers consume embeddings shaped (batch, in_feats) and return logits shaped (batch, num_classes). During training, pass class labels so the margin penalty applies to the target class. Margin warmup is part of the model state: on resume, use the trainer’s restored epoch/step state rather than resetting the output layer schedule.

For a changed speaker inventory, use the task model’s documented output-layer rebuild method. Do not load a classifier head across incompatible class-id mappings without an explicit migration policy.

Metrics and loggers

class hyperion.torch.metrics.metrics.TorchMetric(*args: Any, **kwargs: Any)

Base class for metrics that cannot be objective functions

__init__(weight=None, reduction='mean')

class hyperion.torch.loggers.logger.Logger

Base class for logger objects

params

training params dictionary

__init__() → None

Initializes logger state and distributed rank information.

on_epoch_begin(epoch: int, logs: Dict[str, Any] | None = None, **kwargs: Any) → None

At the start of an epoch

Parameters:

• epoch – index of the epoch

• logs – dictionary of logs

on_epoch_end(logs: Dict[str, Any] | None = None, **kwargs: Any) → None

At the end of an epoch

Parameters:

logs – dictionary of logs

on_val_end(logs: Dict[str, Any] | None = None, **kwargs: Any) → None

At the end of validation

Parameters:

logs – dictionary of logs

on_batch_begin(batch: int, logs: Dict[str, Any] | None = None, **kwargs: Any) → None

At the start of a batch

Parameters:

• batch – batch index within the epoch

• logs – dictionary of logs

on_batch_end(logs: Dict[str, Any] | None = None, **kwargs: Any) → None

At the end of a batch

Parameters:

logs – dictionary of logs

on_model_update(step: int, logs: Dict[str, Any] | None = None, **kwargs: Any) → None

At the end of a model update

Parameters:

• step – index of the step

• logs – dictionary of logs

on_train_begin(logs: Dict[str, Any] | None = None, **kwargs: Any) → None

At the start of training

Parameters:

logs – dictionary of logs

on_train_end(logs: Dict[str, Any] | None = None, **kwargs: Any) → None

At the end of training

Parameters:

logs – dictionary of logs

TorchMetric subclasses are per-batch training measurements; they are not a replacement for the trial-based verification metrics in Metrics and Evaluation API. Logger callbacks receive a shared logs mapping through the displayed lifecycle. Use LoggerList to fan the events out to CSV, progress, TensorBoard, or Weights & Biases loggers. In distributed training, ensure a logger’s output location and rank behavior are intentional before enabling it.

Schedulers and checkpoints

class hyperion.torch.lr_schedulers.lr_scheduler.LRScheduler(optimizer: torch.optim.Optimizer, min_lr: float | Sequence[float] = 0, warmup_steps: int = 0, epoch: int = 0, step: int = 0, update_lr_on_opt_step: bool = True)

Base class for project learning-rate schedulers.

This scheduler supports optional linear warmup from a near-zero value to each parameter group’s base learning rate.

optimizer

Wrapped optimizer.

min_lrs

Per-parameter-group minimum learning rates.

base_lrs

Per-parameter-group base (initial) learning rates.

warmup_steps

Number of optimization steps used for linear warmup.

epoch

Current epoch index.

step

Current optimization-step index.

update_lr_on_opt_step

If True, update LR on each optimizer step; otherwise update LR on epoch boundaries.

__init__(optimizer: torch.optim.Optimizer, min_lr: float | Sequence[float] = 0, warmup_steps: int = 0, epoch: int = 0, step: int = 0, update_lr_on_opt_step: bool = True) → None

Initialize scheduler state.

Parameters:

• optimizer – Wrapped optimizer.

• min_lr – Scalar or per-parameter-group lower bound.

• warmup_steps – Number of optimization steps used for linear warmup.

• epoch – Initial epoch index (for checkpoint resume).

• step – Initial optimization-step index (for checkpoint resume).

• update_lr_on_opt_step – If True, update LR on each optimizer step; otherwise update LR on epoch boundaries.

property in_warmup: bool

Whether the scheduler is currently in the warmup phase.

state_dict() → Dict[str, Any]

Return scheduler state for checkpointing.

The optimizer object itself is excluded.

load_state_dict(state_dict: Mapping[str, Any]) → None

Load scheduler state from state_dict().

Parameters:

state_dict – Serialized scheduler state.

get_warmup_lr() → List[float]

Compute warmup learning rates for each parameter group.

get_lr(step: int) → List[float]

Compute learning rates for a given step/epoch index.

Parameters:

step – Current scheduler index (step or epoch depending on usage).

on_epoch_begin(epoch: int | None = None, **kwargs: Any) → None

Update learning rates at epoch start when configured for epoch updates.

on_epoch_end(metrics: Mapping[str, Any] | None = None) → None

Advance epoch counter at epoch end.

on_opt_step() → None

Update learning rates after an optimization step.

class hyperion.torch.wd_schedulers.wd_scheduler.WDScheduler(optimizer: torch.optim.Optimizer, initial_wd: float | Sequence[float] = 1e-05, warmup_steps: int = 0, epoch: int = 0, step: int = 0, update_wd_on_opt_step: bool = False)

Base class for weight decay schedulers.

This scheduler supports per-parameter-group weight decay scheduling either on optimizer steps or on epoch boundaries.

optimizer

Wrapped optimizer.

initial_wds

Per-parameter-group initial weight decays.

final_wds

Per-parameter-group final/target weight decays.

warmup_steps

Number of warmup steps used by scheduler-specific policies.

epoch

Current epoch index.

step

Current optimization-step index.

update_wd_on_opt_step

If True, update WD on optimizer steps; otherwise on epoch boundaries.

__init__(optimizer: torch.optim.Optimizer, initial_wd: float | Sequence[float] = 1e-05, warmup_steps: int = 0, epoch: int = 0, step: int = 0, update_wd_on_opt_step: bool = False) → None

Initialize scheduler state.

Parameters:

• optimizer – Wrapped optimizer.

• initial_wd – Scalar or per-group initial weight decay.

• warmup_steps – Number of warmup steps used by scheduler policy.

• epoch – Initial epoch index (for checkpoint resume).

• step – Initial optimization-step index (for checkpoint resume).

• update_wd_on_opt_step – Whether to update WD every optimizer step.

property in_warmup: bool

Whether the scheduler is currently in warmup.

state_dict() → Dict[str, Any]

Return scheduler state for checkpointing.

The optimizer object itself is excluded.

load_state_dict(state_dict: Mapping[str, Any]) → None

Load scheduler state from state_dict().

Parameters:

state_dict – Serialized scheduler state.

get_wd(step: int) → List[float]

Compute per-group weight decays for a given step/epoch index.

Parameters:

step – Current scheduler index (step or epoch depending on usage).

on_epoch_begin(epoch: int | None = None, **kwargs: Any) → None

Update WDs at epoch start when configured for epoch updates.

on_epoch_end(metrics: Mapping[str, Any] | None = None) → None

Advance epoch counter at epoch end.

on_opt_step() → None

Update WDs after an optimization step.

Learning-rate and weight-decay schedulers maintain both epoch and optimizer-step counters. update_lr_on_opt_step and update_wd_on_opt_step determine the time axis of a schedule; warmup is measured in optimizer steps. Store scheduler state in the same checkpoint as the optimizer and trainer, otherwise resuming can silently change the effective schedule.

Available learning-rate families include exponential, inverse-power, Noam, cosine, triangular, Adam cosine, and reduce-on-plateau. Weight-decay scheduling currently provides cosine scheduling. Select them via the factories in PyTorch Extension Points.

See also

• PyTorch Extension Points

• Run Resumable, Mixed-Precision, and Distributed Training

• Use Configuration Files