Recognition performance aside, are we real-time?

Among the tools out there, PyTorch offers its profiler to benchmark computational time and memory demands. SpeechBrain wraps this profiler into @profile, @profile_optimiser, @profile_analyst, and @profile_report decorators. While our tutorial suggests how to use them, this recipe helps to figure out a guiding estimate (take it on nominal level) for inference with pretrained models regarding:

• real-time factor
• peak memory

This tool uses the @profile_report decorator which anticipates real-time profiling as:

1. simulate ten batches per duration/batch size setting, ignore first three recordings
2. compute average and standard deviation of the remaining seven profiled recordings
3. report real-time by its upper control limit (µ + 3σ); instead of its expectation only

About 97.5% of attempts to reproduce (with similar hardware, set-up, etc.) reported figures should remain consistent.

In-scope: for researchers and developers to figure out if their systems go somewhat in the right direction.

Out-scope: technology readiness level reporting.

Note: requires PyTorch >= 1.10; the profiler of earlier versions became a legacy profiler.

python profile.py profile.yaml

Specify benchmark configurations in the profile.yaml file

# Which model to profile?
pretrained_model:
  source: speechbrain/asr-wav2vec2-commonvoice-fr  # HuggingFace or local path
  type: EncoderASR  # Pretrained interface

# Which settings should be benchmarked?
profiling_dimensions:
  audio_mockup_secs: [1, 2, 5, 8, 32]  # 1s, 2s, ...
  batch_sizes: [1, 4, 8]  # 1 file per batch, 4 files per batch, ...
  # Fancy to truncate/repeat a real audio?
  example_audio: ../../samples/audio_samples/example2.flac # None -> random data
  # Some audio_mockup_secs x batch_sizes configs can get VRAM intensive
  triangle_only: False  # True -> skip the heavy ones, e.g., 32s x 8 files
  export_logs: True  # export trace logs for visualisation

Note: The choice of a real audio over mock-up/random data is to clarify on the impact of recursive computations whose paths depend on data, e.g., beamforming. (RNN language models are fast on noise: useful for quick inquiry on memory peaks only; not for inquiry on real-time factors.)

How long one test takes depends on its configuration, the pretrained model, and on the machine.

On a 1x V100 GPU, profiling speechbrain/asr-wav2vec2-commonvoice-fr run for ~4 minutes. By contrast, profiling speechbrain/asr-crdnn-rnnlm-librispeech (tracks 80 hypotheses) run for ~24 minutes. (One might consider to track way less hypotheses.)

For low-memory machines, it might be good to benchmark only a few data points of the full duration vs batch size table (triangle_only=True). More demanding settings might not be satisfiable by VRAM.

As an example (only); we report on the speechbrain/asr-wav2vec2-commonvoice-fr pretrained model.
(We used a long LibriSpeech file to get a first picture of the situation.)

As CPU + CUDA, in E-[xponent] notation (real-time: number after the E is negative).

Note: 1E0 = 1x 10^0 = 1.0 is real-time equivalent, every factor below has a negative exponent.

5.15E-02 + 3.62E-02 means

• CPU factor: 5.15E-02 is real-time (factor: 0.0515 < 1)
• CUDA factor: 3.62E-02 is real-time (factor: 0.0362 < 1)
• Total: 1.26E-01 (0.126 < 1) -> real-time

Note: cudaMemcpyAsync is the time-demanding function event on CPU.

On dual-core CPUs.

Note: these values report upper control limits, averages under the impact of deviation. The 16x CPU benchmark appears inconsistent for this setting of durations and batch sizes; a reason could be that more extensive hardware might be suited better for even heavier computations (lower workloads are not processed at full efficiency).

Note: these numbers discern overheads from handling the model itself, the data pipeline, python environments, etc. (the actual VRAM demand will be higher). This overview is purely about the inference step. The profile.py script adds its own overheads on top.

For comparison on 1x V100: speechbrain/asr-crdnn-rnnlm-librispeech.

Note: here, the RNN LM calls up to: 24,731x aten::copy_ (882x for the model above) and 22,4940x cudaLaunchKernel. (This could be because of tensors being created on CPU and then moved to cuda with .to() instead of creating them on the device right away.)

Starting with the PyTorch profiler and benchmark visualisation:

• https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html
• https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/
• https://github.com/pytorch/kineto/tree/main/tb_plugin

Where to go from here:

• https://horace.io/brrr_intro.html (thanks for the xref: @RuABraun !)
• https://pytorch-lightning.readthedocs.io/en/stable/advanced/profiler.html

• Website: https://speechbrain.github.io/
• Code: https://github.com/speechbrain/speechbrain/
• HuggingFace: https://huggingface.co/speechbrain/

Please, cite SpeechBrain if you use it for your research or business.

@misc{ravanelli2024opensourceconversationalaispeechbrain,
      title={Open-Source Conversational AI with SpeechBrain 1.0},
      author={Mirco Ravanelli and Titouan Parcollet and Adel Moumen and Sylvain de Langen and Cem Subakan and Peter Plantinga and Yingzhi Wang and Pooneh Mousavi and Luca Della Libera and Artem Ploujnikov and Francesco Paissan and Davide Borra and Salah Zaiem and Zeyu Zhao and Shucong Zhang and Georgios Karakasidis and Sung-Lin Yeh and Pierre Champion and Aku Rouhe and Rudolf Braun and Florian Mai and Juan Zuluaga-Gomez and Seyed Mahed Mousavi and Andreas Nautsch and Xuechen Liu and Sangeet Sagar and Jarod Duret and Salima Mdhaffar and Gaelle Laperriere and Mickael Rouvier and Renato De Mori and Yannick Esteve},
      year={2024},
      eprint={2407.00463},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2407.00463},
}
@misc{speechbrain,
  title={{SpeechBrain}: A General-Purpose Speech Toolkit},
  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
  year={2021},
  eprint={2106.04624},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  note={arXiv:2106.04624}
}