Inconsistent SoX speed behaviour compared to WavAugment

[pzelasko]

🐛 Bug

With the following effect chain:

def speed(sampling_rate: int) -> List[List[str]]:
    return [
        # Random speed perturbation factor between 0.9x and 1.1x the original speed
        ['speed', RandomValue(0.9, 1.1)],
        ['rate', sampling_rate],  # Resample back to the original sampling rate (speed changes it)
    ]

being applied to a tensor of 16000 samples (with a sampling rate of 16000) I am receiving an output that has a different number of samples than the input. This behavior is inconsistent with what WavAugment does - the shapes are always identical.

This is tested in Lhotse here for WavAugment and here for torchaudio in PR lhotse-speech/lhotse#124.

To Reproduce

Run Lhotse's test suite with torchaudio 0.7 installed. See e.g. the error in Lhotse's CI here: https://github.com/lhotse-speech/lhotse/pull/124/checks?check_run_id=1386141728

Expected behavior

Equal input and output tensor shapes.

Environment

This is my local MacOS env; the other one is GitHub Actions CI in Lhotse.

PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Mac OSX 10.15.7 (x86_64)
GCC version: Could not collect
Clang version: 12.0.0 (clang-1200.0.32.21)
CMake version: version 3.18.4

Python version: 3.7 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.18.1
[pip3] torch==1.7.0
[pip3] torchaudio==0.7.0a0+ac17b64
[conda] blas 1.0 mkl
[conda] mkl 2019.4 233
[conda] mkl-service 2.3.0 py37hfbe908c_0
[conda] mkl_fft 1.2.0 py37hc64f4ea_0
[conda] mkl_random 1.1.1 py37h959d312_0
[conda] numpy 1.18.1 py37h7241aed_0
[conda] numpy-base 1.18.1 py37h3304bdc_1
[conda] pytorch 1.7.0 py3.7_0 pytorch
[conda] torchaudio 0.5.1 pypi_0 pypi

(ignore that last one version, it's some conda+pip quirk... the torchaudio version is 0.7.0 😅)

Additional context

[mthrok]

Hi @pzelasko

How off are they each other?

My initial thought is that this is not a bug but the way WavAugment handles speed is different.
Second thought is that based on the libsox being used, it generates a slightly different results. I noticed this when I was writing tests for the sox effects. (sox command installed via OS utility vs the one cmake builds as a part of torchaudio build process)

apply_effects_tensor and apply_effects_file are tested and confirmed that they return an identical result to sox command for speed as the following.

audio/test/torchaudio_unittest/assets/sox_effect_test_args.json

Lines 73 to 74 in 0e1d814

	{"effects": [["speed", "1.3"]], "input_sample_rate": 4000, "output_sample_rate": 5200}
	{"effects": [["speed", "0.7"]], "input_sample_rate": 4000, "output_sample_rate": 2800}

audio/test/torchaudio_unittest/sox_effect/sox_effect_test.py

Lines 54 to 79 in 0e1d814

	@parameterized.expand(
	load_params("sox_effect_test_args.json"),
	name_func=lambda f, i, p: f'{f.__name__}_{i}_{p.args[0]["effects"][0][0]}',
	)
	def test_apply_effects(self, args):
	"""`apply_effects_tensor` should return identical data as sox command"""
	effects = args['effects']
	num_channels = args.get("num_channels", 2)
	input_sr = args.get("input_sample_rate", 8000)
	output_sr = args.get("output_sample_rate")
	input_path = self.get_temp_path('input.wav')
	reference_path = self.get_temp_path('reference.wav')
	original = get_sinusoid(
	frequency=800, sample_rate=input_sr,
	n_channels=num_channels, dtype='float32')
	save_wav(input_path, original, input_sr)
	sox_utils.run_sox_effect(
	input_path, reference_path, effects, output_sample_rate=output_sr)
	expected, expected_sr = load_wav(reference_path)
	found, sr = sox_effects.apply_effects_tensor(original, input_sr, effects)
	assert sr == expected_sr
	self.assertEqual(expected, found)

audio/test/torchaudio_unittest/sox_effect/sox_effect_test.py

Lines 103 to 128 in 0e1d814

	@parameterized.expand(
	load_params("sox_effect_test_args.json"),
	name_func=lambda f, i, p: f'{f.__name__}_{i}_{p.args[0]["effects"][0][0]}',
	)
	def test_apply_effects(self, args):
	"""`apply_effects_file` should return identical data as sox command"""
	dtype = 'int32'
	channels_first = True
	effects = args['effects']
	num_channels = args.get("num_channels", 2)
	input_sr = args.get("input_sample_rate", 8000)
	output_sr = args.get("output_sample_rate")
	input_path = self.get_temp_path('input.wav')
	reference_path = self.get_temp_path('reference.wav')
	data = get_wav_data(dtype, num_channels, channels_first=channels_first)
	save_wav(input_path, data, input_sr, channels_first=channels_first)
	sox_utils.run_sox_effect(
	input_path, reference_path, effects, output_sample_rate=output_sr)
	expected, expected_sr = load_wav(reference_path)
	found, sr = sox_effects.apply_effects_file(
	input_path, effects, normalize=False, channels_first=channels_first)
	assert sr == expected_sr
	self.assertEqual(found, expected)

[pzelasko]

Yeah, bug might not be the best description - sorry.

I don't think it's actually the speed behaviour, seems like the difference is in how the rate effect is being applied. Consider the following:

import torch
from torchaudio import sox_effects as se
audio = torch.sin(2 * 3.14 * torch.linspace(0, 1, 16000)).unsqueeze(0)

se.apply_effects_tensor(audio, 16000, [['speed', '0.9']])
Out[10]: (tensor([[ 0.0000,  0.0004,  0.0008,  ..., -0.0040, -0.0036, -0.0032]]), 14400)

se.apply_effects_tensor(audio, 16000, [['speed', '0.9']])[0].shape
Out[11]: torch.Size([1, 16000])

se.apply_effects_tensor(audio, 16000, [['speed', '0.9'], ['rate', '16000']])[0].shape
Out[12]: torch.Size([1, 17778])

se.apply_effects_tensor(audio, 16000, [['speed', '0.9'], ['rate', '-q', '16000']])[0].shape
Out[13]: torch.Size([1, 17778])

That last effect chain in WavAugment would have returned a tensor of shape [1, 16000]. Now that I think of it, it's reasonable that the number of samples would have changed... I'll need to check what are they doing (maybe truncating/padding the output?), if you happen to know please share :)

[pzelasko]

OK I think I understand it now. When the output length description is specified in their effect chain invocation, they are truncating or zero-padding the signal (I see that here: https://github.com/facebookresearch/WavAugment/blob/master/augment/speech_augment.h#L120). Sorry for the false alarm! 😉

[mthrok]

Glad you figured. :)