SteerSound

Paper: Steer Unified Audio Editing Toward Reliable Source Selection

SteerSound is an instruction-guided audio editing model supporting sound generation, addition, removal, replacement, pitch modification and volume modification. This repository provides the final SteerSound weights and the components needed for inference.

Model components

The audio VAE and DiT editor are trained by the SteerSound team. The text encoder/tokenizer and vocoder use pretrained weights from the sources below.

Component	Source	Files in this repository
Audio VAE (encoder and decoder)	Trained by us	vae/
DiT audio editor	Trained by us	dit/
T5 text encoder and tokenizer	google-t5/t5-base, using its encoder	text_encoder/
HiFi-GAN vocoder	cvssp/audioldm2 — vocoder	vocoder/

Quick start

Use Linux, Python 3.10 or 3.11, and an NVIDIA GPU. Install a CUDA-enabled PyTorch build and the inference package:

git clone https://github.com/GeWu-Lab/SteerSound.git
cd SteerSound
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install torch==2.7.1 torchaudio==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
pip install -r requirements.txt
pip install -e . --no-deps

import soundfile as sf
from steersound import SteerSoundPipeline

pipe = SteerSoundPipeline.from_pretrained("GeWu-Lab/SteerSound", device="cuda")

# Edit an existing recording.
audio = pipe("Remove the sound of a dog barking.", audio="input.wav")
sf.write("edited.wav", audio, pipe.sample_rate, subtype="PCM_16")

# Generate audio without an input recording.
audio = pipe("Generate the sound of rain.")
sf.write("rain.wav", audio, pipe.sample_rate, subtype="PCM_16")

Command-line inference:

python inference/infer.py \
  --model GeWu-Lab/SteerSound \
  --instruction "Remove the sound of a dog barking." \
  --input input.wav --output edited.wav

Instruction examples

Task	Instruction
Generation	Generate the sound of rain.
Addition	Add the sound of a dog barking.
Removal	Remove the sound of a dog barking.
Replacement	Replace the sound of a dog barking by the sound of a cat meowing.
Volume modification	Increase the volume of the sound of a dog barking by 3.0 dB.
Pitch modification	Reduce the pitch of the sound of a dog barking by 2.0 semitones.

Supply a plain English instruction. The inference code identifies the task and change amount and adds model-control tokens internally. Editing tasks require input audio; generation does not.

For pitch and volume, use Increase or Reduce with an explicit magnitude. Volume changes support ±5 dB; pitch changes support ±4 semitones in increments of 0.5. A temporal instruction can specify a region, for example: Increase the volume of the sound of a dog barking from 2.0 to 4.0 seconds by 3.0 dB. Temporal regions are interpreted by the model from the instruction. Unsupported or ambiguous instructions raise an error.

Audio format and inference settings

Input files are downmixed to mono, resampled to 16 kHz, and truncated or right-padded to 10 seconds. Output is a 10-second mono waveform at 16 kHz. The default runtime uses BF16, PyTorch SDPA and 50 Euler steps, with guidance scale 8.5 for Addition and 3.5 for the other tasks. Use --seed with the CLI or seed=... with the Python API for repeatable sampling.

Model size and GPU memory

Component	Parameters
DiT editor	0.673B
Audio VAE	0.509B
T5 encoder	0.110B
HiFi-GAN vocoder	0.055B
Total	1.348B

GPU memory for inference: approximately 8 GB (BF16, batch size 1, 10-second audio).

Fine-tuning

Pretraining and fine-tuning with standard FM: follow Source audio and preparation to prepare the paper's source audio or your own audio–caption data. This shared guide covers source dataset links, formats, processing steps, and expected outputs. Editing examples are constructed online. Use the standard training guide to pretrain or fine-tune, then evaluate on Complex val/test.

STEER post-training: start from the released SteerSound checkpoint and choose one of these settings:

1. Complex audio editing data: prepare the paper's source audio or your own audio–caption data with the same Source audio and preparation guide. Apply Emphasis or Recovery with online editing-example construction; validate and evaluate on complex_val / complex_test.
2. Selective audio editing data: download the released selective_train, selective_val, and selective_test from the 🤗 dataset. Follow Emphasis / Selective or Recovery / Selective to post-train on the supplied editing pairs and evaluate on the corresponding splits.

Model files

The DiT and VAE weights are stored in BF16. The T5 encoder and vocoder files are stored in FP32 and cast to the runtime dtype when loaded. Architecture and audio settings are in config.json.