AudioGAR: Bridging Reconstruction and Generation

in Latent Audio Generative Models

Xianghong Fang^1,*· Geeyang Tay^1,*· Wentao Ma¹· Tim G. J. Rudner^1,2· Dehan Kong¹

¹University of Toronto²Vijil^*Equal contribution

Can we obtain generation-relevant latents that remain paired with source audio for decoder fine-tuning?

In our paper, we show that AudioGAR constructs intermediate latents between reconstruction and generation, retaining source correspondence at lower noise levels to support generation-aware codec decoder adaptation.

31.3%lower FAD on MusicCaps
1.60 → 1.10

16.8%lower FAD on AudioCaps
1.61 → 1.34

1.5%of original training audio
508.3 audio hours

0.26%of original training cost
10.3 H100-equivalent hours

01 / Motivation

One decoder.
Two latent distributions.

Latent audio models learn in two stages. A codec learns to reconstruct audio, then a generative model learns to produce latents in the codec’s space. The decoder trains on encoder-induced latents, but receives generator-produced latents at inference.

For a source waveform x, reconstruction decodes z_e = E_θ(x) ∼ P_e. Generation decodes z_g ∼ P_g, produced from noise and a text caption. When P_e ≠ P_g, the decoder faces a train-generation mismatch.

Strong reconstruction quality therefore need not transfer to generated audio. AudioGAR adapts the decoder to latents that incorporate the generative process while preserving a source waveform for supervision.

02 / Empirical evidence

The mismatch
is measurable

We construct encoder and generator latents from matched audio-caption examples in AudioX. Linear Discriminant Analysis (LDA) finds a direction that separates the two collections, both at the decoder input and within its feature spaces.

LDA density plots for AudioX on AudioCaps, MusicCaps, and VGGSound-Omni, showing separation of encoder and generated latents at the input and three decoder stages. — Figure 1. Empirical evidence of distribution mismatch (paper §2.2). Rows show AudioCaps, MusicCaps, and VGGSound-Omni; columns show latent space, Conv In, Block 2, and Block 4. Curves are Gaussian-smoothed, normalized densities of one-dimensional LDA scores for encoder latents z_e and generated latents z_g; shading marks overlap. AUC is the area under the receiver operating characteristic curve, and d reports standardized separation.

Latent-space AUC reaches 0.962 on AudioCaps, 0.862 on MusicCaps, and 0.845 on VGGSound-Omni. Separation persists after the latents enter the decoder, including Conv In, Block 2, and Block 4.

Reconstruction quality leaves a generation gap

We compare reconstructed and generated audio across 29 model-dataset combinations. FD uses PANNs embeddings; FAD uses VGGish embeddings. The prefixes r and g denote reconstruction and generation. Lower values indicate closer agreement with the reference audio distribution.

Table 1

Reconstruction-generation performance gap

Reconstruction-generation performance gap across AudioCaps, MusicCaps, and VGGSound-Omni. Lower is better for every metric.
Dataset	Model	rFD ↓	gFD ↓	rFAD ↓	gFAD ↓
AudioCaps	AudioLDM-2-Base	3.19	11.54	1.31	2.04
	AudioLDM-2-Large	3.20	11.53	1.31	1.86
	AudioLDM-S-Full	3.05	21.85	1.27	4.79
	AudioX	7.53	11.83	3.49	1.61
	Stable Audio Open	7.53	30.36	3.49	3.41
	Tango	2.97	14.47	1.14	1.52
	Tango Full	2.97	22.23	1.14	4.10
	Tango Full (FT AudioCaps + MusicCaps)	2.97	9.77	1.14	2.59
	Tango Full (FT AudioCaps)	2.97	10.42	1.14	2.39
	Tango (AF-AC, FT AudioCaps)	2.97	12.08	1.14	2.55
MusicCaps	AudioLDM-2-Large	2.20	18.84	0.66	2.51
	AudioLDM-2-Base	2.20	21.71	0.66	3.82
	AudioX	4.31	9.56	0.95	1.60
	Stable Audio Open	4.31	37.72	0.95	3.42
	Tango	2.18	46.80	0.56	5.64
	Tango Full	2.18	35.67	0.56	5.78
	TangoMusic	2.04	15.20	0.42	1.85
	Tango Full (FT AudioCaps + MusicCaps)	2.18	20.43	0.56	3.61
VGGSound-Omni	AudioLDM-2-Base	1.64	10.03	0.49	1.72
	AudioLDM-2-Large	1.64	9.93	0.49	1.49
	AudioLDM-S-Full	1.63	18.78	0.50	3.44
	AudioX	3.66	9.67	1.91	1.74
	Stable Audio Open	3.66	26.07	1.91	2.68
	Tango	1.62	24.03	0.50	2.88
	Tango Full	1.62	19.01	0.50	3.69
	TangoMusic	1.62	13.43	0.50	2.36
	Tango (AF-AC, FT AudioCaps)	1.62	11.72	0.50	1.46
	Tango Full (FT AudioCaps + MusicCaps)	1.62	13.26	0.50	2.28
	Tango Full (FT AudioCaps)	1.62	19.12	0.50	3.06

All 29 comparisons from the supplied table (paper Table 2). Blue columns show FD; green columns show FAD. FT denotes fine-tuning. Model names link to their checkpoints.

rFD is lower than gFD in all 29 comparisons; rFAD is lower than gFAD in 26. For example, Tango on MusicCaps has rFD 2.18 and gFD 46.80. AudioX on AudioCaps is an FAD exception, with rFAD 3.49 and gFAD 1.61.

The AudioCaps quantile comparison checks whether a few models drive the FD gap. It compares the empirical reconstruction and generation distributions across the evaluated models.

Empirical quantile curves of reconstruction FD and generation FD on AudioCaps, separated across the full quantile range with a median gap of 8.95. — Figure 2. AudioCaps quantiles of rFD and gFD (paper Figure 2). The horizontal axis is FD and the vertical axis is quantile level. The generation median exceeds the reconstruction median by 8.95.

The curves remain separated across the full quantile range. The gap alone does not isolate the decoder’s contribution, because generation also depends on the latent generator. Combined with the LDA evidence, it motivates adapting the decoder to generation-relevant inputs.

03 / Method

Generation-aware
decoder adaptation

Fully generated latents lack a paired source waveform. AudioGAR starts from an encoded waveform, adds controlled noise, and denoises through the frozen generative model. Lower-noise outputs retain the correspondence needed for paired supervision.

We first construct intermediate latents, then use them as decoder inputs while supervising against the original audio. The codec encoder and latent generator remain frozen.

AudioGAR encodes a source waveform, perturbs its latent with Gaussian noise, denoises with the frozen generative model, and decodes the intermediate latent back to audio. — Figure 3. AudioGAR latent construction (paper §3.1). E_θ maps source audio x to z_e. Gaussian noise ε ∼ N(0, I) produces z_noisy^t = a_tz_e + b_tε. The frozen generator denoises this input, conditioned on the source caption, to obtain z_g^t ∼ P_g^t. The decoder reconstructs the waveform from this intermediate latent.

The relative noise level η_t = b_t / (a_t + b_t) controls the transition. η_t = 0 recovers reconstruction; η_t = 1 recovers generation from pure noise. Our experiments use a_t = cos(πt/2) and b_t = sin(πt/2).

01 / Encode

Keep the source pair

Encode a waveform with the pretrained, frozen codec encoder.

02 / Perturb

Choose the noise level

Mix the encoder latent with Gaussian noise using the diffusion schedule.

03 / Denoise

Use the generator

Denoise with the frozen model and the caption paired with the source audio.

04 / Adapt

Fine-tune the decoder

Decode the AudioGAR latent and apply the original codec’s decoder-side objective.

L_AudioGAR = L_codec(x, D_ϕ(z_g^t))

We retain the codec’s reconstruction and adversarial objectives and their loss weights. AudioX updates only the codec decoder; TangoMusic jointly updates the VAE decoder and HiFi-GAN vocoder. Both keep the encoder and latent generator frozen.

AudioGAR improves generation by adapting the decoder to generation-relevant latents with paired source supervision.

04 / Distributional trajectory

How do latents shift
as noise increases?

Latent-FD compares empirical means and covariances under a Gaussian approximation, directly in latent space. We measure the distance from each intermediate distribution P_g^t to both the encoder endpoint P_e and generator endpoint P_g.

On AudioCaps, Latent-FD to encoder latents increases with noise while Latent-FD to generated latents decreases. — Figure 4. AudioGAR bridges the two latent distributions (paper §3.2, Figure 4). Blue curves compare P_g^t with P_e; green curves compare P_g^t with P_g. The horizontal axis is noise level η_t; the vertical axis is Latent-FD. Both datasets trace the same transition toward the generation endpoint.

On MusicCaps, intermediate distributions also move away from encoder latents and toward generated latents as noise increases. — Figure 4. AudioGAR bridges the two latent distributions (paper §3.2, Figure 4). Blue curves compare P_g^t with P_e; green curves compare P_g^t with P_g. The horizontal axis is noise level η_t; the vertical axis is Latent-FD. Both datasets trace the same transition toward the generation endpoint.

On both datasets, increasing noise moves AudioGAR latents farther from P_e and closer to P_g. Lower noise preserves source correspondence; larger noise moves closer to generation while weakening that correspondence.

05 / Evaluation

Better generation.
A small adaptation budget.

AudioGAR improves FD and FAD on AudioX for both audio and music generation. It also transfers to TangoMusic’s decoder and vocoder pipeline. The inference pipeline uses the adapted weights with no additional inference cost.

Table 2

Generation before and after AudioGAR

Reproduced baselines and AudioGAR results from paper Table 3.
Dataset	Model	gFD ↓	gFAD ↓	KL ↓	IS ↑	PC ↑	PQ ↑
AudioCaps	AudioX	11.83	1.61	1.31	12.47	3.16	5.73
AudioCaps	AudioX + AudioGAR	11.62	1.34	1.29	12.06	3.11	5.73
MusicCaps	AudioX	9.56	1.60	1.00	3.65	4.78	6.61
	AudioX + AudioGAR	8.33	1.10	0.99	3.59	4.75	6.56
	TangoMusic	15.20	1.85	1.09	2.85	5.62	7.22
	TangoMusic + AudioGAR	14.25	1.47	1.09	2.83	5.61	7.16

Subset of paper Table 3 using our reproduced baselines. Shading identifies AudioGAR rows. KL: Kullback-Leibler divergence; IS: Inception Score; PC: Production Complexity; PQ: Production Quality. FD and FAD measure distributional agreement; PC and PQ use Meta Audiobox Aesthetics.

On MusicCaps, AudioGAR reduces AudioX FAD by 31.3% and TangoMusic FAD by 20.5%. IS and PC decline slightly, and PQ stays unchanged or declines slightly; the main gains are in FD and FAD.

Table 3

AudioX adaptation cost and data

AudioGAR adaptation relative to original AudioX training, from paper Table 1.
Training	H100-equivalent hours	Relative cost	Audio hours	Relative data
Original AudioX	4,000	100%	33,052.3	100%
AudioGAR adaptation	10.3	0.26%	508.3	1.5%

Adaptation runs for 100k steps on one NVIDIA H100 80GB GPU. Cost and data percentages use original AudioX training as the denominator. The 10.3 hours are the adaptation cost, additional to pretrained AudioX.

06 / Ablation & analysis

What makes adaptation work?

Does the choice of decoder latents matter?

We keep the adaptation procedure fixed and replace encoder latents with AudioGAR latents. Fine-tuning on encoder latents already improves FD and FAD, but AudioGAR yields larger gains on both datasets.

Table 4

Decoder input ablation

FD and FAD subset of paper Table 4, comparing pretrained AudioX with two decoder adaptation inputs.
Decoder setting	AudioCaps FD ↓	AudioCaps FAD ↓	MusicCaps FD ↓	MusicCaps FAD ↓
Pretrained AudioX	11.83	1.61	9.56	1.60
Adapt with encoder latents z_e	11.72	1.55	8.87	1.38
Adapt with AudioGAR latents z_g^t	11.62	1.34	8.33	1.10

FD and FAD subset of paper Table 4. All values evaluate generated audio; lower is better.

On MusicCaps, replacing encoder latents with AudioGAR latents lowers FAD from 1.38 to 1.10. This comparison isolates the benefit of generation-aware inputs beyond decoder fine-tuning alone.

07 / Takeaways

Encoder and generator latents differ across the evaluated audio models, and strong reconstruction alone does not predict strong generation.
AudioGAR constructs intermediate latents that incorporate the generative process while retaining source correspondence at lower noise levels.
Adapting decoder components improves FD and FAD on AudioX and TangoMusic without adding inference cost.
AudioX adaptation uses 508.3 hours of audio and 10.3 H100-equivalent hours, or 1.5% of its original training audio and 0.26% of its training cost.

08 / Citation

Build on AudioGAR

Code ↗Models ↗Paper ↗

@article{fang2026audiogar,
  title   = {AudioGAR: Bridging Reconstruction and Generation in Latent Audio Generative Models},
  author  = {Fang, Xianghong and Tay, Geeyang and Ma, Wentao and Rudner, Tim G. J. and Kong, Dehan},
  journal = {Arxiv},
  year    = {2026}
}