Acoustic Species Identification in the Pantanal

Embeddings come from BirdNET v2.4 [3, 4]. Each file is loaded at 48 kHz mono, split into non-overlapping 3-second chunks, and passed through BirdNET for one 1024-d vector per chunk; one chunk per training file is retained for tractable runtime. Vectors are then L2-normalised as in Section 3 (Listing 1).

def get_embeddings(chunks: list[np.ndarray]) -> np.ndarray:
    # One 1024-d BirdNET embedding per 3-second mono 48 kHz chunk.
    prepared = [np.pad(y, (0, CHUNK_SAMPLES - len(y)))[:CHUNK_SAMPLES]
                for y in chunks]
    inputs = [(chunk, SR) for chunk in prepared]
    result = _birdnet_model.encode_arrays(inputs, device="CPU", ...)
    return result.embeddings[:, 0, :].astype(np.float32)  # (N, 1024)

def l2_normalize(X: np.ndarray) -> np.ndarray:
    # Unit-normalise rows so cosine distance governs classification.
    norms = np.linalg.norm(X, axis=1, keepdims=True)
    return X / np.maximum(norms, 1e-8)

The t-SNE projection of the 20 most populated species (Figure 2) shows mostly compact, well-separated clusters — explaining the high macro-AUC of even a linear classifier — with a crowded centre of acoustically ambiguous classes.

The fine-tuned backbones and from-scratch CNNs instead convert each 3-second clip to a 128 × 128 log-mel spectrogram:

• 128 Mel bands, FFT size 2048, power spectrogram converted to dB with power_to_db;
• time axis zero-padded or truncated to exactly 128 frames;
• the fusion backbones use per-sample min–max scaling to [0, 1]; the from-scratch CNNs use a global mean/standard-deviation normaliser estimated from 500 randomly sampled training clips.

The single-channel image is then resized to each backbone's native resolution — 224 × 224 for the AST/ViT and ConvNeXt fusion models, 260 × 260 for EfficientNet-B2 — while CNN14 consumes the 128 × 128 image directly (Listing 2).

def audio_to_logmel(y, n_mels=128, n_frames=128) -> np.ndarray:
    # 128x128 per-sample-normalised log-mel spectrogram for a 3 s chunk.
    y = np.pad(y, (0, CHUNK_SAMPLES - len(y)))[:CHUNK_SAMPLES]
    hop = CHUNK_SAMPLES // n_frames
    mel = librosa.feature.melspectrogram(
        y=y, sr=SR, n_fft=2048, hop_length=hop,
        n_mels=n_mels, fmin=50, fmax=SR // 2)
    logmel = librosa.power_to_db(mel, ref=np.max)        # to decibels
    logmel = logmel[:, :n_frames]                        # fix time axis
    lo, hi = float(logmel.min()), float(logmel.max())
    return ((logmel - lo) / (hi - lo + 1e-6)).astype(np.float32)

SpecAugment [11] is applied on-the-fly during from-scratch training: a random time mask (up to 30 frames) and frequency mask (up to 15 bins) are zeroed per sample to discourage over-fitting (Listing 3).

def _spec_augment_tensor(mel, max_t=30, max_f=15):
    mel = mel.clone()
    _, f_dim, t_dim = mel.shape
    fill = float(mel.min())
    t = int(torch.randint(1, max_t + 1, ()).item())       # time mask width
    t0 = int(torch.randint(0, max(1, t_dim - t), ()).item())
    mel[:, :, t0:t0 + t] = fill
    f = int(torch.randint(1, max_f + 1, ()).item())        # freq mask width
    f0 = int(torch.randint(0, max(1, f_dim - f), ()).item())
    mel[:, f0:f0 + f, :] = fill
    return mel

The representative spectrograms in Figure 3 — narrow tonal whistles, broadband clicks, harmonic stacks, insect buzzes — show the diversity a single model must generalise across all 206 species.

Three classifiers are trained on the 1024-d L2-normalised embeddings; their key hyperparameters are in Table 1.

The two classical heads are compact (Listing 4); both use balanced class weights so rare species are not ignored.

def fit_lgbm(X, y, n_classes):
    clf = lgb.LGBMClassifier(
        n_estimators=500, num_leaves=127, learning_rate=0.03,
        class_weight="balanced", subsample=0.8, colsample_bytree=0.8,
        min_child_samples=5, reg_lambda=0.1, random_state=42)
    return clf.fit(X, y)

def fit_logreg(X, y):
    clf = LogisticRegression(
        max_iter=2000, C=0.5, class_weight="balanced",
        solver="lbfgs", multi_class="multinomial", random_state=42)
    return clf.fit(X, y)

Mixup (Section 3) is applied to embedding/label pairs with λ ∼ Beta(0.2, 0.2). The implementation (Listing 5) interpolates a batch with a shuffled copy of itself and returns both label targets for a λ-weighted loss.

def _mixup(xb, yb, alpha=0.2):
    # Interpolate pairs of samples and their labels.
    lam = float(np.random.beta(alpha, alpha))
    idx = torch.randperm(xb.size(0), device=xb.device)
    return lam * xb + (1 - lam) * xb[idx], lam, yb, yb[idx]

# inside the training loop:
xb_m, lam, ya, yb_m = _mixup(xb, yb)
logits = model(xb_m)
loss = lam * crit(logits, ya) + (1 - lam) * crit(logits, yb_m)

Fine-tuning takes two forms: replacement classifier heads trained on the frozen BirdNET embedding, and pretrained image backbones fine-tuned on the spectrogram and fused with the embedding.

BirdNET replacement heads.BirdNET v2.4 ships as a TFLite graph whose convolutional feature extractor is inference-only, so its weights cannot be back-propagated. BirdNET-Analyzer's transfer-learning recipe therefore freezes the extractor and trains a new head on the frozen 1024-d embeddings — the REPLACE and REPLACE_HIDDEN modes (Figure 5):

• Linear (REPLACE): BatchNorm → Dropout(0.5) → Linear(1024→C) — a single output layer;
• Hidden (REPLACE_HIDDEN): one Linear(1024→1024) + BatchNorm + ReLU + Dropout(0.5) block before the output layer.

An input BatchNorm standardises the embedding scale. Both heads train for 15 epochs with AdamW (η = 10⁻³, weight decay 10⁻⁴), cosine annealing, a focal loss [9] (γ = 2, label smoothing ε = 0.1) with inverse-frequency class weights, a class-balanced sampler and Mixup [10]; the best validation-AUC checkpoint is restored.

Listing 6 gives the focal loss of Section 3, with label smoothing and inverse-frequency class weights.

class _FocalLoss(nn.Module):
    def __init__(self, weight=None, gamma=2.0, label_smoothing=0.1):
        super().__init__()
        self.weight, self.gamma = weight, gamma
        self.label_smoothing = label_smoothing

    def forward(self, logits, targets):
        ce = F.cross_entropy(logits, targets, weight=self.weight,
                             label_smoothing=self.label_smoothing,
                             reduction="none")
        pt = torch.exp(-ce)                          # prob of true class
        return ((1 - pt) ** self.gamma * ce).mean()

Spectrogram backbone + embedding fusion.Two ImageNet-pretrained backbones are fine-tuned and fused with the BirdNET embedding (Figure 6): an AST-style ViT-Base [5] and a ConvNeXt-Small [6]. Each maps the 224 × 224 spectrogram to a 768-d vector, concatenated with the BatchNorm-standardised 1024-d embedding and passed to a two-layer fusion head. The full stack trains for 8 epochs with AdamW (backbone lr 10⁻⁴, head lr 10⁻³), cosine annealing, the same focal loss and sampler, joint spectrogram/embedding mixup, and mixed precision (Listing 7).

class SpecEmbFusion(nn.Module):
    def __init__(self, backbone_name, emb_dim, n_classes, dropout=0.3):
        super().__init__()
        self.backbone = timm.create_model(
            backbone_name, pretrained=True, num_classes=0, in_chans=1)
        self.emb_bn = nn.BatchNorm1d(emb_dim)
        self.head = nn.Sequential(
            nn.Linear(self.backbone.num_features + emb_dim, 512),
            nn.BatchNorm1d(512), nn.GELU(), nn.Dropout(dropout),
            nn.Linear(512, n_classes))

    def forward(self, spec, emb):
        spec = F.interpolate(spec, size=SPEC_INPUT, mode="bilinear")
        feat = self.backbone(spec)                   # 768-d
        emb = self.emb_bn(emb.float())               # 1024-d
        return self.head(torch.cat([feat, emb], dim=1))

In the Experiment A validation curves (Figure 7), the two BirdNET heads start high and converge within a few epochs, learning only a thin classifier over BirdNET's features, while the AST/ViT and ConvNeXt backbones climb more slowly as they adapt the spectrogram mapping.

Two randomly initialized CNNs are trained on the log-mel spectrogram with no BirdNET signal (Figure 8):

• EfficientNet-B2 [7] (tf_efficientnet_b2, in_chans=1), with a BatchNorm + Dropout(0.4) + Linear head on its 1408-d pooled features;
• PANNs CNN14 [8], the standard six-block VGG-style audio CNN (channels 64→2048), each block two 3×3 conv → BN → ReLU followed by average pooling, then time-pooling, a (max + mean) frequency aggregation, a Linear(2048→2048) + ReLU, and a Dropout(0.5) + Linear output.

Both train for 25 epochs with Adam (η = 10⁻⁴, weight decay 10⁻⁴), cosine annealing, the same focal loss and sampler, and on-the-fly SpecAugment; the best validation checkpoint is restored. Carrying no pretrained features, they serve as a BirdNET-free baseline. Listing 8 shows the PANNs CNN14 backbone block and its (max + mean) frequency aggregation.

class _Cnn14ConvBlock(nn.Module):       # two 3x3 conv -> BN -> ReLU, then avg-pool
    def forward(self, x, pool_size=(2, 2)):
        x = F.relu_(self.bn1(self.conv1(x)))
        x = F.relu_(self.bn2(self.conv2(x)))
        return F.avg_pool2d(x, kernel_size=pool_size)

class Cnn14Bird(nn.Module):             # six blocks: 64 -> 128 -> ... -> 2048
    def features(self, x):
        x = self.bn0(x.transpose(1, 2)).transpose(1, 2)   # normalise mel bins
        for block in (self.block1, ..., self.block6):
            x = F.dropout(block(x), 0.2, self.training)
        x = torch.mean(x, dim=3)                          # pool over time
        x = torch.max(x, dim=2)[0] + torch.mean(x, dim=2) # max + mean over freq
        return F.relu_(self.fc1(F.dropout(x, 0.5, self.training)))

AudioLDM2 encodes the text prompt with a CLAP encoder, runs a denoising diffusion U-Net in the latent space of a mel-spectrogram VAE, and decodes to a waveform with a HiFi-GAN vocoder (Figure 10). The pretrained pipeline runs in half precision on GPU.

Two settings control the pipeline:

• Text prompt: per species code cls, "A field recording of a {cls} bird call in a natural environment";
• Quality filter: following the spec's verification step, a clip is kept only if the target species appears in BirdNET's top-5 predictions with confidence ≥ 0.25. Disabled by default, as it costs an extra BirdNET pass per clip for marginal gains.

The per-class budget adapts to the mean intra-class embedding variance v_c = 1d∑_k=1^dVar_{i: yi=c}[X_ik], so acoustically diverse species receive more samples:

Listing 9 shows the generation loop: each under-represented class is topped up to its budget T_c, and every clip is resampled from AudioLDM2's native 16 kHz to the 48 kHz BirdNET expects.

var = _class_intra_variance(X_tr, y_tr_int, n_classes)
max_var = max(float(var.max()), 1.0)
for cls_str in le.classes_:
    cls_idx = int(le.transform([cls_str])[0])
    cls_target = target + int(SYNTH_EXTRA * (var[cls_idx] / max_var))
    need = max(0, cls_target - int(counts.get(cls_str, 0)))
    prompt = _species_prompt(cls_str)
    for _ in range(need):
        audio = pipe(prompt, num_inference_steps=AUDIOLDM2_STEPS,
                     audio_length_in_s=CHUNK_DUR).audios[0]
        audio = librosa.resample(audio, orig_sr=AUDIOLDM2_SR, target_sr=SR)
        sf.write(str(SYNTH_DIR / f"{cls_str}_{...}.wav"), audio, SR)

AudioLDM2 generated 3,054 synthetic samples, growing the training set from 28,436 to 31,490 examples (+10.7%), including clips for the four singleton species.

All models of Section 5 are retrained on the augmented set with identical architecture and schedule; their validation curves follow the Experiment A pattern.

Three patterns stand out. First, results split along the BirdNET boundary: every BirdNET-based model clears 0.96, while the from-scratch CNNs reach only 0.78–0.86, a 0.12–0.19 gap isolating the value of BirdNET's pretraining. Second, the BirdNET group is tightly clustered (0.960–0.977), so a lightweight one-hidden-layer head beats the heavier fine-tuned backbones for the best single result, 0.9770. Third, AudioLDM2 augmentation improves all nine models, modestly for the strong embedding models (+0.007 to +0.014) but largest where data is scarcest (PANNs CNN14 +0.0416).

Figure 12 shows species with few original examples gain most from augmentation, while a few well-populated species degrade slightly as synthetic clips add distributional shift.