Transformations applied to audio datasets

[msbouanane]

Hello,
thank you very much for this wonderful package.
I'm trying to train my SNN with the N-TIDIGITS audio dataset but I'm not sure about the transformations I should apply to it before passing it through the network.
Anyone has any suggestions?

[biphasic]

Hey,
glad you find it useful. @sheiksadique is currently working on this. Please have a look at this branch https://github.com/synsense/tonic/tree/audio_transforms and consider merging the latest develop branch into that branch before using it. It's based on PyTorch audio and SciPy. We're planning to include that in the next release but it'll still take some time. What transforms are you looking for in particular?

[msbouanane]

I'm not sure about what transforms I should use for N-TIDIGITS. I trained my SNN on the NMNIST before and I used the .ToFrame transform. But for N-TIDIGITS, I don't know what kind of event transformations I need. I tried to train the network without applying any transforms to the dataset but I run into errors. I should note that I'm using snnTorch.

[biphasic]

I see. In principle, the transform you need is a similar one as ToFrame, which adds up all the events in a given time/event count window. Currently, ToFrame supports 3D (time, x, y) data and for audio data the "y" channel is missing, that's why it's complaining. This would need to be changed here. I will open a new issue and tag you. Do you think you'd like to look into this?

[msbouanane]

Yes of course

[biphasic]

thanks a lot. I opened an issue here

[biphasic]

I fixed this issue. You should now be able to use the ToFrame transform and choose a slicing method such as time_window, event_count or else to rasterize your data afaf3e9

[biphasic]

also here is an example file of how I used SHD in the past. This is a PyTorch lightning datamodule but you can see the transforms one might want to apply.

import torch, math
import pytorch_lightning as pl
from tonic import datasets, transforms
from torch.utils.data import DataLoader
import numpy as np
import tonic


class ToRaster():
    def __init__(self, encoding_dim):
        self.encoding_dim = encoding_dim

    def __call__(self, events):
        # tensor has dimensions (time_steps, encoding_dim)
        tensor = np.zeros((events["t"].max()+1, self.encoding_dim), dtype=int)
        np.add.at(tensor, (events["t"], events["x"]), 1)
        return tensor[:250,:]


class HSD(pl.LightningDataModule):
    def __init__(self, batch_size, encoding_dim, dt=4000, num_workers=6, download_dir='./data'):
        super().__init__()
        self.batch_size = batch_size
        self.num_workers = num_workers
        self.download_dir = download_dir
        
        # SHD is recorded using 700 channels, which is a lot so we are going to 
        # downsample that to our liking. Similarly the timestamps in us resolution
        # need to be downsampled.
        self.transform = transforms.Compose([
            transforms.Downsample(time_factor=1/dt, spatial_factor=encoding_dim/700),
            ToRaster(encoding_dim),
        ])
  
    def prepare_data(self):
        datasets.SHD(self.download_dir, train=True)
        datasets.SHD(self.download_dir, train=False)
  
    def setup(self, stage=None):
        self.train_data = datasets.SHD(self.download_dir, train=True, transform=self.transform)
        self.test_data = datasets.SHD(self.download_dir, train=False, transform=self.transform)

    def train_dataloader(self):
        return DataLoader(self.train_data, num_workers=self.num_workers, batch_size=self.batch_size, 
                          collate_fn=tonic.collation.PadTensors(batch_first=True), drop_last=True, shuffle=True)
  
    def val_dataloader(self):
        return DataLoader(self.test_data, num_workers=self.num_workers, batch_size=self.batch_size, 
                          collate_fn=tonic.collation.PadTensors(batch_first=True), drop_last=True)

    def test_dataloader(self):
        return DataLoader(self.test_data, num_workers=self.num_workers, batch_size=self.batch_size, 
                          collate_fn=tonic.collation.PadTensors(batch_first=True), drop_last=True)

[msbouanane]

Thank you very much for your help!
However I still have another inquiry regarding ToFrame Transform and audio datasets when used with snnTorch. When I applied ToFrame to the NMNIST dataset, it returns tensors with shape [ T , dim ] (e.g. (298, 2, 34, 34) with "dim" being the sensor_size tuple). Then when I pass it to the DataLoader it returns tensors with shape [ T , B , dim] (e.g. torch.Size([310, 128, 2, 34, 34])). This makes it easier to pass the dataloader to the snntorch.backprop module to train the network.
However, when I apply ToFrame to an audio dataset like SHD, it doesn't return the TIME dimension in the event tensors but only the dimensions of the sensor_size.
Without the time dimension, it is not possible to train the network using snnTorch's backprop.BPTT.
Here are 2 screenshots to see how the transform responds differently to NMNIST and SHD or NTIDIGITS:

[biphasic]

see answere here: #174 (comment)