Team Members: Neel Shah, Soham Butala, Aaditya Parthasarathy, Aditi Kharkwal, Ameya Bhamare
This project focuses on generating synthetic data for AWS S3 traffic to evaluate new hard drives' performance accurately. By leveraging the complexity of future traffic patterns through synthetic data, we aim to provide a more comprehensive evaluation compared to traditional testing methods.
The motivation behind this project is the need for accurate prediction of future hard drive traffic patterns on AWS S3. Traditional testing methods often fall short in capturing the complex nature of these patterns. Synthetic data generation offers a promising solution by approximating these patterns, thus enabling a more thorough performance evaluation of new hard drives.
The challenge lies in generating synthetic data that mirrors the complexity of actual S3 traffic, adhering to historical data constraints and successfully applying generative models such as GANs (Generative Adversarial Networks) to produce realistic synthetic S3 traffic data.
Our approach involved exploring various data generation methods including:
TabGAN: An evolution of CTGAN for tabular datasets. PAR: Probabilistic AutoRegressive model for learning multivariate time series data. TimeGAN: GAN architecture for time series data. DoppelGANger: Captures the statistical properties of time series data. CT-GAN: Conditional Tabular Generative Adversarial Network for synthesizing realistic tabular data.
Rows: 271 million S3 transactional data points. Columns: 13 (11 categorical, 2 numerical). Security: One-way hashing enforced.
The evaluation of different models led to identifying Gaussian Copula and CTGAN as the best performers for this use case. Notably, Gaussian Copula was found to be significantly faster and more efficient than CT-GAN, both in terms of training speed and data quality.
Modeling Tabular Data using Conditional GAN. (NeurIPS 2019) Tabular GANs for uneven distribution. (arXiv preprint arXiv:2010.00638, 2020) Using GANs for Sharing Networked Time Series Data: Challenges, Initial Promise, and Open Questions.
