A script for training the ConvNextV2 on CIFAR10 dataset using the FSDP technique for a distributed training scheme.
-
Updated
Dec 11, 2023 - Python
A script for training the ConvNextV2 on CIFAR10 dataset using the FSDP technique for a distributed training scheme.
Fast and easy distributed model training examples.
Minimal yet high performant code for pretraining llms. Attempts to implement some SOTA features. Implements training through: Deepspeed, Megatron-LM, and FSDP. WIP
Framework, Model & Kernel Optimizations for Distributed Deep Learning - Data Hack Summit
Fully Sharded Data Parallel (FSDP) implementation of Transformer XL
Add a description, image, and links to the fsdp topic page so that developers can more easily learn about it.
To associate your repository with the fsdp topic, visit your repo's landing page and select "manage topics."