article Mar 20, 2024Optimizing Data Loading and GPU Usage in PyTorchToday, it’s not too uncommon to have data processing tasks—including training neural networks—that require a GPU to finish in a reasonable amount of time. Acquiring one is fairly straightforward thanks to the abundance of cloud providers like AWS, GCP, Paperspace, and so on.[... 445 words]machine-learningpytorchperformance-tuningneural-networks
article Dec 20, 2023Enhance Training Speed with Mixed Precision TrainingIf your data loading is efficient and GPU utilization is almost always at 100%, it is time to consider speeding up the actual computation that happens there. Note that most operations on the GPU involve dealing with floating point variables (activations, gradients, and so on), each having a certain precision – most commonly float32 (or full precision).[... 353 words]machine-learningneural-networksperformance-tuning