The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

Python SQL Parser and Transpiler

AdalFlow: The library to build & auto-optimize LLM applications.

torch-optimizer -- collection of optimizers for Pytorch

On the Variance of the Adaptive Learning Rate and Beyond

PyTorch native quantization and sparsity for training and inference

SAM: Sharpness-Aware Minimization (PyTorch)

Portfolio optimization and back-testing.

The official implementation of “Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training”

Scour - An SVG Optimizer / Cleaner

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

The official implementation of MARS: Unleashing the Power of Variance Reduction for Training Large Models

End-to-end Generative Optimization for AI Agents

TorchOpt is an efficient library for differentiable optimization built upon PyTorch.

Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793

AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights (ICLR 2021)

Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.

Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.

lookahead optimizer (Lookahead Optimizer: k steps forward, 1 step back) for pytorch

optimizer & lr scheduler & loss function collections in PyTorch