Blog

Attention, as a core layer of the ubiquitous Transformer architecture, is a bottleneck for large…

This blog is courtesy of the PyTorch team at Arm. More details can be found here.…

Summary Originally PyTorch, used an eager mode where each PyTorch operation that forms the model…

Over the past year, Mixture of Experts (MoE) models have surged in popularity, fueled by…

Over the past year, we’ve added support for semi-structured (2:4) sparsity into PyTorch. With just…

Summary: With PyTorch distributed’s new asynchronous checkpointing feature, developed with feedback from IBM, we show how…

An efficient decoding Grouped-Query Attention with low-precision KV cache Introduction Generative AI has taken the…

Recently, we demonstrated how FSDP and selective activation checkpointing can be used to achieve 57% MFU…

This post was contributed by Intel AI in partnership with the PyTorch Foundation. In 2017,…

TLDR: We show promising results of up to a 1.46x speedup with <2% drop in accuracy on float32…

Speculative decoding is an optimization technique for inference that makes educated guesses about future tokens…

1.0 Summary We present an optimized Triton FP8 GEMM (General Matrix-Matrix Multiply) kernel TK-GEMM, which…

We are excited to announce the release of ExecuTorch alpha, focused on deploying large language models…

We are excited to announce the release of PyTorch® 2.3 (release note)! PyTorch 2.3 offers…

1.0 Summary We show that by implementing column-major scheduling to improve data locality, we can…

In this blog, we demonstrate the scalability of FSDP with a pre-training exemplar, a 7B…

This post is the fourth part of a multi-series blog focused on how to accelerate…

Overview PyTorch is a Python-based framework for developing deep learning models. It is one of…

TL;DR Leveraging a first principles approach, we showcase a step by step process undertaken to…