memory-optimization

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

A fast and memory-optimized string library for heavy-text manipulation in Python

Flash weight streaming for MLX: run massive models larger than your RAM on Apple Silicon.

This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog post.

NVIDIA Sol-Attn for ComfyUI / Triton kernel on SM89 - SM121, with zero-copy MiniMax H3 nodes: memory-efficient attention, scheduled tau with graph preview, and feed-forward chunking. Measured 1.14–1.44× vs SageAttention and −37% MLP peak VRAM on H3

First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression with near-zero quality loss.

Lightweight Python lazy imports that defer module loading to reduce startup time and initial memory use.

PRISM: O(1) Photonic Block Selection for Long-Context LLM Inference — eliminates the O(N) KV cache scan via photonic broadcast-and-weight similarity engine on TFLN

Unified KV-cache compression for LLM inference: 12 Python-native methods, Debian-tested isolated add-ons, Godzilla KVarN/TriAttention, exact Godzilla/Gigatoken profiles, CUDA weight sharing, and multi-GPU planning.

First open-source KVTC implementation (NVIDIA, ICLR 2026) -- 8-32x KV cache compression via PCA + adaptive quantization + entropy coding

HotPin: routing-guided expert pinning for lossless low-RAM MoE inference on consumer CPU+NVMe - 120B model at 3.84 tok/s in 19 GB RAM, no GPU

Implementation of PagedAttention from vLLM paper - a breakthrough attention algorithm that treats KV cache like virtual memory. Eliminates memory fragmentation, increases batch sizes, and dramatically improves LLM serving throughput.

Run AI models too large for your Mac's memory — at near-full speed. Intelligent expert caching, speculative execution, and 15+ research techniques for MoE inference on Apple Silicon.

🚀 Reclaim GBs of RAM on Linux by auto-freezing idle apps. Zero-config background daemon for GNOME/Wayland. Instant thaw on window focus.

A tiered-memory system design for workloads that don't fit in RAM: measure the working set, pin the hot tier, stream the cold tier from flash. Ships the residency calculator, measurement harnesses, and the build recipes behind it. Predictions validated against public benchmarks.

Automatically reduce the memory size of any pandas dataframe based on downcasting bit types efficiently

RamKleener — CLI tool for quick, safe RAM cleanup: scans processes, applies a 3-tier safety model, and removes memory bloat with user confirmation

GPU Memory Calculator for LLM Training - Calculate GPU memory requirements for training Large Language Models with support for multiple training engines including PyTorch DDP, DeepSpeed ZeRO, Megatron-LM, and FSDP.

Near-optimal vector quantization for LLM KV cache compression. Python implementation of TurboQuant (ICLR 2026) — PolarQuant + QJL for 3-bit quantization with minimal accuracy loss and up to 8x memory reduction.

Automated memory optimization for Pandas DataFrames. Reduces memory by 50-80% and loads CSV 5-10x faster. Drop-in replacement that returns standard DataFrames - works with sklearn, matplotlib, and all your favorite libraries. Safe, fast, and zero refactoring required.