An optimized quantization and inference library for running local LLMs on consumer GPUs, with EXL3 quantization, tensor and expert parallelism, CPU offloading, and OpenAI-compatible serving.
1830An optimized quantization and inference library for running local LLMs on consumer GPUs, with EXL3 quantization, tensor and expert parallelism, CPU offloading, and OpenAI-compatible serving.