NInfer-4090

I wanted a coding agent to run on a local 27B model, on one 24 GB NVIDIA GeForce RTX 4090. A coding agent resends its whole conversation every turn, so the engine needs fast prefill and a cache that returns the earlier prompt. The NInfer engine by Neroued was developed for the RTX 5090. NInfer-3090 by Don-Chad added an SM86 compatibility layer and Qwen3.8 runtime support for the RTX 3090.

I ported NInfer-3090 to sm_89, the RTX 4090, and extended it for agent traffic. Every kernel is tuned for this one model on this one card. The model is a 16.96 GiB artifact with a groupwise Q4/Q5/Q6 body and an 8-bit embedding and output head. A rotated 4-bit KV cache fits the full 262,144-token context on the card. The server speaks OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages with streamed tool calls. It resumes a long agent prompt from the deepest cached point it holds in GPU memory, host RAM, or on disk, and prefills only the new tail. Requests pick a LoRA adapter by name, and requests for different adapters decode in one batch. A decisions API in the same process and on the same weights answers typed questions with probabilities from prefill and a pointer head, without decoding. Caudra uses it as its local decision engine.