What Kimi K3 Costs to Run

Most models are trained in 16-bit precision and experience a reduction in intelligence if you run them in 4-bit precision to save memory. Kimi K3 was trained with native Microscaling Floating Point 4 (MXFP4) quantization-aware training (QAT) with 4-bit release weights by design, so it can be run more efficiently without lossy compression. In Kimi K3’s QAT, every forward pass rounds the weights to 4-bit before using them, so the loss that the model is optimizing includes the quantization difference. Therefore, the model learns to be good at 4-bit, rather than accepting a level of error when running in 4-bit instead of 16-bit.

You can run Kimi K3 in 8-bit or 16-bit precision as well, but the reason to do so would not be for improved quality. It would only be for compatibility reasons. FP4 is for inference, so if you wanted to fine-tune or continue training, you would also need higher precision.

To get a basic estimate of the amount of RAM required to just load a model, you can multiply the total number of parameters by the number of bits of quantization you want to run it at. Then divide that number by 8, because memory is measured in bytes, and there are 8 bits in 1 byte. In the case of Kimi K3, 2.8 trillion parameters x 4 bits divided by 8 is 1.4 terabytes of RAM for the weights alone.

If you would like to run it in 8-bit or 16-bit precision, you will need double or quadruple this amount of RAM. With Kimi K3, you’d only do this to run it on hardware without native FP4 or to use more mature software than the current FP4 kernels. Kimi K2 variants, by comparison, were only 1 trillion parameters, but the full precision weights (BFloat16 (BF16)) required 2 TB of RAM, which is roughly 600 GB more than K3’s native. Then additional RAM is needed to store the key-value (KV) cache, activations, and runtime overhead during inference.

The reason a model as large as Kimi K3 is economical to serve at all is because almost none of it runs on a given token. The efficiency innovations make the cost math work. The MoE sparsity allows only 16 of 896 experts (+ 2 shared) to activate per token, but it uses quantile balancing routing to spread tokens across experts deterministically, so no expert capacity is wasted. Kimi Delta Attention (KDA) keeps a fixed-size attention state instead of per-token KV cache, cutting the KV-cache memory by up to ~75%.

Attention Residuals allow each layer in the neural network to pull in specific outputs from earlier layers with learned weights. Stable LatentMoE is K3’s Mixture of Experts framework where experts operate in a compressed latent space to stay cheap. KDA, Attention Residuals, and Stable LatentMoE work together to increase the overall scaling efficiency by ~2.5x over Kimi K2.

For a single user, running Kimi K3 inference would require an additional 2-15 GB for the KV cache, depending on context, ~30 GB for activations, and ~30 GB for runtime overhead, totaling around 1.5 TB of RAM including the 1.4 TB for the weights. A single GPU is out of the question, leaving us with multi-GPU, and for realistic scaling, multi-node solutions for hosting.

Using basic arithmetic with H100’s 80 GB, and H200’s 141 GB, we would need 19 H100s or 11 H200s. However, even though Hopper (and MI300X) can load 4-bit weights, they dequantize at runtime, keeping the memory savings but losing the speed benefits that K3’s native 4-bit architecture allows. Using the newer generation, Blackwell B200/B300 or AMD MI350X/MI400s to execute MXFP4 (4-bit) natively is a better approach. So self-hosting is a hardware generation decision rather than just a GPU-count decision.

vLLM recommends a minimum of 8x B300s or MI355X/MI350X (with Radeon Open Compute (ROCm)), multi-node for production. Also, because KDA’s unique prefix-caching process doesn’t have per-token KV caching, like standard vLLM, ordinary prefix caching doesn’t work seamlessly with it. The creators of Kimi K3, Moonshot, contributed a custom vLLM implementation that solves this problem. In order to self-host, you need a specific recent vLLM build with the K3-specific code.

Kimi K3’s first-party API is $3.00 input / $0.30 cached input / $15.00 output per 1M tokens, with DigitalOcean live at the same $3/$15. The cheapest native-FP4 single node is ~8x MI350X at ~$4.76/GPU-hour. This is approximately $38/hr and $27,800/month on a reserved GPU Droplet that is running whether or not anyone sends a request. To meet this cost with Serverless Inference, a user would need to use approximately 1.8B output tokens/month, or ~700 output tokens/second sustained around the clock at the API’s $15/1M token rate. In reality, a single-stream tokens per second (TPS) rate is in the tens, maybe hundreds of TPS in ideal conditions, making it virtually impossible to reach that point on a single stream. However, on a node, it is possible to serve at least 40 requests in parallel. With less than the full 1M token context per request, the total requests can be closer to 600.

Heavy users might have ~50M tokens/month costing ~$750 maximum via the API. This is on the higher end considering the more expensive output tokens only. For a single user, it is approximately 40x cheaper to use serverless inference through an API than self-hosting on rented GPUs. However, once you pass 40+ consistent, heavy users at 50M + tokens/month, the math starts to change in favor of self-hosting. These 40 users could be individuals or they could represent an elaborate agent orchestration with 40+ agents directed by a single user.

If we can do 40 concurrent requests on a node, can we use fewer GPUs so we only need enough GPUs for 1 user? The short answer is no, for a few reasons. First, serving models of this size splits each layer across GPUs (tensor parallelism), and the math works well on two, four, or eight GPUs, rather than six (the minimum required to hold the weights + inference requirements for a single user). Therefore, eight is the minimum requirement. Secondly, you generally can’t rent 5-6 of these GPUs. They are rented in packages of one or eight. Even with six, the TPS math would not make it possible to use enough tokens to justify the cost alone.

For some users, the costs are not the main priority. Instead, control, consistency, reliability, data residency, and other factors might make self-hosting more attractive. In that case, it is worth it to look into the Terms of Service for the inference provider you’re using. Many third-party providers don’t store inference data and have excellent data privacy records, so self-hosting on rented GPUs might not be necessary. If you plan to self-host Kimi K3 and offer it commercially as a service, you should also check the Kimi K3 license. It is a custom Kimi K3 license as opposed to Apache or MIT, and it comes with commercial gates above certain revenue ceilings.

If you choose to go with serverless inference, DigitalOcean offers Kimi K3 on the Serverless Inference API as a pay-per-token option at the $3/$15 rate. If you choose to go the self-hosted GPU route, you can use DigitalOcean’s GPU Droplets, Dedicated Inference, or long-term reserved GPUs. With GPU Droplets, you manage the stack, and with Dedicated Inference, DigitalOcean runs a managed vLLM endpoint that you bring the Kimi K3 weights to. The 4-bit native options DigitalOcean provides are the AMD Instinct MI350X and the NVIDIA B300. Availability can change without notice, so be sure to check what is currently available and on which regions before planning your deployment.

If the GPU Droplets and Dedicated Inference options are at capacity, please reach out to the DigitalOcean sales team about dedicated capacity. DigitalOcean also offers Bare Metal GPUs, but does not currently offer FP4 native options in the bare metal format, so they are not the right fit.

With larger open weights models reaching the capabilities of closed frontier models, real infrastructure decisions that used to be invisible behind an API are now becoming part of the strategy for many teams. With a sufficient number of users and high enough tokens per month, self-hosting can provide cost savings, control, and help alleviate privacy concerns. Even after Kimi K3 is no longer a top contender for your team, the self-hosting vs. Serverless Inference decision is one most teams need to make as open-weights models become increasingly capable. The same comparative logic covered in this article applies to these future models as well.

• Kimi K3 Model Card (Hugging Face)
• A Preview of Production-Scale Kimi K3 Support on vLLM
• DigitalOcean GPU Droplets