what on-premises AI actually means
on-premises AI has become a sales term.
vendors use it to mean different things. sometimes it means a cloud deployment in your region. sometimes it means a private cloud tenancy that is logically isolated but physically shared. sometimes it means a hybrid arrangement where inference happens on-site but model updates come from somewhere else. and sometimes it actually means what it says: the model runs on hardware in a building you control, with no dependency on external infrastructure for normal operation.
these are not the same thing. the differences matter enormously in regulated environments. and because the term is used loosely, a lot of organizations think they have implemented on-premises AI when they have implemented something meaningfully different.
i want to describe what genuine on-premises AI infrastructure looks like, what it costs, and what problems it actually solves versus what problems it does not.
the actual architecture
genuine on-premises AI deployment has four layers that all have to work together.
the first is the hardware layer. inference for a model that can do something useful in an enterprise context requires either a reasonably capable GPU or a CPU configuration that has been specifically designed for inference workloads. the GPU path is faster and more expensive. the CPU path is slower and more economical at small to medium scale. the hardware decision is driven by throughput requirements, latency requirements, and budget. there is no universally right answer. there are tradeoffs that have to be mapped to the specific workload.
what is non-negotiable is that the hardware has to be sized for peak load, not average load. a system that handles average throughput fine but degrades badly under peak load is not production-ready. the peak sizing calculation requires understanding the distribution of request volume over time, the latency SLA that the workflow requires, and the model's performance characteristics at different batch sizes. this analysis is underinvested in almost every on-premises deployment i have seen.
the second is the model serving layer. you need a serving framework that handles request batching, manages memory efficiently, and exposes a stable API. vLLM has become the de facto standard for GPU-based LLM serving. for CPU inference, llama.cpp and its derivatives are the most mature options. the serving layer sits between the hardware and the application and has to be configured correctly for the specific model and the specific hardware. this is not a plug-and-play operation. the configuration parameters, batch size limits, memory allocation, and concurrency settings all have to be tuned for the specific environment.
the third is the model itself. and this is where most enterprise on-premises deployments make their most significant mistakes.
the model you run on-premises is almost never the same model you evaluated in a demo or tested via API. it has been quantized. INT8 or INT4 quantization reduces memory requirements significantly and allows larger models to run on smaller hardware, but it changes the model's behavior. not catastrophically, usually, but measurably. the performance on your specific tasks after quantization needs to be evaluated, not assumed to be equivalent to the full-precision version.
additionally, for regulated enterprise applications, the base model almost always needs to be fine-tuned for the specific domain, language requirements, and task types of the deployment. a general-purpose model that has not been adapted to your specific context will perform worse than you expect on the tasks that actually matter and will produce outputs that reflect training data distributions that are not representative of your environment.
the fourth layer is everything else: the api gateway, authentication, audit logging, monitoring, alerting, backup and recovery, update management, and the operational runbooks that govern how all of it is managed. this layer is the most underestimated. organizations evaluate AI vendors on model performance. they do not adequately evaluate the operational requirements of running that model in production, which are substantial and ongoing.
quantization in practice
since quantization is where a lot of the real behavior change happens, it is worth going into more detail.
a 7B parameter model in full float16 precision requires approximately 14GB of VRAM. INT8 quantization reduces this to roughly 7GB. INT4 reduces it further to approximately 3.5GB. this is why quantization matters: it determines whether a given model fits on a given piece of hardware.
but quantization is not lossless. the mechanism is simple in principle: you represent weight values with fewer bits, which means the precision of each weight is reduced, which means the model's behavior changes. in practice, the quality degradation from float16 to INT8 is usually small. the degradation from INT8 to INT4 is larger and more task-dependent. tasks that require fine-grained numerical reasoning, precise instruction following on complex specifications, or consistent behavior across many similar inputs tend to show more degradation under aggressive quantization.
GPTQ and AWQ are currently the best quantization methods for quality preservation. GGUF format with llama.cpp allows fine-grained control over quantization level per layer, which can be used to preserve quality in the attention layers where precision matters most while aggressively quantizing FFN layers where the impact is lower.
the practical implication: when you are choosing what model to run on-premises, you need to evaluate the quantized version of that model on your specific tasks, not the full-precision version. the benchmark numbers in papers and on leaderboards are almost always full-precision. the model you will actually run is not.
the inference throughput math
most people underestimate how fast a correctly configured inference stack can be and how quickly it degrades under load.
a single A100 80GB can handle approximately 2000 tokens per second of output throughput with continuous batching at INT8 precision for a 7B parameter model. that is roughly 40 simultaneous users generating 50 tokens per second each, or 200 simultaneous users generating 10 tokens per second each, depending on your use case distribution. the math changes significantly for larger models: a 70B model on the same hardware gets you roughly 200-400 tokens per second, which is a 5-10x reduction.
the important variable that most throughput calculations ignore is time-to-first-token (TTFT). for interactive applications, the latency until the first token appears matters as much as the overall generation speed. TTFT is dominated by the prefill computation, which processes the input prompt. long prompts, like those in document processing or RAG pipelines with large context windows, dramatically increase TTFT. if your application has long prompts, your effective throughput is lower than the peak token generation rate suggests.
this is why hardware sizing cannot be done based on model size alone. it requires understanding the prompt length distribution, the output length distribution, the concurrency profile, and the latency requirements, and mapping those to hardware capabilities under realistic batch configurations.
what on-premises does not solve
on-premises deployment solves data residency. it solves external dependency for inference. it solves audit logging completeness. it solves some latency concerns.
it does not solve model quality problems. a bad model that runs on-premises is still a bad model. if the model has not been properly fine-tuned for the domain, evaluated on realistic inputs, and calibrated for the uncertainty ranges that the application requires, running it in your own data center does not help.
it does not solve the governance problem in any deep sense. you can log every inference on-premises and still have no systematic understanding of when the model is wrong. logging is not evaluation. you need a monitoring system that has opinions about output quality, not just output volume.
and it does not solve the update problem. models need to be updated. the process for evaluating a new model version, running it in shadow mode against production traffic, comparing its behavior to the current production model, and executing a safe rollback procedure is significant operational work. it does not go away because the inference is on-premises. it just belongs entirely to you now.
the right frame
on-premises AI is not a solution. it is a deployment architecture that enables certain properties: data control, latency predictability, external dependency reduction, audit completeness.
whether those properties justify the infrastructure cost and operational complexity depends entirely on the requirements of the specific application. for consumer applications: almost never. for regulated financial infrastructure: almost always.
the organizations that get this right are the ones that treat on-premises AI as an engineering problem with well-defined requirements, not a procurement decision driven by compliance checklists. the requirements come first. the architecture follows from them. the specific model and stack follow from the architecture.
done in that order, it is manageable. done in reverse, it is expensive and painful and usually produces something that meets the letter of the compliance requirement without actually solving the problem.