VRAM-Aware Workload Allocation
Our compute layer treats accelerator memory as a schedulable production constraint.
More than whether weights fit
VRAM can constrain model size, context, batching, and concurrency. Two requests routed to the same model can have materially different memory profiles because their context length, batch shape, or generation requirements differ.
Allocation model
At runtime, workload placement can consider both the requested model and the current capacity of the available hardware.
- Place
- Queue
- Reroute
- Reject
A workload that would exceed a device’s safe memory envelope should not be sent there merely because that GPU normally serves the model. The alternative may be another GPU pool, a smaller or quantized model, an external endpoint, or delayed batch execution depending on the task.
Headroom matters
Running continuously at the theoretical memory limit makes the system fragile. Small changes in context size, concurrent requests, runtime behavior, or model version can turn a previously successful configuration into an out-of-memory failure.
Production allocation therefore needs operating headroom rather than only maximum packing density.
Memory versus utilization
The most memory-efficient placement is not automatically the fastest placement. Packing multiple workloads onto one accelerator can reduce unused VRAM while increasing queueing or compute contention. Spreading them can improve latency while leaving more memory idle.
The correct allocation depends on the service objective. Interactive workloads may favor latency. Batch workloads may favor total throughput. Private workloads may be constrained to a particular compute pool regardless of global capacity.
Our allocation layer exists to treat VRAM as part of workload routing, not as a static hardware specification engineers consider only when installing a model.
Context is a memory decision
Long context is not free simply because a model advertises a large maximum context window. As active sequence length and concurrent generation increase, runtime memory devoted to attention state can become a significant part of the serving envelope. A workload that fits comfortably at one context length can fail under otherwise identical concurrency with substantially larger contexts.
This makes context policy an infrastructure variable as well as a model-quality variable. The allocator can therefore work together with context construction: remove irrelevant evidence upstream before solving the resulting memory pressure with more hardware.
When several valid paths exist, the system can prefer the route that meets the task requirement with the lowest unnecessary memory occupancy rather than using the maximum context and largest model by default.