Compute Routing
Our infrastructure layer selects where a workload executes after considering the technical and economic constraints of the task.
A different boundary from model routing
A routing decision can choose among local versus external inference, GPU pool, model endpoint, and batch versus realtime execution. Relevant factors include cost, privacy, latency, load, and availability.
Compute routing is related to model routing but operates at a different boundary. Model routing asks what execution capability should perform the task? Compute routing asks where and under what serving path should that capability run now?
Runtime decision
- Local GPU
- Alternate pool
- External endpoint
- Batch queue
The chosen route can change as infrastructure state changes without changing the business-level task contract. A private workload may remain restricted to controlled local infrastructure even when an external endpoint is faster. A latency-sensitive request may use a currently available endpoint while a large archive-processing job is queued for batch execution.
Availability and load
Static routing can waste capacity or create unnecessary outages. If a preferred GPU pool is saturated, the system may have another valid execution path. If an external provider is unavailable, a compatible local model may preserve reduced functionality for selected tasks.
The fallback must still satisfy the task’s permissions and minimum quality requirements.
Economic routing
The cheapest compute path in isolation may not be the cheapest business outcome.
A slower model can reduce inference spend while increasing review volume or degrading an economically important decision. Compute routing therefore operates against workload requirements rather than minimizing raw compute cost.
The objective is policy-constrained placement: execute each workload on infrastructure that satisfies its quality, privacy, latency, availability, and cost requirements at that point in time.
Routing state must remain observable
A workload can produce different latency, cost, or behavior depending on where it was executed. For that reason, the final trace should preserve the selected compute path: local or external, endpoint or pool, model and version, and whether the request ran interactively or through a batch queue.
This becomes important during diagnosis. If quality or latency changes only when a workload spills from a local pool to an alternate endpoint, engineers need to see that correlation rather than treating all executions as equivalent.
Compute routing can also implement safe degradation. During constrained capacity, low-priority work can be delayed while high-priority interactive tasks retain access to scarce compute. The router therefore participates in both performance optimization and production reliability.