The Compute Locus Problem: Architecting for Hybrid AI Inference

The New Compute Locus Problem

The architectural conversation around enterprise AI has fundamentally shifted in the last quarter. We have moved beyond the initial challenge of simply deploying a foundation model behind a secure endpoint. The new, more complex problem is deciding, on a per-request basis, the optimal physical location for inference to occur. This is the Compute Locus Problem: the dynamic selection of an execution venue—cloud, on-premise, or local device—to balance capability, cost, latency, and data security.

The recent announcement of Perplexity AI's hybrid inference orchestrator at Computex 2026 is not an outlier; it is the commercial manifestation of a critical engineering trend. The monolithic model endpoint, served exclusively from a major cloud provider's GPU cluster, is becoming an architectural liability. It is too expensive for simple tasks, too latent for interactive agents, and poses an unacceptable data sovereignty risk for a significant percentage of enterprise workloads. The future of enterprise AI is not a single, powerful model in the cloud, but a fabric of models of varying sizes, distributed across multiple loci and governed by an intelligent orchestration layer.

The Four Axes of Orchestration

A robust hybrid inference orchestrator does not make routing decisions on a single axis. It is a multi-objective optimisation problem, balancing four competing factors in real-time. Your system architecture must be designed to evaluate each request against these axes.

First is Capability. A quantised 3-billion-parameter Phi-3 model running locally cannot summarise a 100-page M&A due diligence document. The orchestrator must first classify the complexity and intent of the request to determine the minimum viable model size required. This often necessitates a small, fast "router" model to perform this initial assessment.

Second is Data Sovereignty. This is the non-negotiable axis for most Australian enterprises. If a request contains sensitive customer data, financial records, or proprietary intellectual property, it must be routed to a trusted compute locus, typically an on-premise GPU cluster or, increasingly, the user's local device. Your orchestration logic must be policy-driven, capable of identifying and quarantining high-risk data payloads from public cloud endpoints.

Third is Latency. For real-time agentic workflows and conversational interfaces, sub-100ms response times are critical. Network round-trips to a cloud endpoint often make this impossible. Local inference on client hardware with dedicated NPUs can deliver tokens in under 50ms, providing a fluid user experience that remote models cannot match. The orchestrator must weigh the user-facing context against the performance profile of each available locus.

Fourth is Cost. Running every query against a frontier model like GPT-5.4 or Claude Opus 4.6 is economically unsustainable. A significant portion of enterprise queries—up to 60% by some internal estimates—are simple tasks like summarisation, classification, or data extraction that can be handled by a much smaller, fine-tuned model at a fraction of the cost. The orchestrator's primary economic function is to divert this traffic away from your most expensive compute resources.

Architecting the Hybrid Inference Orchestrator

Building this system requires three core components: a routing pre-processor, a policy engine, and a robust service mesh.

The entry point is a Router Model. This is not the primary intelligence of your system but a lightweight, low-latency classifier. Its sole purpose is to analyse the incoming prompt and generate metadata for the orchestration logic. It should identify the task type (e.g., classification, summarisation, complex reasoning), estimate token count, and flag potential PII or sensitive keywords. A fine-tuned model from the BERT family or a small mixture-of-experts model is well-suited for this role, offering response times under 20ms.

This metadata is then fed into a Policy Engine. This cannot be a simple set of `if/else` statements hardcoded in your application. To operate at enterprise scale, it must integrate with a dedicated policy management framework like Open Policy Agent (OPA). Rules are defined in a declarative language (Rego for OPA) and can be updated without redeploying the service. This is where you encode your data sovereignty rules, cost thresholds, and capability mappings. For example, a rule might state: `IF data_classification == 'CONFIDENTIAL' AND compute_locus != 'ON-PREM' THEN deny`. This decouples the business logic of routing from the mechanics of the service itself.

Finally, a Service Mesh like Istio or Linkerd manages the physical routing of the request to the chosen endpoint. Whether the target is a Kubernetes service hosting vLLM on-prem, a SageMaker endpoint in AWS, or an API that triggers local execution on a client device, the service mesh provides a unified control plane. It handles service discovery, mTLS for secure communication, load balancing, and crucial failover logic. If the local model fails to respond, the mesh can be configured to automatically retry the request against a more robust, centralised endpoint, providing systemic resilience.

A Practitioner's Roadmap

This is not a theoretical exercise for 2027. The hardware and software components to build a version 1.0 orchestrator exist today. Senior technical leaders should be initiating projects in three areas immediately.

First, benchmark everything. You cannot orchestrate what you have not measured. Establish rigorous, repeatable benchmarks for your primary models across every potential compute locus. Measure tokens per second, time-to-first-token, and cost-per-million-tokens for your cloud endpoints (e.g., Azure AI), your on-premise hardware (e.g., Triton Inference Server on H100s), and a reference local device (e.g., ONNX Runtime on a new Snapdragon X Elite). This data is the foundation of your routing logic.

Second, make quantisation a core competency. For local and edge inference to be viable, aggressive model optimisation is non-negotiable. Your team must develop expertise in post-training quantisation frameworks like Activation-aware Weight Quantization (AWQ) and GPTQ. The performance gains are profound; a 4-bit quantised Llama-3-8B model can deliver performance on par with its bfloat16 counterpart for many tasks, while fitting comfortably into device memory. The emergence of fp8 precision on platforms like NVIDIA's Blackwell architecture will make this even more critical.

Third, build the router PoC now. Do not attempt to build the all-encompassing orchestrator from day one. Start by taking the top five most frequent query types from your production logs. Build a simple classification model to distinguish between them and write the initial policy logic to route the two simplest types to a smaller, cheaper model. This iterative approach allows you to build the muscle memory and foundational components—the policy engine, the service mesh configuration—that will scale to a truly hybrid system.