Beyond the Lakehouse: Architecting the Data Plane for Hybrid AI Factories

The recent strategic alliance between Cloudera and VAST Data, announced on July 14, 2026, is more than a partnership; it is a market signal. Their stated goal of creating a unified "AI factory" for hybrid environments directly confronts the most significant, and often overlooked, bottleneck in production AI: data delivery. For years, we have architected our platforms around compute constraints. Now, as GPU capacity becomes more accessible, the pendulum has swung. The primary impediment to scaling AI workloads is no longer the availability of silicon, but the architectural limitations of getting data to it efficiently.

The standard cloud data lakehouse paradigm, built on object stores like Amazon S3, was designed for the sequential, high-throughput patterns of SQL-based analytics. It was not designed for the high-concurrency, low-latency, and often random-access I/O patterns demanded by distributed model training, fine-tuning, and large-scale inference. The result is GPU starvation—multi-million dollar clusters sitting idle, waiting for data. Architecting the next generation of AI-native platforms requires a fundamental rethink of the data plane.

Why are cloud-native lakehouses hitting an I/O wall?

Cloud-native storage like S3 and its equivalents struggle with AI workloads because their design prioritises scale and cost-efficiency for analytics over the high-performance I/O characteristics required for model training. This mismatch manifests in several critical areas. First is latency. The network round-trip from a GPU instance to an object storage bucket introduces milliseconds of latency, an eternity for a processor designed to operate in nanoseconds. When a training job using PyTorch's DataLoader needs to fetch thousands of small files for each batch, this latency accumulates, leaving the GPU waiting.

Second is throughput throttling and consistency. Object stores enforce API request limits to ensure multi-tenant stability. A distributed training job with hundreds of parallel workers can easily exceed these limits, leading to throttled performance and failed data fetches. Furthermore, the eventual consistency models of some object storage services can introduce complexities for workloads that require immediate read-after-write access, a common pattern in data preprocessing pipelines that generate intermediate artefacts.

Finally, metadata performance becomes a severe constraint at scale. A training dataset may consist of billions of small files. Listing and accessing these files puts immense pressure on the object store's metadata layer, which was never intended to function as a high-performance file system index. These limitations combined mean that while the theoretical bandwidth of cloud storage is high, the practical, realised performance for AI workloads is often a fraction of what is required to keep expensive compute resources fully saturated.

What are the core principles of a high-performance AI data plane?

An effective AI data plane is engineered to close the distance between data and compute, prioritising data locality, parallel I/O, and intelligent caching to maximise GPU utilisation. The objective is to create a storage fabric that behaves like a local file system for the GPU cluster, even when it is disaggregated and spans multiple physical locations. This is a significant departure from the lakehouse model, where storage and compute are intentionally and loosely coupled.

Three principles are paramount. First, **data locality via intelligent tiering**. The platform must automatically stage the working set of data for a given AI job onto high-performance storage—ideally NVMe flash—that is network-adjacent to the GPU cluster. This is not manual data copying; it is a policy-driven, transparent caching layer that understands the access patterns of the workload.

Second, **a massively parallel architecture**. The system must eliminate serial bottlenecks. This means a control plane that can handle millions of metadata operations per second and a data plane that can service simultaneous requests from thousands of clients without contention. Solutions like VAST's Disaggregated, Shared-Everything (DASE) architecture exemplify this approach, providing a single, scalable namespace built on a high-performance network fabric.

Third, **separation of metadata and physical layout**. Open table formats like Apache Iceberg (version 1.5.0 and later) are critical here. Iceberg provides a consistent, high-performance metadata layer that catalogues the data, manages schema evolution, and enables time travel. This allows the underlying physical storage—the "data plane"—to be optimised independently for performance without breaking the logical structure and governance of the data lakehouse.

How do we architect for hybrid AI workloads?

A hybrid architecture is the logical endpoint for serious enterprise AI, and it is achieved by decoupling the logical data representation from the physical storage fabric. This allows an organisation to deploy specialised storage nodes where they are most effective—on-premises for low-latency training on sensitive data, and in the cloud for elastic, large-scale inference workloads—while maintaining a single, unified view of the data.

The blueprint looks like this: Apache Iceberg, managed by a platform like Cloudera Data Platform (CDP) or a governance layer like Unity Catalog, serves as the universal metadata and table format. This provides the abstraction layer. Beneath this, you deploy a high-performance storage fabric, like the VAST Data Platform, that can present a consistent namespace across both your on-premises data centre and your chosen cloud provider(s). A training job running on an on-premises GPU cluster reads from local storage nodes. An inference job running in AWS can read from nodes deployed in the same region. Critically, the Iceberg table definition remains the same. The data access is location-aware and optimised by the underlying platform, not the application developer.

This architecture allows compute to follow the data, or vice versa, based on cost, performance, and regulatory requirements. You can fine-tune a model on-premises using private, regulated data, then push the resulting model artefact and its associated Iceberg tables to the cloud for global, scalable serving. This eliminates the prohibitively expensive and slow process of lifting and shifting entire petabyte-scale datasets across the corporate firewall or between clouds.

What does this mean for Australian organisations?

For Australian organisations, particularly those in finance, healthcare, and the public sector, a hybrid AI architecture provides a pragmatic solution to the dual pressures of innovation and regulation. It offers a direct path to leverage global, cloud-hosted frontier models while ensuring that sensitive customer and corporate data used for fine-tuning or Retrieval-Augmented Generation (RAG) remains within sovereign boundaries, in alignment with the Privacy Act.

This control over data locality is essential for building trustworthy AI systems. Frameworks such as the NSW AI Assessment Framework (AIAF) place a strong emphasis on data governance, security, and accountability. A hybrid data plane provides the auditable, physical controls necessary to satisfy these requirements, demonstrating that sensitive data never leaves Australian shores. This approach supports a robust responsible AI posture by design, rather than as an afterthought. Furthermore, by building sovereign AI infrastructure, organisations contribute to the goals of the National AI Centre and the broader push to develop onshore AI capability.

The architectural pattern is clear: a unified metadata layer with Iceberg, governed by a centralised catalogue, sitting atop a distributed, hybrid data plane. This is the foundation required to build the AI factories of tomorrow. As specialists in agentic AI engineering, we at Precision Data Partners work with organisations to design and implement these high-performance data platforms, ensuring their AI investments deliver tangible results by eliminating the data delivery bottleneck once and for all.