The Data Wars: Snowflake vs. Databricks – A Battle for the Cloud Frontier
It is November, 2024 and the war rages on with no apparent end in sight. In the shadowy depths of the digital battleground, where innovation and ambition collide, two tech titans—Snowflake and Databricks—are waging a relentless war for dominance in the lucrative realm of cloud data and AI. This is not a conflict fought with bullets or bombs but with cutting-edge algorithms, disruptive features, and global conferences that serve as their war rooms.
This past week marked the latest skirmishes in the ongoing Data Wars, as the two giants continued their global campaigns. Databricks rallied its forces with a conference in Tokyo, a critical waypoint on their World Tour, while Snowflake launched its own offensive in Sydney, bringing their battle for cloud supremacy to the southern hemisphere. So, let’s take a look at the latest weapons in their arsenal and what exciting new capabilities are now available.
Snowflake
The following announcements were made this last week during the Snowflake BUILD conference.
The announcement of the general availability (GA) of Snowflake Notebooks, which is now accessible across AWS, Azure and GCP commercial regions.
Multimodal support for Meta 3.2 Models was also announced, enabling the processing and integration of various data types—such as text, images, and audio—within a single framework. This enhancement gives organisations the ability to build more comprehensive AI applications that can analyse and interpret diverse data sources simultaneously large language models (LLMs) like Meta’s Llama 3.2.
Snowflake's Integrated LLM Evaluation and Monitoring feature was announced, which provides users with built-in tools to assess and monitor large language models (LLMs) directly within the Snowflake platform, offering over 20 metrics, including relevance, groundedness, stereotype, and latency, enabling comprehensive evaluation during both development and production phases.
Snowpark Container Services is now GA for AWS and Azure commercial regions, with some exceptions. This feature of Elastic Scaling of Containers with GPU Support enables developers to deploy, manage, and scale containerised applications directly within the Snowflake platform, using Snowflake-managed infrastructure, meaning organisations can run sophisticated applications and models securely within Snowflake, eliminating the need to move data outside the platform and ensuring robust data governance and security.
Databricks
These are some of the latest announcements from the Databricks World Tour of late:
Predictive Optimisation Enabled by Default: Databricks has enabled predictive optimisation by default for all new accounts. This feature automates maintenance operations for Unity Catalog managed tables. By automating these tasks, Databricks reduces the manual effort required for table maintenance, enhancing data integrity and operational efficiency.
Enhanced Serverless Compute for Workflows: This provides users with greater control over performance and cost optimisation. This improvement allows organisations to tailor their compute resources to specific workload requirements, balancing performance needs with budget constraints effectively.
Open Sourcing of Unity Catalog: This move by Databricks promotes interoperability and flexibility, enabling organisations to integrate diverse tools and platforms while maintaining consistent governance and metadata management.
So on one side stands Snowflake, the white-armoured sentinel of data warehousing, its enterprise-grade fortresses promising seamless integration and unmatched performance. Opposing it is the fiery insurgent, Databricks, the lakehouse champion fusing data lakes and warehouses, armed with its open-source philosophy and a spark of rebellious ingenuity. The war for the future of data has only just begun.
It will be interesting to see how things pan out for both of these incredibly powerful platforms in the months and years ahead. At Precision Data Partners, we’re technology agnostic and will resolve to find the best solution for our customers based on their technological requirements. If you have any questions about which technology is best for you, contact us below and we would be happy to discuss.
