The data transformation sweeping through the artificial intelligence industry marks a significant shift toward integrated, user-friendly platforms and innovative growth strategies.

Spearheaded by companies such as Databricks Inc., this evolution is characterized by rapid expansion and strategic acquisitions designed to bolster capabilities. This transformative period is driven by the increasing need to manage diverse workloads, enhance programmatic functions and address complex issues surrounding data governance, security and compliance. As the industry adapts to these demands, it sets the stage for a new era of technological advancement and efficiency.

Analyst Howie Xu talks with theCUBE about Databricks and data transformation.

“It’s always easy to have a PowerPoint to say, ‘Hey, here’s all the reports,’ but when people really do it, it’s always a lot of work: governance, security, silos and then who owns what, data lineage, compliance, dedup, all sorts of problems,” said Howie Xu (pictured), analyst and a member of the CUBE Collective community. “I see so many companies solving so many problems; it’s a renaissance.”

Xu spoke with theCUBE analysts John Furrier, Rob Strechay and Savannah Peterson at the Data + AI Summit, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed data transformation and the rapid growth and competitive dynamics between Databricks and Snowflake Inc., highlighting Databricks’ strategic innovations and acquisitions that are driving the evolution and integration of data and AI technologies.

The competitive data transformation landscape: Databricks vs. Snowflake

The competitive landscape between Databricks and Snowflake is intense, with both companies vying for dominance in the data transformation and AI space. Databricks’ recent acquisitions, such as Mosaic ML and Tabular, have positioned it well for future growth, according to Furrier.

“Databricks brought game to this thing,” he said. “They brought uniform general availability on the Delta Lake, Unity Catalog, open source, Mosaic AI adding more tools, LakeFlow up on the pipelines, and then serverless taking away all the infrastructure so there’s no knobs and clusters to manage it.”

The ability for Databricks to integrate data engineering, business intelligence and SQL functionalities is seen as a key advantage. Even still, there are some challenges, Xu pointed out.

“I do think the next few years, there will be some layers of technology to lower down or actually allow you to switch easier,” he said. “We have been talking about data gravity for the last decade. Electricity is a scarce resource. The coating is the scarce resource. You want data to move there. It’s not going to be a wholesale move. It’s about can you do that incrementally, can you virtualize the data layer?”

Innovation and integration: The path forward

Innovation and integration are at the heart of the industry’s progress. The separation of compute from data to support diverse workloads and engines is a trend that is expected to drive significant advancements. This shift will enable more programmatic capabilities and foster innovation across various sectors, according to Furrier.

“Separating compute from data will allow the ability to support diversity of workloads and various engines … a lot of different things that were once diverse and messy can now be connected and that’s going to enable programmatic cool things,” he said.

As the industry continues to evolve, the focus on simplifying processes and enhancing user experience remains crucial. Unity Catalog is one example of an important innovation at the data layer that could revolutionize data management for enterprises, potentially allowing complete data estate overhauls at a significantly reduced cost.

“I’ve already had two different organizations say to me, at two different parties this week, ‘I’m really excited to get my hands on the open-source Unity Catalog to go and see what I can actually do with it and how much of it’s actually open source,’” Strechay said. “I think that’s the key … what features that they have on top of Unity actually land in the open source versus what pieces are actually Databricks built on top of Unity.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE Research’s coverage of the Data + AI Summit:

Photo: SiliconANGLE