With data being the backbone of artificial intelligence, data engineering fits into the picture because it addresses how the data pipelines are laid out.

Through Airflow, Astronomer Inc. enhances the data engineering concept by managing complex data pipelines because not having control of these pipelines means operating on shaky ground, which hinders the realization of AI objectives, according to Andy Byron (pictured, left), chief executive officer of Astronomer.

“When we look at what Astronomer does and what Airflow does, really it does three things on top of allow the data engineer to have access to delivering and building pipelines,” Byron said. “One is it allows companies to centralize more of their work and collaboration between software engineering, data engineering and MLOps. The second is the security and governance around that. Astronomer and Airflow allows companies to bring not only a centralized environment but a highly governed and secure environment as well.”

Byron and Steven Hillion (right), senior vice president of data and artificial intelligence at Astronomer, spoke with theCUBE industry analyst John Furrier at the “Supercloud 5: The Battle for AI Supremacy” event, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how data engineering is setting the ball rolling for AI as this sector continues to be top of mind for enterprises and how Astronomer and Airflow fit into the picture.

Airflow creates a line of business

Astronomer continues to incorporate more innovations into the Airflow product. For instance, more features have been embedded in Airflow to allow data engineers to build pipelines using large language models, and this helps develop a line of business based on the value generated, Byron pointed out.

“We’ve created our own AI and ML algorithms that we’re using internally,” he noted. “As these companies start to use Airflow to start to generate more data and deliver it into these big data sets and ML and AI applications, it starts to become a line of business. In that line of business, we’re empowering more and more engineers to get the job done and deliver more business value.”

Since most traditional machine learning models are non-deterministic, this hinders transparency. Airflow tackles this pain point by offering a view of the lineage of the different data sources, according to Hillion.

“My favorite example is the Texas Rangers; I think that’s a sports team,” he explained. “I think they’ve been doing pretty well lately, and a lot of their success, we can justifiably claim, is driven through Airflow and through the Astro platform that’s running it for them. They’re getting data off the field; they’re getting medical data. My other favorite is the firm Laurel that uses generative AI models to summarize the work that lawyers do, accountants do and other professional services do.”

AI and ML projects require architecting data for seamless data flows. As a result, Airflow enables flawless access to information when building ML applications and language models, according to Byron.

“That’s an open-source project that we’re the primary contributors towards,” he stated. “We also make our commercial product that has a lot of higher value add feature sets above the Airflow product itself, and those feature sets are front and center with helping companies deliver on AI and ML initiatives.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the “Supercloud 5: The Battle for AI Supremacy” event:

Photo: SiliconANGLE