Description

Building and Deploying LLM applications with Apache Airflow

Discover how to build reliable, scalable LLM applications using Apache Airflow in Kaxil Naik's insightful talk. Explore design patterns that integrate proprietary enterprise data to create value-driven AI applications. From orchestrating structured and unstructured data to interacting with tools like OpenAI's GPT-4 and HuggingFace, learn how Airflow enables seamless machine learning workflows at scale.

Kaxil Naik, Senior Director of Engineering at Astronomer and a leader in data engineering, shares real-world examples and future enhancements for Airflow, including proposed new providers for LLM integration. Perfect for AI, data science, and engineering professionals seeking cutting-edge solutions for natural language processing (NLP) and AI-driven workflows.

Instructors Bio

Kaxil Naik

Sr. Director of Engineering at Astronomer

Kaxil Naik is a seasoned technology leader and a prominent figure in the data engineering community and open-source communities, best known for his extensive contributions to Apache Airflow, the leading platform for data orchestration. As Senior Director of Engineering at Astronomer, Kaxil leads key initiatives to enhance both the Apache Airflow project and the Astronomer platform, driving innovation to meet the evolving demands of modern data-driven enterprises. Kaxil has been instrumental in shaping the future of Airflow, spearheading both Airflow 2.0 and Airflow 3.0 initiatives, and leading significant contributions such as DAG Serialization, Scheduler HA, and Secrets Backend. His deep involvement in the Airflow community includes founding and organizing various Airflow Meetups across the world (London, Bangalore, Hyderabad, etc), including co-organizing the Airflow Summit and fostering global collaboration on Airflow’s roadmap. With a Master’s in Data Science & Analytics from Royal Holloway, University of London, Kaxil’s career spans Data Science, Big Data, and DevOps. He began working with Airflow in 2017 as a Big Data consultant at Data Reply and has since become a leading voice in the open-source data community. His ongoing work at Astronomer ensures that Airflow continues to set the standard in data orchestration, and his vision continues to shape how organizations approach modern AI and ML-driven workflows.

Outline:

  • Chat about Chatbots IRL

  • Airflow for RAG

  • Real Use Cases

  • Running this in pr GitHub Code