About the job
P-59
At Databricks, we are driven by a mission to empower data teams to tackle some of the most pressing challenges facing our world. Whether it's transforming transportation or accelerating medical innovations, we achieve this by creating and operating the foremost data and AI infrastructure platform. Our customers leverage deep data insights to enhance their business operations. Founded by engineers with a relentless commitment to customer satisfaction, we eagerly embrace every opportunity to address technical challenges—from designing cutting-edge UI/UX for data interaction to scaling our services across millions of virtual machines. And this is just the beginning.
In modern data analysis, advanced techniques such as machine learning surpass the capabilities of conventional SQL query engines. As a software engineer on the Runtime team at Databricks, you will be instrumental in developing next-gen distributed data storage and processing systems that not only outperform specialized SQL query engines in relational query performance but also offer the flexibility and programming abstractions necessary to support a diverse range of workloads, from ETL processes to data science applications.
Key projects you might work on include:
- Apache Spark™: Contribute to the leading open-source framework for big data.
- Data Plane Storage: Deliver reliable, high-performance services and client libraries for managing vast amounts of data on cloud storage solutions like AWS S3 and Azure Blob Store.
- Delta Lake: A sophisticated storage management system that merges the scalability and cost-effectiveness of data lakes with the performance and reliability of data warehouses, offering features such as ACID transactions and time travel to simplify complex data engineering architectures.
- Delta Pipelines: Simplify the management of extensive data engineering pipelines, allowing customers to effortlessly orchestrate, deploy, test, and upgrade thousands of data pipelines while minimizing operational burdens.
- Performance Engineering: Develop the next-generation query optimizer and execution engine that is fast, tuning-free, scalable, and robust.

