About the job
At Databricks, we are committed to empowering data teams to tackle the world’s most challenging issues—from pioneering the next generation of transportation to fast-tracking medical innovations. We achieve this by developing and operating the premier data and AI infrastructure platform, enabling our clients to leverage deep data insights for enhanced business performance.
Join our Spark™ Structured Streaming team, where we are at the forefront of integrating Stream Processing capabilities into the Spark™ and Databricks Intelligence Platform. As the stream processing landscape evolves, we aim to not only create cutting-edge technology but also to deliver a leading managed service that empowers customers to efficiently handle their Stream Processing workloads.
We are looking for an experienced Engineering Leader to drive the development of Spark™ Structured Streaming initiatives, incorporating both open-source and Databricks-specific elements. Your primary goal will be to elevate Spark™ Structured Streaming into a premier Stream Processing engine by introducing advanced features such as enhanced state management and new operators, while also optimizing performance from both latency and throughput perspectives through innovative architectural designs.

