Staff Software Engineer - Distributed Data Systems

P-186 At Databricks, we are passionate about empowering data teams to tackle some of the world's most challenging problems, from detecting security threats to advancing cancer drug development. Our mission is to build and operate the premier data and AI infrastructure platform, allowing our customers to concentrate on the high-value challenges central to their missions. Since our inception in 2013 by the original creators of Apache Spark™, Databricks has transformed from a small office in Berkeley, California, into a global powerhouse with over 1,000 employees. Today, thousands of organizations—from startups to Fortune 100 companies—trust Databricks with their mission-critical workloads, solidifying our status as one of the fastest-growing SaaS companies worldwide. Our engineering teams are dedicated to developing highly technical products that address real-world needs. We continuously push the limits of data and AI technology while ensuring the resilience, security, and scale essential for our customers' success on our platform. We manage one of the largest-scale software platforms, comprising millions of virtual machines that generate terabytes of logs and process exabytes of data daily. At this scale, we routinely encounter cloud hardware, network, and operating system faults, and our software must adeptly shield our customers from these challenges. In modern data analysis, sophisticated techniques such as machine learning extend far beyond the traditional SQL query engine's roll-up and drill-down capabilities. As a software engineer on the Runtime team at Databricks, you will contribute to building next-generation distributed data storage and processing systems that not only surpass specialized SQL query engines in relational query performance but also offer the expressiveness and programming abstractions necessary to support varied workloads, ranging from ETL to data science. Here are some example projects you might work on: Apache Spark™: Develop the de facto open-source standard framework for big data. Data Plane Storage: Provide reliable and high-performance services and client libraries for storing and accessing massive amounts of data on cloud storage backends like AWS S3 and Azure Blob Store. Delta Lake: A storage management system that merges the scale and cost-efficiency of data lakes with the performance and reliability of data warehouses, along with the low latency of streaming. Its higher-level abstractions, including ACID transactions and time travel, significantly simplify real-world data engineering architecture. Delta Pipelines: Managing a single data engineering pipeline can be challenging. The objective of the Delta Pipelines project is to streamline this process.

P-59 At Databricks, we are driven by a mission to empower data teams to tackle some of the most pressing challenges facing our world. Whether it's transforming transportation or accelerating medical innovations, we achieve this by creating and operating the foremost data and AI infrastructure platform. Our customers leverage deep data insights to enhance their business operations. Founded by engineers with a relentless commitment to customer satisfaction, we eagerly embrace every opportunity to address technical challenges—from designing cutting-edge UI/UX for data interaction to scaling our services across millions of virtual machines. And this is just the beginning. In modern data analysis, advanced techniques such as machine learning surpass the capabilities of conventional SQL query engines. As a software engineer on the Runtime team at Databricks, you will be instrumental in developing next-gen distributed data storage and processing systems that not only outperform specialized SQL query engines in relational query performance but also offer the flexibility and programming abstractions necessary to support a diverse range of workloads, from ETL processes to data science applications. Key projects you might work on include: Apache Spark™: Contribute to the leading open-source framework for big data. Data Plane Storage: Deliver reliable, high-performance services and client libraries for managing vast amounts of data on cloud storage solutions like AWS S3 and Azure Blob Store. Delta Lake: A sophisticated storage management system that merges the scalability and cost-effectiveness of data lakes with the performance and reliability of data warehouses, offering features such as ACID transactions and time travel to simplify complex data engineering architectures. Delta Pipelines: Simplify the management of extensive data engineering pipelines, allowing customers to effortlessly orchestrate, deploy, test, and upgrade thousands of data pipelines while minimizing operational burdens. Performance Engineering: Develop the next-generation query optimizer and execution engine that is fast, tuning-free, scalable, and robust.

At Databricks, we are committed to empowering data teams to tackle some of the world's most significant challenges—ranging from revolutionizing transportation to accelerating medical breakthroughs. Our mission is realized through the development and management of the foremost data and AI infrastructure platform, enabling our clients to derive deep insights from their data to enhance business outcomes. Founded by engineers and driven by a passion for customer success, we eagerly embrace each opportunity to confront technical hurdles, from innovating next-generation UI/UX for data interaction to scaling our services across millions of virtual machines. Our journey has just begun.As a Senior Staff Software Engineer, you will be a pivotal member of the executive leadership team within the Data Platform organization. You will utilize cutting-edge AI developer tools and methodologies to construct the Data Intelligence Platform, which will facilitate automated decision-making across Databricks. This role involves collaboration with distinguished Tech Leads, product teams, data scientists, and others. You will shape the future of various tools, including logging, orchestration, data transformation, metric stores, governance platforms, and data consumption layers.

Who We AreNuro is at the forefront of self-driving technology, dedicated to making autonomy a reality for everyone. Established in 2016, we have developed the world’s most scalable driver, merging advanced AI with automotive-grade hardware. Our innovative platform, the Nuro Driver™, is designed to facilitate a diverse array of applications—from robotaxis and commercial fleets to personal vehicles. With a proven track record from years of deployment, Nuro provides automakers and mobility platforms a clear path to commercial-scale autonomous vehicles, fostering a safer, more interconnected future.About the RoleWe are seeking experienced Software Engineers to expand and optimize Nuro's extensive cloud and data center computing infrastructure. This system is crucial for various business applications across the company, including machine learning data pipelines, data discovery, simulation, and time-series multimedia log processing. In this role, you will create a scalable, efficient, and reliable system that connects core infrastructure with high-level job requests, enhancing large-scale computations for autonomy and other engineering teams.About the WorkDesign and develop scalable, reliable frameworks, such as BATES, to manage and execute large workloads in both cloud and on-premise clusters.Collaborate with application teams across Nuro to provide robust, generic tools and APIs that streamline the development of mission-critical applications.Enhance workload efficiency and turnaround time by analyzing application context, identifying inefficiencies and bottlenecks, prioritizing tasks, and implementing innovative solutions.

Who We Are Nuro is a pioneering self-driving technology company dedicated to democratizing autonomy for everyone. Established in 2016, we are developing the world’s most scalable driver, integrating advanced AI with automotive-grade hardware. Nuro licenses its flagship technology, the Nuro Driver™, enabling a diverse range of applications from robotaxis and commercial fleets to privately owned vehicles. With a proven track record in self-driving deployments, Nuro provides automakers and mobility platforms with a clear pathway to commercial-scale autonomous vehicles, fostering a safer, more interconnected future. About the Role We are seeking talented senior engineers to enhance and scale Nuro’s expansive computing infrastructure within cloud and data centers. This infrastructure is vital for numerous essential business applications, including machine learning data pipelines, data discovery, simulation, and time-series multimedia log processing. In this role, you will design and implement a scalable, efficient, and reliable system that connects fundamental infrastructure with high-level job requests, making large-scale computations more accessible for autonomy and other infrastructure engineers. About the Work Design and build scalable, dependable frameworks, such as BATES, to orchestrate and manage large-scale workloads in both cloud and on-premise clusters. Collaborate with application teams across Nuro to develop robust, generic tools and APIs that expedite the creation of mission-critical applications. Enhance workload efficiency and turnaround time by deeply understanding application contexts, identifying inefficiencies and bottlenecks, prioritizing tasks, and implementing innovative solutions. About You You possess extensive experience in creating and developing large-scale distributed applications (e.g., Kubernetes). You are proactive in recognizing opportunities for system improvements and driving positive transformations. You have the capability to translate high-level requirements into detailed, cohesive system designs. You are eager to gain a deep understanding of application logic across Nuro and are passionate about going the extra mile. You are highly collaborative and dedicated to delivering the best possible user experience. You embody startup spirit: your track record shows you thrive in the dynamic, often ambiguous environment of a startup.

Join our innovative team at ServiceNow as a Senior Staff Software Engineer specializing in Agentic AI Systems. In this critical role, you will lead the design, development, and implementation of cutting-edge AI solutions that enhance our customer experience and drive operational efficiency. You will work collaboratively with cross-functional teams to create scalable and robust systems that utilize advanced machine learning techniques.

Waymo is a leader in autonomous driving technology dedicated to becoming the most trusted driver in the world. Originating from the Google Self-Driving Car Project in 2009, our focus has been on creating the Waymo Driver—an exceptional driver that enhances mobility access while significantly reducing traffic-related fatalities. The Waymo Driver not only supports our fully autonomous ride-hailing service but can also be adapted to various vehicle platforms and applications. With over ten million rider-only trips completed, our technology has safely navigated more than 100 million miles on public roads and processed tens of billions of miles in simulation across over 15 U.S. states.The Perception Data team at Waymo plays a critical role in shaping the strategy and technical direction for all data utilized in training and evaluating the Waymo Driver's perception capabilities. We manage the complete data lifecycle, developing automated systems and 'infrastructure-as-product' solutions that convert vast amounts of driving sensor data into high-quality training datasets. Our team tackles complex challenges, including active learning loops and open-vocabulary modeling, bridging raw data with advanced machine learning techniques.By integrating data ingestion, curation, and evaluation into a unified ecosystem, we facilitate the swift development of foundational models and next-generation perception systems. We work closely with Machine Learning, Infrastructure, and Evaluation teams to address intricate data challenges, ensuring our models can effectively interpret the long-tail of rare events. Ultimately, our contributions lay the groundwork for the Waymo Driver to operate safely in diverse environments.

Role Overview Anduril Industries is hiring a Staff System Modeling Engineer for Warfighter Systems in Mountain View, California. This role centers on applying advanced modeling methods to improve military systems and support mission success. Work here shapes the effectiveness of defense solutions and has a direct impact on operational outcomes.

P-188 At Databricks, we are on a mission to fundamentally transform the data lifecycle, simplifying processes from data ingestion to ETL, Business Intelligence (BI), and extending to Machine Learning (ML) and Artificial Intelligence (AI) through a unified platform. We envision a future where traditional data warehouse architectures are replaced by innovative solutions like the Lakehouse architecture (CIDR 2021 paper), which integrate data warehousing and advanced analytics, effectively addressing critical challenges such as data staleness, reliability, cost of ownership, data lock-in, and limited use-case support. To turn this vision into reality, we are developing a next-generation decoupled query engine and structured storage system designed to surpass specialized data warehouses in relational query performance while maintaining the versatility of general-purpose systems like Apache Spark™. This system is intended to support a wide array of workloads, from ETL processes to data science applications. As a member of our team, you will engage in one or more of the following areas, contributing to the design and implementation of cutting-edge systems that redefine industry standards: Query compilation and optimization Distributed query execution and scheduling Vectorized execution engine Data security measures Resource management strategies Transaction coordination processes Efficient storage structures (encodings, indexes) Automatic physical data optimization techniques

The RoleAre you a talented software engineer eager to leverage your extensive experience in developing and enhancing production systems at the forefront of AI technology? If so, we invite you to consider our Staff Software Engineer position on the Natural Language Understanding team within our 'agent lab'. In this pivotal role, you will contribute significantly to our mission of extending the capabilities of AI agents to execute tasks reliably on a large scale.As part of the Moveworks AI Assistant platform, you will lead the innovation in areas such as agent orchestration, sandboxed file systems, latency optimization, agent memory, and the advancement of LLMs (Large Language Models). Our team thrives on rapidly developing scalable infrastructure and tackling complex product and engineering challenges to maximize value for our customers. If you aspire to achieve the pinnacle of your career while working with a passionate and impactful team, we would love to connect with you.What You Will Do:Lead and tackle challenging engineering projects to create and enhance dependable AI agent systems.Define and implement cutting-edge distributed systems, ensuring their scalability and production readiness, while integrating the latest advancements in machine learning and AI.Set technical direction and influence the engineering roadmap to deliver enduring value to our clients.Establish high standards for code quality, including robustness, extensibility, and performance.Collaborate with senior experts across various domains, including machine learning, security, product development, and customer success, to build world-class enterprise AI solutions.Mentor and develop team engineers, focusing on technical design, execution, and excellence.Work closely with machine learning engineers to define and develop end-to-end architecture for outstanding AI systems.

ECL is leading a technological revolution with the launch of the world’s first hydrogen-powered, off-grid green data center, achieving an impressive uptime of 99.999%. As the demand for high-performance data centers grows to support AI and machine learning, ECL is at the forefront of delivering innovative solutions. Our cutting-edge modular architecture allows for scalable construction projects ranging from 1-2MW, designed to cater to the needs of larger data center installations. Join us and contribute to a completely green power generation system amidst the rapid advancements in artificial intelligence and machine learning.This is a unique opportunity to be part of a groundbreaking team poised for significant growth. ECL was founded by industry veterans from renowned companies such as Facebook, Microsoft, LinkedIn, Cisco, HPE, and Bloom Energy.

Job Title: Senior Staff Software Engineer - Enzyme Location: Mountain View, California At Databricks, we are driven by a passion for empowering data teams to tackle some of the world’s most intricate challenges—from revolutionizing transportation to accelerating medical advancements. We achieve this by developing and managing the leading data and AI infrastructure platform, enabling our customers to harness deep data insights to elevate their businesses. The Lakeflow Engineering team at Databricks oversees our comprehensive ETL product line, including innovative solutions like Materialized Views, Structured Streaming, and Delta Live Tables. We operate one of the largest data engineering platforms globally, processing exabytes of data daily for our extensive client base. We are in search of a committed technical leader to guide the Materialized Views engineering team. This team is tasked with creating next-generation Materialized View features that enhance both ETL workloads and query acceleration. In this role, you will engage in various areas to design and implement advanced systems that surpass current state-of-the-art: Incrementally maintaining materialized views Optimizing query performance Managing resources efficiently Developing effective storage structures Automating physical data optimization Your Impact: Address significant business needs at a large scale through your software engineering expertise. Deliver a robust, scalable, and fault-tolerant architecture for Materialized Views. Conduct low-level systems debugging, performance measurement, and optimization on large production clusters. Shape architecture design, influence product roadmaps, and assume ownership of new projects. Introduce tools that enhance automation and operational efficiency of services. Utilize your extensive experience to help preempt and resolve production issues. Plan and lead complex technical projects collaborating with multiple teams across the organization.

Join our dynamic team at LinkedIn Corporation as a Senior Staff Software Engineer specializing in Systems Infrastructure. In this pivotal role, you will be responsible for designing, implementing, and maintaining robust infrastructure systems that support our growing platform. You will collaborate with cross-functional teams to drive innovation and optimize our systems for scalability and performance.

P-1124 Summary At Databricks, we are driven by a passion for empowering data teams to tackle some of the most challenging problems in the world, spanning from security threat detection to advancements in cancer drug development. We achieve this by constructing and managing a top-tier data and AI infrastructure platform, enabling our customers to concentrate on their high-value missions. Our engineering teams are dedicated to creating highly technical products that address significant needs in the world. We continuously push the limits of data and AI technology while maintaining the resilience, security, and scalability essential for our customers’ success on our platform. Customers place their trust in Databricks with their most sensitive data, and our Trust & Safety team is committed to establishing the world’s most reliable data analytics and machine learning platform. Security Engineering plays a pivotal role within Trust & Safety, working diligently to safeguard customer data from malicious threats. We are seeking seasoned leaders like you to craft the vision and define the strategy for this critical area. The impact you will have: Enhance the safety of Databricks for our customers by identifying and addressing critical vulnerabilities in our infrastructure and services. Attract industry-leading talent to our team. Serve as a representative for the security engineering discipline throughout the organization, advocating for a data-driven approach. Represent Databricks at academic and industry conferences and events. What we look for: A minimum of 7 years of experience in Data Security or related fields, with expertise in at least two of the following areas: Cryptography, Kubernetes Security, Web Security, Governance, Privacy, Trust, Safety, Authentication, Identity Management, Access Control, Key Management, Inter-Service Authentication, Secure Application Frameworks, Detection & Response. Proven experience in building systems at large-scale internet companies is highly desirable. ...

P-1126 Summary At Databricks, we are passionate about empowering data teams to tackle the world's most pressing challenges, ranging from security threat detection to groundbreaking cancer drug development. Our mission is to create and maintain the premier data and AI infrastructure platform that enables our customers to concentrate on high-value tasks central to their objectives. Our engineering teams are dedicated to developing innovative products that address significant real-world needs. We continually strive to advance the frontiers of data and AI technology while ensuring our operations are resilient, secure, and scalable to guarantee our customers' success on our platform. Customers place their trust in Databricks with their most critical data, and the Trust & Safety department's mission is to establish the most reliable data analytics and machine learning platform globally. Our Security Engineering team plays a pivotal role in safeguarding customer data from malicious threats. We seek seasoned leaders like you to craft the vision and strategy for this vital domain.

At Databricks, we are dedicated to empowering data teams to tackle some of the most significant challenges in the world—from realizing the next mode of transportation to accelerating medical innovations. Our mission is to build and operate the premier data and AI infrastructure platform, enabling our customers to harness deep data insights for business enhancement. Our engineering teams create technical products that address real-world needs. We continually push the limits of data and AI technology while ensuring the security and scalability essential for our customers' success on our platform. We manage one of the largest-scale software platforms, operating millions of virtual machines that generate terabytes of logs and process exabytes of data daily. Our software is designed to seamlessly shield our customers from cloud hardware, network, and operating system faults. As a member of the Observability team, you will craft observability solutions that deliver insights into the health and performance of our products and infrastructure.

Join Databricks as a PhD Systems Software Engineer where you will leverage your advanced academic training to contribute to innovative software solutions. You will collaborate with a team of talented engineers and researchers to push the boundaries of technology and deliver high-quality systems that meet our clients' needs.

Waymo is seeking a Staff Software Engineer focused on DevAI to help advance artificial intelligence for autonomous vehicles. This position is based in Mountain View, CA. Role overview This role centers on designing and building AI solutions that support Waymo's self-driving technology. Collaboration with teams across engineering, product, and research will be a core part of daily work. The goal is to strengthen AI capabilities and improve system performance. What you will do Design and develop AI-driven features for autonomous vehicle systems Work closely with colleagues from different disciplines to integrate and refine AI models Contribute to the ongoing improvement of system performance and reliability Impact This position plays a key part in shaping how AI supports the future of transportation at Waymo.

Who We AreGatik is at the forefront of transforming middle-mile logistics through our cutting-edge autonomous transportation-as-a-service (ATaaS) solution. Our focus is on delivering safe and consistent services while enhancing freight movement efficiency and alleviating congestion. In 2021, we made history by launching the world's first fully driverless commercial transportation service in partnership with Walmart, with our Class 3-7 autonomous trucks actively operating in key markets like Texas, Arkansas, and Ontario, Canada.Our proprietary Level 4 autonomous technology, Gatik Carrier™, is meticulously engineered to facilitate safe and efficient freight transportation between pick-up and drop-off points. With robust capabilities tailored for both highway and urban settings, Gatik Carrier™ represents a comprehensive solution that seamlessly integrates advanced software and hardware to optimize logistics operations for our clients.About the RoleWe are seeking a Senior/Staff Systems Engineer to lead the Verification and Validation processes for Gatik’s autonomous vehicle systems. This position is primarily located at our headquarters in Mountain View, CA, with additional opportunities available in Fort Worth, TX.What You Will DoEstablish and implement model-based system engineering (MBSE) practices and methodologies for effective requirements management, ensuring clear communication of functional and performance requirements across stakeholders and system levels.Provide advanced systems expertise in integrating architectural changes and component revisions, ensuring optimal performance and efficiency in autonomous vehicle operations.

Who We Are Nuro is a pioneering self-driving technology company dedicated to making autonomy accessible to everyone. Established in 2016, Nuro is revolutionizing transportation with the most scalable autonomous driver, integrating advanced AI with automotive-grade hardware. Our flagship technology, the Nuro Driver™, is licensed to cater to various applications, including robotaxis, commercial fleets, and personal vehicles. With years of proven self-driving deployments, Nuro provides automakers and mobility platforms a clear pathway to commercial-scale autonomous vehicles, fostering a safer, more connected future. About the Role We are in search of a Senior/Staff Software Engineer to take on a pivotal technical leadership role in Nuro’s Machine Learning (ML) Data engine. You will be positioned at the crucial intersection of Autonomy, Machine Learning, and Infrastructure, serving as an architect for the systems that power our autonomy AI models. In this role, you’ll collaborate with the Autonomy team to implement the technical strategy to convert vast amounts of autonomy data into high-value training signals for decision-making. Your responsibilities will include designing and constructing data products for autonomy researchers, developing queries for rare 'needle-in-a-haystack' scenarios, and automating labeling and data ingestion workflows. You will work closely with Autonomy ML researchers to grasp their data needs, cooperate with infrastructure teams to establish the appropriate data interfaces and APIs, and develop robust tools for data selection, simulation, and introspection, all capable of handling data at scale. If you thrive on solving complex problems with practical solutions that impact the real world, we invite you to join us. About the Work Data Pipeline Architecture: Design and implement scalable data ingestion and processing pipelines that transform data streams into targeted training datasets. Lead efforts to enhance data quality, identify anomalies, and manage out-of-distribution examples to ensure effective model training and deployment. Cross-Functional Leadership: Collaborate with autonomy teams and data infrastructure teams to create effective ML data pipelines and products for ML engineers. ML Tooling & Introspection: Build infrastructure and visualization tools that enable ML researchers to easily introspect data, identify model failure modes, request new data samples, and comprehend shifts in data distribution. Labeling Operations Integration: Work closely with the data operations team to establish quality standards for labeling operations.