Senior Machine Learning Engineer at Databricks | San Francisco

At Databricks, we are driven by our mission to empower data teams to tackle the world's most challenging problems—whether it's realizing the next mode of transportation or accelerating medical advancements. Our dedication is to build and operate the premier Data and AI infrastructure platform, enabling our clients to harness deep insights from data to enhance their operations. We are on the lookout for a Senior Engineering Manager to join our Photon team, playing a pivotal role in defining the next generation of ultra-fast, scalable data processing solutions. In this role, you will lead a team of exceptional engineers working on Photon—the innovative query engine that powers Databricks' groundbreaking Lakehouse Platform. This is a unique opportunity to drive innovation on a grand scale, optimize data systems for tangible results, and collaborate with top experts in query optimization, execution, storage, and large-scale analytics. If you are passionate about empowering teams, architecting groundbreaking technologies, and accelerating analytics for millions of users, we invite you to join our team.

Join Databricks as a Senior Backend Software Engineer and contribute to solving some of the world's most challenging problems. We empower data teams through our state-of-the-art data and AI infrastructure platform, enabling businesses to leverage deep data insights for transformative growth. At Databricks, founded by engineers and driven by a customer-first philosophy, we tackle technical challenges with enthusiasm — from designing next-gen UI/UX for data interfacing to scaling our services across millions of virtual machines.In this role, you will collaborate closely with your team to develop scalable infrastructure and innovative products for the Databricks platform.

P-150 At Databricks, we are dedicated to empowering data teams to tackle some of the world's most challenging problems, ranging from security threat detection to the development of cancer treatments. We achieve this by creating and managing the leading data and AI infrastructure platform, allowing our clients to concentrate on the high-value challenges central to their missions. Founded in 2013 by the original creators of Apache Spark™, Databricks has evolved from a modest office in Berkeley, California, to a global enterprise with over 1000 employees. Our platform is trusted by thousands of organizations, from startups to Fortune 100 companies, establishing us as one of the fastest-growing SaaS firms worldwide. Our engineering teams design and develop highly technical products that meet significant real-world needs. We continually push the limits of data and AI technology while maintaining the resilience, security, and scalability essential for our customers' success on our platform. We operate one of the largest-scale software platforms, comprising millions of virtual machines that generate terabytes of logs and process exabytes of data daily. Given our scale, we frequently encounter cloud hardware, network, and operating system faults, and our software must effectively shield our customers from any disruptions. As a Software Engineer focused on backend development, you will collaborate closely with your team and product management to prioritize, design, implement, test, and operate microservices for the Databricks platform and products. This role includes writing software in Scala/Java, building data pipelines using Apache Spark™ and Apache Kafka, integrating with third-party applications, and interacting with cloud APIs such as AWS, Azure, CloudFormation, and Terraform. Below are examples of teams you can join: Data Science and Machine Learning Infrastructure: Develop services and infrastructure that bridge the gap between machine learning and distributed systems. Our technology supports flagship collaborative workspaces, notebooks, IDE integrations, and project management tools. We also facilitate large-scale machine learning through environment management, distributed training, and lifecycle management via MLflow. Compute Fabric: Construct the resource management infrastructure that supports all big data and machine learning workloads on the Databricks platform in a robust, flexible, secure, and cloud-agnostic manner. This software oversees millions of virtual machines.

The Applied AI team at Databricks is at the forefront of pioneering GenAI-powered products. In recent years, we have successfully launched the Databricks Assistant, AI/BI Genie, and Agent Bricks, collaborating with product teams to significantly enhance LLM quality for these offerings. These innovations are utilized by hundreds of thousands of Databricks users daily. We are dedicated to solving complex challenges such as code suggestion, error detection and correction, text-to-SQL generation, automatic pipeline generation, and knowledge QA.As we continue to evolve our GenAI products, we are looking for multiple GenAI Engineers at various experience levels to lead the next phase of our development. Our goals for 2025 include improving LLM quality, broadening GenAI capabilities across Databricks products, and reinforcing our platform architecture to facilitate seamless AI interactions on a large scale.

Join Databricks as a Senior Product Manager for Unity Catalog, where we are dedicated to empowering data teams to tackle the world's most pressing challenges. Our innovative data and AI infrastructure platform is designed to help customers harness deep insights to enhance their business operations. Founded by engineers and driven by customer feedback, we eagerly take on technical challenges, from crafting next-gen UI/UX for data interaction to scaling our services across millions of virtual machines.In this role, you will oversee all product activities for Unity Catalog, the foundational metadata and governance layer for the Lakehouse. This includes implementing fine-grained security, governance capabilities, and productivity features like data lineage, search, discovery, auditing, and compliance for our customers' data and AI assets.Your responsibilities will involve coordinating product vision to implementation, engaging with large enterprise clients to ascertain their needs, developing strategic product roadmaps, collaborating with engineering to bring products to life, and liaising with various internal stakeholders to ensure successful product outcomes.Your Impact:Design and implement new governance and security features as part of Unity Catalog.Define and develop platform services that empower Databricks’ data & AI product teams, including ML/AI, Data Science & Engineering, and Data Warehousing, to seamlessly integrate workload-specific governance enhancements.Drive the lifecycle of new features from ideation through customer engagement to launch, including marketing strategies and ensuring broad customer adoption across all segments.Provide insights on the data governance landscape, customer needs, and market trends to engineering and leadership teams through close collaboration with customers and partners.

Are you a dedicated Developer Advocate with a passion for bridging the gap between technology and the developer community? If you thrive on connecting with others, hosting engaging events, and diving into the nuances of application development and developer experience, we want to hear from you!As a Senior Developer Advocate at Databricks on the Neon DevRel team, you'll play a pivotal role in linking our engineering teams with an expanding community of application and agent developers utilizing Postgres and Neon. Your technical acumen will be essential in maintaining and enhancing our SDKs, educating users, and shaping the future of Agent and Developer Experience (AX & DX) for Neon.Developer experience will be central to your responsibilities. You will help enhance our TypeScript SDK and CLI, manage Neon’s MCP server, and develop other developer-oriented tools, all while keeping ahead of the evolving AI landscape. Your expertise in DevTools and AX will be shared through presentations, meetups, blog articles, videos, and community interactions, empowering developers to build more effectively with Neon and demonstrating our commitment to exceptional Developer Experience (DX) and Agent Experience (AX).Collaboration will be key as you work closely with the neon.com community, product teams, and engineering staff, ensuring that developer feedback directly influences product direction. You'll also partner with fellow advocates, program managers, and the broader Databricks DevRel organization to create a cohesive and impactful strategy for agent experiences.The ideal candidate will embody the core values of our DevRel team: a profound enthusiasm for application development, strong empathy for developer needs, and the ability to convey complex concepts with clarity and effectiveness.In this role, you’ll foster and expand the Neon community through top-notch technical content and hands-on experiences, continuously feeding insights back into the product to further enhance the developer experience.

At Databricks, we are driven by a commitment to empower data teams in overcoming the world's most complex challenges, from realizing the next generation of transportation to expediting groundbreaking medical innovations. Our mission is fulfilled through the development and operation of the premier data and AI infrastructure platform, enabling our clients to harness profound data insights to enhance their enterprises. Founded by engineers with a customer-centric mindset, we eagerly embrace every chance to confront technical challenges – whether it's crafting next-gen UI/UX for data interaction or scaling our services and infrastructure across a multitude of virtual machines. About Our Team: Delivering a stellar platform necessitates an outstanding developer experience. The Databricks Repos team lays the groundwork for source control and workflow, facilitating how data engineers, machine learning experts, and analysts manage code, collaborate with peers, and deploy data and AI applications into production. Our team enhances the version control experience within Databricks, delineating user interactions with Git providers, code change management, lineage tracking, pull request evaluation, and code promotion into jobs, pipelines, and deployments. We are dedicated to crafting a seamless, reliable, AI-native workflow that merges the best practices of Git-style development with the specific requirements of data and AI teams. Our goal is to offer an effortless journey from authoring to production across notebooks, SQL, Python, and Lakeflow, all within a unified platform. In this role, you will spearhead the vision and roadmap for Databricks Repos, determining how developers create, test, review, and deploy code within our ecosystem. Collaborating with the Workflows, Notebooks, Lakeflow, Data Management, and broader Developer Experience teams, you will deliver an integrated and user-friendly path to production. Your Impact: Design the comprehensive production experience for data and AI teams, encompassing authoring, Git integration, and CI/CD processes. Define integration strategies for Repos with GitHub, GitLab, and Azure DevOps, promoting seamless branching, pull requests, reviews, conflict resolution, and deployment workflows. Collaborate with engineers to develop AI-assisted code management features, including automated code suggestions, recommended diffs, merge assistance, and deployment validation. Act as the voice of the customer, translating user feedback into workflow enhancements that simplify source control and release management. Boost developer productivity and reliability by aligning Repos with Databricks Workflows, Lakeflow, and deployment mechanisms. Facilitate the growth of Databricks adoption.

About This Role Join Databricks as a Software Engineer focused on GenAI inference, where you will play a pivotal role in designing, developing, and enhancing the inference engine that drives our Foundation Model API. Collaborating at the intersection of research and production, you will ensure our large language model (LLM) serving systems are optimized for speed, scalability, and efficiency. Your contributions will span the entire GenAI inference stack, from kernels and runtimes to orchestration and memory management. What You Will Do Participate in the design and implementation of the inference engine, collaborating on a model-serving stack tailored for large-scale LLM inference. Work closely with researchers to integrate new model architectures or features such as sparsity, activation compression, and mixture-of-experts into the engine. Optimize latency, throughput, memory efficiency, and hardware utilization across GPUs and other accelerators. Build and maintain tools for instrumentation, profiling, and tracing to identify bottlenecks and inform optimization efforts. Develop scalable routing, batching, scheduling, memory management, and dynamic loading mechanisms for inference workloads. Ensure reliability, reproducibility, and fault tolerance in inference pipelines, including A/B launches, rollback, and model versioning. Integrate with federated and distributed inference infrastructure, orchestrating across nodes, balancing load, and managing communication overhead. Engage in cross-functional collaboration with platform engineers, cloud infrastructure, and security/compliance teams. Document and share insights, contributing to internal best practices and open-source initiatives as appropriate.

P-1398 Note: This is a hybrid role requiring approximately 3 days in the office in Plano, TX. Mission: As a Staff Data & AI Technical Solutions Engineer, you will take the lead in developing and guiding others in crafting innovative Data & AI technical solutions for various customer issues. This includes an in-depth exploration of production data pipelines, AI workflows, streaming data ingestion, and more. You possess extensive expertise in Data & AI architectures, having encountered a diverse array of use cases while successfully managing intricate production environments in compliance with Service Level Agreements (SLAs). You will work collaboratively with product engineering and customer account teams to deliver exceptional results for customers, especially in critical production downtime scenarios. Your role is to empower customers with the knowledge and skills necessary to create efficient architectures and dependable services, leveraging Databricks’ platform to achieve their strategic goals. Reporting to a Technical Support Engineering manager, you are recognized as a technical authority within a leading global support engineering organization, celebrated for your leadership, technical knowledge, and ability to resolve even the most challenging customer situations. The impact you will have: Lead and direct technical solutions for a wide range of complex challenges reported by Databricks customers, including escalated and critical cases. Conduct comprehensive code-level analysis and systems architecture assessments of customer workloads to resolve issues related to Databricks products, including Spark core internals, Spark SQL, Delta, DLT, and Model Serving. Act as a customer support advisor, adept at diffusing escalations during incidents directly with customer stakeholders, swiftly identifying mitigations and solutions for advanced use cases, and strategically helping to avert future customer issues. Contribute to simplifying Databricks products and enhancing customer production environments' stability by influencing Engineering and Backline Support teams to identify areas for product improvements. Establish yourself as a deep technical expert and leader in the Databricks platform and production systems, mentoring others to share your expertise. What we look for: A minimum of 8 years of experience in designing, building, testing, and maintaining data pipelines, with relevant past experience in Spark being mandatory. Proficiency in SQL databases or data warehouses, among other technical skills.

At Databricks, we are committed to empowering organizations to unlock the full potential of data and artificial intelligence. Our goal is to create the premier platform for the development, deployment, and management of AI applications at scale, enabling our customers to leverage intelligent solutions to revolutionize their businesses.About Our Team:The Databricks AI team is pioneering how businesses utilize artificial intelligence. Our mission involves creating foundational capabilities that allow customers to build agents and models, manage intricate workflows, and integrate AI seamlessly into their data and applications. As the AI landscape evolves rapidly, our approach requires innovative thinking and the adaptability to navigate these changes. We are not merely creating features; we are fundamentally reshaping how the world interacts with AI.Your Impact:Shape the future of enterprise AI: Define and advance the vision of how Databricks empowers customers to harness generative AI, agents, and groundbreaking workloads.Transform breakthroughs into products: Collaborate with top-tier engineering and research teams to convert state-of-the-art AI innovations into reliable tools for millions.Expand possibilities for customers: Work closely with data and AI leaders to identify novel use cases and design products that make the previously unfeasible achievable and replicable.Be the voice of vision and execution: Inspire the roadmap from concept to launch, steering engineering efforts and articulating how Databricks AI transforms customer capabilities.Build for scale and longevity: Establish the strategies and principles that will guide Databricks AI for years to come, even as the industry undergoes rapid changes.

At Databricks, our passion lies in empowering data teams to tackle some of the most challenging problems globally, from detecting security threats to advancing cancer drug development. We achieve this by creating and operating the world's premier Data Intelligence Platform, enabling our customers to concentrate on high-value challenges central to their missions. Founded in 2013 by the original creators of Apache Spark, Databricks has evolved from a small office in Berkeley, California, into a global powerhouse with over 1000 employees. We are trusted by thousands of organizations—from small startups to Fortune 100 companies—with their mission-critical workloads, establishing us as one of the fastest-growing SaaS companies worldwide. Our engineering teams are dedicated to developing highly technical products that address real-world needs. We continuously push the limits of data and AI technology while maintaining the resilience, security, and scalability crucial for our customers' success on our platform. Operating one of the largest-scale software platforms, our infrastructure comprises millions of virtual machines that generate terabytes of logs and process exabytes of data daily. At this scale, we routinely encounter cloud hardware, network, and operating system faults, and our software is designed to shield our customers from such issues effectively. As a Data Scientist on the Data Team, you will play a pivotal role in fostering a data-driven culture within Databricks by addressing product and business challenges. The Data Team serves as an internal, production 'customer' that utilizes Databricks and influences the future trajectory of our products. Your Impact: Steer the direction of key data science initiatives including segmentation, recommendation systems, forecasting, product analytics, churn prediction, and insights. Collaborate closely with Engineering, Product Management, Sales, and Customer Success to discern product usage patterns and trends, facilitating data-driven decisions, recommendations, and forecasts. Manage stakeholder expectations in your focus area—gather evolving requirements, define project OKRs and milestones, and effectively communicate progress and results to non-technical audiences. Mentor and support junior data scientists within the team, assisting with project planning, technical decisions, and conducting code and documentation reviews. Advocate for the data science discipline across the organization, amplifying our commitment to becoming more data-driven. Develop self-service internal data products to simplify data access within the company. Represent Databricks at academic and industry conferences and events.

Location: San Francisco Bay Area, California REQ ID FEQ127R103 Mission Join our team as a Solutions Architect, where you will play a pivotal role in empowering large enterprises to harness the full potential of their data. Collaborating closely with the Enterprise Account Executive (AE), you will define and implement the technical strategy for our most significant accounts, driving deeper and wider adoption of machine learning (ML) and artificial intelligence (AI) solutions. Your expertise in value-based selling, technical account management, and leadership will be key in maximizing client success. You will work alongside a dedicated team that builds proofs of concept and showcases Databricks' innovative products. A strong technical understanding of ML and AI applications across various use cases relevant to the High Tech and Manufacturing sectors is essential. Outcomes Act as the primary technical voice for Databricks across multiple clients. Guide customers through a transformational journey, facilitating their evaluation and adoption of Databricks solutions. Implement tailored technical strategies within accounts based on a deep understanding of their needs. Foster a network of technical champions within client organizations. Align technical strategies with Databricks solutions effectively. Provide structured mentorship to enhance the skills of team members. Earn respect from peers through your experience, insights, and contributions. Competencies: Expertise in establishing and leading virtual teams to achieve success within client accounts. Experience managing large global accounts with an annual recurring revenue (ARR) exceeding $1 million. Ability to build relationships with executives and key influencers. Skill in presenting persuasive viewpoints to decision-makers, steering them toward successful outcomes. Strong technical background in big data, data science, and cloud technologies. Proven ability to drive data-driven business transformation. Programming experience in Python, R, Scala, or Java. Preferred: Databricks Certification.

Join Databricks as a Senior Manager in Production Engineering, where you will lead a team of talented engineers in developing and optimizing our groundbreaking data platform. You will drive innovation, enhance productivity, and ensure the seamless operation of our production environments.Your strategic vision and technical expertise will be pivotal in shaping the future of our engineering practices. You will collaborate closely with cross-functional teams to align engineering goals with business objectives, ensuring our solutions are both efficient and scalable.

Join Databricks as a Senior Engineering Manager specializing in AI Runtime. In this pivotal role, you will lead a talented team in developing innovative solutions that drive the future of artificial intelligence. Your expertise will guide the architecture and implementation of AI runtime components, delivering scalable and efficient systems that empower our clients to harness the full potential of AI technologies.Your responsibilities will include fostering a collaborative environment, mentoring team members, and ensuring the team's alignment with our strategic vision. You will collaborate closely with cross-functional teams to define and execute the roadmap for AI runtime, ensuring high performance and reliability.

At Databricks, we are dedicated to empowering data teams to tackle some of the world's most challenging problems. Our Databricks Free Edition provides a platform where students, enthusiasts, and aspiring professionals can engage with data and AI directly, building the skills and confidence needed to shape the future. We are in search of a Senior Product Manager to be a key player in steering the Databricks Free Edition initiative. In this role, you will work closely with a diverse team of product managers, engineers, and designers to craft experiences that make learning data and AI both engaging and accessible.

Join us in shaping the future of data and AI with Databricks SQL!At Databricks, we empower enterprises and startups globally by delivering cutting-edge Data & AI solutions. Our Databricks SQL (DBSQL) product stands at the forefront of next-generation AI-powered cloud data warehousing, enabling businesses to efficiently query, visualize, and make real-time decisions at scale. Recognized as the fastest-growing data warehouse globally, DBSQL is revolutionizing the industry.We are seeking a dynamic, strategic, and customer-focused Product Manager to drive innovation, define our roadmap, and guide DBSQL through its next phase of expansion.

At Databricks, we are dedicated to empowering data teams to tackle the world's most challenging issues, from realizing the next generation of transportation to expediting medical advancements. Our mission involves constructing and managing the premier data and AI infrastructure platform, enabling our clients to leverage profound data insights to enhance their operations. The Workspace Platform team is embarking on an ambitious journey to scale our customer base by 100x and support the evolution of agentic AI workloads. Our objective is to create a unified, consistent, and foundational Shared Platform that enhances the overall Databricks Workspace experience. As a Senior Engineering Manager within the Workspace Platform team, you will spearhead the development of a cohesive infrastructure that underpins vital customer-facing functionalities across various Databricks products. These include Content Discovery (similar to Google Search), Content Organization (akin to Google Drive), collaborative code editing, and repository management (comparable to GitHub). This is a high-impact opportunity to lead a team of approximately 20 software engineers in developing platform features, intuitive workspace experiences, and essential partner integrations that are pivotal to Databricks' growth and user adoption.

Join Databricks as a Senior Manager in Commissions FP&A, a pivotal role where you will lead the GTM Finance team's commission planning, forecasting, and analytics. You will collaborate closely with the Sales Compensation Strategy and Sales Strategy teams during the annual planning and design cycles. Additionally, you will engage with Accounting to streamline the monthly accrual process and work with GTM Finance partners to deliver insightful analyses on compensation plan success.This strategic position offers a unique opportunity to influence one of the most critical investment areas in our go-to-market strategy.Your Impact:Collaborate with the Sales Compensation Strategy team to align annual compensation plans with business objectives.Oversee the cost modeling for compensation plans, designing quota guardrails, accelerator rates, and managing overassigned quotas to balance commission payouts with performance goals.Develop and uphold a comprehensive commissions forecast, utilizing actuals and anticipated top-line performance to assess budget variances and provide analytical insights to partners.Manage the monthly commissions accrual process, delivering key variance insights to Finance leadership.Conduct ad hoc analyses, offering recommendations for mid-year investments and modifications to compensation plan designs.

Role Overview The Business Intelligence Manager at Databricks shapes how business users interact with data and AI on the Databricks platform. This role focuses on building secure, interactive applications that help teams turn raw data into practical insights. The position is based in New York or San Francisco. What You Will Do Build: Design and develop frontend interfaces and API backends. This includes setting up FastAPI to route user queries to model-serving endpoints, and delivering solutions from data dashboards to enterprise chat tools using the Mosaic AI Agent Framework. Architect: Create secure, scalable application architectures that support custom SaaS offerings and meet go-to-market needs. Scale: Develop applications that connect to Databricks SQL through the Statement Execution API or Databricks SDK. Set up CI/CD pipelines with Declarative Automation Bundles (DABs) to automate deployments across development, staging, and production. Key Technologies Streamlit React FastAPI Databricks Apps serverless infrastructure Locations New York San Francisco, California