Qualifications
Key Responsibilities:
Lead the development and deployment of machine learning-based search and discovery relevance models and systems integrated with Databricks' products and services.
Design and implement automated ML and NLP pipelines for data preprocessing, query understanding, rewriting, ranking, retrieval, and model evaluation, allowing for rapid experimentation and iteration.
Collaborate with product managers and cross-functional teams to spearhead technology-first initiatives that drive innovative business strategies and product roadmaps for the search and discovery experience.
Contribute to the development of a robust framework for evaluating improvements in search ranking—both offline and online.
What We’re Looking For:
A Bachelor’s degree (Master's or PhD preferred) in Computer Science or a related field.
Over 6 years of experience in developing search relevance systems at scale, either in production environments or impactful research settings.
Proven experience applying Large Language Models (LLMs) to enhance search relevance.
Familiarity with one or more of the following areas:
Query understanding
Natural Language Processing (NLP)
Text mining
Recommendation systems
Personalization techniques
Discovery mechanisms
Conversational AI
A strong grasp of computer science fundamentals.
Contributions to popular open-source projects.
About the job
Join the Applied AI team at Databricks, where we are leading the charge in developing cutting-edge AI/ML-driven products. Our customers generate a vast array of assets like tables, notebooks, dashboards, datarooms, pipelines, SQL queries, and ML models on our platform, often numbering in the hundreds of millions. Navigating through these assets is crucial for our users, enhancing their productivity and effectiveness.
As our Search product evolves, we are on the lookout for talented Machine Learning Engineers at various levels—from junior to senior—to enhance our Search Quality. By 2026, our focus will be on elevating search rankings, refining query understanding, establishing robust evaluations, and expanding asset coverage to facilitate seamless search experiences at scale.
About Databricks
At Databricks, we are at the forefront of innovation, developing state-of-the-art AI solutions that redefine user interaction with data and our products. You will have the opportunity to work alongside a talented team dedicated to transforming the industry.
