Qualifications
Key Responsibilities:Design, develop, and maintain scalable data pipelines using Azure Databricks.Implement large-scale data processing solutions utilizing PySpark.Optimize Spark jobs for performance, scalability, and cost efficiency.Integrate data from various sources, encompassing both structured and unstructured datasets.Collaborate with cross-functional teams to gather data requirements and deliver robust solutions.Conduct data cleansing, transformation, and aggregation to support analytics use cases.Monitor and troubleshoot data workflows to ensure reliability and availability.Enforce best practices for data governance, security, and compliance.Create and maintain documentation for data architecture, pipelines, and processes.Continuously enhance data engineering processes through automation and optimization.What Makes You a Great Fit:Extensive hands-on experience with Azure Databricks.Proficiency in PySpark for distributed and large-scale data processing.Strong understanding of Spark architecture and performance tuning.Experience with Azure cloud services and data storage solutions.Thorough knowledge of ETL/ELT processes and data transformation techniques.Ability to manage large datasets in distributed environments.Solid grasp of data modeling and data warehousing concepts.Capability to troubleshoot complex data processing issues.Strong analytical and problem-solving skills.Excellent communication and collaboration abilities.
About the job
Join Our Team as a Senior Data Engineer!
We are actively seeking an experienced Azure Databricks Engineer with a proven track record in developing and managing large-scale data processing solutions. In this pivotal role, you will be entrusted with the design, development, and optimization of data pipelines leveraging Azure Databricks and PySpark, all aimed at enhancing our advanced analytics, reporting, and business intelligence capabilities.
Your collaboration with data engineers, analysts, and business stakeholders will be crucial in transforming raw data into scalable, high-performance data solutions. The ideal candidate will possess a robust background in distributed data processing, with expertise in cloud-based data platforms and performance optimization within Azure ecosystems. This position demands proficiency in managing end-to-end data workflows and familiarity with modern data engineering best practices.
About Weekday's Client
Weekday's Client is a leading organization dedicated to harnessing the power of data for transformative business solutions. We prioritize innovation, collaboration, and excellence in our data engineering practices.