About the job
We are seeking a highly skilled Senior DevOps Engineer to join our innovative team at Tyme. The ideal candidate will possess extensive experience in managing workloads on EC2 clusters using DataBricks or similar technologies, ensuring seamless data connectivity and availability through Data Mesh architecture. You will play a crucial role in implementing least privileged access across AWS accounts and the Databricks data platform, while also operating distributed computing engines such as Spark. Success in this role requires a solid understanding of storage technologies, data orchestration, CI/CD pipelines, and operational best practices with Spark and other data warehousing tools.
Your Responsibilities
- Oversee, plan capacity, and operate workloads on EC2 clusters using DataBricks/EMR to guarantee efficient and reliable data processing.
- Partner with stakeholders to design and implement a Data Mesh architecture across multiple enterprise entities.
- Employ Infrastructure as Code (IaC) tools like CloudFormation or Terraform to define and manage user access to data and compute resources on the data platform.
- Establish role-based access control (RBAC) mechanisms using IaC templates to uphold least privilege principles and secure resource access.
- Work collaboratively with cross-functional teams to design, implement, and enhance data pipelines and workflows.
- Utilize distributed engines like Spark for efficient processing and analysis of large data sets when necessary.
- Develop and maintain operational best practices for Spark and other data warehousing tools to ensure system stability and performance.
- Implement and oversee storage technologies for effective data storage and retrieval according to business requirements.
- Quickly troubleshoot and resolve platform-related issues to minimize downtime and disruptions.
- Stay informed on emerging technologies and industry trends to continuously improve the data platform infrastructure.
- Document processes, configurations, and changes to maintain thorough system documentation.

