Qualifications
Main Responsibilities:
Build and maintain core batch and near-real-time pipelines.
Model and evolve Silver and Gold layers for analytical use cases.
Optimize performance and reliability in distributed workloads.
Implement and monitor Data Quality through expectations.
Perform advanced optimizations, including shuffle management, skew handling, partitioning, and caching.
Implement incremental loads, manage slowly changing dimensions (SCD), and perform normalization/denormalization for analytics.
Contribute to engineering standards, best practices, and code reviews.
Requirements:
Degree in Systems Engineering or similar fields with 3 to 5 years of experience as a Data Engineer, demonstrating strong proficiency in Spark.
Solid experience in:
PySpark or Scala
Delta Lake
Partitioning and optimization using OPTIMIZE, Z-ORDER
Incremental loads and SCD management
Data Quality with expectations
Advanced SQL
Desired certifications (not mandatory):
Databricks Data Engineer Associate/Professional
AWS (Data, Architect, or equivalent)
Apache Spark certifications
About the job
NEORIS, now a part of EPAM, is a leading Digital Accelerator that empowers companies to embrace the future. With over 20 years of experience as Digital Partners for some of the largest corporations worldwide, we are a vibrant team of more than 4,000 professionals across 11 countries. In our multicultural startup-like culture, we cultivate innovation and continuous learning to deliver high-value solutions for our clients.
We are currently seeking a talented individual for the position of Senior Data Engineer. The ideal candidate will possess extensive experience with Spark and Delta Lake to construct and optimize core data pipelines (both batch and near-real-time), model Silver/Gold layers, and ensure the reliability and performance of our data solutions at scale.
About NEORIS
NEORIS, part of EPAM, is a global leader in digital transformation, leveraging two decades of experience to guide major corporations into the future. We emphasize a culture of innovation and continuous improvement among our diverse team of professionals.