About the job
The Exciting Opportunity
This position plays a vital role in architecting and enhancing our platform to meet business demands while optimizing our systems. In this role, you will have the opportunity to develop new data pipelines, manage platforms hosted on data streams for both batch and real-time loading, and create real-time visualizations.
Key Responsibilities:
- Maintain and enhance our existing data platform.
- Develop processes to ingest data from Kafka, APIs, and databases using AWS MSK Connect.
- Design and maintain real-time data processing applications utilizing frameworks such as Spark Structured Streaming and Kafka Streams.
- Implement transformations on data streams.
- Participate in data modeling adhering to standards like Inmon, Kimball, and Data Vault.
- Ensure data quality by verifying consistency and accuracy.
- Stay current with research and advancements in technology to improve our data platform.
- Possess an investigative mindset to troubleshoot issues creatively and manage incidents effectively.
- Take full ownership of assigned projects and tasks while collaborating within a team environment.
- Document processes thoroughly and conduct knowledge-sharing sessions.
What We're Looking For:
Essential Qualifications:
- Proven experience with modern cloud database technologies, especially Snowflake.
- Expertise in orchestrating data pipelines using Airflow.
- Proficient in AWS Glue.
- Familiarity with Apache Iceberg.
- Strong experience with SQL and Data Integration Tools.
- Proficiency in programming languages such as Python or Scala.
- Knowledge of AWS Services like S3, Lambda, API Gateways, DMS, and RDS.
- Development experience in Microsoft and Linux/Cloud environments.
- Exceptional analytical and problem-solving skills.

