Qualifications
Key Responsibilities
Design, develop, and maintain scalable, reliable, and high-throughput data ingestion pipelines for structured and semi-structured data.
Implement secure and efficient data lake and SQL-based storage architectures tailored for performance and cost-effectiveness.
Create and maintain internal tools and frameworks for data ingestion using Python, Golang, and SQL.
Collaborate with Cloud, Edge, Product, and AI teams to establish data contracts, schemas, and retention policies.
Utilize AWS cloud infrastructure (including Argo Workflows, S3, Lambda, Glue, Kinesis, Athena, and RDS) to facilitate comprehensive data workflows.
Apply Infrastructure-as-Code (IaC) methodologies using Terraform to manage data platform infrastructure.
Monitor data pipelines for quality, latency, and failures using tools such as CloudWatch, SumoLogic, or DataDog.
Continuously optimize storage, partitioning, and query performance across large-scale datasets.
Engage in architecture reviews to ensure adherence to security, compliance, and best practices.
Skills and Qualifications
5+ years of professional experience in software engineering or data engineering.
Proficient programming skills in Python and Golang.
In-depth knowledge of SQL and contemporary data lake architectures (e.g., using Parquet, Iceberg, or Delta Lake).
Practical experience with AWS services, including but not limited to: S3, Lambda, Glue, Kinesis, Athena, and RDS.
Expertise in using Terraform for automating infrastructure deployment and management.
Experience in real-time or batch data ingestion at scale and in designing fault-tolerant ETL/ELT pipelines.
Familiarity with event-driven architectures and messaging systems such as Kafka or Kinesis.
Strong debugging and optimization skills.
About the job
Join brightai, a rapidly growing company revolutionizing business operations through the integration of AI, IoT, and cloud-native services into scalable, real-time platforms. As a Data Platform Engineer, you will be instrumental in constructing and sustaining the data infrastructure that drives our innovative products, services, and insights.
Become a part of a diverse team dedicated to ingesting, processing, and managing extensive streams of sensor and operational data from a wide variety of devices, including drones, robots, industrial systems, and smart environments.
About brightai
brightai is at the forefront of transforming business operations by leveraging cutting-edge technologies such as AI, IoT, and cloud-native solutions. Our mission is to create scalable and real-time platforms that empower businesses to operate more efficiently and effectively.