About the job
We are seeking a passionate Full-Stack Software Engineer focused on data pipelines. In this dynamic position, you will play a key role in the development of ingest pipelines, scalable REST APIs, and facilitate comprehensive data exploration and insights. Our platform is designed for large-scale data ingestion, complex querying, and interactive data analysis. While your primary emphasis will be on the data-pipeline layer, collaboration with various teams is essential to ensure seamless functionality and optimal performance. If you are eager to enhance team processes and tools for quicker integration of new data sources, we want to hear from you!
Key Responsibilities
Design and implement robust data-processing workflows.
Oversee the entire data-processing lifecycle, including collection, discovery, analysis, cleaning, modeling, transformation, enrichment, and validation.
Create and uphold data models and JSON Schemas to ensure data integrity and consistency.
Work collaboratively with analysts and engineers to fulfill data requirements.
Optimize data storage and retrieval in Elasticsearch, Dgraph, MongoDB, and Redis.
Manage dataflow orchestration using Apache NiFi.
Mentor colleagues on best practices in data processing and software engineering.
Utilize AI platforms for hybrid automated/manual data transformation, code generation, and schema management.
Collaborate with analysts, product owners, and engineers to ensure operational solutions meet needs.
Propose and execute process improvements for more efficient delivery of new data sources.

