About the job
Data Engineer (Python)
Company Overview
Orcrist Technologies is at the forefront of innovation with the Orcrist Intelligence Platform (OIP), a cutting-edge data intelligence system built on Kubernetes. Our platform is available as a SaaS solution or can be deployed on-premises, including air-gapped setups. We manage both streaming and batch data pipelines that empower search functionalities, machine learning enrichment, and investigative workflows for our mission-critical clientele.
Role Summary
As a Data Engineer, you will play a pivotal role in quickly validating new data initiatives from inception to deployment, ensuring they are adoptable and scalable. In this innovative environment, you will prototype effective connectors and pipelines, generate performance assessments, and create handoff packages for productization by our Foundation or delivery team.
Key Responsibilities
- Prototype ingestion and connector patterns (batch and streaming) utilizing NiFi, Kafka, Kafka Connect/Streams, and Change Data Capture approaches.
- Design schemas and data models that are both prototype-grade and easily adoptable, ensuring semantic clarity and a disciplined approach to evolution.
- Develop incremental lakehouse datasets using Hudi, Iceberg, and Delta patterns, producing outputs for real-world latency and throughput evaluations.
- Implement data quality and provenance considerations early in the process, incorporating checks, metadata hooks, and operational basics.
- Containerize and deploy prototypes on Kubernetes, providing minimal runbooks and configurations for seamless adoption.
- Create adoption artifacts including schemas, reference implementations, technical design notes, and a backlog for integration.
Qualifications
- Minimum of 3 years of experience in data engineering with a proven track record of delivering real-world data pipelines beyond ad-hoc scripts.
- Proficient in Python and SQL, skilled in building transformations, validation tools, and pipeline integration code.
- Solid understanding of streaming and Change Data Capture fundamentals, along with experience in the Kafka ecosystem.
- Familiar with lakehouse architectures and query layers (e.g., Hudi, Iceberg, Delta, Trino, Hive, Postgres) and their role in making datasets accessible.
- Comfortable working in Kubernetes and container environments and adept at documenting technical decisions clearly.
- Must be eligible to work in Germany; EU/NATO citizenship is preferred, and export-control screening will apply.
Preferred Qualifications
- Experience with data quality tools such as Great Expectations or metadata/lineage platforms (OpenMetadata, DataHub, Atlas).
- Experience with on-premises or air-gapped deployments and awareness of governance and policy for regulated environments.
- Proficiency in German (B1+) and familiarity with OSINT, GEOINT, or multi-INT data structures.
What We Offer
- A modern data stack with real-world constraints: Kafka, NiFi, and more.

