Join a forward-thinking Berlin-based scale-up that operates remotely, providing innovative market intelligence and software solutions tailored for the automotive sector. As the company embarks on a new growth journey, they are seeking a seasoned Scraping Infrastructure Engineer to bolster their international, high-impact team.If you are passionate about designing and maintaining robust, large-scale scraping systems adept at navigating complex anti-bot and blocking challenges, this position is a perfect match. You will oversee the complete lifecycle of high-volume scraping pipelines, concentrating on the infrastructure, tools, and strategic defenses that ensure precise, consistent, and rapid data collection at scale.Key ResponsibilitiesInfrastructure Design & Strategy: Architect, develop, and sustain the foundational infrastructure for extensive, asynchronous data extraction systems.Advanced Resilience Engineering: Formulate, implement, and iteratively enhance sophisticated anti-blocking strategies, IP rotation, fingerprint management, and anti-bot evasion techniques to guarantee exceptional reliability and uptime in the face of modern web defenses.Operational Excellence & Monitoring: Establish comprehensive monitoring, alerting, and logging frameworks to proactively address issues, optimize performance, and maintain data integrity across the platform.Core Development: Create, test, and deploy highly reliable, fault-tolerant web scraping components utilizing advanced Python tools, including Scrapy, Playwright, Selenium, and Requests.Integration & Pipelines: Oversee and automate high-volume data ingestion processes and seamless connections with both internal and external REST APIs.DevOps & Automation: Champion DevOps best practices, including infrastructure management with Docker, familiarity with Nomad (a plus), and CI/CD pipelines.Collaboration & Mentorship: Collaborate with fellow engineers to establish standards, refine core infrastructure tools, and mentor junior team members.QualificationsCore Experience: Extensive hands-on experience in high-volume web scraping and data extraction employing Python.Anti-Blocking Expertise: In-depth practical knowledge of anti-bot solutions, including...
Mar 20, 2026