Qualifications
Key Responsibilities:Design & Development: Create, test, and implement robust web scraping scripts and crawlers utilizing advanced Python tools (such as Playwright, Selenium, Requests, BeautifulSoup, etc.). Scalability: Design and maintain asynchronous scraping systems capable of extensive, large-scale data extraction. Resilience: Develop, monitor, and optimize advanced anti-blocking strategies and proxy rotation to guarantee high reliability and uptime. Integration: Oversee and automate data ingestion pipelines and ensure seamless integration with external REST APIs. Operational Excellence: Troubleshoot, monitor, and continually enhance scraper performance, reliability, and data quality. Collaboration: Work alongside other engineers to improve our core scraping infrastructure, tooling, logging, and monitoring systems. DevOps Support: Assist with DevOps tasks, including Docker, CI/CD, and managing Linux environments. Requirements:Core Experience: Demonstrated hands-on experience in high-volume web scraping and data extraction using Python. Technical Depth: Strong understanding of HTML parsing, browser automation techniques, and asynchronous programming. Frameworks: Proficient with top web scraping frameworks (e.g., Playwright, Scrapy, or Selenium). Web Knowledge: In-depth knowledge of REST APIs, HTTP protocols, and effective proxy management. Database Skills: Familiarity with both SQL and NoSQL databases.
About the job
Join our client, a Berlin-based scale-up that operates with a remote-first culture, delivering advanced market intelligence and innovative software solutions specifically tailored for the automotive sector. As they embark on an exciting growth journey, we are seeking a skilled Python Web Scraping Developer to enhance their dynamic and impactful international team.
If you're passionate about solving intricate data extraction problems, developing highly scalable web crawlers, and ensuring that large-scale scraping systems operate seamlessly in production, this opportunity is perfect for you. You will take charge of the complete lifecycle of our high-volume scraping pipelines, ensuring the data we gather is accurate, consistent, and delivered rapidly.
About onhires
Our client is a pioneering scale-up based in Berlin, focusing on delivering state-of-the-art market intelligence and software solutions tailored for the automotive industry. They embrace a remote-first working environment, promoting flexibility and innovation within their teams.
