About the job
Join our dynamic Crawler team at Perplexity as a Senior Backend Software Engineer. In this pivotal role, you will be responsible for designing, developing, and managing systems that handle the ingestion, processing, and management of vast web-scale data to support our cutting-edge search technologies. This is a high-impact engineering position that requires a blend of backend and frontend expertise within our data acquisition stack.
Key Responsibilities
Lead the development of large-scale web crawlers, ingestion pipelines, and data processing systems.
Construct, maintain, and optimize vital backend and frontend components for crawler services, including storage solutions, data retrieval functions, and UI dashboards for effective data management.
Work closely with Search and Infrastructure teams to ensure efficient and high-quality ingestion and processing of billions of web pages.
Architect and implement full-stack features and scalable distributed systems capable of handling high loads and real-time data operations.
Experiment with innovative approaches to rapidly enhance system performance, usability, and reliability.
Qualifications
At least 5 years of software development experience with a solid understanding of data structures and algorithms in languages such as Python, C++, Rust, or Go.
Experience in developing large-scale web crawlers is highly preferred.
Demonstrated expertise in building, deploying, and optimizing high-load, distributed services.
Strong knowledge of cloud infrastructure, particularly hands-on experience with Kubernetes (K8s) and AWS.
A passion for writing clean, efficient, and scalable code.

