Data Engineer (PySpark) - Expert in Cloudera Data Platform

Job Title: Data Engineer (PySpark)________________________________________About the RoleWe are in search of a talented Data Engineer with significant expertise in PySpark and the Cloudera Data Platform (CDP) to bolster our data engineering team. In this role, you will be tasked with the design, development, and maintenance of scalable data pipelines that gua…

Join gsstech-group as a talented Data Engineer specializing in PySpark and the Cloudera Data Platform (CDP). In this pivotal role, you will be responsible for designing, developing, and maintaining high-quality data pipelines that are both scalable and efficient. Your contributions will ensure optimal data performance and availability across our organization.This position requires extensive hands-on experience in big data technologies, cloud-native environments, and advanced data processing frameworks. You will collaborate with diverse teams to create reliable data solutions that facilitate actionable business insights.Key Responsibilities1. Data Pipeline DevelopmentCraft and sustain scalable ETL/ELT data pipelines utilizing PySpark on CDP.Guarantee data integrity, reliability, and performance optimization.2. Data IngestionBuild ingestion frameworks to gather data from various sources including relational databases, APIs, streaming platforms, and file systems.Load both structured and unstructured data into Data Lake and Data Warehouse environments.3. Data Transformation & ProcessingProcess, cleanse, and transform extensive datasets using PySpark.Create reusable data processing components.4. Performance OptimizationOptimize Spark jobs and Cloudera components for peak performance.Enhance memory, partitioning, and execution strategies.Minimize ETL runtime and elevate cluster efficiency.5. Data Quality & ValidationEstablish data validation checks and monitoring systems.Maintain comprehensive data quality and governance standards.6. Automation & OrchestrationAutomate workflows using Apache Oozie, Apache Airflow, or similar orchestration tools.Support CI/CD integration for data pipelines.7. Monitoring & SupportOversee pipeline health and troubleshoot any issues that arise.Provide ongoing production support and drive continuous improvement initiatives.

Join gsstech-group as a Data Engineer specializing in PySpark and Data Modeling. In this critical role, you will leverage your expertise to design, construct, and maintain robust data pipelines that facilitate data analysis and reporting.As a Data Engineer, you will collaborate closely with data scientists and analysts to ensure that our data architecture supports the needs of the business. Your work will directly contribute to the optimization of our data management processes and the strategic use of data across the organization.

Job Title: Lead Data Scientist / Expert in Data Insights.Roles and Responsibilities:Demonstrated proficiency in Business Intelligence tools, particularly Tableau and MicroStrategy, ensuring a continuous flow of actionable data.Strong understanding of the transformative impact of data analytics on business operations, highlighting its advantages.Innovate by developing decision analytics solutions utilizing advanced methodologies and techniques.Design user-friendly processes and reporting models that empower business units to access and manage their data in real-time.Oversee engagement methodologies to ensure compliance, mitigate risks, and maintain the highest quality standards, collaborating with teams to implement necessary adjustments.

Role OverviewJoin us at gsstech-group as a Senior Technology Engineer, where you will play a pivotal role in enhancing the stability, automation, and operational excellence of our Data Science Platform (DSP).This is a hands-on engineering position, not a support role. You will be responsible for driving automation, orchestration, and reliability across a robust hybrid cloud ecosystem (OpenShift + AWS/Azure/GCP).Your expertise will be essential in ensuring the seamless operation of DSP. You will tackle issues head-on—if something fails, scales inadequately, or necessitates manual intervention, it will be your responsibility to implement a permanent solution.

Join our innovative technology team as the Vice President of Data Platforms. We are looking for an accomplished leader to guide our AI and machine learning strategies from our Dubai office. In this pivotal role, you will oversee the development and implementation of cutting-edge data platforms that drive our organization forward in the digital landscape.Your Responsibilities:Formulate and execute a comprehensive AI/ML strategy aligned with our business goals.Manage the design, development, and upkeep of scalable data platforms and AI/ML infrastructure.Lead and nurture a talented team of data scientists, engineers, and analysts, encouraging a culture of creativity and excellence.Work collaboratively with cross-functional teams to identify and deploy AI/ML solutions that enhance business outcomes.Establish and uphold best practices for data governance, security, and compliance across all AI/ML projects.Keep informed of the latest trends in AI/ML and data technologies, providing strategic recommendations for adoption.Build and maintain relationships with key technology partners and vendors in the AI/ML industry.Communicate AI/ML strategies, progress, and outcomes to executive leadership and stakeholders.Facilitate the integration of AI/ML capabilities into current products and services.Optimize resource allocation and manage budgets for AI/ML initiatives.

We are looking for a talented and experienced Senior Data Engineer to join our dynamic Data Engineering team in Dubai. The successful candidate will play a pivotal role in designing, constructing, and enhancing scalable data pipelines, while also facilitating advanced analytics and machine learning applications. This position demands a deep understanding of big data technologies, data processing frameworks, and feature engineering to effectively support machine learning models in a live production setting.Key ResponsibilitiesCollect and assess business and technical requirements for effective data solutions.Conduct Exploratory Data Analysis (EDA) to reveal data patterns and assess data quality.Design, develop, and maintain high-performance, scalable data pipelines.Ingest both structured and unstructured data from various sources.Transform and process extensive datasets using PySpark and Python.Apply feature engineering techniques to enhance machine learning models.Optimize Spark jobs for peak performance and cost efficiency.Ensure the quality, integrity, and security of data across all pipelines.Collaborate closely with Data Scientists, Analytics Teams, and Delivery Leads.Participate in an Agile environment alongside cross-functional teams.Effectively communicate with stakeholders, providing valuable technical insights.Utilize Git for version control and manage CI/CD workflows.Required Skills & ExpertiseProven experience in Python and PySpark.In-depth knowledge of Apache Spark (performance tuning and optimization).Comprehensive understanding of the Hadoop ecosystem.Advanced skills in SQL.Experience in building end-to-end data and ML pipelines.Strong grasp of data modeling and warehousing concepts.Experience with feature engineering for ML applications.Familiarity with Git and version control systems.Exposure to cloud platforms (AWS/Azure/GCP) is advantageous.QualificationsBachelor’s or Master’s degree in Computer Science, Engineering, or a related field.8–10+ years of experience in Data Engineering / Big Data.Experience in the banking or financial services sector is preferred.Soft SkillsExceptional problem-solving and analytical skills.Strong communication and stakeholder management abilities.Capability to work both independently and collaboratively within a team.Adaptability to fast-paced Agile workflows.

Join our innovative team at mexdigital as an AI Data Engineer. In this role, you will be instrumental in shaping our data-driven solutions, leveraging cutting-edge AI technologies to enhance business processes and deliver impactful insights. Collaborate with cross-functional teams to design, implement, and maintain scalable data pipelines and machine learning models that drive our projects forward.

Welcome to SFORS! As a prominent player in the global financial markets, we pride ourselves on being at the forefront of proprietary trading and pre-market strategies. With over 20 years of experience, our success is driven by a relentless commitment to talent development, cutting-edge trading technologies, sophisticated risk management models, and strategic trading methodologies, all aimed at ensuring our traders achieve remarkable success.We are seeking a talented Data Engineer to join our dynamic team in Dubai. In this role, you will be instrumental in designing, developing, and maintaining robust data pipelines and infrastructure that support our analytics and machine learning initiatives. Your expertise will guarantee the seamless flow of data across various systems, uphold data modeling best practices, ensure stringent data security measures, and empower our teams with high-quality, structured data.

Join our dynamic team at gsstech-group as a Senior Data Engineer. In this pivotal role, you will design, build, and enhance scalable data pipelines that empower advanced analytics and machine learning initiatives.Collaborating closely with Data Scientists, Analytics Delivery Leads, and interdisciplinary teams, you will transform raw data into impactful insights within a fast-paced Agile environment.Key ResponsibilitiesPartner with stakeholders to collect and evaluate data requirements.Conduct Exploratory Data Analysis (EDA) to discern data patterns and quality.Architect and implement robust, scalable, and high-performance data pipelines.Ingest, process, and convert extensive structured and unstructured datasets.Apply feature engineering techniques that bolster machine learning models.Enhance Spark jobs for optimal performance, scalability, and cost-effectiveness.Maintain data quality, integrity, and security across all pipelines.Work collaboratively with Data Scientists and Analytics teams to deploy ML pipelines.Engage in Agile ceremonies and contribute to ongoing improvements.Effectively communicate technical solutions to both technical and non-technical stakeholders.Required Skills & QualificationsBachelor’s or Master’s degree in Computer Science, Engineering, or a related discipline.8–10 years of experience in Data Engineering and Big Data ecosystems.Proficient programming skills in Python.Practical experience with PySpark / Apache Spark, including performance optimization.Strong knowledge of the Hadoop ecosystem.Advanced skills in SQL.Experience with data pipeline development and ETL frameworks.Familiarity with Machine Learning pipelines and feature engineering.Knowledge of version control systems (Git).Excellent problem-solving and analytical abilities.Good to HaveExperience with cloud platforms (AWS / Azure / GCP).Understanding of data warehousing solutions.Familiarity with workflow orchestration tools (e.g., Airflow).Experience in the banking or financial services sector.Key CompetenciesStrong collaboration and communication abilities.Capability to thrive in a fast-paced Agile setting.Detail-oriented with an ownership mindset.Proficient in stakeholder management and coordination.

DeepLight AI is a premier consultancy specializing in artificial intelligence and data solutions, renowned for our extensive expertise in deploying intelligent enterprise systems across diverse industries, particularly in financial services and banking. Our team merges profound knowledge in data science, statistical modeling, AI/ML technologies, workflow automation, and systems integration with a pragmatic grasp of intricate business operations.The Data Engineer will play a pivotal role in designing, implementing, and optimizing data pipelines and infrastructure to bolster our innovative AI systems. This role involves close collaboration with our cross-functional team to ensure the effective collection, storage, processing, and analysis of large-scale data, unlocking valuable insights and fostering innovation across various sectors.

Envision Employment Solutions is actively seeking talented Senior Data Management Engineers to join our esteemed partner, a prominent global leader in IT Services and Consulting.Job Overview:We are in search of a Senior Data Management Engineer who possesses extensive knowledge in data architecture, integration, and transformation, with a particular focus on leveraging Informatica tools. The successful candidate will be responsible for designing and implementing comprehensive enterprise data solutions that guarantee data accessibility, quality, governance, and scalability across various business domains.Your role will involve collaboration with cross-functional stakeholders to effectively translate business requirements into secure, reliable, and high-performance data systems.Key Responsibilities:Design, develop, and maintain robust ETL processes and data pipelines utilizing Informatica (PowerCenter, IDQ, or Cloud Data Integration).Optimize data integration workflows for both structured and unstructured data sources.Ensure the highest standards of data quality, integrity, governance, and security across enterprise platforms.Engage in data modeling and architectural discussions.Work closely with data architects, business analysts, and application teams to align data solutions with business objectives.Document data flows, transformations, and system architecture comprehensively.Identify and resolve performance bottlenecks and complex data-related challenges.Support initiatives related to enterprise data governance and metadata management.Qualifications:Bachelor’s degree in Computer Science, Information Systems, or a related field.3 to 5 years of professional experience in data engineering or management roles.Proficient hands-on experience with Informatica (PowerCenter, IDQ, or Cloud Data Integration).Strong SQL skills and familiarity with relational databases (e.g., Oracle, SQL Server).Solid understanding of data warehousing concepts and enterprise data architecture.Experience in data governance and metadata management.Knowledge of cloud data platforms (AWS, Azure, or GCP) is advantageous.Excellent analytical and troubleshooting abilities.Fluent communication skills in English.Benefits and Work Environment:Competitive basic salary.Social insurance coverage.Comprehensive family medical insurance (AXA).Location: Dubai.Work Model: Hybrid – 2 days in the office and 3 days remote. Depending on project requirements, you may be required to work from the client’s premises more frequently.Working Hours: 9 AM to 6 PM.Days Off: Saturday and Sunday.

Oversee and lead a team of proficient data analysts.Develop data mappings essential for creating data marts and lakes.Design and implement user-friendly interfaces and dashboards for effective data interaction and visualization.Conduct thorough testing and validation of software and data products to guarantee accuracy and dependability.Engage in collaboration with fellow developers, data analysts, and stakeholders to ensure software alignment with business objectives.Provide analytical support to Business Performance and Strategic Analytics for departments and decision-makers.Core Skills:Expertise in Data Product Development and Data Mart design.Knowledge of Machine Learning, Deep Learning, Time-Series Analysis, and Optimization techniques.Understanding of Retail, Business, and Wholesale Banking products.Strong communication and interpersonal skills.Proficient in people management with a focus on multicultural awareness.Technical Skills:Proficient in Python (along with R, SAS, and Spark) and SQL, DAX.Strong coding abilities in languages such as Python, R, and Java.Familiarity with cloud data technologies.Competencies:Advanced proficiency in SQL and Python; experience with SAS/R/Spark is an advantage.Experience in deployment across various databases and server/cloud environments including AWS and Azure.Thorough understanding of banking functionalities is a significant advantage.Excellent communication and documentation skills to effectively convey information to stakeholders.Familiarity with tools such as Power BI, Tableau, or Qlik is beneficial.

Join our dynamic team at dev2 as a Senior Java Engineer specializing in Big Data. In this role, you will leverage your expertise in Java and big data technologies to design, develop, and maintain scalable data solutions. You will collaborate with cross-functional teams to deliver high-quality products that meet our clients' needs.

Join our dynamic team at ghobashgroup as a Junior Data & AI Engineer, where you will be instrumental in leveraging data-driven solutions to enhance our AI capabilities. This role is tailored for Emirati nationals who are eager to kickstart their careers in the exciting fields of data science and artificial intelligence.

Modeling Expertise:Proven experience in designing and implementing propensity models focused on cross-selling and upselling opportunities.In-depth knowledge of Next Best Action / Next Best Offer systems.Solid understanding of supervised learning methodologies, uplift modeling, and causal inference techniques.Programming & Engineering:Proficient in Python (required); knowledge of R or Scala is advantageous.Experience with distributed computing frameworks such as Spark, Dask, or Ray.Skilled in developing production-level ML pipelines using tools like Airflow, MLflow, or Kubeflow.Strong familiarity with software engineering best practices including version control, CI/CD, and thorough testing.Data Infrastructure:Experience with cloud platforms (AWS/GCP/Azure) and data warehouses (Snowflake, BigQuery, Redshift).Excellent SQL skills; adept at optimizing queries and handling large datasets.MLOps & Deployment:Proven ability to deploy models to production using APIs or streaming technologies (Kafka, Flink).Experienced in model versioning, experiment management, and deployment practices with MLflow, SageMaker, or Vertex AI.Monitoring & Observability:Capable of establishing model monitoring, drift detection, and alerting systems utilizing Prometheus, Grafana, Evidently, or custom dashboards.Familiarity with logging frameworks and performance profiling for ML services.GenAI (Preferred):Experience with Large Language Models, embeddings, prompt engineering, and vector databases (e.g., FAISS, Pinecone).Ability to incorporate GenAI into decision-making systems or customer-facing applications.Leadership & Collaboration:Proven leadership skills to guide senior data scientists while maintaining a hands-on approach.Adept at collaborating with product managers, engineers, and business stakeholders.

ABOUT US | Whiteshield is a premier global advisory firm strategically positioned at the crossroads of strategy, public policy, and digital transformation. We collaborate with governments and leading organizations to craft and execute initiatives aimed at enhancing economic competitiveness, fostering digital enablement, and ensuring sustainable growth.Our approach uniquely blends strategic advisory services with pragmatic execution. We empower senior decision-makers to navigate intricate regulatory, economic, and technological landscapes, converting aspirations into well-structured programs, delivery frameworks, and quantifiable outcomes.Unlike conventional consultancies that focus merely on recommendations, we remain actively engaged throughout the implementation process, ensuring that transformation plans are not only conceptualized but also effectively realized.WHERE YOU FIT IN | We are developing robust data platforms and architectures that serve as the backbone of national and enterprise transformation initiatives, facilitating decision-making, AI systems, and large-scale digital services.At Whiteshield, data is not an afterthought; it is a core component. From constructing interoperable data ecosystems across various institutions to enabling AI-ready infrastructures, our work operates at the intersection of engineering, analytics, and strategic foresight.We seek a Data Architect capable of bridging both vision and execution. You will delineate how data is structured, governed, and utilized, ensuring it is reliable, scalable, and poised to produce tangible impacts.You thrive in complex, multi-stakeholder environments, adeptly translating ambiguous requirements into clear data models, architectures, and standards. You possess a deep understanding of how contemporary data platforms bolster AI, analytics, and operational systems, and how to design them effectively.Your role will involve close collaboration with engineers, analysts, policymakers, and client stakeholders to build data ecosystems that are not only technically robust but also user-friendly, trustworthy, and aligned with strategic objectives.WHAT YOU WILL DO | Design and implement scalable, secure, and high-performing data architectures across both cloud and hybrid environments.Establish enterprise data models, standards, and governance frameworks to ensure consistency and interoperability.Architect modern data platforms (data lakes, warehouses, lakehouses) to support analytics, AI, and operational use cases.Lead client engagements as a trusted advisor on data strategy, architecture, and platform design.Translate business and policy objectives into structured data solutions and pipelines.Establish data governance practices, including data quality, lineage, cataloguing, and more.

Join our dynamic team at gsstech-group as a meticulous and analytical Data Analyst specializing in the insurance sector. We are looking for a candidate with robust experience in data migration projects and Master Data Management (MDM). In this pivotal role, you will contribute to transformative projects related to policy, claims, underwriting, and customer data, ensuring exceptional data quality, integrity, and adherence to governance standards.The successful candidate will possess strong analytical capabilities, comprehensive knowledge of the insurance business domain, and practical experience in data mapping, reconciliation, validation, and MDM processes.Key Responsibilities1. Insurance Domain Data Analysis· Analyze insurance data across Policy Administration, Claims, Underwriting, Billing, and Reinsurance systems.· Collaborate with business stakeholders to identify data requirements and regulatory obligations.· Interpret complex insurance data models and business rules.2. Data Migration· Execute source-to-target data mapping and document transformation logic.· Carry out data profiling, cleansing, validation, and reconciliation activities.· Identify data gaps, inconsistencies, and quality issues during the migration process.· Assist in migrating from legacy systems to contemporary core insurance platforms.· Validate migrated data through User Acceptance Testing (UAT) and post-migration audits.3. Master Data Management (MDM)· Support the implementation and ongoing maintenance of MDM frameworks.· Manage various data sets and data sources effectively.· Define and monitor data quality Key Performance Indicators (KPIs).· Ensure standardization, de-duplication, and creation of golden records.· Collaborate with governance teams to enforce data standards and policies.4. Reporting & Insights· Develop comprehensive dashboards and reports utilizing tools such as Microsoft Power BI and Tableau.· Conduct ad-hoc analyses to support underwriting and claims performance.· Present actionable data insights to business and leadership teams.

Join our dynamic team at mexdigital as a Data Analyst. We are looking for a detail-oriented individual who is passionate about transforming data into actionable insights. You will be responsible for analyzing complex data sets, generating reports, and presenting findings to stakeholders. This role is ideal for someone who thrives in a fast-paced environment and is eager to contribute to data-driven decision-making.

Role ObjectiveThe Data Manager will be responsible for architecting and managing the organization's data migration and management systems. The primary objective is to guarantee data accuracy, accessibility, and security while establishing a unified data repository that empowers leadership with actionable insights for data-driven decision-making.Key ResponsibilitiesStrategy & Architecture: Formulate comprehensive data strategies across the enterprise, including the development and implementation of data lakes, data warehouses, and master data management (MDM) frameworks.Data Governance & Compliance: Create and enforce policies for data management and sharing, ensuring adherence to both local and international regulations (e.g., UAE Data Protection Law, GDPR, PDPL).Data Quality Assurance: Develop methodologies for quality data collection to maintain the adequacy, accuracy, and legitimacy of all datasets.Security & Protection: Establish and execute secure procedures for data management and analysis, focusing on technical aspects such as encryption, access control, and breach prevention.Reporting & Visualization: Collaborate with stakeholders to convert complex data into valuable insights via dashboards (Power BI/Qlik).Team Leadership: Direct and mentor a team of data analysts and database administrators, promoting a culture of data-driven decision-making within the department.Develop and implement data management policies and procedures.Design, maintain, and optimize databases and data systems.Oversee data collection, storage, backup, and recovery processes.Conduct data analysis and produce reports for management.Collaborate with IT and business teams to fulfill data requirements.Monitor the performance of data systems and address issues as they arise.Ensure data integrity, quality, and compliance with security standards.Manage data governance frameworks and associated documentation.