Data Engineer Team Lead  

As a Lead Data Engineer, you will:

• Manage the growth and delivery of the Data Engineering team as well as internal communications on data flows and architecture.
• Design, build and maintain a highly performance, robust and scalable end-to-end data processing pipeline including, but not limited to data transformation, data integration, auditing, logging, normalization, modeling, and error handling.
• Design and build standard data definition, dictionary, and management platform to ensure quality, consistency and robustness in tracking and reporting process.
• Ensure high data integrity and quality from various data sources that are aligned to industry best practices.
• Build and design monitoring system on current data stack.
• Clearly articulate pros and cons of different technologies, solutions and platforms based on different use cases.
• Document use cases, solutions and recommendations.
• Collaborate with DE Product Manager to define upcoming initiatives and resource requirements.
• Collaborate with cross functional teams (Engineer, DevOps, Information Security, Principal Architect, etc) to resolve data quality and technical issues.

What we need:

Essential:

• Proficient in designing and implementing all aspects of Datalake concepts: Ingestion, Processing, Data Cleansing, and Workflow Management (Airflow).
• General proficiency in performing data transformations via ELT/ETL framework.
• Proficiency in CI/CD methodologies.
• Proficient in Python language and complex SQL.
• Experience in writing Apache Beam or Apache Spark including its optimization techniques.
• Highly proficient in different types of data ingestion operating concepts: batching or streaming and its tools, including (but not limited to) Apache Airflow, Pub/Sub, Kafka.
• Experience in Cloud Environment, preferably GCP.
• Generally proficient in building monitoring systems for end-to-end aspects of a big data cluster: Ingestion Pipeline, Data Quality, Data Security, and Workflow Management Server.
• Experience in handling multiple data sources and Cloud trackers: MongoDB, MySQL, Postgres, Firebase, and Google Analytics.
• Capable of navigating and working effectively in a DevOps model including leveraging related technologies: Kubernetes, BitBucket, Docker, etc.

Desired:

• Proficiency in one of these programming languages: Java, Scala, GO, or other programming languages.
• Ability to work with stream-processing systems: Storm, Spark-Streaming, etc
• Experience with Big Data tools: Hadoop, Spark, Kafka, etc