Senior Data Engineer  

This role exists to:

Implement & maintain methods to improve data reliability and quality by combine raw information from different sources to create consistent and machine-readable formats for the usage of Data Analysts, Data Scientists & other internal or external teams within a cloud-based data ecosystem.

Core Responsibilities:

• Design, build and maintain a highly scalable pipeline (end-to-end) for large volumes of complex data processing and analysis within a cloud-based ecosystem.
• Build and develop large data warehouses, high-performance data processing pipelines, and ETL tools.
• Maintain standard data definition and dictionary to ensure data quality, consistency, and robustness in tracking and reporting processes.
• Ensure high data integrity and quality from various data sources that are aligned to industry best practices.
• Clearly articulate the pros and cons of different technologies, solutions, and platforms based on different use cases.
• Document use cases, solutions, and recommendations.

Essential:

• Solid experience and knowledge on Google Cloud Platform (or any of the major cloud solution providers such as AWS, Azure, AliCloud) for data management: Composer, Pubsub, Google Storage, Dataflow, Big Query and Data Fusion, or equivalent
• Proven knowledge & delivery working with structured & unstructured data with Airflow, or a diverse set of data technologies: Redshift, Elastic Search, PostgreSQL, Kubernetes
• High fluency in Python, SQL or equivalent
• Knowledge of version control tools such as Bitbucket, GitHub & GitLab.

Desired:

• Ability to work with stream-processing systems: Storm, Spark-streaming, etc
• Experience with big data tools: Hadoop, Spark, Kafka, etc