Senior Data Engineering
Compas.co.id
- Jakarta Selatan, IndonesiaJakarta Selatan, DKI Jakarta, Indonesia, 28, Jalan Haji Abdul Majid Raya, Cipete Utara Kel., Kebayoran BaruJakarta SelatanDKI JakartaIndonesiaIndonesia
- IDR 15,000,000.00 - IDR 20,000,000.00 / monthIDR150000002000000015000000MONTH
- Full timeFULL_TIME
Expired 2 years ago
2022-09-29T17:00:00+00:00Job closed.
Job Description
- Overseeing the collection, storage, management, quality, and protection of data.
- Gather and process raw data at scale (including writing scripts, web scraping, calling APIs, writing SQL queries, etc.)
- Engineering costs optimization
- Lead, motivate, and manage data engineers teams.
- Effectively communicate the status, value, and importance of data collection to executive members and staff.
- Target: - Client data monitoring and notifications
- Crawler stability > 90%
- Mean time to repair crawler incident < 3 days
- Scalable proxy
Minimum Qualifications
- Min 4 years programming experience in python, NodeJs and PHP
- Min 2 years experience for web crawling and data cleaning
- Min 2 years experience in queuing and message broker (Kafka, RabbitMQ, Google Cloud Pub/Sub, NATS). min. 1 platform.
- Min 2 years experience with the UNIX environment.
- Min 1 year experience in GCP Product like Bigquery, Dataflow, PubSub, GCS
- Min 1 year experience in protecting web scrapers against site ban, IP leak, browser crash, CAPTCHA and proxy failure.
- Min 1+ years experience with container technology (Kubernetes, Docker)
- Bachelor's degree in information technology or related fields
Required Skills
- PHP
- Node.js
- Python
Jobs Summary
- Job Level
- Mid-Senior Level / Manager
- Job Category
- IT and Software
- Educational Requirement
- Bachelor's degree graduate
- Recruiter response to application
- Often
- Office Address
- 28, Jalan Haji Abdul Majid Raya, Cipete Utara Kel., Kebayoran Baru
Feel secure when applying: look for the verified icon and always do your research on a company. Avoid and report situations when employers require payment or work without compensation as part of their application process.
About Compas.co.id
Compas is developed by the team that brought Telunjuk.com, a technology company based in Jakarta, Indonesia. We focus on e-commerce business intelligence tools that provide actionable insights to empower you to make strategic decisions for your greater online business.
Compas is developed because we care about your online business development that needs more improvement through data validation. You will be given the market insights from the data that has been collected in our system. We receive multiple insights which were provided by our sophisticated crawler engine, ensuring that we provide you with strong and reliable data, because data will lead your online business into high credibility. We also help you in comparing the performance of your online business and other competitors. Here we are to support you to be on top of the competition.