Senior Data Engineer

Job Description and Requirements

We are seeking a highly motivated Data Engineer to join our team. The successful candidate will have the responsibility of designing, implementing, and maintaining the data infrastructure that helps with our Observability and SRE platforms. The candidate should have a proven background in data engineering, distributed systems, and monitoring tools, as well as a passion for building scalable and reliable data pipelines and dashboards.

Key Responsibilities:

• Design and implement the data infrastructure that helps with our Observability and SRE platforms.
• Partner with other teams to ensure that data infrastructure is effectively integrated with other systems and processes.
• Ensure that the data infrastructure is scalable, reliable, and can handle large volumes of data.
• Develop custom scripts and tools to extend the functionality of the data pipeline.
• Perform root cause analysis on incidents related to data pipeline and develop solutions to prevent similar incidents from occurring in the future.
• Provide guidance and mentorship to junior members of the team.
• Act as a SME on data engineering, providing guidance to other teams across the organization.
• Continuously evaluate and implement new tools and technologies to improve the data infrastructure.

Qualifications:

• Excellent Python programming skills.
• Knowledge of Linux OS, Networking, NFS and HPC technologies.
• Experience with distributed data queuing and data processing systems such as Kafka, Snowflake or Spark.
• Deep expertise of ETL and data ingestion technologies such as Logstash.
• Experience with monitoring tools such as Elasticsearch, Prometheus and other alike
• Software programming lifecycle knowledge and experience in CI/CD, source code tools and testing.
• Knowledge of cloud computing platforms such as Azure, AWS, or GCP.
• Familiarity with containerization technologies such as Docker and Kubernetes.
• Excellent problem-solving skills and attention to detail.
• Ability to work collaboratively with other teams and stakeholders.
• Excellent communication skills, both verbal and written.
• Ability to drive and mentor junior members of the team.
• Exceptional knowledge of data engineering best practices and trends.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 8+ years of experience in data engineering, distributed systems, or a related field.