Hadoop Administrator Our client provides shared technology services, original research and community programs that help organizations meet the ever-evolving needs of their users, institutions, and communities. With office locations around the globe, their employees are dedicated to offering premier services and software.
They are an employee centric company that believes strongly in work-life balance. They are looking for people who are passionate around solving problems through innovation and engineering best practices! You will be part of the Data Platform Engineering team responsible for administration, infrastructure planning, and scaling of their Big Data Hadoop and relational database technologies. This position will be responsible for the administration and support of multiple open-source Hadoop NoSQL environments with a secondary role supporting Snowflake and Kafka on-premise and within AWS.
You will work closely with infrastructure and development teams to ensure business applications are highly available and performing within agreed on service levels. This position will work closely with the development teams to tune and troubleshoot applications to maintain high performing environments. Additionally, you will provide guidance for best practices and standards for the data platforms to your development partners.
Responsibilities: Install, manage, maintain, and tune Hadoop/HBase clusters Develop and enforce operational best practices to maintain availability and responsiveness of HDFS/Hbase cluster resources Work in embedded development teams to understand usage patterns and tune application and Hadoop cluster interaction to provide reliable performance and limit impact to other data users Participate in on-call support rotation Review existing software/hardware architecture and identify opportunities for improvement in the areas of scalability, maintainability, and performance Monitor Hadoop cluster connectivity and security Understand and review Hadoop log files HDFS support and maintenance. Qualifications: Hands on experience with the Hadoop stack (HBase, Yarn, Spark/MapReduce, Kafka) Tuning and hardening of data platforms, using open source/native tools and/or internal solutions Ability to deploy a Hadoop cluster, add and remove nodes, manage jobs, monitor critical components of the cluster, configure name-node high availability, configure backups. Strong understanding and support of the Linux operating system (Red Hat preferred) Strong knowledge of scripting and automation tools (eg Ansible, Bash, Python, GitLab) Familiarity reading and compiling open source code Ability to read, understand, and write java code Expertise in typical system administration duties such as capacity management, performance tuning, system dump analysis, and server hardening (security) 5-7+ years relevant IT experience Additional Skills/Experience desired: Experience with other database technologies (PostgreSQL, Cassandra, MongoDB, MySQL) Experience supporting a production multi-site enterprise environment Experience working within a Scrum/Agile framework How this role differs from a Standard Hadoop Data Platform Engineer or Administrator: They are not using Cloudera/Hortonworks in their environment.
Instead, they compile and deploy their own Hadoop version, which includes Hadoop, HDFS, HBase, Spark, Thrift, and Kafka. This role requires a deep understanding of Hadoop technologies beyond the ability to use the Cloudera/Hortonworks provided GUI, in order to understand the underlying components and configurations. It also requires a strong skillset in Linux, shell scripting and java to be able to support and troubleshoot issues within the data platform.
This is not a position that focuses on moving or transforming data from one data platform to another
