Senior DevOps / Big Data Engineer

Senior DevOps / Big Data Engineer

Marcus Donald are now looking for a Senior DevOps / Data Engineer to join a client based in Canary Wharf on a contractual basis.

In this role you will be responsible for the configuration and on-going management of cloud PaaS/IaaS, working with Big Data platforms such as Hortonworks based on Hadoop clusters configured on cloud data centres such as those of Azure or AWS. 

You will ensure availability of the platform to services teams and their service users. You will also carry out continuous improvements to platforms, including system development, processes design and automation and diagnostics improvements. Assurance of platform security, privacy and resilience will be a main focus.

Key Skills as a Senior DevOps / Big Data Engineer

• Agile software engineering practices (CI/CT/CD)
• Azure
• Microservices Architecture and APIs
• Hadoop ecosystem using Hortonworks or Cloudera
• Cloudbreak, Hive, Ambari, Sqoop, Oozie, SPARK, Atlas, Ranger, HBase, HDFS, YARN and ELK
• Experience connecting cloud infrastructure to on-prem networks
• VPNs, encryption & key/certificate management, endpoint protection, virtual firewalls/ACLs/NSGs, setting up bastion nodes, etc
• VPNs, tunnelling, AD connectors, routing
• Authentication technologies (LDAP, OAuth, 2FA, SAML, and SSO via Kerberos)
• Distributed system design and dependency management
• Troubleshooting
• Bash, Python, Java or Ruby
• Monitoring tools, approach and implementation
• Linux system administration

Key Responsibilities as a Senior DevOps / Big Data Engineer

• Ensure principles of privacy, security and resilience are assessed and designed into solutions
• Implementation and on-going administration of Hadoop infrastructure
• Propose and deploy new hardware and software environments required for Hadoop and to expand existing environments
• Building and supporting technical stack as well as evaluating technology for use in the stack
• Setup new Hadoop users, setting up Linux users, Kerberos principles and testing HDFS and Hive access for new users
• Working with the infrastructure, network, database, application and business intelligence teams to guarantee high data quality and availability
• Monitor and maintain Hadoop ecosystem connectivity and security to guarantee confidentiality, integrity and availability
• Integrating technology to provide end to end architecture

If you feel you would be suitable for this DevOps Engineer position, please apply via this advert or contact Oliver directly on 0203 328 0400.

Senior DevOps / Big Data Engineer