Principal / Senior Data Engineer (Data Platforms)

Job description

Overview2 weeks ago Be among the first 25 applicantsGet AI-powered advice on this job and more exclusive features.Simple Machines.
Data Engineered to LifeTMSimple Machines is a leading independent boutique technology firm with a global presence, including teams in London, Sydney, San Francisco, and New Zealand.
We specialise in creating technology solutions at the intersection of data, AI, machine learning, data engineering, and software engineering.
Our mission is to help enterprises, technology companies, and governments better connect with and understand their organisations, their people, their customers, and citizens.
We are a team of creative engineers and technologists dedicated to unleashing the potential of data in new and impactful ways.
We design and build bespoke data platforms and unique software products, create and deploy intelligent systems, and bring engineering expertise to life by transforming data into actionable insights and tangible outcomes.Principal / Senior Data Engineer (Data Platforms) – Wellington, New ZealandSimple Machines.
Data Engineered to LifeTMRequirementsThe Principal / Senior Data Platforms Engineer at Simple Machines is a dynamic, hands-on role focused on building real-time data pipelines and implementing data mesh architectures to enhance client data interactions.
This position blends deep technical expertise in modern data engineering methods with a client-facing consulting approach, enabling clients to effectively manage and utilise their data.
Within a team of top-tier engineers, the role involves developing greenfield data solutions that deliver tangible business outcomes across various environments.Technical ResponsibilitiesDeveloping Data Solutions: Implement and enhance data-driven solutions integrating with clients' systems using state-of-the-art tools such as Databricks, Snowflake, Google Cloud, and AWS.
Embrace modern data architecture philosophies including data products, data contracts, and data mesh to ensure a decentralized and consumer-oriented approach to data managementData Pipeline Development: Develop and optimise high-performance, batch and real-time data pipelines employing advanced streaming technologies like Kafka, and Flink.
Utilise workflow orchestration tools such as Dataflow and AirflowBig Data Processing & Analytics: Utilise big data frameworks such as Apache Spark and Apache Flink to address challenges associated with large-scale data processing and analysis.
These technologies are crucial for managing vast datasets and performing complex data transformations and aggregationsCloud Data Management: Implement and oversee cloud-specific data services including AWS Redshift, S3, Google BigQuery, and Google Cloud Storage.
Leverage cloud architectures to improve data sharing and interoperability across different business unitsSecurity and Compliance: Ensure all data practices comply with security policies and regulations, embedding security by design in the data infrastructure.
Incorporate tools and methodologies recommended for data security and compliance, ensuring robust protection and governance of data assetsConsulting ResponsibilitiesClient Advisory: Provide expert advice to clients on optimal data practices that align with their business requirements and project goalsProfessional Development, Training & Empowerment: Keep up with the latest industry trends and technological advancements, continually upgrading skills and achieving certifications in the technologies Simple Machines implements across its client base.
Educate client teams and enable their efficient utilisation and maintenance of these solutionsIdeal Skills and ExperienceCore Data Engineering Tools & Technologies: Demonstrates proficiency in SQL and Spark, and familiarity with platforms such as Databricks and Snowflake.
Well-versed in various storage technologies including AWS S3, Google Cloud BigQuery, Cassandra, MongoDB, Neo4J, and HDFS.
Adept in pipeline orchestration tools like AWS Glue, Apache Airflow, and DBT, as well as streaming technologies like Kafka, AWS Kinesis, Google Cloud Pub/Sub, and Azure Event HubsData Storage Expertise: Knowledgeable in data warehousing technologies like BigQuery, Snowflake, and Databricks, proficient in managing various data storage formats including Parquet, Delta, ORC, Avro, and JSON to optimise data storage and retrievalData Modelling Expertise: Proficient in data modelling, understanding the implications and trade-offs of various methodologies and approachesInfrastructure Configuration for Data Systems: Competent in setting up data system infrastructures, favouring infrastructure-as-code practices using tools such as Terraform and PulumiProgramming Languages: Proficient in Python and SQL, with additional experience in programming languages like Java, Scala, GoLang, and Rust considered advantageousCI/CD Implementation: Knowledgeable about continuous integration and continuous deployment practices using tools like GitHub Actions and ArgoCD, enhancing software development and quality assuranceAgile Delivery and Project Management: Skilled in agile, scrum, and kanban project delivery methods, ensuring efficient and effective solution developmentConsulting and Advisory Skills: Experienced in a consultancy or professional services setting, offering expert advice and crafting customised solutions that address client needs.
Effective in engaging stakeholders and translating business requirements into practical data engineering strategiesProfessional Experience and QualificationsProfessional Experience: At least 8+ years of data engineering or equivalent experience in a commercial, enterprise, or start-up environment.
Consulting experience within a technology consultancy or professional services firm is highly beneficialEducational Background: Degree or equivalent experience in computer science or a related fieldRight to Work: Must have full New Zealand working rights and reside in Wellington
#J-*****-Ljbffr

Required Skill Profession

Other General