Job title: Data Engineer
Location: London, E14
Salary: £55,000 - £65,000 depending on experience
Contract type: Permanent
Hours: 37.5, Monday – Friday
WFH policy: Employees are required to attend the office 2 days/week
Flexible working: Informal flexible work patterns subject to line manager discretion including a 9-day fortnight.
As the Data Engineer, you will own data ecosystems and will work with developers, solution architects, technical BAs, data scientists, and other SMEs to define the optimum data architecture for the business. You will build the required ETL/ELT pipelines for the collection, preparation, and storage of data in a form that is optimised, secure and reliable.
You will have a specific focus on developing the data inputs that are utilised by the organisation’s internal forecasting model of the GB power markets; designing a framework that scales effectively along with the source data.
Design data infrastructure to support modelling and data analytics
Build and maintain ETL/ELT pipelines to make data accurate and easy to use
Work to ingest and transform data sets from a variety of data sources
Explore ways to enhance data quality and reliability
Assist with the establishment of a data culture across the organisation
Drive better data governance through the creation and embedding of principles and processes e.g. Logical Data Model & Flow Diagram, Data Dictionary, Data Semantic Layers
Set service level indicators and monitor execution of data workflows and configure alerts
Apply dimensional data modelling concepts and practices to develop conceptual, logical and physical data models to support insight delivery
Identify data quality issues through data profiling, analysis, and stakeholder engagement
LCCC reserves the right to assign other duties commensurate with the role as required.
Skills Knowledge and Expertise:
At least 2 years of designing data infrastructure to support modelling and data analytics
Experience building, modelling, and maintaining data pipelines
Strong experience in SQL optimisation, performance tuning, data modelling and SQL/database design skills.
Hands on experience within the Azure Data ecosystem, with Azure Databricks, Data Factory, Data Lake, and Synapse. Certifications are a plus.
Strong competence in Python or Scala, ideally PySpark experience