Modern Data Platform

Data is ingested into a Raw data layer 'As-Is'. Where possible we will use low-code solutions such as FiveTran or Stich to reduce the technical complexity and Total Cost of Ownership. However, where these solutions are not a good fit, due to cost or complexity of the connection we will use either traditional ETL tools, such as Talend or custom cloud functions.

Through stakeholder workshops and data profiling we will establish target data models for the prepared layer. Depending on the requirements the target data models will typically use star schema design, optimised for leading business intelligence tools.

An agile data modelling approach known as Business Event and Analysis Modelling (BEAM) will be utilise to capture the data model, initially in the form of a Business Event Matrix:

dbt enables us to encapsulate the transformation logic in a series of data models, providing the following benefits:

• Easy to use for non engineers (shared data knowledge between engineering and non engineering teams)
• Extremely flexible data model (recreate data easily, backfills are easy)
• Transformations are at a data warehouse level, allowing for massively parallel processing
• Built in testing for data quality
• Online, searchable data catalog and lineage
• Reusable macros
• Low learning curve enables analysts to add their own models
• Production run using dbt cloud or through Airflow trigger.

Business intelligence tools can struggle when analysing very large data sets. We can work with your data analysts and data scientists to build aggregate data models within the warehouse to improve reporting performance.