Building a data warehouse: your systems in one data model

When

When does a mid-sized company need a data warehouse?

You need a data warehouse as soon as a recurring analysis requires data from more than one system and your team currently brings that data together by hand. Until then, a report straight from the ERP is often enough.

The sales report by product group, for example, needs orders from the ERP, quotes from the CRM and invoices from accounting. Controlling builds an Excel file for it every month that only one person understands.

In cases like this, a data warehouse solves the merging once and for all, with a data model that records the relationships between customers, orders and items. datadice first opens up the sources your project needs; more are added when a new project calls for them.

For your IT management

Which platform does datadice use and does it fit our standards?

datadice works with the platforms you have already set; at Thomann that is Google Cloud with BigQuery. We define the architecture once your IT has explained which cloud, which databases and which authorisation systems apply.

For Thomann, datadice built on Google Cloud: the data warehouse runs in BigQuery, the data pipelines are documented and versioned with Dataform, and the dashboards run in Looker Studio. For your project, your data protection requirements, your operating model and the question of who maintains the platform later are what decide.

Before the build, you have an architecture sketch, the list of source connections and an access concept. If a source system changes, the affected connections can be found precisely because every data pipeline is documented.

Comparison

Data warehouse, data lake or Excel?

A data warehouse stores prepared, structured data according to a fixed model and is suited to reports and key figures. A data lake stores raw data in many formats and pays off when large volumes of unstructured data arise.

Excel remains the right tool for one-off analyses; a monthly analysis with several sources depends there on the person who maintains the file.

For your finance management

What costs arise after the build?

After the build, there are platform costs and maintenance effort. datadice agrees both with you before the start, separately from the costs of the build.

With cloud services such as BigQuery, the ongoing platform costs depend on storage and queries, that is, on data volume and usage. On top of that come maintaining the definitions and adjustments when a source system changes.

We agree in advance which tasks your team takes on and what datadice provides in operation. Your finance management thus gets an overview of one-off costs, ongoing costs and internal effort.