The star from lesson 4.13.1 was filled in one go, from a database that stood still. In production, the Kladde changes every minute, and the star should catch up every night, without rereading everything each time, without losing an appointment and without counting one twice when a run aborts in the middle. This lesson builds that load. You already know some of its rules: idempotence and restart from the import in K3, “at least once” delivery from lesson 4.6.1, numbers that are assigned before the commit, from lesson 4.4.5.
ETL or ELT
Both abbreviations describe the same three steps in a different order:
- ETL (extract, transform, load): read data from the source, transform it on the way, then load it into the target ready to use. That is how classic data warehouses worked, when compute in the target was expensive.
- ELT (extract, load, transform): load the data into the target raw first, then transform it there, with the target's compute. That is how a lakehouse works: raw data lands in a layer, and Spark or SQL shapes the star from it, layer by layer.
ELT is the normal case today, because the raw data is kept and a transformation can be rerun at any time. But there is one reason
that forces ETL, and it concerns the Kladde: What the target must never see has to go before the load. The person who booked in
appointments.Bookings does not belong in OneLake, not even in a raw layer that “only the data team” sees. So the load reads only
the columns the star needs. That is a transformation before the load, and it is data protection by design from lesson 4.8.3.