In lesson 4.13.2, you wrote the Kladde's star into a lakehouse with four cells of Spark SQL, and you did not ask what Spark actually does. This lesson catches up on that. It is an overview, not a Spark course: you should be able to judge when Spark is the right tool, what happens in a notebook when you run a cell, and how you can tell from the execution plan that a query is getting expensive. Fabric notebooks speak several languages; the course uses Spark SQL, because you already speak SQL.
What Spark is for
Apache Spark computes in a distributed way: one program, the driver, breaks the work into tasks, and many executors on several machines carry them out at the same time, each on a part of the data. The data usually lies as files in a data lake, in Fabric as Delta tables in OneLake. That pays off when one of three conditions holds:
- The data volume is too large for a database to read in reasonable time, or too large to keep permanently in a database.
- The data is not tables: files in many formats, logs, semi-structured JSON that first has to be put into shape.
- The transformation is hard: many steps, joins of large volumes, calculations you do not want to squeeze into one SQL query.
And it does not pay off when the data fits in a database and the query runs there in seconds. That is the case with the Kladde: a few tens of thousands of bookings per year and office are computed by SQL Server faster than a Spark cluster starts. Anyone who uses Spark anyway pays with startup time, the capacity's compute and a second language in the team. For small volumes, Fabric explicitly offers the way without Spark: besides the Spark kernel, notebooks have a Python kernel and a T-SQL kernel, and the warehouse computes with T-SQL.
For the Kladde, Spark is therefore a tool you should know, not one it needs. In lesson 4.13.5, you write this trade-off into an ADR.