Row-major layout is better than column-major for:

Writes — Appending individual examples is fast. The rule is row-major for many writes, column-major for many column-based reads.

Storing the number 1000000 as text takes 7 bytes. As an int32 it takes:

4 bytes — 32 bits. Binary formats are more compact, which is most of why Parquet beats CSV on size.

The book's interviews.csv, at 17,654 rows by 10 columns, went from 14 MB as CSV to:

6 MB as Parquet — AWS separately reports Parquet as up to 2× faster to unload and consuming up to 6× less storage in S3.

The book's framing of structured versus unstructured data is that it decides:

Which side of the pipe assumes the structure — The code that writes assumes it for structured data; the code that reads assumes it for unstructured.

A data lake stores raw data before processing. A data warehouse stores data that is:

Already processed into ready-to-use formats — And a lakehouse combines the flexibility of a lake with the data management of a warehouse.

ELT becomes less attractive as data grows because:

Searching a massive amount of raw data is inefficient — And as infrastructure standardises, committing to a schema becomes feasible again.

ML-SYSTEMS

intermediate

Data Engineering for ML

ML Engineer path · lesson 66 of 80

Watch a row-major read touch every column to return four of them, price the same file as CSV and Parquet, and see stateful stream processing reuse 29 days of work a batch job throws away every night.

The idea

A filing cabinet versus a card index. If you usually want everything about one person, file by person. If you usually want one field about everyone, file by field. Choose wrong and every query walks past ninety-nine things it did not want.

How to use

On the first tab, toggle the layout and drag how many features you need — row-major touches all of them regardless.

Data serialization is converting a data structure or object state into a format that can be stored or transmitted and reconstructed later. The choice that matters most for ML is not the format name — it is whether it is row-major or column-major.

click to switch layout

Features you actually need

4 of 12

columns touched

read and discarded

33%

read efficiency

Row-major must read all the columns and then filter. Even knowing the row sizes it is slow, because you jump around memory and cannot take advantage of caching. What row-major wins is writes — appending individual examples is fast.

The rule: row-major when you do a lot of writes; column-major when you do a lot of column-based reads. And a trap worth knowing — pandas is built around DataFrame, which is column-major (inspired by R), while a NumPy ndarray is row-major by default. People coming from NumPy access DataFrames by row and find them slow. Measured: iterating a DataFrame by column takes 0.07 seconds; by row, 2.41 seconds.

Common formats

Format	Binary / text	Human-readable	Example uses
JSON	Text	Yes	Everywhere
CSV	Text	Yes	Everywhere
Parquet	Binary	No	Hadoop, Amazon Redshift
Avro	Binary primary	No	Hadoop
Protobuf	Binary primary	No	Google, TensorFlow (TFRecord)
Pickle	Binary	No	Python, PyTorch serialization

Binary is more compact. Storing 1000000 as text takes 7 characters = 7 bytes; as an int32 it takes 32 bits = 4 bytes. The book's worked case: interviews.csv, 17,654 rows × 10 columns, went from 14 MB as CSV to 6 MB as Parquet. AWS reports Parquet is up to 2× faster to unload and consumes up to 6× less storage in S3 than text formats.

CSV also loses numbers

It serialises nontext characters poorly — writing a float can lose precision, with 0.12345678901232323 arbitrarily rounded to 0.12345678901. For a feature store that is silent data corruption, and it will not show up until a model trained on the rounded values behaves slightly differently from one trained on the originals.

Key insight

Modern computers process sequential data far more efficiently than nonsequential data, which is the whole reason layout matters. The choice is not about the file extension — it is a prediction about how the data will be read, made at the moment you write it.

Row-major or column-major?

Sort

Sort each trait into the storage layout it describes.

Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

Row-major

Drop items here

Column-major

Drop items here

6 pieces left to place

Test your knowledge

20 questions

Question 1 of 20

Score: 0

CSV is row-major, which means consecutive elements in a row are:

The short version

In more depth

Data Engineering for ML

Row-major vs column-major

CSV is row-major (consecutive elements in a row are adjacent in memory); Parquet is column-major. Modern computers process sequential data far more efficiently than nonsequential data.

Row-major is better for accessing examples, and allows faster writes — appending individual rows.

Column-major is better for accessing features. Ride-sharing case: 1,000 features stored, 4 wanted (time, location, distance, price). Row-major must read all of them and filter — and even knowing the row sizes it's slow, because you jump around memory, unable to take advantage of caching.

Trap: pandas' DataFrame is column-major; a NumPy ndarray is row-major by default. Measured: iterating a DataFrame by column takes 0.07 s; by row, 2.41 s.

Text vs binary

1000000 as text = 7 characters = 7 bytes; as int32 = 4 bytes. interviews.csv (17,654 × 10) went from 14 MB as CSV to 6 MB as Parquet. AWS: Parquet is up to 2× faster to unload and 6× less storage in S3. CSV also serialises floats poorly — 0.12345678901232323 → "0.12345678901".

Structured vs unstructured

"Structured data means that the code that writes the data has to assume the structure. Unstructured data means that the code that reads the data has to assume the structure."

Data lakes store raw data before processing; data warehouses store data already processed. A lakehouse is the hybrid (Databricks, Snowflake). ETL transforms before loading; ELT stores raw first — fast arrival, but less attractive as data grows.

OLTP and OLAP are going stale

ACID: Atomicity (payment fails ⇒ no driver assigned) · Consistency · Isolation (two users can't book the same driver) · Durability. Non-ACID systems are sometimes called BASE.

Three reasons the terms are dated: the separation was a technology limitation that is closing (CockroachDB, Iceberg, DuckDB); storage and processing are now decoupled (BigQuery, Snowflake); and "online" is overloaded — it now means online (immediately available), nearline (made online quickly without human intervention), offline (requires human intervention).

Three modes of dataflow

1.Databases — needs a shared database, and read/write is slow.

2.Services (request-driven) — synchronous; a service that is down can take down everything depending on it. REST for public APIs, RPC within an organisation.

3.Real-time transport (event-driven) — a broker so each service talks only to it. Pubsub (Kafka, Kinesis) vs message queues (RocketMQ, RabbitMQ).

"Request-driven architecture works well for systems that rely more on logic than on data. Event-driven architecture works better for systems that are data-heavy."

Batch vs stream

Batch features are static (a driver's rating); streaming features are dynamic (drivers available right now). Most problems need both.

Stream processing's real strength is stateful computation: for engagement over a 30-day trial, a batch job recomputes 30 days every day, while a stream job computes only the new data and joins it with the older computation. Engines: Flink, KSQL, Spark Streaming. "Batch is a special case of streaming."