Understanding Foundation Models

Question 1

The pretraining objective for a foundation language model is:

Accepted Answer

Predicting the next token — Everything a base model does comes from next-token prediction over a very large corpus. Post-training changes what it is fitted to, not the objective.

Question 2

In the relation C ≈ 6ND, the symbols N and D stand for:

Accepted Answer

Parameters and training tokens — N is the parameter count and D is the number of training tokens; their product times a constant of about six gives training FLOPs.

Question 3

Compute-optimal training suggests roughly how many training tokens per parameter?

Accepted Answer

About 20 — The compute-optimal point sits near 20 tokens per parameter, which is why size and data should be scaled together rather than parameters alone.

Question 4

Spending a fixed compute budget on a model four times larger than compute-optimal produces:

Accepted Answer

An under-trained model that is both worse and more expensive to serve — The data budget must shrink to pay for the parameters, so the model has more capacity than the data can teach it to use — and you carry that capacity at every inference call.

Question 5

Why do production models often exceed the compute-optimal token count?

Accepted Answer

Because training compute is paid once but inference compute is paid per call forever — Over-training a smaller model is a worse deal at training time and a much better one at serving time, since fewer parameters are carried through every future request.

Question 6

A base model asked "How do I reset my password?" is most likely to respond with:

Accepted Answer

More questions of the same kind, continuing the document — A base model continues text. It has seen that string inside FAQ lists, so it writes more of the list — it has no concept of being addressed.

Question 7

Supervised finetuning primarily teaches a model to:

Accepted Answer

Respond to instructions rather than continue text — SFT fits the model to demonstrations where a question is followed by an answer, installing the conversational behaviour that pretraining does not produce.

Question 8

Preference tuning (RLHF or DPO) is trained on:

Accepted Answer

Pairs of responses ranked against each other — Comparative judgements are far more reliable from humans than absolute scores, so preference data is collected as "A is better than B" pairs.

Question 9

The parameter count of a model predicts which quantity most tightly?

Accepted Answer

Serving memory footprint — Memory is roughly parameters times bytes per parameter. Capability depends far more on data quality, token budget and post-training than on size.

Question 10

A mixture-of-experts model with 400B total parameters may serve like a much smaller model because:

Accepted Answer

Only a subset of parameters is activated per token — Sparse activation means FLOPs per token track the active parameters, not the total — so total size stops predicting cost and latency.

Key insight

How a chat model gets made

Test your knowledge

The pretraining objective for a foundation language model is:

The short version

In more depth

Understanding Foundation Models

Training data determines the shape of capability

Scaling laws

Post-training

Reading a model size

Interview Questions

References & Further Reading

Related topics

Love this visualization?