Multimodal LLMs (Vision)

Question 1

How does an image enter a vision-language model?

Accepted Answer

Sliced into patches, each embedded as a token-like vector — A vision encoder patchifies the image and embeds each patch — "visual words" that join the token sequence.

Question 2

What does the projection (adapter) layer do in a VLM?

Accepted Answer

Maps patch embeddings into the language model's text-embedding space — It aligns the two spaces so a sun-colored patch and the word "sun" land near each other — the glue between encoder and LLM.

Question 3

Why can a VLM answer questions about an image without a separate vision pipeline?

Accepted Answer

Attention operates over image and text tokens in one sequence — One shared token space means one attention mechanism reasons across modalities — perception and language in a single call.

Question 4

Text smaller than a patch (e.g. 6-pixel type) is misread because:

Accepted Answer

It was blurred into its tile at encode time — the information is gone — The encoder is the bottleneck: no prompt can recover pixels the patch embedding never kept. Crop or raise resolution instead.

Question 5

Roughly what does a 1024×1024 photo cost in tokens (typical provider pricing)?

Accepted Answer

several hundred to ~1 000 tokens — Providers price by tiles/patches; a full photo lands in the hundreds-to-~1k range — think "a page or two of prose" per image.

Question 6

Sending a 30-page PDF as page images primarily risks:

Accepted Answer

Consuming tens of thousands of context tokens before the question — Each page is ~1-1.6k image tokens; 30 pages ≈ 50k tokens of window and budget consumed by pixels alone.

Question 7

Which task profile favors a VLM over a classical OCR pipeline?

Accepted Answer

Long-tail varied documents with no per-template payoff — VLMs handle layout implicitly with zero per-template work; classical OCR+parsers stay cheaper on high-volume uniform documents.

Question 8

Why are VLMs the "eyes" of computer-use agents?

Accepted Answer

They read screenshots to find elements and infer state, enabling click decisions — Screenshot in → element location and state out is precisely the perception step a UI-driving agent needs.

Question 9

"Hallucinated detail" in vision models means:

Accepted Answer

Confidently describing content the image cannot actually resolve — Asked about something below resolution or out of frame, the model may confabulate plausibly — an honesty failure, not a perception one.

Question 10

A mitigation that exposes confabulation about an image is:

Accepted Answer

Asking the model to quote exactly the text it can see — Evidence-first prompting ("quote what you see, then answer") makes unsupported claims visible and checkable.

Key insight

How an image reaches the LLM

Test your knowledge

How does an image enter a vision-language model?

The short version

In more depth

Multimodal LLMs (Vision)

The pipeline

Engineering consequences

Interview Questions

References & Further Reading

Related topics

Love this visualization?