Multimodal prompting with Gemini

What you'll learn

• Build an image prompt as an ordered list of parts that says what the pictures hold.
• Explain why PDFs, audio and video go in by uploaded handle instead of inline bytes.
• Price a page, an image and a second of audio in tokens before you call the model.
• Design prompts that fix the output shape and refuse anything the file does not contain.

T3.1 wrote prompt patterns for text alone: roles, few-shot examples, chain of thought. This lesson keeps those patterns and adds the part they never needed: the evidence arrives as a picture, a document or a recording.

A Larkspur customer sends a screenshot of the pricing page and asks How much outbound traffic does the starter plan include? A text-only assistant cannot see the screenshot. A multimodal model can, and it can still answer badly: it can describe the page instead of reading it, or estimate a value from a bar that carries no printed number. The prompt has three jobs here. It must tell the model what kind of image it holds, ask the exact question, and give it a way to say the answer is not on the page.

An image prompt is a list of parts: one or more images, then the question as text. The notebook sends two scanned pages on US wildfires to Gemini this way, with the pictures first and the question last:

Figure 1: An image prompt is an ordered list. The pictures travel inline with the request; nothing is uploaded first.

The notebook's code runs gemini-2.5-flash at temperature 0, although its Markdown cells still describe Gemini 1.5 Flash. The model name changes over time; the shape of the request stays.

Three choices that carry over

The wildfire prompt makes three choices worth copying for any image question:

Table 1: The wildfire prompt's choices, applied to the Larkspur question.

The third choice is a trade-off. Lesson 2 of T3.2 sends both wildfire questions in one GPT-4o call. Separate calls pay for the images twice, but each answer can be graded by itself.

The Larkspur prompt adds one line the wildfire prompt lacks: "If the value is not printed on the page, say so." Without it the model meets a bar with no label and estimates its height. Call that failure guessing from the picture: an answer that looks like a reading of the page but is a measurement of pixels.

An image as the question

An image can also be the thing you ask about. The notebook's last example sends a video, a still image and the prompt "When is the moment in the image happening in the video? Provide a timestamp." For a support team that is a customer's screenshot of an error, matched against a recorded walkthrough to find the step where they are stuck.

What an image costs

Images cost tokens like text. Gemini's token documentation counts an image of 384 pixels or less on both sides as 258 tokens, and tiles larger images into 768 by 768 tiles of 258 tokens each [1]. A large screenshot therefore costs a multiple of a small one. Documents and recordings are priced the same way, by page and by second, and they travel differently.

PDFs, audio and video go through the File API: you upload the file once and get back a file handle, a reference the model can read. The handle takes the image's place in the contents list. The notebook uploads a Congressional Research Service PDF on wildfire statistics, an interview recording, three videos and one still image this way.

The upload happens once. The notebook reuses one audio handle for three prompts:

Figure 2: One upload, three prompts. Each call sends the handle, not the recording's bytes.

Sending the bytes again with every prompt is the re-upload tax: the same file crosses the network three times for three questions. A handle avoids it, and uploaded files stay available for 48 hours [2].

Three prompts over one recording

Each of the three prompts fixes the output shape and keeps the model to what is in the recording:

Table 2: The notebook's three audio prompts. The notebook ships without outputs, and the answers depend on your run.

For Larkspur, the transcript prompt turns a recorded support call into turns labelled Customer and Agent. A follow-up question such as "What refund window did the agent quote?" can then be checked against the handbook's 30 days. A summary that states the wrong window is the expensive error here, because a customer will act on it.

What a document or a call costs

The token rates come from Gemini's documentation: 258 tokens per PDF page [3] and 32 tokens per second of audio [1]. With Larkspur sizes:

Table 3: The token price of two Larkspur files, before the question is read.

A 10-minute call costs almost twice the whole handbook. Every question about that call pays the 19,200 again, because the handle saves transfer, not tokens.

When to upload

Gemini accepts PDFs up to 50 MB or 1,000 pages [3]. Its Files API guide says to upload whenever the whole request, files and prompt together, is larger than 100 MB [2]. Upload below that size too when the same file serves several prompts, as the audio handle does in Figure 2:

• Inline: small images sent once, such as the pricing screenshot.
• Uploaded handle: PDFs, recordings and anything you will ask about more than once.

Video goes in the same way, and Gemini samples it into frames by itself. Lesson 2 of T3.2 samples video by hand and shows what the sampling rate decides.

1. Google, Understand and count tokens, Gemini API documentation, checked November 2026.
2. Google, Files API, Gemini API documentation, checked November 2026.
3. Google, Document understanding, Gemini API documentation, checked November 2026.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• An image prompt is an ordered list of images followed by the question. Say what the images hold, ask the model to use all of them, and give it a way to say a value is not printed.
• Upload PDFs, recordings and video once through the File API and reuse the handle across prompts; upload anything that takes a request over 100 MB.
• Media cost tokens on every call: 258 per small image or PDF page and 32 per second of audio. Fix the output shape in the prompt and forbid anything that is not in the file.