Chat message patterns

Chat models expect conversations as a list of dictionaries. Each dictionary uses role and content keys. The content key holds the user message passed to the model. Large language models accept text and tools and multimodal models combine text with images, videos, and audio.

Transformers uses a unified format where each modality type is specified explicitly, making it straightforward to mix and match inputs in a single message.

This guide covers message formatting patterns for each modality, tools, batch inference, and multi-turn conversations.

Text

Text is the most basic content type. It’s the foundation for all other patterns. Pass your message to "content" as a string.

message = [
    {
        "role": "user",
        "content": "Explain the French Bread Law."
    }
]

You could also use the explicit "type": "text" format to keep your code consistent when you add images, videos, or audio later.

message = [
    {
        "role": "user",
        "content": [{"type": "text", "text": "Explain the French Bread Law."}]
    }
]

Tools

Tools are functions a chat model can call, like getting real-time weather data, instead of generating it on its own.

The assistant role handles the tool request. Set "type": "function" in the "tool_calls" key and provide your tool to the "function" key. Append the assistant’s tool request to your message.

weather = {"name": "get_current_temperature", "arguments": {"location": "Paris, France", "unit": "celsius"}}
message.append(
    {
        "role": "assistant", 
        "tool_calls": [{"type": "function", "function": weather}]
    }
)

The tool role handles the result. Append it in "content". This value should always be a string.

message.append({"role": "tool", "content": "22"})

Multimodal

Multimodal models extend this format to handle images, videos, and audio. Each input specifies its "type" and provides the media with "url" or "path".

Image

Set "type": "image" and use "url" for links or "path" for local files.

message = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
            {"type": "text", "text": "What pastry is shown in the image?"}
        ]
    }
]

Video

Set "type": "video" and use "url" for links or "path" for local files.

message = [
    {
        "role": "user",
        "content": [
            {"type": "video", "url": "https://static01.nyt.com/images/2019/10/01/dining/01Sourdough-GIF-1/01Sourdough-GIF-1-superJumbo.gif"},
            {"type": "text", "text": "What is shown in this video?"}
        ]
    }
]

Audio

Set "type": "audio" and use "url" for links or "path" for local files.

message = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "url": "https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac"},
            {"type": "text", "text": "Transcribe the speech."}
        ]
    }
]

Mixed multiple

The content list accepts any combination of types. The model processes all inputs together, enabling comparisons or cross-modal reasoning.

message = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
            {"type": "video", "url": "https://static01.nyt.com/images/2019/10/01/dining/01Sourdough-GIF-1/01Sourdough-GIF-1-superJumbo.gif"},
            {"type": "text", "text": "What does the image and video share in common?"},
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
            {"type": "image", "url": "https://assets.bonappetit.com/photos/57e191f49f19b4610e6b7693/master/w_1600%2Cc_limit/undefined"},
            {"type": "text", "text": "What type of pastries are these?"},
        ],
    }
]

Batched

Batched inference processes multiple conversations in a single forward pass to improve throughput and efficiency. Wrap each conversation in its own list, then pass them together as a list of lists.

messages = [
    [
        {"role": "user",
        "content": [
                {"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
                {"type": "text", "text": "What type of pastry is this?"}
            ]
        },
    ],
    [
        {"role": "user",
        "content": [
                {"type": "image", "url": "https://assets.bonappetit.com/photos/57e191f49f19b4610e6b7693/master/w_1600%2Cc_limit/undefined"},
                {"type": "text", "text": "What type of pastry is this?"}
            ]
        },
    ],
]

Multi-turn

Conversations span multiple exchanges, alternating between "user" and "assistant" roles. Each turn adds a new message to the list, giving the model access to the full conversation history. This context helps the model generate more appropriate responses.

message = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
            {"type": "text", "text": "What pastry is shown in the image?"}
        ]
    },
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "This is kouign amann, a laminated dough pastry (i.e., dough folded with layers of butter) that also incorporates sugar between layers so that during baking the sugar caramelizes."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://static01.nyt.com/images/2023/07/21/multimedia/21baguettesrex-hbkc/21baguettesrex-hbkc-videoSixteenByNineJumbo1600.jpg"},
            {"type": "text", "text": "Compare it to this image now."}
        ]
    }
]

Update on GitHub