Multimodal

Early chatbots only read and wrote text. Multimodal models handle several kinds of input and output: you can show one a photo of a rash, a chart or a broken washing machine, talk to it out loud, or ask it to make an image.

Under the hood, the model turns pictures and sound into the same kind of pieces (tokens) it uses for text, so it can reason across them: read the numbers off a chart, then explain what they mean. It can also misread a picture as confidently as it misreads a sentence, so a blurry photo or a cropped screenshot gets a confident wrong answer just as easily as a vague question does.