Multimodal models: contrastive pretraining, fusion, and any-to-any generation

Why this subject matters now

Until roughly 2021, computer vision and natural language processing were separate fields with separate pretraining recipes: ImageNet-style supervised classification on one side, masked or causal language modeling on the other. Two results collapsed the boundary. CLIP (Radford et al., 2021, OpenAI) and ALIGN (Jia et al., 2021, Google) showed that a pair of encoders trained contrastively on hundreds of millions of noisy web image-text pairs learns a representation that classifies images zero-shot, retrieves across modalities, and transfers better than supervised pretraining at comparable scale. Flamingo (Alayrac et al., 2022, DeepMind) then showed that a frozen language model can be taught to consume those visual features through a thin cross-attention interface, and LLaVA (Liu et al., 2023, Wisconsin-Madison and Microsoft Research) showed the interface can be as small as one linear layer if the instruction data is right. From there the field split into two engineering philosophies that this page compares throughout: bolt a vision encoder onto a pretrained language model through an adapter, or tokenize everything, images, audio, text, into one discrete stream and train a single model natively, the route publicly described for Gemini (Google DeepMind, 2023) and taken openly by Chameleon (Meta, 2024).

The practitioner's questions are now quantitative. How many tokens does an image cost at a given resolution, and what does that do to the context budget and the KV cache? Why does a contrastive model need a batch of 32k and a temperature of 0.01, and what breaks if the temperature is left unclamped? Why does a vision-language model confidently describe objects that are not in the image, and why do benchmark scores move several points when answer options are reordered? Each of these has a concrete arithmetic or gradient-level answer, and the answers are what this page derives. The sibling pages own the neighboring territory: the language-model page owns the decoder pipeline, FLOPs accounting, and KV-cache arithmetic that this page reuses, and the diffusion page owns continuous image generation; here the generation story is told through discrete tokens, which is the route autoregressive multimodal models actually take.

Core theory

Contrastive pretraining: the InfoNCE objective

The data is a stream of pairs \( (x_i, y_i) \): an image and the alt-text that accompanied it on some web page. An image encoder \( f \) and a text encoder \( g \) map each side to \( \R^d \), and both outputs are L2-normalized, so \( u_i = f(x_i)/\lVert f(x_i)\rVert \) and \( v_i = g(y_i)/\lVert g(y_i)\rVert \) live on the unit sphere and their inner product \( s_{ij} = u_i\T v_j \in [-1, 1] \) is a cosine similarity. The training signal is co-occurrence: \( (u_i, v_i) \) came from the same web page, every other pairing in the batch presumably did not. The InfoNCE loss, introduced for contrastive predictive coding by van den Oord et al. (2018) and applied symmetrically by CLIP, treats each row of the batch similarity matrix as a classification problem: given image \( i \), pick its caption out of the \( N \) candidates in the batch,

and each column as the mirror problem, given a caption, pick its image,

The two directions are not redundant. The row softmax normalizes over captions and never compares two images with each other; the column softmax normalizes over images and never compares two captions. Using both makes each embedding serve simultaneously as a query and as a key, which is what a retrieval system needs, and empirically the symmetric loss trains better-conditioned spaces than either direction alone. Note also what the loss does not require: no labels, no bounding boxes, no human annotation beyond what the web already produced. That is why the recipe scales to the 400M pairs of CLIP's WIT dataset and the 1.8B noisy pairs of ALIGN, and why data curation, not architecture, became the competitive axis (the DataComp benchmark of Gadre et al., 2023, exists to study exactly this).

Why "InfoNCE"? For a batch of \( N \) pairs the row loss is a lower-bound estimator of the mutual information between the two views: van den Oord et al. show \( I(u; v) \ge \log N - \L_{\text{i}\to\text{t}} \). The proof is short. The optimal critic in the softmax is the density ratio \( p(v\mid u)/p(v) \); substituting it and taking expectations, the loss becomes an expectation of \( \log\!\big(1 + (N{-}1)\,\E\,[p(v)/p(v\mid u)]\big) \)-type terms bounded below by \( \log N - I(u;v) \). Two practical consequences follow directly from the bound. First, the bound saturates at \( \log N \): a batch of 32,768 can certify at most \( \log_2 32768 = 15 \) bits of mutual information per pair, which is one reason large batches help. Second, the loss only ever compares similarities within a batch, a fact that matters for the modality gap below.

The temperature, derived from the gradient

Everything interesting about \( \tau \) falls out of one differentiation. Fix a row \( i \), write \( p_{ij} = \exp(s_{ij}/\tau)\big/\sum_k \exp(s_{ik}/\tau) \) for the row softmax, and let \( \ell_i = -s_{ii}/\tau + \log\sum_j \exp(s_{ij}/\tau) \) be the per-row loss. Differentiating with respect to a similarity, using \( \partial \log\sum_j e^{s_{ij}/\tau} / \partial s_{ik} = p_{ik}/\tau \),

Three facts are visible at once. First, every gradient carries a factor \( 1/\tau \): halving the temperature doubles the effective learning rate on the similarities, so \( \tau \) and the learning rate are entangled. Second, the push on a negative pair is proportional to \( p_{ij} \), the softmax weight that negative currently receives. With small \( \tau \) the softmax is sharp and nearly all of the repulsive gradient concentrates on the few hardest negatives, the confusable ones; with large \( \tau \) the repulsion spreads uniformly over the batch. Temperature is therefore a hard-negative-weighting dial, not a cosmetic rescaling. Third, the attractive and repulsive terms balance: \( \sum_j \partial \ell_i / \partial s_{ij} = 0 \), so the loss only shapes relative geometry within the row, never absolute positions.

CLIP makes \( \tau \) learnable, and the gradient explains what the optimizer does with it. Differentiating \( \ell_i \) with respect to \( \tau \) directly,

Once training has made the positive similarity \( s_{ii} \) larger than the softmax-average similarity of the row, which happens almost immediately, this derivative is positive, so gradient descent drives \( \tau \) down, monotonically sharpening the softmax. Nothing in the loss stops it: as \( \tau \to 0 \) the loss approaches a margin objective on the single hardest negative and keeps decreasing. Left alone, \( 1/\tau \) grows without bound, the logits \( s_{ij}/\tau \) blow past the fp16 range inside the \( \exp \), and the \( 1/\tau \) factor in every gradient destabilizes the encoders. CLIP therefore parameterizes the scale as a learnable log-temperature, optimizing \( t' = \log(1/\tau) \) so that the scale \( e^{t'} \) is positive by construction, initializes it at \( \log(1/0.07) \approx 2.659 \), and clamps the scale at 100, equivalently \( \tau \ge 0.01 \). The clamp is not decorative. The released OpenAI CLIP checkpoints report a logit scale of essentially exactly 100, that is \( \tau = 0.01 \), the clamp boundary, exactly as this derivation predicts: training pushed the temperature to the floor and the clamp held it there. That single reported number is the empirical fingerprint of the runaway the gradient above describes.

The modality gap

A natural expectation is that contrastive training fuses the two modalities into one cloud: matched image and text embeddings nearly coincident on the sphere. Measurement says otherwise. Liang, Zou, and colleagues (2022) showed that in trained CLIP-family models all image embeddings occupy one narrow cone and all text embeddings another, with a roughly constant offset vector between the cone centers, the modality gap. The mean image-text cosine of matched pairs in CLIP sits far below 1, while the loss is nonetheless near its floor. Two mechanisms produce this. First, the cone effect: at random initialization a deep encoder's outputs already concentrate in a narrow cone (each layer's nonlinearities and normalizations contract angular spread), and two differently-initialized encoders start in two different cones. Second, and this is the part the derivation above makes precise, InfoNCE has no force that closes the gap: the loss depends only on the similarities \( s_{ij} \) compared within a row or column, and the softmax is shift-invariant, so adding a constant to every logit in a row changes nothing. A uniform cross-modal offset that raises or lowers all \( s_{ij} \) by nearly the same amount is thus invisible to the objective; only the contrast between matched and mismatched pairs is trained. Liang et al. further showed the equilibrium gap size depends on the temperature, with low \( \tau \) preserving a larger gap, and that artificially shrinking or enlarging the gap by translating one modality's embeddings can shift downstream zero-shot accuracy and fairness metrics in either direction: the gap is a property of the objective's invariances, not a bug with a one-line fix. Practical consequence: pipelines that mix modalities in one vector index, or that threshold absolute cosine values, must calibrate per modality pair; a matched image-text cosine of 0.3 can be a strong match while a text-text cosine of 0.3 is a weak one.

SigLIP: the sigmoid loss, and why batch size decouples

The softmax in InfoNCE creates a systems problem: every device needs the full \( N \times N \) similarity matrix's row and column sums, so the standard implementation all-gathers all \( N \) embeddings to every device. At CLIP's batch of 32,768 that is manageable; the desire to push batch size further, and to remove the two all-gathers, motivated Zhai, Mustafa, Kolesnikov, and Beyer (2023, Google) to replace the softmax with independent binary classifications. SigLIP scores every pair in the batch as pair-or-not-pair with a logistic model,

with a learnable inverse-temperature \( t \) and, crucially, a learnable bias \( b \). The gradient with respect to one similarity, using \( \frac{d}{dx}\log\sigma(x) = \sigma(-x) \), is

and the decoupling is right there: the gradient on pair \( (i,j) \) depends only on that pair's own similarity, not on any normalizer over the batch. Each pair is its own two-class problem. Batch size no longer changes the definition of the loss, it only changes how many negative pairs are sampled per positive (\( N^2 - N \) negatives to \( N \) positives), and the implementation can chunk: each device holds its own embeddings, passes its text block around a ring of devices, and accumulates loss terms blockwise, with no moment at which the full matrix or the full gathered batch must exist anywhere. Zhai et al. found quality saturates near batch 32k anyway, so the practical win is efficiency and robustness at ordinary batch sizes rather than million-size batches.

The bias \( b \) fixes the class imbalance that the sigmoid formulation creates. At initialization with \( s_{ij} \approx 0 \), a bias-free loss gives every one of the \( N^2 - N \) negatives the same loss \( \log 2 \) as each positive, so the negative side outweighs the positive side by a factor of \( N - 1 \) and the first optimizer steps collapse all similarities downward. SigLIP initializes \( t = 10 \) and \( b = -10 \): on the 3x3 matrix of Problem 1 those values give pairwise logits \( t s_{ij} + b \) of \( -1, -2, -3 \) on the diagonal and \( -6 \) to \( -10 \) off it, hence per-pair losses of 1.313, 2.127, and 3.049 on the positives against at most 0.0025 on any negative (computed in python): the positives dominate the gradient from step one, the imbalance is neutralized, and \( b \) is subsequently learned. A trained SigLIP checkpoint settles on a large positive inverse temperature \( t \) and a substantially negative bias \( b \), and its calibrated pairwise probabilities behave unlike CLIP's softmax: each caption gets an independent probability, they do not sum to 1, and an image matching none of the offered captions can score near zero on all of them, which a softmax cannot express.

InfoNCE (softmax, per row/column)          SigLIP (sigmoid, per pair)

  images ──► u_1..u_N ─┐                     images ──► u_1..u_N ─┐
  texts  ──► v_1..v_N ─┤                     texts  ──► v_1..v_N ─┤
                       ▼                                          ▼
        S = U Vᵀ  (N x N, gathered)             S_block = U_dev V_ringᵀ
        row softmax  + col softmax              per-pair  log σ(±(tS+b))
        needs global normalizers                no normalizer: chunkable
        batch size defines the loss             batch size = #negatives only
          

The vision encoder: ViT patchification arithmetic

Both encoders in a CLIP-family model, and the vision side of essentially every VLM, are Vision Transformers (Dosovitskiy et al., 2021, Google). The design choice that matters for all the budget arithmetic on this page is patchification: an image of resolution \( R \times R \) with patch size \( P \) becomes \( N = (R/P)^2 \) tokens, each token the flattened \( P^2 \cdot C \) pixel values of one patch pushed through a single linear layer into the model width \( d \). A learned position embedding is added per patch, a class token may be prepended, and from there the model is a plain pre-norm transformer encoder with bidirectional attention, no causal mask, since an image has no autoregressive order. The consequences of \( N = (R/P)^2 \) are quadratic twice over: doubling resolution quadruples the token count, and since self-attention is quadratic in tokens, it multiplies attention FLOPs by sixteen. Every resolution strategy in the AnyRes section below is a negotiation with this arithmetic.

From encoder to language model: the connector design space

A contrastive encoder produces a grid of features; a language model consumes a sequence of \( d_{\text{LM}} \)-dimensional embeddings. Everything between them is "the connector", and the design space is essentially one question: how many tokens does an image get, and who pays for compressing it. Three archetypes cover the field.

Linear projection (LLaVA). Take every patch feature from the penultimate layer of a frozen CLIP ViT-L/14 encoder and multiply by one learned \( d_v \times d_{\text{LM}} \) matrix (LLaVA-1.5 upgraded this to a two-layer MLP with GELU). At the 336-pixel resolution LLaVA-1.5 uses, that is \( (336/14)^2 = 576 \) visual tokens inserted directly into the LM's input sequence. No information is discarded and nothing decides what is salient; the language model's own attention does all the reading. The cost is context: 576 tokens per image, growing with resolution, each occupying KV cache in every layer (the arithmetic is Problem 3). The result that made this respectable is that it works: with two-stage training, first the projection alone on image-caption pairs, then projection plus LM on instruction data, LLaVA matched or beat far heavier connectors.

Cross-attention resamplers (Flamingo's perceiver, Qwen-VL's adapter). Fix a budget of \( M \) learned latent vectors and let them attend over however many visual features the encoder produced. The mechanics deserve deriving because the asymmetry is the entire point. Cross-attention with latent queries \( Z \in \R^{M \times d} \) and visual features \( X_v \in \R^{N_v \times d} \) computes

with \( QK\T \in \R^{M \times N_v} \): the score matrix is rectangular, \( M \times N_v \), not \( N_v \times N_v \). FLOPs are \( 2 M d^2 \) for queries, \( 4 N_v d^2 \) for keys and values, and \( 4 M N_v d \) for scores and the weighted sum: linear in \( N_v \), never quadratic, because the visual features are only ever keys and values, never queries. The output is \( M \) vectors regardless of input size, so a variable number of frames or tiles compresses to a fixed token bill. Flamingo's perceiver resampler uses \( M = 64 \) latents over an arbitrary number of video frames, then feeds the resampled features into gated cross-attention layers interleaved into a frozen 70B-class LM, each gate a \( \tanh(\alpha) \) scalar initialized at \( \alpha = 0 \) so the pretrained LM is exactly recovered at step zero and vision fades in as \( \alpha \) is learned. The original Qwen-VL used the same idea as a single cross-attention layer compressing each image's ViT grid to 256 tokens.

Q-Former (BLIP-2). Between the archetypes sits Li et al.'s (2023, Salesforce) Q-Former: 32 learned queries processed by a small BERT-initialized transformer whose blocks self-attend among the queries and cross-attend into the frozen image encoder. It is pretrained before ever meeting the LM, with three objectives sharing the tower under different attention masks: image-text contrastive (queries versus text, CLIP-style), image-text matching (a binary head over fused query-text states), and image-grounded text generation. The queries thus learn to extract the text-relevant 32-token summary of an image, and only then is a single linear layer trained to hand that summary to a frozen OPT or FLAN-T5. The economy is extreme, 32 tokens against LLaVA's 576, and BLIP-2 beat 80B-parameter Flamingo on VQA with a fraction of the trainable parameters. The price is an information bottleneck chosen before the question is known: 32 tokens cannot carry every small text string or spatial relation in a dense document, which is why OCR-heavy and fine-grained benchmarks later pushed the field back toward projection-style connectors at higher resolution.

                        connector design space (tokens per image / trainable interface)

  frozen ViT grid ──► linear / MLP ────────────────► 576+ tokens ──► LM input   (LLaVA)
                       keeps everything, LM reads it, context pays

  frozen ViT grid ──► M latent queries ──► XAttn ──► 64 tokens  ──► gated XAttn (Flamingo)
                       fixed bill, linear in N_v, gate tanh(0) = no-op at init

  frozen ViT grid ──► 32 queries + BERT tower ─────► 32 tokens  ──► frozen LM   (BLIP-2)
                       pretrained bottleneck: ITC + ITM + generation masks
          

Early, late, and hybrid fusion

The taxonomy that organizes all of the above: where do the modalities first interact. Late fusion is CLIP itself: two towers that never exchange information until the final dot product. It is the cheapest possible interaction, which is why it powers retrieval at billion-image scale, embeddings can be precomputed and indexed, but a single inner product cannot express "count the chairs" or bind attributes to objects, so late fusion caps out at matching. Early fusion puts raw or lightly-encoded tokens of both modalities into one transformer from layer 1 and lets every layer attend across modalities; Chameleon and Fuyu-style models are the pure form. Maximum expressivity, maximum cost: every image token pays attention against every text token in every layer, and the model must be trained jointly from early on. Hybrid fusion is everything in between, and describes most deployed VLMs: substantial unimodal encoding first (a full ViT, most of an LM), interaction afterward, either by inserting projected visual tokens into the LM sequence (LLaVA: hybrid leaning early, interaction in every LM layer after insertion) or by periodic cross-attention (Flamingo: interaction only at the inserted gated layers). The engineering tradeoff is KV-cache and FLOPs versus grounding fidelity, and the fact that both LLaVA-style insertion and Flamingo-style cross-attention remain competitive tells you the field has not found a dominant point on that curve.

Native multimodal training versus adapter bridging

Orthogonal to fusion depth is a training question: is the model multimodal from pretraining, or made multimodal afterward. Adapter bridging, the LLaVA/BLIP-2/Flamingo route, freezes most of both pretrained towers and trains the thin interface, plus optionally the LM, on a comparatively tiny paired corpus (LLaVA-1.5's entire recipe is roughly 1.2M examples). It is astonishingly cheap, preserves the LM's text ability by construction, and inherits both towers' blind spots: whatever CLIP's encoder never represented, no adapter can recover, and the LM's text-only priors remain fully intact, which the hallucination section will blame for asserting absent objects. Native training, the route publicly described for Gemini (interleaved text, image, audio, and video tokens from the start of pretraining) and demonstrated openly by Chameleon, mixes modalities in the pretraining stream itself. It buys uniform treatment, any-to-any potential, and no frozen-encoder ceiling, at the price of frontier-scale pretraining cost, delicate optimization (Chameleon reports that naive training diverged from softmax logit drift, and stabilizing it required query-key normalization and reordered layer norms), and a real risk of degrading text-only quality since image tokens now consume a share of the token budget. The industry's current equilibrium is pragmatic: open efforts (LLaVA, InternVL, Qwen-VL) mostly bridge, frontier labs increasingly pretrain natively, and the gap between the two recipes is one of the live empirical questions of the field.

Resolution: AnyRes tiling and native-resolution ViTs

A 336-pixel input reads a scene but not a receipt. Raising the encoder's native resolution is expensive, position embeddings must be re-interpolated and attention cost grows with \( T^2 \), so LLaVA-NeXT popularized tiling, usually called AnyRes: choose a grid from a candidate set (1x2, 2x2, 2x3, ...), split the image into 336-pixel tiles, encode each tile independently through the same frozen encoder, and append a downscaled full-image overview so global layout survives the cut. Token cost is \( 576 \times (\text{tiles} + 1) \): a 2x2 grid costs \( 576 \times 5 = 2880 \) tokens, a 3x3 grid \( 576 \times 10 = 5760 \), already past a 4k context on its own (worked in Problem 3). Tiling also fractures objects that cross tile boundaries, which the overview only partly repairs. The cleaner alternative is to make the ViT itself resolution-native: NaViT (Dehghani et al., 2023, Google) packs patches from images of arbitrary aspect ratio into sequences, and Qwen2-VL adopts the same idea for VLMs, feeding each image as \( \lceil H/28 \rceil \times \lceil W/28 \rceil \) tokens after a 2x2 merge of 14-pixel patches, so a 1344x896 image costs \( 48 \times 32 = 1536 \) tokens with no tiling artifacts, positions handled by a multimodal rotary embedding (M-RoPE) that factors position into temporal, height, and width axes. InternVL's line similarly scales a purpose-trained 6B-parameter vision encoder (InternViT-6B) with dynamic tiling up to high resolutions. The common lesson: resolution is a token-budget decision, and the connector and the position scheme, not the encoder weights, are usually the binding constraint.

Video: factorized spatio-temporal attention

Video multiplies the token bill by frame count: \( F \) frames of \( N_p \) patches is \( T = F N_p \) tokens, and joint attention over all of them costs \( 4 T^2 d = 4 F^2 N_p^2 d \) FLOPs per layer, 16 frames of 256 patches is 4096 tokens and 68.7 GFLOPs per layer at \( d = 1024 \), before the MLP. Factorized attention, systematized for video by ViViT (Arnab et al., 2021, Google), splits each layer into spatial attention within each frame (\( F \) independent \( N_p \)-token problems) followed by temporal attention within each patch position (\( N_p \) independent \( F \)-token problems). The saving is derived as Problem 5: the ratio of joint to factorized attention FLOPs is exactly \( F N_p / (F + N_p) \), about 15x for the configuration above, at the cost that no single attention operation ever relates patch \( a \) of frame 1 to a different patch \( b \) of frame 2 directly; such relations must compose across layers. For VLM-style video understanding the coarser tools come first: sample frames sparsely, encode each with the image encoder, pool or resample per frame (Flamingo's perceiver treats frames as extra rows of keys and values at fixed latent cost), and only recent long-video models spend attention across full token grids.

Audio, briefly

Audio enters the same machinery through a spectrogram. The standard encoder recipe, exemplified by Whisper (Radford et al., 2022, OpenAI), converts a waveform to a log-mel spectrogram (80 mel bins, 25 ms windows, 10 ms hop), applies a pair of strided convolutions that downsample time, and runs a transformer encoder over the result, so 30 seconds of audio becomes 1500 encoder states; a VLM-style connector can then resample those states into LM tokens exactly as it does patch features. For generation the discrete route mirrors the VQ story of the next section: neural codecs such as SoundStream (Zeghidour et al., 2021, Google) quantize audio into residual vector-quantized token stacks at 50-75 tokens per second, which an autoregressive model can emit and a decoder can render back to a waveform. GPT-4o's publicly described design, one model consuming and producing text, audio, and image tokens in a single stream, is this recipe taken to its conclusion; the latency win over the older transcribe-then-respond-then-synthesize pipeline comes from deleting the two boundary crossings, and the paralinguistic win (hearing tone, emitting laughter) comes from the fact that a transcript is a lossy projection that the token stream never takes.

Discrete tokenization: VQ-VAE and the straight-through estimator

To let one autoregressive transformer emit images, audio, and text from a single vocabulary, the continuous signal must become a sequence of integers. The vector-quantized variational autoencoder (van den Oord et al., 2017, DeepMind) does this. An encoder \( E \) maps an image to a grid of latent vectors \( z_e(x) \in \R^{h \times w \times d} \); a codebook \( \{e_1, \dots, e_K\} \subset \R^d \) of \( K \) learned vectors replaces each latent by its nearest codebook entry,

and a decoder \( D \) reconstructs the image from the quantized grid. Each spatial position now carries an integer in \( \{1, \dots, K\} \): the image has become a short sequence of tokens, typically \( 16 \times 16 = 256 \) or \( 32 \times 32 = 1024 \) of them, from a vocabulary the model shares with text.

The obstacle is the gradient. The map from \( z_e \) to \( z_q \) is an \( \argmin \), piecewise constant, so \( \partial z_q / \partial z_e = 0 \) almost everywhere and no gradient reaches the encoder through the quantizer. The straight-through estimator resolves this by defining the forward value and the backward gradient separately. Write

where \( \mathrm{sg}[\cdot] \) is the stop-gradient operator, identity in the forward pass and zero in the backward pass. The forward value is exactly \( e_{k(x)} \), since the two \( z_e \) terms cancel numerically; the backward value has \( \partial z_q / \partial z_e = 1 \), because the bracketed term contributes no gradient, so the reconstruction gradient \( \partial \L_{\text{rec}} / \partial z_q \) is copied unchanged onto \( z_e \). It is a biased estimator, the true Jacobian of a nearest-neighbor assignment is not the identity, but it points the encoder in a direction that lowers reconstruction error, and empirically it trains. The codebook itself gets no gradient from this path (stop-gradient blocked it), so VQ-VAE adds two explicit terms:

The codebook term pulls the chosen code \( e_k \) toward the encoder output it must represent, with the encoder held fixed; its gradient is \( 2(e_k - z_e) \), plain k-means-style centroid updating, and many implementations replace it by an exponential moving average of assigned encoder vectors instead. The commitment term, weighted by \( \beta \approx 0.25 \), does the reverse, pulling the encoder output toward its chosen code with the code held fixed, so the encoder cannot outrun the codebook by inflating \( \lVert z_e \rVert \) and leaving codes stranded. The tension between the two terms is the whole design: without commitment the encoder's outputs drift and most codes go unused, the failure mode called codebook collapse that Problem 5 quantifies.

Two extensions matter in practice. VQGAN (Esser, Rombach, and Ommer, 2021, Heidelberg) keeps the VQ-VAE structure but replaces the pixel L2 reconstruction with a perceptual loss plus a patch discriminator, so the decoder produces sharp textures instead of the blurry mean that L2 rewards, and then trains an autoregressive transformer over the resulting code grid; this is the template every discrete-token image generator, and Chameleon and Parti among the multimodal models, inherits. Residual vector quantization, standard in neural audio codecs such as SoundStream and used by some image tokenizers, quantizes the residual repeatedly: set \( r_0 = z_e \), and for \( m = 1, \dots, M \) pick the nearest code from the m-th codebook, \( k_m = \argmin_j \lVert r_{m-1} - e^{(m)}_j \rVert \), subtract it, \( r_m = r_{m-1} - e^{(m)}_{k_m} \), and sum: \( z_q = \sum_{m=1}^{M} e^{(m)}_{k_m} \). With \( M \) codebooks of \( K \) entries this represents \( K^M \) distinct points using only \( MK \) stored vectors, an exponential effective vocabulary at linear storage cost, at the price that the autoregressive model must now predict \( M \) tokens per spatial position.

Contrastive versus generative pretraining

Contrastive and generative objectives learn different things from the same image-text data, and the difference is legible in the losses. InfoNCE only ever needs enough information to tell the matched caption from the other \( N - 1 \) in the batch; once an embedding is discriminative at that granularity, the loss is satisfied, and features that do not help intra-batch discrimination, exact counts, fine spatial relations, small text strings, are free to be discarded. That is why CLIP is extraordinary at retrieval and zero-shot classification and mediocre at reading a price tag. A generative objective, whether image-conditioned caption generation (the autoregressive \( -\sum_t \log p(y_t \mid y_{<t}, x) \)) or masked prediction, forces the model to reconstruct detail it would otherwise drop, because every token of the caption must be produced from the visual evidence. The cost is that generation supervises a conditional density, not an aligned metric space, so a purely generative model has no clean cross-modal similarity to index for retrieval. The field's response is to combine them: CoCa (Yu et al., 2022, Google) trains one encoder-decoder with a contrastive loss on pooled features and a captioning loss on the decoder simultaneously, and BLIP's three-objective Q-Former does the same at the interface. The practitioner's rule of thumb is that retrieval and zero-shot recognition want contrastive alignment, grounding and generation want a generative signal, and modern recipes pay for both.

Grounding and the hallucination problem

The characteristic failure of vision-language models is object hallucination: the model confidently describes objects, or answers yes to their presence, when they are not in the image. The mechanism follows directly from the adapter architecture. A bridged VLM keeps a language model that was pretrained on text alone and therefore carries strong priors over what co-occurs in the world, dining tables come with chairs, streets come with cars, and those priors compete with the visual tokens for control of the output. When the visual signal is weak, few tokens, low resolution, a frozen encoder that never represented the object, or the instruction data over-rewarded fluent affirmative answers, the language prior wins and the model asserts the statistically likely object rather than the observed one. Rohrbach et al. (2018, Berkeley) first quantified this for captioning with the CHAIR metric, the fraction of mentioned objects absent from the ground-truth annotation. The now-standard probe is POPE (Li et al., 2023, Renmin University and Microsoft), which turns hallucination into balanced yes/no questions, does this image contain a <object>, with the negatives sampled three ways: at random, from frequently-occurring objects, and from objects that co-occur with what is actually present. The adversarial and popular splits are much harder than the random one, and a model that scores well by answering yes too often is caught because the balanced negatives make the trivial constant-yes policy score exactly 50%. Measuring grounding this way, rather than by caption fluency, is what made the problem improvable; mitigations include higher-resolution and more visual tokens, contrastive decoding that subtracts the language-only prior, and instruction data with hard negative questions.

How CLIP embeddings condition image generation

The contrastive representation this page derives is also the standard conditioning signal for text-to-image diffusion, which is where the two sibling topics meet. There are two distinct ways a CLIP-family encoder feeds a generator. In latent diffusion (Rombach et al., 2022, Heidelberg and Runway), the text prompt is run through a frozen text encoder, CLIP ViT-L/14's text tower in Stable Diffusion 1.x, an OpenCLIP ViT-bigG plus CLIP ViT-L pair in SDXL, producing a sequence of token embeddings, and the denoising U-Net or DiT cross-attends into that sequence at every block: the text embeddings are the keys and values, the noisy image latents are the queries, exactly the rectangular cross-attention derived for the connector above. In unCLIP, the architecture behind DALL-E 2 (Ramesh et al., 2022, OpenAI), the conditioning is instead a single CLIP image embedding: a diffusion prior maps the CLIP text embedding to a CLIP image embedding, and a decoder diffuses an image conditioned on that embedding, so the contrastive space is literally the interface between understanding and generation. The denoising mathematics, the forward process, the variational bound, and why classifier-free guidance sharpens samples, is derived on the diffusion page; here the relevant point is that the CLIP objective's output, a normalized vector on a sphere, is precisely what those cross-attention layers were built to consume, and that the modality gap discussed above is why the diffusion prior in unCLIP is needed at all rather than feeding the text embedding directly.

Evaluation and its pitfalls

Multimodal systems are measured along axes that do not reduce to one number. Cross-modal retrieval reports Recall@k on Flickr30k and COCO (Problem 4); zero-shot recognition reports top-1 on ImageNet and its distribution-shift variants; visual question answering reports accuracy on VQAv2, GQA, and the OCR-heavy TextVQA and DocVQA; captioning reports CIDEr; and broad capability is probed by MMMU, MMBench, and SEED. The pitfalls are severe enough that a headline number is nearly meaningless without the protocol. Benchmark contamination is pervasive, because the web-scale pretraining corpora overlap the public test sets. Multiple-choice VQA is sensitive to option ordering and to the exact answer format, so scores move several points when the letters are permuted or when a model that knows the answer phrases it in words the exact-match scorer does not accept. VQAv2's accuracy is soft, crediting an answer that matches at least three of ten human annotators, which rewards short canonical strings and penalizes correct-but-verbose ones. And retrieval Recall@k saturates: on an easy benchmark R@5 and R@10 approach 1 for every competent model, so only R@1 discriminates. The defensible way to read a leaderboard is to hold the protocol fixed, report several axes, and treat any single-benchmark claim, especially on a saturated or contaminated one, as weak evidence.

Implementation

The symmetric CLIP loss is a dozen lines once the encoders exist. The subtleties are all in the details the derivation flagged: L2-normalize before the dot product so similarities are cosines, scale by the clamped \( \exp \) of a learnable log-temperature rather than dividing by \( \tau \), and take cross-entropy in both directions against the identity permutation as labels. In distributed training the embeddings are all-gathered so every row and column sum is complete, which is the cost SigLIP removes. The PyTorch and JAX versions below are the reference formulation; both reproduce the hand computation of Problem 1 exactly.

import torch
import torch.nn.functional as F

def clip_loss(img_emb, txt_emb, logit_scale):
    # img_emb, txt_emb: (N, d) raw encoder outputs
    # logit_scale:      scalar = exp(learnable t'), clamped so tau >= 0.01
    u = F.normalize(img_emb, dim=-1)             # (N, d) onto unit sphere
    v = F.normalize(txt_emb, dim=-1)             # (N, d)
    logits = logit_scale * (u @ v.t())           # (N, N) = cosine / tau
    labels = torch.arange(u.shape[0], device=u.device)
    loss_i2t = F.cross_entropy(logits, labels)       # rows: image picks caption
    loss_t2i = F.cross_entropy(logits.t(), labels)   # cols: caption picks image
    return 0.5 * (loss_i2t + loss_t2i)

# reproduce Problem 1 exactly (similarities given directly, logit_scale = 1/tau)
S = torch.tensor([[0.9, 0.1, 0.3], [0.2, 0.8, 0.0], [0.4, 0.3, 0.7]])
logits = S / 0.5                                  # (N, N)
lab = torch.arange(3)
print(0.5 * (F.cross_entropy(logits, lab) + F.cross_entropy(logits.t(), lab)))
# tensor(0.4989)

import jax.numpy as jnp
import optax

def clip_loss(img_emb, txt_emb, logit_scale):
    # img_emb, txt_emb: (N, d) raw encoder outputs
    u = img_emb / jnp.linalg.norm(img_emb, axis=-1, keepdims=True)   # (N, d)
    v = txt_emb / jnp.linalg.norm(txt_emb, axis=-1, keepdims=True)   # (N, d)
    logits = logit_scale * (u @ v.T)                                 # (N, N)
    labels = jnp.arange(u.shape[0])
    xe = optax.softmax_cross_entropy_with_integer_labels
    loss_i2t = xe(logits, labels).mean()          # rows
    loss_t2i = xe(logits.T, labels).mean()        # cols
    return 0.5 * (loss_i2t + loss_t2i)

S = jnp.array([[0.9, 0.1, 0.3], [0.2, 0.8, 0.0], [0.4, 0.3, 0.7]])
logits = S / 0.5
lab = jnp.arange(3)
xe = optax.softmax_cross_entropy_with_integer_labels
print(0.5 * (xe(logits, lab).mean() + xe(logits.T, lab).mean()))
# 0.4989190

The ViT patch embedding is the point where the arithmetic of Problem 2 becomes code. A convolution whose kernel size equals its stride is exactly a per-patch linear projection: each non-overlapping \( P \times P \) window is flattened and mapped to width \( d \), producing \( (R/P)^2 \) tokens. A class token is prepended and a learned position table is added. Both frameworks express the same operation; the only real difference is the memory layout, channels-first in PyTorch and channels-last in the Flax convention.

import torch
import torch.nn as nn

class PatchEmbed(nn.Module):
    def __init__(self, img=224, patch=16, in_ch=3, dim=768):
        super().__init__()
        self.n = (img // patch) ** 2                  # 196 patches
        # stride == kernel  =>  each patch is one linear projection to dim
        self.proj = nn.Conv2d(in_ch, dim, kernel_size=patch, stride=patch)
        self.cls = nn.Parameter(torch.zeros(1, 1, dim))
        self.pos = nn.Parameter(torch.zeros(1, self.n + 1, dim))

    def forward(self, x):                 # x: (B, 3, 224, 224)
        x = self.proj(x)                  # (B, 768, 14, 14)
        x = x.flatten(2).transpose(1, 2)  # (B, 196, 768)
        cls = self.cls.expand(x.shape[0], -1, -1)
        x = torch.cat([cls, x], dim=1)    # (B, 197, 768)
        return x + self.pos               # add learned positions

pe = PatchEmbed()
print(pe(torch.randn(2, 3, 224, 224)).shape)   # torch.Size([2, 197, 768])
print(sum(p.numel() for p in pe.parameters())) # 742656 (embed block only)

import jax
import jax.numpy as jnp
import flax.linen as nn

class PatchEmbed(nn.Module):
    patch: int = 16
    dim: int = 768

    @nn.compact
    def __call__(self, x):                # x: (B, 224, 224, 3)  NHWC
        B = x.shape[0]
        x = nn.Conv(self.dim, (self.patch, self.patch),
                    strides=(self.patch, self.patch))(x)   # (B, 14, 14, 768)
        x = x.reshape(B, -1, self.dim)                     # (B, 196, 768)
        cls = self.param('cls', nn.initializers.zeros, (1, 1, self.dim))
        cls = jnp.broadcast_to(cls, (B, 1, self.dim))
        x = jnp.concatenate([cls, x], axis=1)              # (B, 197, 768)
        pos = self.param('pos', nn.initializers.normal(0.02),
                         (1, x.shape[1], self.dim))
        return x + pos

model = PatchEmbed()
params = model.init(jax.random.PRNGKey(0), jnp.zeros((2, 224, 224, 3)))
print(model.apply(params, jnp.zeros((2, 224, 224, 3))).shape)  # (2, 197, 768)

The two numeric demos that follow are self-contained and reproduce the hand arithmetic of Problems 1 and 5, including the SigLIP initialization check and the VQ straight-through estimator. They depend on nothing but NumPy and a few lines of PyTorch, and print the same values worked above.

import numpy as np

S = np.array([[0.9, 0.1, 0.3],
              [0.2, 0.8, 0.0],
              [0.4, 0.3, 0.7]])
logits = S / 0.5                       # tau = 0.5

def ce_rows(z):                        # softmax cross-entropy, diagonal targets
    z = z - z.max(1, keepdims=True)    # numerical stabilization
    p = np.exp(z); p /= p.sum(1, keepdims=True)
    return -np.log(np.diag(p))

li2t = ce_rows(logits).mean()          # 0.5024  image -> text
lt2i = ce_rows(logits.T).mean()        # 0.4954  text  -> image
print(li2t, lt2i, 0.5 * (li2t + lt2i)) # 0.5024 0.4954 0.4989

# SigLIP at init t = 10, b = -10: positives dominate from step one
t, b = 10.0, -10.0
Z = 2 * np.eye(3) - 1                   # +1 on the diagonal, -1 off it
Lsig = -np.log(1 / (1 + np.exp(-(Z * (t * S + b)))))
print(np.diag(Lsig))                   # [1.313 2.127 3.049]  (positives)
print(Lsig[~np.eye(3, dtype=bool)].max())  # 0.00248            (any negative)

import torch

def vq(z_e, codebook, beta=0.25):
    # z_e: (B, d) encoder outputs;  codebook: (K, d) learned codes
    d2 = (z_e.pow(2).sum(1, keepdim=True)          # (B, 1)
          - 2 * z_e @ codebook.t()                 # (B, K)
          + codebook.pow(2).sum(1))                # (K,)  -> squared distances
    k = d2.argmin(1)                               # (B,) nearest code index
    z_q = codebook[k]                              # (B, d) quantized
    codebook_loss = (z_q - z_e.detach()).pow(2).sum(1).mean()
    commit_loss   = (z_e - z_q.detach()).pow(2).sum(1).mean()
    z_q_st = z_e + (z_q - z_e).detach()            # value z_q, gradient 1
    return z_q_st, k, codebook_loss + beta * commit_loss

# codebook perplexity from assignment counts (Problem 5)
counts = torch.tensor([40., 30., 20., 10., 0., 0., 0., 0.])
p = counts / counts.sum()
H = -(p[p > 0] * p[p > 0].log()).sum()
print(H.item(), H.exp().item())        # 1.2799  3.596  (perplexity, max is 8)

How it is done in practice

The gap between the derivations above and a deployed system is mostly scale, data, and the arithmetic of serving. Contrastive pretraining runs at batches of tens of thousands across hundreds of accelerators, and because InfoNCE needs a complete similarity matrix, every step all-gathers all embeddings to every device; the embeddings are small (one vector per example) so the gather is cheap relative to the encoder forward pass, but it is a synchronization point, which is a large part of why SigLIP's chunkable sigmoid loss is attractive at the largest scales. The competitive axis is data, not architecture: CLIP trained on 400M curated pairs, ALIGN on 1.8B noisy ones, and the DataComp benchmark (Gadre et al., 2023) exists precisely to show that filtering and balancing the corpus moves zero-shot accuracy more than encoder changes do.

On the serving side the binding constraint is the one Problem 3 quantifies: visual tokens are expensive because they inflate the KV cache that must be streamed from HBM at every decode step. The H100 in this repository sustains about 2.93 TB/s of bf16 memory bandwidth and, on large matmuls, up to 744.6 bf16 TFLOPS, so a visual encoder forward is compute-cheap (Problem 2 bounded a ViT-B/16 image below 50 microseconds of pure math) while the decoder's per-step cost is bandwidth-bound in the KV cache. That asymmetry is why production VLMs invest in token reduction, patch merging in Qwen2-VL's 2x2 merge, pixel-shuffle downsampling in InternVL, and Q-Former or perceiver compression, rather than in faster encoders. Attention itself is served with fused kernels: the same H100 shows a 27-fold speedup and a 61-fold peak-memory reduction for FlashAttention over a naive score-matrix implementation at sequence length 8192, which is what makes the long visual-plus-text contexts of tiled high-resolution VLMs tractable at all.

The current research frontier

The live disagreement is between bridging and native multimodality. The open bridging line, LLaVA (Wisconsin-Madison and Microsoft), InternVL (Chen et al., 2024, Shanghai AI Laboratory), and the Qwen-VL family (Alibaba), has pushed adapter recipes to strong document and chart understanding by raising resolution and scaling instruction data, and Qwen2-VL's native-resolution ViT with M-RoPE shows the encoder side is not settled either. The native line, publicly described for Gemini (Google DeepMind) and demonstrated openly by Chameleon (Meta, 2024), trains one transformer over interleaved discrete tokens for all modalities, and reports that the naive version is unstable, requiring query-key normalization and reordered layer norms to stop the softmax-logit drift that the temperature analysis on this page predicts. On the objective side, SigLIP's sigmoid loss (Zhai et al., 2023, Google) is displacing softmax InfoNCE where efficiency matters, and the contrastive-versus-generative question is being answered by combining both, as CoCa (Yu et al., 2022) and BLIP-2 (Li et al., 2023, Salesforce) do. Grounding and hallucination remain unsolved: POPE-style probing showed the problem is architectural, rooted in the frozen language prior, and the competing fixes, higher resolution, contrastive decoding, and harder instruction data, all help without closing it. The unified any-to-any direction, one model that both reads and draws, is the most active frontier, and whether it arrives through discrete tokenization (the VQGAN and Chameleon route) or through continuous-latent generation attached to an autoregressive backbone is genuinely open.

Open source to read

• openai/CLIP is the original reference; open clip/model.py to see the learnable logit_scale, its clamp at \( \log(100) \), and the symmetric loss exactly as derived here.
• mlfoundations/open_clip is the open reproduction that trained the checkpoints most systems use; src/open_clip/loss.py holds both the gathered InfoNCE loss and the SigLIP sigmoid loss with its ring exchange.
• google-research/big_vision is the SigLIP and NaViT home; read the SigLIP loss and the patch-packing code for native-resolution training.
• haotian-liu/LLaVA is the canonical projection-connector VLM; open the multimodal projector and the two-stage training script to see how few lines the connector is.
• salesforce/LAVIS contains BLIP-2 and the Q-Former; the Q-Former module shows the three-objective pretraining with its attention masks.
• huggingface/transformers has production implementations of CLIP, SigLIP, BLIP-2, LLaVA, Qwen2-VL, and InternVL side by side, which is the fastest way to compare connectors and position schemes.
• facebookresearch/chameleon is the open native mixed-modal model; read the tokenizer and the query-key-normalized attention that stabilizes joint training.

Common misconceptions

"CLIP's temperature is just a scale factor you can tune away." The gradient shows it is a hard-negative-weighting dial: the \( 1/\tau \) factor multiplies every similarity gradient and controls how sharply the repulsion concentrates on the hardest negatives. It is entangled with the learning rate, it runs away toward zero if unclamped, and CLIP clamps it at 100 for exactly that reason.

"Contrastive training pulls matched image and text embeddings together on the sphere." It pulls them together only relative to mismatched pairs. The softmax is shift-invariant, so a uniform cross-modal offset is invisible to the loss, and trained CLIP models exhibit a persistent modality gap with matched cosines far below 1.

"A bigger contrastive batch always helps." The InfoNCE bound saturates at \( \log N \), and SigLIP's authors found quality plateaus near a batch of 32k. Beyond that, larger batches buy diminishing mutual-information headroom, and the sigmoid loss exists partly because the softmax's all-gather, not the batch size itself, was the real bottleneck.

"More visual tokens are strictly better for a VLM." Tokens are context, and context is KV cache that must be streamed every decode step (Problem 3). A 3x3 AnyRes grid costs 5760 tokens and several gigabytes of cache per image; token-frugal connectors like BLIP-2's 32 queries exist because the marginal token is expensive, not free.

"Hallucination is a data-quantity problem that scale fixes." POPE-style probing localized it in the frozen language prior competing with weak visual evidence, an architectural cause. Scale helps, but resolution, contrastive decoding, and hard-negative instruction data address the mechanism more directly, and none fully closes it.

"The VQ straight-through estimator computes the true gradient of the quantizer." It does not. The quantizer is an \( \argmin \) with zero gradient almost everywhere; the estimator simply copies the decoder gradient past it and is deliberately biased. It works because it points the encoder downhill in reconstruction error, not because it is exact.

Self-check

References

1. Prince, S. J. D. Understanding Deep Learning. MIT Press, 2023. udlbook.github.io.
2. Bishop, C. M., and Bishop, H. Deep Learning: Foundations and Concepts. Springer, 2024. bishopbook.com.
3. van den Oord, A., Li, Y., and Vinyals, O. Representation Learning with Contrastive Predictive Coding. 2018. arXiv:1807.03748.
4. Radford, A., Kim, J. W., Hallacy, C., et al. Learning Transferable Visual Models From Natural Language Supervision (CLIP). 2021. arXiv:2103.00020.
5. Jia, C., Yang, Y., Xia, Y., et al. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (ALIGN). 2021. arXiv:2102.05918.
6. Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid Loss for Language Image Pre-Training (SigLIP). 2023. arXiv:2303.15343.
7. Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT). 2020. arXiv:2010.11929.
8. Jaegle, A., Gimeno, F., Brock, A., et al. Perceiver: General Perception with Iterative Attention. 2021. arXiv:2103.03206.
9. Alayrac, J.-B., Donahue, J., Luc, P., et al. Flamingo: a Visual Language Model for Few-Shot Learning. 2022. arXiv:2204.14198.
10. Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. 2023. arXiv:2301.12597.
11. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual Instruction Tuning (LLaVA). 2023. arXiv:2304.08485.
12. Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved Baselines with Visual Instruction Tuning (LLaVA-1.5). 2023. arXiv:2310.03744.
13. Bai, J., Bai, S., Yang, S., et al. Qwen-VL: A Versatile Vision-Language Model. 2023. arXiv:2308.12966.
14. Wang, P., Bai, S., Tan, S., et al. Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. 2024. arXiv:2409.12191.
15. Chen, Z., Wu, J., Wang, W., et al. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. 2024. arXiv:2312.14238.
16. Chameleon Team (Meta AI). Chameleon: Mixed-Modal Early-Fusion Foundation Models. 2024. arXiv:2405.09818.
17. van den Oord, A., Vinyals, O., and Kavukcuoglu, K. Neural Discrete Representation Learning (VQ-VAE). 2017. arXiv:1711.00937.
18. Esser, P., Rombach, R., and Ommer, B. Taming Transformers for High-Resolution Image Synthesis (VQGAN). 2021. arXiv:2012.09841.
19. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-Resolution Image Synthesis with Latent Diffusion Models. 2022. arXiv:2112.10752.
20. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical Text-Conditional Image Generation with CLIP Latents (unCLIP / DALL-E 2). 2022. arXiv:2204.06125.
21. Yu, J., Wang, Z., Vasudevan, V., et al. CoCa: Contrastive Captioners are Image-Text Foundation Models. 2022. arXiv:2205.01917.
22. Liang, W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. 2022. arXiv:2203.02053.
23. Dehghani, M., Mustafa, B., Djolonga, J., et al. Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution. 2023. arXiv:2307.06304.
24. Arnab, A., Dehghani, M., Heigold, G., et al. ViViT: A Video Vision Transformer. 2021. arXiv:2103.15691.
25. Radford, A., Kim, J. W., Xu, T., et al. Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). 2022. arXiv:2212.04356.
26. Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. SoundStream: An End-to-End Neural Audio Codec. 2021. arXiv:2107.03312.
27. Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, K. Object Hallucination in Image Captioning (CHAIR). 2018. arXiv:1809.02156.
28. Li, Y., Du, Y., Zhou, K., et al. Evaluating Object Hallucination in Large Vision-Language Models (POPE). 2023. arXiv:2305.10355.
29. Gadre, S. Y., Ilharco, G., Fang, A., et al. DataComp: In search of the next generation of multimodal datasets. 2023. arXiv:2304.14108.