NeurIPS 2026 · Evaluations & Datasets Track

OpenView: Empowering MLLMs with
Out-of-view VQA

Qixiang Chen1, Cheng Zhang1, Chi-Wing Fu2, Jingwen Ye1, Jianfei Cai1

1Monash University    2The Chinese University of Hong Kong

Examples of contextual and directional out-of-view questions, and model performance before and after OpenView training

Examples of out-of-view VQA. OpenView synthesizes multi-choice out-of-view VQA data, featuring both contextual and directional question types. Empowered with OpenView, several MLLMs can largely improve their performance on out-of-view VQA, making their capability closer to human performance.

Abstract

Recent multimodal large language models (MLLMs) show great potential in natural image understanding. Yet, they perform well, mainly on reasoning in-view contents within the image frame. This paper presents the first study on out-of-view (OOV) understanding, i.e., the ability to reason about objects, activities, and scenes beyond the visible frame of a perspective view. Our technical contributions are threefold. First, we design OpenView, a four-stage pipeline that leverages panoramic imagery for large-scale and diverse multi-choice VQA synthesis, where full-scene coverage enables flexible view framing while preserving global scene awareness. Second, we curate OpenView-Dataset, a high-quality synthetic dataset from diverse real-world panoramas to empower MLLMs upon supervised fine-tuning. Third, we build OpenView-Bench, a benchmark that jointly measures choice and rationale accuracy for interpretable and diagnosable evaluation. Experimental results show that despite having a large gap from human performance in OOV VQA answer selection, multiple MLLMs, upon empowered by OpenView, can consistently boost their performance, uplifted from 48.3% to 62.3% on average.

Formulation

Out-of-view (OOV) understanding

We introduce a new MLLM understanding task: reasoning about objects, activities, and scenes beyond the visible frame of a perspective view. It requires models to integrate visible cues, such as scene structure, camera viewpoint, and object appearance, with contextual priors and world knowledge. Since multiple hypotheses about unseen regions may be consistent with the observed image, we formulate it as single-view multi-choice VQA, i.e., comparative plausibility reasoning among a small set of candidate hypotheses.

Perspective view of a museum exhibit about tsunamis
Input: a single perspective view
An example from OpenView-Dataset; rationales abridged.
Contextual · Culture scene
Which of the following sets of objects would most likely appear outside of this view?
  1. A
    Interactive touch screens and additional infographic panels. Rationale:Highly plausible in this educational museum setting, complementing the existing display format.
    candidate
  2. B
    A large aquarium with marine life displays. Rationale:Could relate to the tsunami theme, but seems too large and unrelated to the exhibit style.
    candidate
  3. C
    Artisanal souvenirs and gift shop merchandise. Rationale:Plausible, but less consistent next to an educational exhibit than more educational material.
    candidate
  4. D
    Sports equipment like basketball hoops and soccer nets. Rationale:Lacks connection to the educational theme about natural disasters.
    candidate
  5. E
    None of the above. Rationale:Unlikely, since A presents a strong case.
    compositional

Every option comes with a rationale. Models are scored jointly on the selected answer and its rationale: choice, rationale, and joint accuracy.

Questions come in two types

Contextual questions

Assess whether objects, attributes, or conditions are plausible beyond the visible region.

“Which type of feature would be least plausible to find outside the current view?”

Directional questions

Emphasize spatial continuity under explicit changes in camera view direction.

“After tilting the camera down about 40°, what would most likely appear in view?”
Interactive Demo

Can you reason beyond the frame?

Like the models, you see only a single perspective view. Pick a scene and answer its question. The view then expands into the full 360° panorama so you can check your answer.

This crop is all the model sees.
Rationale & full panorama
Full panorama

Full 360° panorama. The red frame marks the perspective view you were shown.

Method

The OpenView pipeline

Cropping subviews from ordinary images leaves question design without the full scene context. Panoramas instead provide full 180°×360° views of the target environments, enabling flexible view selection while preserving global scene awareness. OpenView is an automatic four-stage pipeline: a data preprocessor for panorama filtering, a visual analyzer for semantic and spatial annotation, a proposal generator for constructing VQAs, and a proposal refiner for quality assurance and augmentation.

OpenView four-stage pipeline
Dataset & Benchmark

OpenView-Dataset

Generated with the OpenView pipeline, OpenView-Dataset contains over 158k realistic multi-choice OOV VQAs from 16k filtered panoramas, spanning a broad range of real-world scenarios.

158k
synthetic OOV VQAs
16k
filtered panoramas
11
scene categories
5
public panorama sources
Distribution of the first four words of questions

Question patterns exhibit high linguistic consistency, guiding the model to focus on OOV understanding.

Scene category distribution across indoor and outdoor environments

Scene category distribution across indoor and outdoor environments, showing broad coverage of real-world scenarios.

Word cloud of options

The option word cloud reveals diverse object and scene terms.

Word cloud of rationales

The rationale word cloud highlights recurring reasoning phrases.

Word length distribution of questions Word length distribution of options Word length distribution of rationales

Word length distributions for questions, options, and rationales.

OpenView-Bench

A manually-verified benchmark curated from panoramas explicitly excluded from training, comprising 1,155 high-quality OOV VQAs with balanced task difficulty, scene coverage and answer distribution.

1,155
manually verified VQAs
77
panoramas, 7 per
scene category
580 / 575
contextual / directional
86.0%
human choice accuracy
(Fleiss' κ = 0.82)
Distribution of scene categories across VQAs in OpenView-Bench

Distribution of scene categories across VQAs in OpenView-Bench.

Distribution of correct answers across the five options

Answer distribution across the five options.

Results

Evaluation on OpenView-Bench

A substantial gap is evident between humans and MLLMs. While humans surpass 83% choice accuracy across both task types, model accuracies still fall short of this level. Considering rationale evaluation reveals a lower joint accuracy, particularly in directional tasks.

Choice accuracy

The proportion of correctly-selected options among all the questions.

Rationale accuracy

The reliability of the model in justifying the correctly-chosen answers through reasonable option-wise explanations, judged by GPT5-mini against human-revised rationales.

Joint accuracy

The proportion of questions for which both the selected answer and its rationale are correct.

ModelRank ContextualDirectionalOverall
ChoiceRationaleJoint ChoiceRationaleJoint ChoiceRationaleJoint
Human-level–83.02––89.58––86.00––
GPT-4o†–64.14––41.74––52.99––
GPT-5†–63.79––44.52––54.20––
Proprietary models
GPT-4o-mini963.1093.9959.3148.3587.0542.0955.7690.9950.74
GPT-4o671.7283.8960.1755.8389.4149.9263.8186.3055.07
Gemini-3-flash877.5978.8961.2162.4370.1943.8270.0475.0352.55
GPT-5270.8691.7365.0057.0495.4354.4363.9893.3759.74
GPT-5.4170.5299.0269.8359.3096.4857.2164.9497.8763.56
Open-source models
LLaVA-NeXT-Mistral-7B1742.0739.3416.5529.0429.948.6935.5835.5212.64
GLM-4.1V-9B-Base1456.3887.4649.3142.7879.1033.8449.6183.8941.62
GLM-4.1V-9B-Thinking1152.4192.1148.2745.0490.7340.8648.7491.4744.58
InternVL3.5-8B1256.9090.0051.2145.3979.6936.1751.1785.4543.72
InternVL3.5-8B‡1549.4882.5840.8642.2685.6036.1745.8983.9638.53
Qwen3-VL-8B-Instruct1058.1094.9655.1748.3590.6543.8353.2593.0149.53
Qwen3-VL-8B-Thinking767.4191.0561.3849.5791.2345.2258.5391.1253.33
Ovis2.5-9B1666.2165.8943.6343.6566.1428.8754.9865.9836.28
MiniCPM-V 4.5368.7993.2364.1357.0489.3350.9562.9491.4757.57
Qwen2.5-VL-3B-Instruct1857.9313.998.1044.7012.065.3951.3413.156.75
Qwen2.5-VL-7B-Instruct1361.0381.3649.6545.3976.6334.7853.2579.3542.25
Qwen2.5-VL-32B-Instruct466.3895.5863.4553.2294.1250.0959.8394.9356.80
Qwen2.5-VL-72B-Instruct568.1095.7065.1751.1392.5247.3159.6594.3456.27

Results are reported as percentages, with models ranked by the overall joint accuracy. The best results among all models and within open-source models are highlighted in red and light red, respectively. †Description-based test setting. ‡Thinking mode enabled.

What makes OOV difficult?

We analyze the responses of six representative models from different performance tiers and categorize the errors into two primary types.

Breakdown of error cases across models
Left: Breakdown of error cases across models.
An example contextual test case with model answers
Right: An example test case. Baseline MLLMs misinterpret the scene, while fine-tuned models correctly infer the unseen activity.

Perception limitation

Missed evidence, where key objects or cues present in the frame are overlooked, and hallucination, where models generate descriptions inconsistent with the actual image content.

Reasoning limitation

The dominant failure mode, accounting for approximately 70% of all errors across models: option analysis errors in assessing each candidate against the visual context, and logic errors where final selections contradict the model's own preceding analysis.

OpenView-Dataset guides MLLMs with consistent improvements

We fine-tune four open-source MLLMs on OpenView-Dataset. Supervised fine-tuning yields substantial gains across all models; notably, the overall joint accuracy of LLaVA-NeXT-Mistral-7B increases from 12.64% to 54.63%. After fine-tuning, Qwen3-VL-8B-Instruct achieves the strongest overall OOV performance among open-source models.

Model ContextualDirectionalOverall
ChoiceRationaleJoint ChoiceRationaleJoint ChoiceRationaleJoint% gain
LLaVA-NeXT-Mistral-7B42.0739.3416.5529.0429.948.6935.5835.5212.64↑ 332%
+ OpenView-Dataset71.5590.8465.0049.5789.1244.1860.6190.1454.63
InternVL3.5-8B56.9090.0051.2145.3979.6936.1751.1785.4543.72↑ 18%
+ OpenView-Dataset66.0390.8659.9948.0089.1342.7857.0690.1451.43
Qwen2.5-VL-7B-Instruct61.0381.3649.6545.3976.6334.7853.2579.3542.25↑ 39%
+ OpenView-Dataset72.7693.3667.9356.0087.8949.2264.4290.9958.62
Qwen3-VL-8B-Instruct58.1094.9655.1748.3590.6543.8353.2593.0149.53↑ 24%
+ OpenView-Dataset77.0793.2971.9057.2289.3651.1367.1991.6261.56

The final column reports the relative improvement in overall joint accuracy. Gains are highlighted in blue and minor rationale drops in gray.

Cross-paradigm generalization to MindCube

Training on single-view plausibility comparison also improves multi-view 3D spatial reasoning on MindCube, indicating that the visual reasoning skills induced by OpenView-Dataset are transferable beyond its training paradigm.

ModelOverallRotationAmongAround
Qwen2.5-VL-7B-Instruct31.0237.2831.0627.02
+ OpenView-Dataset37.7339.5038.6527.72
Δ+6.71+2.22+7.59+0.70
Qwen3-VL-8B-Instruct29.2335.2529.4823.27
+ OpenView-Dataset41.5035.6243.0529.86
Δ+12.27+0.37+13.57+6.59

Performance on the MindCube benchmark. Fine-tuning on OpenView-Dataset improves performance across all question types.

Application

Informative descriptions for panorama outpainting

We prompt Qwen3-VL-8B-Instruct and its fine-tuned variant to describe the conditioned image and iteratively infer adjacent views by rotating 45° to the right, forming a closed viewing loop. The resulting multi-view descriptions condition image&text-conditioned panorama generation with MVDiffusion. Drag the slider to compare the two.

Outpainting by base model
Outpainting by fine-tuned model
⇔
+ OpenView-Dataset
Base model

The base model often repeats references to objects from the conditioned view, whereas the fine-tuned model produces more informative and spatially coherent descriptions by introducing plausible unseen objects, resulting in more complete scene interpretations.

BibTeX

If you find OpenView useful or relevant to your research, please consider citing our paper 😊

@article{chen2025openview,
  title={OpenView: Empowering MLLMs with Out-of-view VQA},
  author={Chen, Qixiang and Zhang, Cheng and Fu, Chi-Wing and Ye, Jingwen and Cai, Jianfei},
  journal={arXiv preprint arXiv:2512.18563},
  year={2025}
}