MVPNet

Abstract

Active perception in vision-based robotic manipulation aims to move the camera toward more informative observation viewpoints, thereby providing high-quality perceptual inputs for downstream tasks. Most existing active perception methods rely on iterative optimization, leading to high time and motion costs, and are often tightly coupled with specific task formulations, which limits their transferability. In this paper, we propose a general one-shot multimodal active perception framework for robotic manipulation. The framework comprises a dataset construction pipeline and an optimal viewpoint prediction network. First, task-oriented optimal viewpoints are obtained through systematic candidate-view sampling, task-adaptable viewpoint quality evaluation, and multi-mode clustering of high-quality viewpoint candidates. Large-scale training datasets are constructed via domain randomization using the resulting viewpoint labels. Subsequently, an optimal viewpoint prediction network is developed, which leverages multimodal self-attention to align and fuse visual and geometric features, directly predicting the required camera pose adjustments. Experiments in both simulation and real-world scenarios show that MVPNet achieves better robotic grasping performance than state-of-the-art active perception baselines under the same single-viewpoint-adjustment setting. The same framework is further extended to barcode recognition and object-centric view quality optimization, validating its generality across heterogeneous perception objectives.

Main Contribution

We propose a data-driven multimodal active perception framework that directly predicts the task-oriented optimal observation viewpoint, enabling improved perception with only a single viewpoint adjustment and generalization across tasks. The main contributions of our work are summarized as follows:

A general one-shot multimodal active perception framework is proposed, comprising a dataset construction pipeline and an optimal viewpoint prediction network. This framework enables the unified modeling of diverse task requirements, thereby extending its applicability to a broader range of task scenarios.
A dataset construction pipeline for task-oriented optimal observation viewpoints is established, in which optimal viewpoints are defined through task-specific viewpoint quality evaluation functions. Using the resulting viewpoint labels, large-scale datasets are constructed via domain randomization.
An optimal observation viewpoint prediction network is developed. By applying self-attention across the concatenated multimodal features, this network effectively aligns and fuses 2D visual and 3D geometric information to predict the required camera pose adjustment.
The proposed framework is instantiated in viewpoint-constrained robotic grasping and evaluated against state-of-the-art active perception methods, demonstrating its effectiveness in improving downstream task performance. It is further extended to barcode recognition and object-centric view quality optimization, validating its generality across heterogeneous perception objectives.

Framework Overview

Overall framework of the proposed method, illustrated with robotic grasping in viewpoint-constrained environments: (a) sampling and evaluating candidate viewpoints to obtain the optimal viewpoint for each object, followed by dataset construction via domain randomization; (b) training MVPNet based on the constructed dataset; and (c) deploying the trained network and conducting comparative evaluations.

Network Architecture

First, the current observation is obtained and preprocessed together with the natural language description of the target object. Subsequently, modality-specific encoders are employed to extract features, which are then aligned and fused using a Transformer. Finally, an MLP maps the fused representation to the camera pose adjustments.

Synthetic Dataset Construction

Simulated Experiments

Real-World Evaluation for Robotic Grasping

Generality Evaluation Across Perception Objectives

BibTeX

@article{qin2026general,
                title={A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Learning to Predict Optimal Viewpoint},
                author={Qin, Deyun and Liu, Zezhi and Luo, Hanqian and Liang, Xiao and Fang, Yongchun},
                journal={arXiv preprint arXiv:2601.13639},
                year={2026}
              }

A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Task-Oriented Optimal Viewpoint Prediction

The "Focus-then-Execute" active perception paradigm.