A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Learning to Predict Optimal ViewpointActive perception in vision-based robotic manipulation aims to move the camera toward more informative observation viewpoints, thereby providing high-quality perceptual inputs for downstream tasks. Most existing active perception methods rely on iterative optimization, leading to high time and motion costs, and are tightly coupled with task-specific objectives, which limits their transferability. In this paper, we propose a general one-shot multimodal active perception framework for robotic manipulation. The framework enables direct inference of optimal viewpoints and comprises a data collection pipeline and an optimal viewpoint prediction network. Specifically, the framework decouples viewpoint quality evaluation from the overall architecture, supporting heterogeneous task requirements. Optimal viewpoints are defined through systematic sampling and evaluation of candidate viewpoints, after which large-scale training datasets are constructed via domain randomization. Moreover, a multimodal optimal viewpoint prediction network is developed, leveraging cross-attention to align and fuse multimodal features and directly predict camera pose adjustments. The proposed framework is instantiated in robotic grasping under viewpoint-constrained environments. Experimental results demonstrate that active perception guided by the framework significantly improves grasp success rates. Notably, real-world evaluations achieve nearly double the grasp success rate and enable seamless sim-to-real transfer without additional fine-tuning, demonstrating the effectiveness of the proposed framework.
We propose a data-driven multimodal active perception framework that directly predicts the optimal observation viewpoint, enabling improved perception with only a single relook, and generalizes across different tasks. The main contributions of our work are summarized as follows:
Overall framework of the proposed method, illustrated with robotic grasping in viewpoint-constrained environments: (a) sampling and evaluating candidate viewpoints to obtain the optimal viewpoint for each object, followed by dataset construction via domain randomization; (b) training the MVPNet based on the constructed dataset; and (c) deploying the trained network and conducting comparative evaluations.
First, the current observation is obtained and preprocessed together with the natural language description of the target object. Subsequently, modality-specific encoders are employed to extract features, which are then aligned and fused using a Transformer. Finally, an MLP maps the fused representation to the camera pose adjustments.
......