SenseWeather: Multimodal Sensing for Robust Robotic Perception under Extreme Conditions
Fucai Ke, Sukai Huang, Junyi Hao, Kal Backman, Naim Rastgoo, Gholamreza Haffari, Hamid Rezatofighi
Arxiv · 2026
PDFResearch
Pushing the boundaries of how machines perceive and interact with the world.
2D Visual Perception
This research area focuses on 2D visual understanding from images and video. It includes object detection, semantic and panoptic segmentation, human/object pose estimation, multi-object tracking, optical flow, and depth estimation.
- SenseWeather: Multimodal Sensing for Robust Robotic Perception under Extreme Conditions · Arxiv
- DPL: Decoupled Prototype Learning for Enhancing Robustness of Vision-Language Transformers to Missing Modalities · IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- AerOSeg++: Scale-Aware and Texture-Guided Open-Vocabulary Segmentation with SAM Features for Remote Sensing Images · ACM Transactions on Multimedia Computing, Communications and Applications
- dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3 · Arxiv
- Marginalized Generalized IoU (MGIoU): A Unified Objective Function for Optimizing Any Convex Parametric Shapes · Annual Conference on Artificial Intelligence (AAAI)
3D Vision and Robotics Perception
This research area studies 3D scene understanding and robotic perception from images, videos, and point clouds. It includes SLAM, NeRF, 3D Gaussian splatting, dynamic scene reconstruction, point cloud reasoning, and 3D detection and mapping for robotics.
- Feedforward Novel View Synthesis for Heterogeneous Cameras · Advances in Neural Information Processing Systems (NeurIPS)
- Hier-SLAM++: Neuro-Symbolic Semantic SLAM with a Hierarchically Categorical Gaussian Splatting · Asian Conference on Computer Vision (ACCV)
- CalibAnyView: Beyond Single-View Camera Calibration in the Wild · Arxiv
- Asset-Driven Sematic Reconstruction of Dynamic Scene with Multi-Human-Object Interactions · Arxiv
- Hier-SLAM: Scaling-up Semantics in SLAM with a Hierarchically Categorical Gaussian Splatting · IEEE International Conference on Robotics and Automation (ICRA)
Reasoning
This area develops models and systems that interpret visual information through structured, compositional, multi-step, and tool-aware reasoning. It includes visual reasoning, visual grounding, neuro-symbolic reasoning, long-video understanding, and benchmark construction for evaluating reasoning in realistic robotic settings.
- JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments · Advances in Neural Information Processing Systems (NeurIPS)
- Seek-and-View Reasoning for Multi-View Spatial Understanding · Arxiv
- VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations · European Conference On Computer Vision (ECCV)
- VQ-VA World: Towards High-Quality Visual Question-Visual Answering · IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- An Empirical Study on How Video-LLMs Answer Video Questions · IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Planning
This line of work studies decision-making and action selection for autonomous agents operating in complex environments. It includes active mapping and planning, goal-directed navigation, multi-agent and multi-UAV planning, and planning under uncertainty for search, tracking, and robotic autonomy.
- What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities · Arxiv
- PING: A Physics-Informed Neuro-Symbolic Generator for Continuous-Time Planning · International Conference on Automated Planning and Scheduling (ICAPS)
- A Probabilistic Framework for Hierarchical Goal Recognition · International Conference on Principles of Knowledge Representation and Reasoning (KR)
- Neurosymbolic Active Goal Recognition in Partially Observable Environments · International Conference on Autonomous Agents and Multiagent Systems (AAMAS, Extended Abstract)
- Probabilistic Active Goal Recognition · International Conference on Principles of Knowledge Representation and Reasoning (KR)
Robotics & Embodied Systems
This research area targets integrated robotic and embodied AI: full systems that combine perception, reasoning, and planning on physical platforms—UAV and field robots, search-and-track missions, and human-facing deployment. It emphasizes social and in-the-wild human–robot interaction, aerial and conservation robotics, neuro-symbolic mission-level autonomy, and robotics-oriented datasets (pose, activity, multimodal sensing).
- JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments · Advances in Neural Information Processing Systems (NeurIPS)
- ARIS: Agentic and Relationship Intelligence System for Social Robots · Advances in Neural Information Processing Systems (NeurIPS, Workshop)
- Mini-BEHAVIOR-Gran: Revealing U-Shaped Effects of Instruction Granularity on Language-Guided Embodied Agents · The Conference on Empirical Methods in Natural Language Processing (EMNLP)
- SenseWeather: Multimodal Sensing for Robust Robotic Perception under Extreme Conditions · Arxiv
- JRDB-Pose3D: A Multi-person 3D Human Pose and Shape Estimation Dataset for Robotics · Arxiv
Efficient AI & Autonomy
This research area develops efficient multimodal AI, distributed inference, and agentic and multi-agent architectures for real-time perception, reasoning, decision-making, and adaptive autonomy on resource-constrained robotic platforms.
Distributed AI Systems
This research area designs distributed systems that orchestrate AI workloads, models, agents, and resources across robots, edge devices, and cloud infrastructure, enabling scalable, collaborative, resilient, and energy-aware robotic operations.
Safe & Trustworthy AI
This research area makes foundation models and embodied agents safer, more reliable, and more accountable. It includes machine unlearning and concept erasure, privacy-preserving learning, model and data protection, hallucination diagnosis in vision-language models, and safety of vision-language-action models for robots that work around people.
Other ML Topics
It includes broader representation learning, optimization, multimodal learning, and adjacent ML topics that still support the lab's wider research agenda.
- Sample-Efficient Learning from Agent Experience · Arxiv
- A Simple State Space Model Excels at Multivariate Time Series Classification · Arxiv
- ECORE: Energy-Conscious Optimized Routing for Deep Learning Models at the Edge · IEEE International Conference on Pervasive Computing and Communications (PerCom)
- Deep learning classification integrating embryo images with associated clinical information from ART cycles · Scientific Reports
- Acoustic-to-Hyper-Spectral: Hyper-Spectral Image Construction from Frequency Spectrums through Simulated Annealing (Student Abstract) · Proceedings of the AAAI Conference on Artificial Intelligence
Community Impact
Datasets & Benchmarks
Dataset Benchmark
JRDB: JackRabbot Dataset and Benchmark
JRDB is one of VL4AI's flagship robotic perception benchmarks. It anchors later work on pose estimation, trajectory forecasting, social understanding, and reasoning in crowded human environments.
Dataset Benchmark
MOT20: A Benchmark for Multi-object Tracking in Crowded Scenes
MOT20 is the benchmark anchor for VL4AI's crowded-scene multi-object tracking work.
Dataset Benchmark
Social Motion Forecasting Benchmark
The SoMoF benchmark grouped VL4AI's trajectory and pose forecasting work around a public evaluation target.
Dataset Benchmark
Completion3D: Stanford 3D Point Cloud Completion Benchmark
Completion3D is a benchmark reference for structural point-cloud completion work such as TopNet.
Dataset Benchmark
LISC: Leukocyte Images for Segmentation and Classification
The LISC database contains hematological images taken from peripheral blood of healthy subjects. The database was released to enable comparative evaluation of nucleus and cytoplasm segmentation methods as well as recognition of different white blood cells in hematological images.
Archived Dataset Link
