Shuangyu Xie

PostDoc @ UCB, Ph.D. @ TAMU

photo.jpg
email google scholar linkedin x / twitter

I am a Postdoctoral researcher in the AUTOLab at UC Berkeley, working with Prof. Ken Goldberg. Prior to joining UC Berkeley, I obtained my Ph.D. and M.S. degree from Texas A&M University, advised by Prof. Dezhen Song. My research aims to make robots more capable and resilient in unstructured environments, spanning applications in manipulation, mobile manipulation and agricultural robotics.

Research Highlights

My recent work explores how Vision-Language Models and robotics can advance each other: real robot experience provides the grounded, embodied data that VLMs need, while VLMs bring the open-world semantic understanding that robots need to perceive, reason, and act. I pursue this along three threads:

  • Learning VLMs from robot experience. Robo2VLM (NeurIPS 2025, Spotlight) turns large-scale, in-the-wild robot manipulation datasets into grounded visual question-answering data, using the robot’s own sensing and kinematics to supervise VLM training and evaluation.
  • Evaluating VLMs for robotics. RoboVista (RSS 2026) is the first robot-at-work benchmark that evaluates VLMs across diverse real-world robot deployments—manipulation, driving, and surgical scenes—with a public leaderboard.
  • Understanding and curating robot data with VLMs. RoboSQ (ICRA 2026) is a semantic query engine that uses natural language to curate high-quality, task-aligned training data from large robot datasets, and RoboSegNet (CVPR 2026) fuses vision and kinematics to segment robot trajectories into meaningful sub-tasks, zero-shot.

News

Apr 20, 2026 RoboVista has been accepted to RSS 2026 — the leaderboard is now online, and the benchmark is available on Hugging Face!
Feb 26, 2026 RoboSegNet has been accepted to CVPR 2026!
Jan 29, 2026 RoboSQ has been accepted to ICRA 2026!
Sep 20, 2025 Robo2VLM has been accepted to NeurIPS 2025 as a Spotlight — the dataset has 29k+ downloads on Hugging Face!

Selected Publications

* indicates co-first author

  1. robovista.png
    RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
    Shuangyu Xie*, Kaiyuan Chen*, Ziyang Chen, Simeon Adebola, Yixuan Huang, Zehan Ma, Tianshuang Qiu, Wentao Yuan, Dhruv Shah, Pannag R. Sanketi, and Ken Goldberg
    In Robotics: Science and Systems (RSS), 2026
    Website
  2. robosegnet.png
    Learning Multi-Task Robot Trajectory Segmentation from Visual and Kinematic Streams
    Kaiyuan Chen*, Shuangyu Xie*, Andrew Goldberg, and Ken Goldberg
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026
    Website
  3. robosq.png
    RoboSQ: Semantic Queries for Task-Aligned Robot Training Data
    Kaiyuan Chen*, Shuangyu Xie*, Kush Hari, Andrew Goldberg, Kavish Kondap, and Ken Goldberg
    In IEEE International Conference on Robotics and Automation (ICRA), 2026
    Website
  4. weed_cluster.png
    Energy Efficient Planning for Repetitive Heterogeneous Tasks in Precision Agriculture
    Shuangyu Xie, Ken Goldberg, and Dezhen Song
    In IEEE International Conference on Robotics and Automation (ICRA), 2025
  5. robo2vlm.png
    Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets
    Kaiyuan Chen*, Shuangyu Xie*, Zehan Ma, and Ken Goldberg
    In Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025
    Spotlight
  6. botany-bot.png
    Botanybot : Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats
    Simeon Adebola, Chung Min Kim, Justin Kerr, Shuangyu Xie, Prithvi Akella, Jose Luis Susa Rincon, Eugen Solowjow, and Ken Goldberg
    In IROS, 2025
  7. capm.png
    Coupled Active Perception and Manipulation Planning for a Mobile Manipulator in Precision Agriculture Applications
    Shuangyu Xie, Chengsong Hu, Di Wang, Joe Johnson,  Muthukumar Bagavathiannan, and Dezhen Song
    International Conference on Robotics and Automation (ICRA), 2024
  8. flamer_robot_platform.png
    System and Algorithm for Robotic Weed Flaming Using a Quadruped Robot and a 6-Axis Manipulator
    Di Wang, Chengsong Hu, Shuangyu Xie, Joe Johnson, Hojun Ji, Yingtao Jiang, Muthukumar Bagavathiannan, and Dezhen Song
    International Conference on Intelligent Robots and Systems (IROS), 2024
  9. Dimenet.png
    Dynamic Intrinsic Parameter Rectification Network for Cameras with Optical Image Stabilization in Desktop Augmented Reality Applications
    Shuangyu Xie, Di Wang, Shu-Hao Yeh, Wei Yan, and Dezhen Song
    IEEE International Conference on Automation Science and Engineering (CASE), 2024
  10. pretouch.png
    A Pretouch Perception Algorithm for Object Material and Structure Mapping to Assist Grasp and Manipulation Using a DMDSM Sensor
    Fengzhi Guo, Shuangyu Xie,  Di Wang,  Cheng Fang,  Jun Zou, and  Dezhen Song
    International Conference on Intelligent Robots and Systems (IROS), 2023
  11. weed_spray.png
    Algorithm and System Development for Robotic Micro-Volume Herbicide Spray Towards Precision Weed Management
    Chengsong Hu, Shuangyu Xie, Dezhen Song, J. Alex Thomasson, Robert G. Hardin IV, and Muthukumar Bagavathiannan
    IEEE Robotics and Automation Letters, 2022
  12. weed_detection.png
    Toward Robotic Weed Control: Detection of Nutsedge Weed in Bermudagrass Turf Using Inaccurate and Insufficient Training Data
    Shuangyu Xie, Chengsong Hu, Muthukumar Bagavathiannan, and Dezhen Song
    IEEE Robotics and Automation Letters, 2021