About

Chuang Gan

Principal Research Staff Member

Website Google Scholar

Who they work with

Categories

Computer Vision Graph Deep Learning Multimodal Learning Neuro-Symbolic AI

I am a principal research staff member at MIT-IBM Watson AI Lab. I am also a visiting research scientist at MIT, working closely with Prof. Antonio Torralba and Prof. Josh Tenenbaum. Before that, I completed my PhD with the highest honor at Tsinghua University, where I was supervised by Prof. Andrew Chi-Chih Yao. I primarily focus on video understanding, including representation learning, neural-symbolic visual reasoning, audio-visual scene analysis and embodied intelligence. My research work have been recognized by Microsoft Fellowship, Baidu Fellowship, and media coverage from CNN, BBC, The New York Times, WIRED, Forbes, and MIT Tech Review.

Publications with the MIT-IBM Watson AI Lab

Masked Motion Encoding for Self-Supervised Video Representation Learning

CVPR

EC^2 : Emergent Communication for Embodied Control

CVPR

Learning Situation Hyper-Graphs for Video Question Answering

CVPR

Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos

CVPR

Mod-Squad: Designing Mixtures of Experts As Modular Multi-Task Learners

CVPR

3D Concept Learning and Reasoning from Multi-View Images

CVPR

Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention

CVPR

PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification

ICLR

FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation

ICLR

SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments

ICLR

DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics

ICLR

Hyper-Decision Transformer for Efficient Online Policy Adaptation

ICLR

Planning with Large Language Models for Code Generation

ICLR

3D Concept Grounding on Neural Fields

NeurIPS

Learning Neural Acoustic Fields

NeurIPS

Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation

NeurIPS

Learning Active Camera for Multi-Object Navigation

NeurIPS

Learning Physical Dynamics with Subequivariant Graph Neural Networks

NeurIPS

On-Device Training Under 256KB Memory

NeurIPS

SNAKE: Shape-aware Neural 3D Keypoint Field

NeurIPS

Music Gesture for Visual Sound Separation

Prompting Decision Transformer for Few-shot Policy Generalization

ICML

Finding Fallen Objects Via Asynchronous Audio-Visual Integration

CVPR

Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction

CVPR

AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation

CVPR

Network Augmentation for Tiny Deep Learning

ICLR

ComPhy: Compositional Physical Reasoning of Objects and Events from Videos

ICLR

RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation

ICLR

Linking Emergent and Natural Languages via Corpus Transfer

ICLR

Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics

ICLR

DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools

ICLR

Temporal and Object Quantification Networks

IJCAI

STAR: A Benchmark for Situated Reasoning in Real-World Videos

NeurIPS

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

NeurIPS

Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language

NeurIPS

Memory-efficient Patch-based Inference for Tiny Deep Learning

NeurIPS

PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning

NeurIPS

When does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

NeurIPS

Curious Representation Learning for Embodied Intelligence

ICCV

FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations

ICLR

HAT: Hardware-Aware Transformers for Efficient Natural Language Processing

Computation and Language Machine Learning

On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learning

ICLR Machine Learning

Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules

CVPR Computer Vision

PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics

ICLR Machine Learning

Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning

ICLR Computer Vision

Learning Task Decomposition with Ordered Memory Policy Network

ICLR

RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning.

AAAI Computer Vision

Augmenting Policy Learning with Routines Discovered from a Single Demonstration

AAAI Machine Learning

MVFNet: Multi-View Fusion Network for Efficient Video Recognition

AAAI Computer Vision

AGENT: A Benchmark for Core Psychological Reasoning

ICML Artificial Intelligence

Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning

EMNLP Computation and Language

Adversarial Option-Aware Hierarchical Imitation Learning

ICML Machine Learning

Tiny Transfer Learning: Towards Memory-Efficient On-Device Learning

Efficient AI

MCUNet: Tiny Deep Learning on IoT Devices

Deep Learning Efficient AI

Foley Music: Learning to Generate Music from Videos

ECCV Computer Vision

DataMix: Efficient Privacy-Preserving Edge-Cloud Inference

ECCV

Dense Regression Network for Video Grounding

CVPR

CLEVRER: The first video dataset for neuro-symbolic reasoning

Neuro-Symbolic AI Computer Vision

Once for All: Train One Network and Specialize it for Efficient Deployment

Efficient AI ICLR

Deep Audio Priors Emerge From Harmonic Convolutional Networks

Audio Processing Deep Learning

Location-aware Graph Convolutional Networks for Video Question Answering

AAAI

Self-supervised Moving Vehicle Tracking with Stereo Sound

Multimodal Learning Computer Vision

The sound of motions

Computer Vision

TSM: Temporal Shift Module for Efficient Video Understanding

Efficient AI Computer Vision

Graph Convolutional Networks for Temporal Action Localization

Graph Deep Learning Computer Vision

Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement

NeurIPS

Imitation learning from observations

Deep Learning

Visual Concept-Metaconcept Learning

Neuro-Symbolic AI

Cross-channel Communication Networks

NeurIPS Deep Learning

Watch, Reason and Code: Learning to Represent Videos Using Program

Computer Vision

Defensive Quantization: When Efficiency Meets Robustness

Efficient AI Robustness

Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question Answering

Computer Vision Time Series

The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

Neuro-Symbolic AI Computer Vision

Weakly Supervised Dense Event Captioning in Videos

Computer Vision

StNet: Local and Global Spatial-Temporal Modeling for Action Recognition

Computer Vision Time Series

Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

Neuro-Symbolic AI Computer Vision

Unsupervised Domain Adaptation for 3D Keypoint Estimation via View Consistency

ECCV

Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation

Generative Models

Geometry Guided Convolutional Neural Networks for Self-Supervised Video Representation Learning

CVPR

Sparse, Smart Contours to Represent and Edit Images

CVPR

Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification

CVPR

The Sound of Pixels

Multimodal Learning Computer Vision