Multimodal Spatial Reasoning & Object Memory
I am developing lightweight object-centric semantic 3D memory for wearable and embodied spatial intelligence. The system converts sparse egocentric RGB/depth/pose observations into localized object records, so an agent can answer spatial queries without storing or processing every frame as a dense map.
Current pipeline
- Sparse Habitat/HM3D or egocentric exploration with VLM-guided object-category selection.
- YOLO-World detection, NanoSAM masks, DepthPro depth, field-of-view geometry, and visibility-aware object scoring.
- Batched or crop-level object VLM descriptions, CLIP/DINOv3 embeddings, and cross-view association.
- Queryable object records with labels, masks, depth, global coordinates, object-view links, and object-object spatial relations.
- Wearable/robot-side object memory with edge/cloud support for heavier perception, identity consolidation, and prioritized map updates.
Supported tasks
- View retrieval, object localization, instance clustering, and spatial search.
- Agentic grounding for robots or wearable assistants operating under sensing, power, bandwidth, and compute constraints.
Haozhe Lei (Graduate Student Member, IEEE) is a Ph.D. candidate in Electrical and Computer Engineering at New York University, advised by Professor Sundeep Rangan in NYU WIRELESS. He received the B.E. degree in electrical engineering and automation from China Agricultural University in 2019 and the M.S. degree in computer engineering from NYU in 2022. His research develops uncertainty-aware wireless intelligence for embodied autonomy and adaptive 6G systems, combining algorithms, digital twins, and physical testbeds to transform sparse RF and multimodal observations into calibrated spatial beliefs and closed-loop decisions.
His current work includes MC-CLE and LOCUS-DT for posterior RF localization, PIRL and wireless digital-twin priors for zero-shot robot navigation, MCMB-HDT for closed-loop multi-band handset adaptation, and object-centric graph memory for multimodal spatial reasoning. He also builds FR3/mmWave RFSoC/Pi-Radio channel-sounding systems with TurtleBot4 and Jackal UGV platforms. He received the 2023 Ernst Weber Fellowship from the NYU Tandon School of Engineering.