https://panshark.github.io/tags/multimodal-ai/Multimodal AIHugoBlox Kit (https://hugoblox.com)en-usThu, 01 Jan 2026 00:00:00 +0000https://panshark.github.io/media/icon_hu_702a800cd775dbac.pnghttps://panshark.github.io/tags/multimodal-ai/https://panshark.github.io/projects/multimodal-spatial-reasoning/Thu, 01 Jan 2026 00:00:00 +0000https://panshark.github.io/projects/multimodal-spatial-reasoning/<p>I am developing lightweight object-centric semantic 3D memory for wearable and embodied spatial intelligence. The system converts sparse egocentric RGB/depth/pose observations into localized object records, so an agent can answer spatial queries without storing or processing every frame as a dense map.</p> <p><strong>Current pipeline</strong></p> <ul> <li>Sparse Habitat/HM3D or egocentric exploration with VLM-guided object-category selection.</li> <li>YOLO-World detection, NanoSAM masks, DepthPro depth, field-of-view geometry, and visibility-aware object scoring.</li> <li>Batched or crop-level object VLM descriptions, CLIP/DINOv3 embeddings, and cross-view association.</li> <li>Queryable object records with labels, masks, depth, global coordinates, object-view links, and object-object spatial relations.</li> <li>Wearable/robot-side object memory with edge/cloud support for heavier perception, identity consolidation, and prioritized map updates.</li> </ul> <p><strong>Supported tasks</strong></p> <ul> <li>View retrieval, object localization, instance clustering, and spatial search.</li> <li>Agentic grounding for robots or wearable assistants operating under sensing, power, bandwidth, and compute constraints.</li> </ul>