Grounding Dynamic Manipulation Concepts on Multimodal Sensing

Curated from Weiyu's research knowledge base · shared 2026-06-25

Start here → Grounding Dynamic State-Change Concepts on Multimodal Sensing for Closed-Loop Manipulation

The research-idea brainstorm. Core thesis: existing work grounds static properties (hardness, material) on non-visual sensing and uses them for one-shot selection; manipulation is transformation, so the concepts that matter are dynamic — object state and state-transitions (“hot enough now”, “became slippery after wetting”, “coming to a boil”) — and the signal that reveals them often lives in a non-visual modality (thermal, acoustic, tactile/friction). The proposal: ground named, learned, transferable state-change concepts on multimodal non-visual sensing, monitor them over time, and use their value to drive reasoned closed-loop correction.

The doc includes: a four-type taxonomy of dynamic concepts; the BLADE connection (ground the effects/verbs, not the properties/adjectives); three framing spines (planner-predicates / policy-correction / perception-benchmark); a novelty-landscape table positioning against the closest prior work; a task × modality × annotation catalog (the design matrix across all sensory modalities, with single-sensor vs. inferred tagging and a grounded/novel coverage map); five hero benchmark tasks; and open research questions.

What's in this folder

• The brainstorm: knowledge/brainstorm/grounding_dynamic_state_change_concepts_multimodal.html
• 91 paper summaries in knowledge/paper/ — everything the brainstorm and the survey reference: the language + multimodal-sensing corpus (tactile / audio / force / thermal for manipulation) plus the dynamic-state-change / closed-loop-correction related-work anchors.
• 91 source PDFs in raw/papers/ — linked from each summary's header.
• 1 literature survey in knowledge/search/ — the language + multimodal (non-visual) sensing for robotic manipulation targeted search: the full 73-candidate landscape (10 clusters) the corpus was built from, with per-paper metadata and arXiv links.

How to read it

• Read the brainstorm top-to-bottom; every concept and paper mention links to the relevant summary.
• Each summary follows the same shape: one-liner, problem & motivation, method, setup, results, limitations & open questions, why-it-matters, quotable lines, and related papers.
• The survey checklist is browsable by cluster (touch–language, tactile foundation models, multisensory VLAs, audio/acoustic, audio-visual anchors, force/proprioception, thermal, binding LMs, datasets/sims, surveys); its paper links go to arXiv.

Paper-to-paper links within this set resolve. A few links in the brainstorm's “Related” sections and in summary footers point to papers or index pages outside this curated package and will not open. Summary text is faithful to each source; where a value was not reported in a paper it is marked not reported rather than guessed.