CIS-RAM 2026
CIS-RAM 2026 · FIRST AUTHOR
DR-TSLM
Connecting words and gestures to the objects and places they refer to.
- Research focus
- Multimodal human–robot interaction
- Inputs
- Speech and video
- Outputs
- Object masks and target locations
01 · THE QUESTION
What do “this” and “here” refer to?
Please pick up this object and place it here.
The words alone leave the object and destination open to interpretation. A hand gesture supplies the missing reference, but it has to be read at the right moment. DR-TSLM connects the spoken expression with its visual context to identify what the person means.
02 · THE METHOD
From speech to spatial targets
The system combines temporal alignment with spatial grounding. Instead of treating a video as one static scene, it uses the timing of spoken references to select relevant gesture frames.
Locate the reference
Whisper provides word-level timestamps. The language model identifies expressions referring to an object or a location.
Align the gesture
The corresponding time window selects up to five video frames. Temporal position encoding preserves their order for the visual-language model.
Ground the target
The model predicts the referred object or place. Segmentation produces an object mask, while location references produce pixel coordinates for the robot interface.
03 · IN THE WORKSPACE
From an instruction to a robot action
The physical demonstrations connect reference resolution with a robot arm. They show both a directly named object and an object indicated through speech and gesture.
These sequences illustrate system behavior; they are separate from the quantitative reference-resolution benchmark below.
04 · EVALUATION
Timing matters for reference resolution
0.92Temporal deictic reasoning accuracy
DR-TSLM-3B · manuscript Table II
DR-TSLM-3B · manuscript Table II
The model identifies the object referred to in a video using the accompanying text prompt. Removing temporal position encoding reduces accuracy to 0.70, a difference of 22 percentage points.
Correctness in this benchmark is judged with GPT-5. The metric measures reference resolution; it does not measure complete robot pick-and-place success.
| Model configuration | Accuracy |
|---|---|
| DR-TSLM-3B | 0.92 |
| Without temporal position encoding | 0.70 |
Data behind the study
- 1,200
- Deictic-expression text instances
- 1,000
- Annotated scenes selected from 100DOH
- 100
- Speech–gesture pairs from 10 participants
These are the sizes of the study’s three datasets, rather than a shared test-set denominator. Results and figures on this page follow the research manuscript.
Current scope
The method focuses on spoken deictic expressions paired with gestures. Implicit intentions and richer three-dimensional references remain directions for further work.
PUBLICATION