🐚 The World of Visuo-Tactile World Models

• Tactile VAE is the so-called perception layer. It takes raw, high dimensional data and compresses it into low-dimensional representations through a spatio-temporal encoder, using 3D causal convolutions. Being "causal" (the causal kernels) means it only looks at the past and present, never the future, so it can run in real-time. Tactile VAE also uses Implicit Neural Representation (INR) as an implicit deformation decoder for the robot to be able to query the exact 3D deformation at any coordinate.

  • By inputting a spatial coordinate $(x, y)$ and a latent code $z$, the model predicts the continuous deformation $d$ at that specific point.

• INR is important as it allows the model to treat gel as a continuous physical surface as opposed to being limited by the discreteness of the pixel resolution of the sensor. This is primarily done during training. During inference, the low-dimensional representations produced by the encoder are the primary pieces of information passed to the next modules.

• The Visuo-Tactile World Model is the prediction layer. Having two separate paths, one for video and one for touch, it deploys a Diffusion Transformer (DiT) mechanism to create frames of the predicted tactile and visual states. The multi-model conditioner takes the current video, touch and desired action to be taken by the robot as conditions for the prediction of the states. The model uses projected 2D actions as the primary conditioning for the action, as the researchers found this generalizes better than raw 3D coordinates. During training, a weight map is used to force the model to be more accurate during high-intensity contact moments, i.e. the dynamic-aware loss concept.

• The adaptive visuo-tactile fusion policy is where the decision process over the next robot's action takes place. Here, instead of processing tactile signals as static snapshots, the Latent Tactile Differential (LTD) encoder captures the first-order time derivative of the tactile state. It performs channel-wise concatenation of three distinct vectors:

  • $f_t^c$ : The encoded latent representation of the immediate observed state.
  • $f_t^p$: The temporally aggregated future state generated by the World Model.
  • $f_t^p - f_t^c$ : The explicitly computed delta between expectation and observation.

• The differential term tells the robot if a contact event is about to happen.

• There is also a gating mechanism involved. First, an MLP classifies the likelihood of physical interaction by thresholding tactile deformation magnitudes. Then, the gating network outputs normalized scalar weights $W_t$ (tactile) and $W_v$ (visual), where $W_t + W_v = 1$. The final state representation is a weighted combination given by $f_{vt} = \text{concat}(W_v \odot f_v, W_t \odot \tilde{f}_t)$. When the robot's end-effector obscures the target object and visual occlusion takes place, the gating network shifts the policy's dependence toward the tactile manifold to maintain stable control.

• To ensure the action trajectory is smooth and physically consistent, it's treated as a joint probability distribution over a short time horizon $H$.

• The reflexive latent tactile controller is the reflex layer, operating at 60Hz, much faster than the 4Hz planner, to handle emergencies. This module bridges the gap between feedforward planning (what's known as the Slow Policy) and real-time execution (Fast Policy). It constantly checks the real-time touch feedback. It compares the real touch to the predicted touch from the world model by using Nearest-Neighbor Interpolation to upsample the predicted tactile latents (as the world model operates at a lower frequency). If they don't match (e.g., the robot missed the object or hit it too hard), it calculates a Refined Action $a_r$ to fix the mistake as fast as possible.

• The final command sent to the robot to be executed is the sum of the planner's goal and the reflex's correction: $a_{total} = a_{planner} + \lambda \cdot a_{reflexive}$.

The DreamTacVLA was an attempt to include tactile sensing into Vision-Language-Action (VLA) models, because these are completely blind to physical contact (*it is similar to Meta's VT-WM, but here the world model is a module inside a VLA, rather than the whole system).

• High-resolution GelSight tactile images are spatially aligned with wrist-camera and third-person camera views, so the model learns that what the tactile sensor sees corresponds to a specific location in what the camera sees
• They also trained a latent tactile world model to predict future tactile states (the so-called dreaming part).
• The model drafts an action from vision, predicts what touch will feel like, then refines the action based on the predicted contact.

VTAM originated from a Video-Action Model where contact-rich tasks fail to be dealt with properly because force and contact transitions aren't visible in pixels.

• They augmented a pretrained video transformer with tactile streams via lightweight modality transfer finetuning.

Honourable Mentions:

• AnyTouch, which is a representation learning framework, not a world model.

  • It takes tactile images from multiple sensor types (like GelSight, DIGIT, DuraGel, Tac3D) and maps them into a shared embedding space using contrastive learning and masked autoencoders.
  • The goal is a universal tactile encoder that works regardless of which sensor you're using.
  • No future predictions. Just perception basically.

• HapticVLA is built on the idea that you can use tactile data during training to teach a VLA model about contact physics, but during inference you don't need the tactile sensor at all.

References

[1] C. Higuera, S. Arnaud, B. Boots, M. Mukadam, F. R. Hogan, and F. Meier, "Visuo-tactile world models," arXiv preprint arXiv:2602.06001, 2026.

[2] Y. Zheng, S. Gu, W. Li, Y. Zheng, Y. Zang, S. Tian, X. Li, C. Hao, C. Gao, S. Liu, H. Li, Y. Chen, S. Yan, and W. Ding, "OmniVTA: Visuo-tactile world modeling for contact-rich robotic manipulation," arXiv preprint arXiv:2603.19201, 2026.