One-liner. TacEx is a modular framework that embeds a state-of-the-art GPU-accelerated soft-body contact simulator (GIPC) plus two GelSight image/marker simulators (Taxim + FOTS) inside NVIDIA Isaac Sim / Isaac Lab, so you can get GelSight Mini RGB images and marker-displacement fields as observations inside a GPU-parallelized RL stack — closing the gap where contact-rich tactile RL had no precise, easy-to-use simulator that lived in a modern robot-learning framework.
RL for contact-rich manipulation is bottlenecked by the lack of stable, reliable contact simulation that couples soft-body deformation with tactile sensing. Existing tactile simulators each use a different physics engine, sensor model, robot, and host simulator, which makes comparison and interoperability hard, and none plug into a feature-rich modern stack (photorealistic rendering, ROS, GPU-parallel physics, teleoperation, standard RL libraries). TacEx's bet is to stop building yet another standalone simulator and instead integrate the best existing components into Isaac Sim / Isaac Lab, gaining all of Isaac's infrastructure for free while keeping the tactile pipeline modular so the user can pick which simulations to enable per task.
A GelSight simulation needs three components (App. A): physics (contact properties), optical (perceived RGB image), and marker (marker motion field reflecting gel deformation). TacEx mixes-and-matches an implementation for each, orchestrated by Isaac Sim's loop (Fig 1, Fig 3).
Physics simulation — three gelpad options. The gelpad can be simulated as (i) a PhysX rigid body with compliant contact (fastest, for prototyping); (ii) a PhysX FEM-based soft body; or (iii) a GIPC soft body. For (iii) the authors ported the GIPC solver — a fully GPU-based, inversion-free Gauss–Newton optimization of an IPC barrier energy — and wrote Python bindings. GIPC is chosen because IPC guarantees intersection- and inversion-free simulation regardless of material parameters or deformation severity, simulates static and dynamic friction, and reduces the need to fine-tune sim parameters (helpful for domain-randomization-based Sim2Real). GIPC additionally permits soft-to-soft contact.
GIPC–Isaac integration (App. C). Isaac Sim handles scene setup, robot simulation, rendering; the gelpad is attached to the sensor case and moves kinematically with the robot (PhysX). The non-attached gelpad vertices are handled by GIPC. Per step: do a PhysX step (robot moves), recompute the attachment points, then call the GIPC solver to compute new positions for the remaining gelpad vertices and other GIPC objects, then update the Isaac USD meshes (via the USDRT/Fabric API for fast vertex updates) and render. To build GIPC objects, an asset's USD triangle mesh is extracted and a tetrahedral mesh is generated with Wildmeshing. Attachment points are found by sphere-ray-casting each tet point against the sensor-case collider via the PhysX scene-query interface; their constant offsets to the sensor-case pose are precomputed and saved as USD properties.
Optical simulation — Taxim. Generate height maps with Isaac Sim cameras, smooth with pyramid Gaussian kernels, then use a polynomial look-up table to map surface normals to RGB; add shadows for realism (a GPU-accelerated Taxim implementation is used).
Marker simulation — FOTS. From the same height maps, FOTS
models marker displacements with exponential functions for normal/shear/twist loads;
contact centers are computed from the height maps and the z-rotation of
objects relative to the gelpads is extracted from Isaac Sim. The full sensor-output
pipeline (object height map → indentation depth → gelpad deformation →
RGB + marker flow) is in Fig 4.
Positioning vs. concurrent work. The authors state TacEx is closest to the concurrent TacSL, which also brings visuotactile simulation into Isaac Sim, but TacEx differs by using GIPC for FEM-based soft-body simulation (instead of a simplified soft contact model) and by being explicitly modular.
This is a feasibility-and-speed paper, not an accuracy or success-rate paper. Headline qualitative findings:
Simulation-speed benchmarks (ms per frame, ball-rolling experiment):
| Tactile sim (Table 1) | 1 env | 8 envs | 16 envs | 18 envs |
|---|---|---|---|---|
| height-map gen (USD cameras) | 1.37 | 0.43 | 2.88 | 3.51 |
| optical (GPU Taxim + shadows) | 5.90 | 2.58 | 5.73 | 5.93 |
| marker (FOTS, CPU, no parallel.) | 4.49 | 1.76 | 5.05 | 5.23 |
| PhysX physics (Table 2) | 1 | 16 | 64 | 128 | 256 | 1024 |
|---|---|---|---|---|---|---|
| rigid gelpad | 3.69 | 0.24 | 0.067 | 0.036 | 0.021 | 0.0093 |
| soft gelpad | 4.71 | 0.45 | 0.18 | 0.13 | OOM | OOM |
GIPC soft-body physics (Table 3) costs 24.95 ms at 1029 verts / 3717 tetra, rising to 221.61 ms at 12509 verts / 66563 tetra — i.e., accuracy via GIPC is roughly an order of magnitude slower per frame than PhysX rigid contact and does not parallelize across many envs within the 12 GB VRAM budget.
From the authors:
What I noticed reading it:
This is off the core BLADE method axis — it is a sensor/simulator infrastructure paper with no language, no planning abstractions, and no predicate learning — so I won't manufacture a tight connection. Its relevance is infrastructural and thesis-supporting:
is_grasped,
is_inserted, is_screwed_tight, surface_is_rough
— are not visually evaluable. TacEx is the plumbing that would let a
BLADE-style system learn classifiers for such tactile predicates in simulation:
GelSight RGB + marker fields as the observation channel a tactile predicate
classifier would read. BLADE today crops faucet pixels into a vision classifier;
a tactile-predicate extension would crop gelpad images/marker fields into a tactile
classifier, and would need exactly this kind of GPU-parallel tactile sim to train.Tangentially it also informs the "structure-vs-scale" framing: TacEx is squarely on the side of accurate physics simulation as a data source, complementary to the learned-world-model alternatives.
A crucial bottleneck for applying RL to tactile-rich manipulation tasks is the lack of stable and reliable contact simulation that includes soft-body interaction and tactile sensing. — §1 Introduction / p.1
Our approach is closest to the concurrent work TacSL [26] which also incorporates visuotactile simulation into Isaac Sim, however in contrast to TacSL, we leverage GIPC [18] for FEM-based soft-body simulation (instead of a simplified soft contact model). — §1, Related Work / p.2
A limitation of our current work is that it only contains qualitative experiments and demonstrations in simulation. Our tactile simulation [...] lacks experiments that investigate whether they can be used for Sim2Real or not. — §4 Conclusion and Future Work / p.4
Papers cited that should likely be ingested next:
Newly ingested in 2026-06-24 batch — directly relevant: