Experiment Video

Abstract

Distributed tactile sensing is hard to scale over large robot surfaces because dense arrays increase wiring, cost, and fragility. Sound of Touch is an active acoustic tactile sensing method that uses continuously excited tensioned strings as sensing elements. Two contact microphones observe spectral changes caused by contact. From short-duration audio, the system estimates contact location and normal force, and detects slip. A physics-grounded string vibration simulator explains how contact position and force shift vibration modes. Experiments show millimeter-scale localization, reliable force estimation, and real-time slip detection.

Method Overview

The sensor uses a steel string, dual EBow-style electromagnetic drivers for continuous excitation, and dual contact microphones. Contact at position x and force F changes effective string length and tension, producing structured resonance shifts. A split-string simulator predicts these trends and supports design analysis, while a real-world inference model maps audio to contact state tuple (contact, location, force, slip).

Hardware and System

The prototype is built on an aluminum frame with guitar hardware and a 10 cm sensing range. Microphone signals are independently amplified and streamed through a Focusrite Scarlett 4i4 interface at 44.1 kHz using a low-latency JACK setup for real-time inference.

Learning Pipeline

Audio windows are transformed into spectral inputs and encoded using a frozen Audio Spectrogram Transformer (AST). Lightweight task-specific heads estimate contact and slip (binary classification), and location and force (regression). The model is optimized jointly with masked losses so regression is trained only when contact is present.

Results

Contact and slip classification reached 100% accuracy on the evaluated test set. Detailed regression metrics are shown below.

Contact Location Estimation

Contact Force Estimation

Contact

For questions, contact Xili Yi.