Acustico: Surface Tap Detection and Localization using Wrist-based Acoustic TDOA Sensing

Motivation

As computing devices become increasingly ubiquitous, there is a pressing need for input technologies that can be always available. While wearable devices like smartwatches and smartglasses enable always-available touch input, it comes at the cost of small physical size, which limits user’s input due to the fat finger problem. One potential solution is to exploit the surfaces in the environment around us, which provide a large area for accurate and comfortable input.

In this paper, we present Acustico, a passive acoustic sensing approach that enables tap detection and 2D tap localization on uninstrumented surfaces using a wrist-worn device. This is achieved using a novel application of acoustic time differences of arrival (TDOA) analysis. To overcome the challenges posed by the wristband form factor (e.g., sensors have to be close to each other), we adopt a sensor fusion approach by exploiting the propagation speed difference of “surface wave” (i.e., vibrations through surface) and “sound wave” (i.e., vibrations through air) to better estimate the TDOA and localize the tap. Through a careful design procedure, we come up with an optimal sensor configuration that meets wristband constraints and has better sensing performance.

Prototype

Our wristband prototype consists of two highly sensitive accelerometers (Model 352A24, PCB Piezotronics) and two MEMS microphone breakouts (INMP401, Sparkfun) with built-in amplifiers (gain = 67dB). Four sensors are situated on the bottom of the wrist with a mechanical structure. The accelerometers were connected to an ICP signal conditioner (Model 482C24, PCB Piezotronics) with a gain of 10. Since the TDOAs we targeted are on the order of microseconds, we needed a sampling rate of at least 1MHz. We therefore used the PicoScope 2406B for sampling the amplified signals from both accelerometers and microphones and set its sampling rate to 1MHz with the raw data streaming into a desktop via USB.

Applications

We built four demo applications to showcase the potential use cases of Acustico. The first three applications show how Acustico can be used with AR devices (e.g., Microsoft HoloLens) to enrich the input expressiveness. The last application provides a coherent “mouse experience” by combining Acustico with an optical flow sensor.

AR Applications. The first application we built is a dial pad for AR devices. User can simply tap at different locations to enter a phone number and call the person he wants to contact (Figure (a)). Similarly, we also developed a calculator for AR devices. User can input the numbers on any surfaces nearby and get the calculation results (Figure (b)). Our third application is a whack-a-mole AR game. User can hit moles by taping on the surface, which offers an immersive gaming experience with corresponding haptic tactile feedback (Figure (c)).

Optical Flow + Acustico for Surface-wide Localization. Acustico focuses on tap localization relative to the wristband. We combine the Acustico with an optical flow sensor underneath to additionally track the forearm motion on the surface. This combination can be used to expand any of the applications demonstrated above to support tap localization over larger surface-wide AR interfaces. We further implement a mouse experience (Figure (e)) where the user simply moves his or her wrist to control the pointer position and taps on left/right region with respect to the wristband to invoke left/right click.