A Vision-Based Tactile Sensing System for Multimodal Contact Information
  Perception via Neural Network
In general, robotic dexterous hands are equipped with various sensors for
acquiring multimodal contact information such as position, force, and pose of
the grasped object. This multi-sensor-based design adds complexity to the
robotic system. In contrast, vision-based tactile sensors employ specialized
optical designs to enable the extraction of tactile information across
different modalities within a single system. Nonetheless, the decoupling design
for different modalities in common systems is often independent. Therefore, as
the dimensionality of tactile modalities increases, it poses more complex
challenges in data processing and decoupling, thereby limiting its application
to some extent. Here, we developed a multimodal sensing system based on a
vision-based tactile sensor, which utilizes visual representations of tactile
information to perceive the multimodal contact information of the grasped
object. The visual representations contain extensive content that can be
decoupled by a deep neural network to obtain multimodal contact information
such as classification, position, posture, and force of the grasped object. The
results show that the tactile sensing system can perceive multimodal tactile
information using only one single sensor and without different data decoupling
designs for different modal tactile information, which reduces the complexity
of the tactile system and demonstrates the potential for multimodal tactile
integration in various fields such as biomedicine, biology, and robotics.
Weiliang Xu, Guoyuan Zhou, Yuanzhi Zhou, Zhibin Zou, Jiali Wang, Wenfeng Wu, Xinming Li
A Vision-Based Tactile Sensing System for Multimodal Contact Information Perception via Neural Network

###### Abstract

In general, robotic dexterous hands are equipped with various sensors for acquiring multimodal contact information such as position, force, and pose of the grasped object. This multi-sensor-based design adds complexity to the robotic system. In contrast, vision-based tactile sensors employ specialized optical designs to enable the extraction of tactile information across different modalities within a single system. Nonetheless, the decoupling design for different modalities in common systems is often independent. Therefore, as the dimensionality of tactile modalities increases, it poses more complex challenges in data processing and decoupling, thereby limiting its application to some extent. Here, we developed a multimodal sensing system based on a vision-based tactile sensor, which utilizes visual representations of tactile information to perceive the multimodal contact information of the grasped object. The visual representations contain extensive content that can be decoupled by a deep neural network to obtain multimodal contact information such as classification, position, posture, and force of the grasped object. The results show that the tactile sensing system can perceive multimodal tactile information using only one single sensor and without different data decoupling designs for different modal tactile information, which reduces the complexity of the tactile system and demonstrates the potential for multimodal tactile integration in various fields such as biomedicine, biology, and robotics.

 multimodal tactile sensing, vision-based tactile sensor, deep neural network, grasped object perception

## I Introduction

The ability of humans to manipulate objects with dexterity relies on the thousands of sensory receptors in hands that detect various types of tactile information, including vibration, pressure, and friction [1-4]. This multimodal tactile information is also essential for robots to achieve dexterous manipulation. To enable accurate measurement of external physical stimuli in dexterous manipulation tasks, several tactile sensing mechanisms such as piezoelectric, capacitive, cantilever-based, and resistive sensing systems have been developed [5-16]. However, obtaining multimodal tactile information remains a significant challenge for tactile sensing systems [17-19]. One of the main reasons is that most conventional tactile sensing systems are based on point measurements, resulting in low spatial resolution [20]. Additionally, due to the limitations of sensing principles, numerous multimodal tactile perception systems rely on different sensing mechanisms to achieve decoupling of various tactile modes, making it difficult to achieve decoupled sensing of multiple tactile modes within a single system [21-23]. This restricts their applications in compact sensor structures and multimodal tactile sensing tasks.

To address these issues, among different types of tactile sensors, vision-based tactile sensors have been extensively investigated [24-30], which use an elastomer and a complementary metal-oxide-semiconductor (CMOS) sensor to convert tactile information into visual representations. By modifying the design of the elastic design, these sensors can detect different modalities of tactile information, such as force, contact location, and contact shape, enabling high-resolution, multimodal tactile information sensing [31-36]. These systems utilize the characteristics of vision-based tactile sensors to capture different modalities of tactile information (such as force, contact position, and contact posture, etc.) in different physical domains. By establishing effective mapping relationships specific to each physical domain, visual representations are transformed into corresponding tactile information, and appropriate algorithms and models are designed to achieve accurate feature extraction and recognition. However, as the dimensions of different tactile modalities increase, it becomes necessary to adopt different optical designs and a suitable decoupling method for different contact information. As shown in **Table I**, to realize the simultaneous recognition of multiple tactile information, the researchers have tried to combine a variety of structural designs and data processing techniques, such as the combination of elastomer markers or finite element analysis to provide feedback on multidimensional forces, the combination of structured light design or photometric stereo methods to achieve three-dimensional reconstruction, and combining thermochromic materials to realize temperature sensing. This implies more complex challenges in system integration and data decoupling to ensure the accurate extraction and recognition of multimodal tactile features. Therefore, a primary challenge for vision-based tactile sensing systems is
how to effectively process and decouple multimodal tactile information with visual representations to achieve accurate feature extraction and recognition.

In the face of the significant demand and challenges in extracting multimodal tactile information from visual representations, we expanded the tactile perception technology based on vision-based tactile sensors to achieve multimodal tactile perception with the use of reflective-only optical design, as shown in **Fig. 1**. Vision-based tactile sensing system accesses tactile images with high-density contact information, and deep neural networks can be used to achieve the recognition of multimodal tactile information. Compared with the traditional tactile sensing system, the proposed vision-based tactile sensing system has the ability to sense high-resolution and high-density tactile information which can characterize contact information for different modalities with only one sensor. Based on the output of the tactile image from the vision-based tactile sensor, a deep neural network to decouple the tactile images was proposed to obtain the contact object classification, position, pose angle, and normal force information. In contrast to traditional methods, the proposed approach does not require different algorithmic processes and optical designs for different modalities of contact information. Instead, it only requires the tactile images to recognize contact information for different modalities. Moreover, since the proposed method is data-driven and the extracted features are common to other tasks, it is possible to further extend the recognition of more modalities of tactile information by simply adding output dimensions without redesigning the data processing methods. The proposed system provides a new tactile solution with high resolution and integration to be applied to multimodal sensing tasks such as biomedical, biology, and robotics.

## II Design of the vision-based sensor

### _Design and mechanism of the vision-based sensor_

The structure of the proposed vision-based tactile sensor design is shown in **Fig. 2(a)**. The sensor consists of three main components: a soft contact interface with a reflection layer, a shell, and a CMOS image sensor with a light source. The soft contact interface consists of a reflection layer made of aluminum film manufactured by a sputter coater (Quorum Q150T S Plus), a deformation layer made of polydimethylsiloxane (PDMS Sylgard 184, manufactured by Dow), and a support layer made of acrylic, which converts the external physical contact information into optical information. The CMOS sensor equipped with a wide-angle lens is responsible for capturing optical information. To keep the system compact, three pairs of white Light Emitting Diodes (LEDs) in the CMOS peripheral circuitry provide a light source to illuminate the contact interface. The shell is 3D printed (Ultimaker S3 3D printer) and has interfaces for installation with the CMOS sensor and the contact interface. In the fabrication process, an acrylic sheet was fixed on a model made from the 3D printer to manufacture the support layer first. Then 2 mL of a mixture of uncured PDMS and curing agent at a ratio of 10:1 by mass was poured over the support layer to create a deformation layer, which was then placed in a vacuum chamber to remove air bubbles and allowed to cure naturally. Once the deformation layer was fully cured, the sample was plasma cleaned to improve the adhesion of the coating. A reflection layer was then manufactured by sputtering an Al film onto the sample surface. Finally, the assembly of the CMOS sensor with LED module, the shell made of 3D printing, and the soft contact interface

Fig. 1: Schematic diagram of the vision-based multimodal tactile sensing system. (a) Robotic dexterous manipulation tasks. (b) Schematic diagram of the vision-based tactile sensor. (c) Schematic diagram of the information flowing.
completed the overall design of the vision-based tactile sensor with an overall size of 35\(\times\)35\(\times\)30 mm. However, if the three components were assembled directly, there would be overexposure on the tactile image due to the excessive light intensity at the light source projection on the reflection layer (as shown in **Fig. 2(b)**), thus introducing system noise. Therefore, a shell with a diffusion layer was designed, as shown in **Fig. 2(c)**. When the light passes through the plane where the diffusion layer is located, it is attenuated before reaching the reflection layer, which eliminates the overexposure phenomenon.

The working principle of vision-based tactile sensors is to capture the optical changes of the surface caused by contact using a CMOS camera to characterize the contact characteristics as visual representations. Here, assuming that the inner surface of the sensor is a diffuse reflecting surface, the surface can be described using a height function \(z=f(x,y)\)[26]. In this case, the reflection intensity at the point \((x,y)\) can be represented as:

\[I(x,y)=R(\partial f/\partial x,\partial f/\partial y)E(x,y), \tag{1}\]

where \(R\) is a reflective function defined based on the surface gradient \(\partial f/\partial x\) and \(\partial f/\partial y\), and \(E\) represents the intensity of the incident light. When an object contacts the sensor, the surface of the sensor deforms, causing a change in the reflection function \(R\) of the sensor surface [26]. This change is characterized by local intensity variations in the image, allowing us to obtain tactile information about the deformation of the contact surface. As shown in **Fig. 3**, when an external force \(F\) is applied to the flexible deformation layer with a reflection film, the optical reflectance under the reflection film changes. The embedded CMOS sensor at the bottom can sense the changes in the reflection field caused by different contact situations due to deformation, and then convert the tactile information into visual representations as tactile images. Information about different contact objects, contact positions, and contact postures can be obtained by analyzing the distribution patterns of reflection field intensity in the visual representation of the tactile image. For different contact loads, the normal force can be estimated based on the reflection field intensity and the area of the reflection field pattern in the visual representation. These details will be discussed in detail in the following sections.

### _Design of experiments for vision-based sensor_

As shown in **(1)**, the intensity of the incident light is closely related to the decoupling of the final tactile information. By regulating the area size and the relative position of the diffusion layer, the light intensity on the inner surface of the reflection layer can be regulated to improve the system resolution of the sensor. Here, two different types of diffusion layers were designed by regulating their relative position. The first type is shown in **Fig. 4(a)** and is called Type 1 (T1), where the diffusion layer is located at a position 18 mm above the CMOS sensor, and the light enters the surface of the reflection layer directly to illuminate the reflection layer. The second type is shown in **Fig. 4(b)** and is called Type 2 (T2), where the diffusion layer is located at 10 mm above the CMOS sensor. The light enters the edge of the deformation layer, illuminating the inner surface of the reflection layer by a light-guided action of the deformation layer. In addition, with these two types, four different diffusion layer patterns were designed as shown in **Fig. 4(a)** and **Fig. 4(b)** by adjusting the light transmission area size. To ensure that the diffusion layer does not affect the CMOS imaging, with the camera lens diameter of 15 mm as the standard, a circular pattern with 15 mm diameter called Category 1 (C1), a square pattern with 15 mm side length called Category 2 (C2), a rectangular pattern with 15mm and 24mm side length called Category 3 (C3), and a rectangular with 15mm and 33mm side length called Category 4 (C4) were fabricated. We built a resolution test platform to compare the resolution images of each design (**Fig. 5(a)**). Vision-based tactile sensors use CMOS to capture optical information to extract contact information. Due to its sensing properties, the upper limit of tactile sensing resolution is theoretically the imaging resolution of a CMOS imaging system, which is 7.13 lp/mm for the horizontal stripe and 8.98 lp/mm for the vertical stripe. Therefore, a resolution test target based on the USAF 1951 resolution test target with 100 um gradients was proposed (**Fig. 5(b)**, left). Using the proposed

Fig. 4: (a) Four different diffusion pattern designs of type 1. (b) Four different diffusion pattern designs of type 2.

Fig. 3: Illustration of the principle of the tactile sensor.

Fig. 2: (a) Schematic diagram of the vision-based tactile sensor. (b) The tactile image without diffusion layer design. (c) The tactile image with diffusion layer design.
resolution test target as a quantitative standard enables a direct comparison of the resolution performance of different designs and their relationship to the theoretical upper image resolution limit.

The results are shown in **Fig. 5(c)** and **Fig. 6**, where **Fig. 6(a)** shows the comparison of the resolution performance of four different designs of T1 type with the original CMOS imaging system, and **Fig. 6(b)** shows the comparison of the resolution performance of four different designs of T2 type with the original CMOS imaging system. It can be found that the average resolution of both horizontal and vertical stripes of the T1 type design is better than that of T2. The T1 type makes the light from the light source reach the inner surface directly, while the T2 type makes the light reach the inner surface through the action of the deformation layer. So, the T1 design makes the light reaching the inner surface of the reflection layer more intense, thereby enhancing the resolution of the system to a certain extent. Similarly, for the same design type, when the area of the design pattern is larger, the intensity of the light reaching the inner surface of the reflection layer is more intense. Therefore, it can be found that the average resolution of the C1 pattern design is the best in both T1 and T2 designs. **Fig. 5(b)** shows the resolution test target and the imaging resolution images of the CMOS camera, which represents the theoretical limit of spatial resolution for vision-based tactile sensors. **Fig. 5(c)** shows the tactile resolution images of TIC1 and T2C4, which have the best and the worst resolution performance respectively. To conclude, the main influence on the designs of diffusion layers is the light field distribution of the light reaching the reflection layer. By regulating the relative positions of the diffusion layers and the pattern areas, the T1C1 design has a better performance with an average resolution of 7.13 lp/mm, which means that this design can resolve line pairs at the hundred-micron level comparable to the human tactile perception capability [37] and is the closest to the theatrical maximum tactile sensing resolution. Therefore, the T1C1 type will be used for the following experiments.

## III Design of the neural network

### _Network structure_

In robotic grasping tasks, multimodal information sensing is of interest. The current multimodal tactile systems for the recognition of different information are often difficult to form an organic combination in hardware and often discrete in recognition data processing, which brings inconvenience to multimodal feature extraction to a certain extent. When an object touches the vision-based tactile sensor, the visual representations often contain multimodal information such as the pose angle, location, and normal force of the contact object, so the key to the problem lies in how to decouple the information of different tactile modes from the visual representations. Here, a multimodal information recognition method based on neural network is introduced as shown in **Fig. 7(a)**, to accomplish contact part classification, object location recognition, posture angle recognition, and normal force recognition.

The structure of the proposed network with reference to the idea of target detection algorithms is shown in **Fig. 7(b)**. The original image size of 640\(\times\)640 is passed through the backbone network to obtain 3 feature maps of different dimensions of size 80\(\times\)80, 40\(\times\)40, and 20\(\times\)20. This can be seen by dividing the original image into a total of 8,400 regions of size 8\(\times\)8, 16\(\times\)16, and 32\(\times\)32, which are used to detect small, medium, and large objects respectively. Then, the three feature maps of different dimensions are fused by FPN (Feature Pyramid Network) [38], and finally, the outputs of four different contact information in the 8400 regions are obtained after passing through a decoder with four channels (each channel has two 3 \(\times\) 3 convolutional layers and one 1 \(\times\) 1 convolutional layer) respectively. The first channel outputs the probability of positive samples and the coordinates and size of the predicted box in each region. The second channel outputs object category information, the third channel outputs pose angle information, and the fourth channel outputs normal

Fig. 5: (a) The resolution test platform for the vison-based tactile sensor. (b) The resolution image of the CMOS camera. (c) The resolution image of the vision-based tactile sensor output using the T1C1 (left) and T2C4 (right) type.

Fig. 6: (A) Resolution performance of four different designs of type T1 and the original CMOS imaging system. (B) Resolution performance of four different designs of T2 type and the original CMOS imaging system.
force information. After the tactile images are processed by the network, we can obtain the contact part class, pose angle, location, and normal force information of different target objects. The PC used in training and testing the neural network has a GPU of NVIDIA GeForce RTX 3080, a CPU of Intel Xeon Gold 5222, and a system of Windows Server 2019.

### _Loss function_

The proposed network is based on the idea of target detection algorithms [39-44], whose core idea is to divide the input image into \(s\times s\) small grids in the backbone. Each grid is responsible for predicting the class and the location of the object whose center falls on it. However, to introduce the recognition of force and pose angle, the Decoder was altered, which is responsible for outputting the final prediction information. According to the idea of Decoupled Head in YOLOX, replacing the coupled detection head with the decoupled head can decouple the classification problem from the regression problem, which can effectively improve the model performance [40]. In this regard, we designed the decoder with four parallel channels, each of which is responsible for predicting class, location, normal force, and pose angle information, respectively. Correspondingly, the loss function (**Eq. 2**) can be divided into four blocks corresponding to four outputs. Firstly, the Binary Cross-Entropy loss (_BCE_ ) between the possibility of positive samples and the real situation of the 8400 regions is calculated, and then the normal force regression loss, the classification identification loss, the pose angle loss, and the loss of prediction frames of the \(B\) samples filtered by simOTA are calculated. Here, to address the boundary problem in the pose angle recognition task, the angle values were encoded using the _CSL_ (Circular Smooth Label) technique [45] to convert the regression task into an angle classification task at a fine granularity of 1 \({}^{\circ}\). Also, the _smoothL1_ (Smooth L1 Loss function) [46] was introduced to allow the network less sensitive to outliers and anomalies in the force regression task. Eventually, the loss function of the network is expressed as follows,

\[Loss=\sum_{i=1}^{B}\left\{BCE\big{(}cls_{pre},cls_{gt}\big{)}+BCE \big{[}\theta_{pre},CSL\big{(}\theta_{gt}\big{)}\right]\] \[+smoothL1\big{(}f_{pre},f_{gt}\big{)}\] \[+\Big{[}1-IoU\big{(}box_{pre},box_{gt}\big{)}^{2}\Big{]}\Big{\}}\] \[+\sum_{i=1}^{B}BCE\big{(}obj_{pre},obj_{gt}\big{)}, \tag{2}\]

where \(B\) represents the total number of samples filtered by simOTA [40]. 8400 represents the output dimension of the backbone. \(obj_{pre}\) represents the probability that the prediction is a positive sample, and \(obj_{gt}\) is 0 or 1 (where 0 represents a negative sample and 1 represents a positive sample). \(cls_{pre}\) and \(cls_{gt}\) represent the predicted and the real class information respectively. \(\theta_{pre}\) and \(\theta_{gt}\) represent the predicted and the real pose angle information respectively. \(f_{pre}\) and \(f_{gt}\) represent the predicted and the real normal force information respectively. \(CSL\) represents the _CSL_ encoding function with the Gaussian function as the window function. \(smoothL1\) is a robust \(L1\) loss that is less sensitive to outliers [46]. \(IoU\big{(}\)Intersection over Union) evaluates the overlap of the predicted box \(box_{pre}\) and the true box \(box_{gt}\).

### _Definition of the pose angle error_

In the pose angle prediction task, the predicted pose angles range from 0\({}^{\circ}\) to 180\({}^{\circ}\), and the actual angle numerical difference does not represent the actual angle error, such as the difference between 1\({}^{\circ}\) and 179\({}^{\circ}\) is 178\({}^{\circ}\) in numerical value while the actual difference is only 2\({}^{\circ}\). In this regard, as

Fig. 7: (a) Architecture of the proposed data processing process. (b) The structure of the proposed network.
mentioned above, the CSL technique was used to solve this problem during the training period. In the evaluation phase, we use the definition of rotation matrix to get the error of the pose angle shown as follows,

\[\Delta\theta=\arccos\left[\left|\frac{tr\left(R_{pre}R_{gt}{}^{-1}\right)}{2} \right|\right]\,, \tag{3}\]

where \(R_{gt}\) represents the rotation matrix of the ground truth angle. \(R_{pre}\) represents the rotation matrix of the predicted angle. \(tr\) represents the trace of the matrix. \({R_{gt}}^{-1}\cdot R_{pre}\) can characterize the rotation matrix of the predicted value angle to the true value angle. Here, we use the definition of the rotation matrix to calculate the true value of the pose angle, and the derivation process is as follows.

As shown in **Fig. 8**, we define the actual and predicted angle as \(\theta_{1}\) and \(\theta_{2}\), and the actual frame \(A_{1}\) and predicted frame \(A_{2}\) of the target can be regarded as rotated by \(\theta_{1}\) and \(\theta_{2}\) along the x-axis respectively. So, \(A_{1}\) and \(A_{2}\) can be expressed as follows,

\[A_{1}=\begin{bmatrix}x_{1}\\ y_{1}\end{bmatrix}=\begin{bmatrix}cos\theta_{1}&-sin\theta_{1}\\ sin\theta_{1}&cos\theta_{1}\\ sin\theta_{1}&cos\theta_{1}\\ \end{bmatrix}\begin{bmatrix}x_{0}\\ y_{0}\end{bmatrix}=C_{0}^{1}A_{0}\,, \tag{4}\] \[A_{2}=\begin{bmatrix}x_{2}\\ y_{2}\end{bmatrix}=\begin{bmatrix}cos\theta_{2}&-sin\theta_{2}\\ sin\theta_{2}&cos\theta_{2}\\ \end{bmatrix}\begin{bmatrix}x_{0}\\ y_{0}\end{bmatrix}=C_{0}^{2}A_{0}\,. \tag{5}\]

At the same time, \(A_{2}\) can be also considered as \(A_{1}\) rotated by \(\Delta\theta\), which is why \(A_{2}\) can also be expressed as follows,

\[A_{2}=\begin{bmatrix}x_{2}\\ y_{2}\end{bmatrix}=\begin{bmatrix}cos\Delta\theta&-sin\Delta\theta\\ sin\Delta\theta&cos\Delta\theta\end{bmatrix}\begin{bmatrix}x_{1}\\ y_{1}\end{bmatrix}=C_{1}^{2}A_{1}\,. \tag{6}\]

By associating **(4)**, **(5)**, and **(6)**, we can find that the rotation matrix from \(A_{1}\) to \(A_{2}\) can be expressed as follows,

\[C_{1}^{2}=C_{0}^{2}C_{1}^{0}=\begin{bmatrix}cos\Delta\theta&-sin\Delta\theta \\ sin\Delta\theta&cos\Delta\theta\end{bmatrix}\,. \tag{7}\]

Therefore, the absolute error between the actual angle and the predicted angle is finally obtained as **(3)**.

## IV Experimental Evaluation

For accurate robot manipulation tasks, it is necessary to obtain multimodal contact information such as category, pose, location, and forces applied to the target object during contact, etc. In this regard, this section demonstrates the performance of the proposed system in recognizing multimodal contact information through experiments. The experimental setup, as shown in **Fig. 9(a)**, consists primarily of a digital force gauge, a Raspberry Pi, and a translation platform. The sensor is fixed on the translation platform to control the contact location and pose angle with the probe, while the digital force gauge with a probe is mounted on the vertical translation stages. The vertical translation stage is then controlled to establish contact between the probe and the sensor, at which point the force gauge starts recording data and sends a rising edge signal to the Raspberry Pi, which then drives the camera to take a picture for a synchronized, real-time measurement of the tactile sensing system and the force gauge.

As shown in **Fig. 9(b)**, to investigate the relationship between visual representation and normal forces, we collected more than 3,000 samples of five spherical contact probes with diameters of 10 mm, 15 mm, 20 mm, 25 mm, and 30 mm. For pose angle analysis, we used a strip-shaped contact probe for collecting different posture data with more than 2,000 samples. We developed a contact probe with a contact interface that aligns with the proposed vision-based tactile sensor, allowing us to simulate scenarios where a robot grasps a screw and when it grasps five different objects.

### Visual representations characteristics of different contact situations in tactile images

Based on the mechanism and design optimization of the vision-based tactile sensor we developed a multimodal tactile sensing system. The vision-based tactile sensor converts tactile information into high-density, high-resolution visual representations as tactile images, which can be decoupled as multimodal tactile feature information from redundant visual information. In order to investigate the ability of the multimodal vision-based tactile sensing system to distinguish between different modes of tactile information, we used different types of contact probes as test objects and analyzed their visual representations characteristics separately.

First, in this study, an induction system based on single-point contact was used to investigate the recognition of positive force. Five spherical contact probes of varying diameters were used in conjunction with a vision-based tactile sensor to apply different loads, as shown in **Fig. 10(a)**.

Fig. 8: Diagram of angle error calculation.

Fig. 9: (a) The experimental setup of multimodal contact information recognition. (b) Photos of different contact probes.
Because the spherical probes are geometrically isotropic, there is no issue of pose recognition during contact. By using probes of different diameters, the relationship between force conditions and tactile image features can be explored in an intuitive manner. As shown in **Fig. 10(a)**, three contact probes were selected to demonstrate that the normal force applied to the target object is related to the area and light field distribution of the tactile image. Under the same contact force, the tactile image produced by a larger diameter contact probe has a larger contact area and shallower shading. For example, in the load range of 0 to 3 N, as the diameter of the contact probe increases, the tactile image has a larger contact area, and the contrast of the edge shading gradually decreases. When the same diameter probe was used for contact, the tactile image produced under higher load conditions had a larger contact area and stronger contrast. For instance, when using a 10 mm diameter contact probe for contact, as the load increases, the resulting tactile image has a larger contact area, the intensity of the edge reflection light field gradually decreased, and the contrast of its edge shadow gradually increased. Furthermore, the proposed vision-based tactile sensing system can characterize different contact poses. The five spherical probes used in the previous section have geometric isotropy, so there is no difference in the contact surface for different contact poses. Therefore, a strip probe (**Fig. 10(b)**) was introduced to apply different contact poses to the sensor. For ease of subsequent data processing and discussion, a five-parameter method [45], with an angle range of 0\({}^{\circ}\) to 180\({}^{\circ}\), was used to define posture information. For different contact postures, the visual representation of the tactile image also presents different posture angles, and the distribution of the reflection light field is related to the object's contact posture angle. Additionally, as shown in the comparison between **Fig. 10(a)** and **Fig. 10(b)**, the proposed sensing system can also characterize different contact positions and object information, such as shape and texture features. Tactile images are generated when different objects are in contact with the sensor at different positions, and different object categories can be distinguished by their unique shape and texture features of the light field distribution. The position information of the objects on the contact surface can also be clearly distinguished.

### _Integrating neural network enables tactile sensing for different modes._

To validate the accuracy of the system in recognizing forces and poses through neural network system, we collected more than 3,000 samples with varying loads and more than 2,000 samples with different pose angles and evaluated the ability to distinguish different modalities of tactile information. Where all the samples were divided into training and test sets in a ratio of nine to one and 10 % of the samples in the training set were taken as the validation set for tuning the hyper-parameters. Furthermore, distinguishing different tactile information modes requires the network to learn more features from a single tactile image. Therefore, we compare three different backbone networks, a lightweight backbone ShuffleNet [47], a common backbone for target detection CSPnet [48], and a transformer-based image processing backbone Swin Transformer [49]. All three backbone networks compress the original image by a factor of 8, 16, and 32, so they can obtain feature maps with three different dimensions of 80\(\times\)80, 40\(\times\)40, and 20\(\times\)20 for decoder input.

**Table II** shows the networks evaluation for pose angle and normal force, respectively. The neural network design with Swin Transformer as the backbone network has an MAE (Mean Absolute Error) of 1.1 N for normal force recognition and 0.45 \({}^{\circ}\) for pose angle recognition; with ShuffleNet as the backbone network has an MAE of 0.7 N for normal force recognition and 0.67 \({}^{\circ}\) for pose angle recognition; and with CSPnet as the backbone network has an average absolute error of 0.2 N for normal force recognition and 0.41 \({}^{\circ}\) for pose angle recognition. The result shows that the model with CSPnet backbone outperforms the ShuffleNet and the Swin Transformer. For angle recognition, the proposed neural network subdivides angles from 0\({}^{\circ}\) to 180\({}^{\circ}\) into 180 categories with a subdivision of 1\({}^{\circ}\). Therefore, the discriminative limit of the model is to distinguish 1\({}^{\circ}\) of angular difference. The average accuracy of angle recognition for the three networks with different backbones is within the systematic error. For

Fig. 10: Tactile images of different contact situations (the scale in the figure is 10 mm). (a) Tactile images of different normal forces. (b) tactile images of different contact postures.
normal force recognition, theoretically the larger the number of model parameters the better the recognition accuracy is, so the network with CSPnet as the backbone performs better than ShuffleNet. In addition, the network with Swin Transformer as the backbone lacks some of the inductive biases inherent to convolutional neural networks, and therefore does not generalize well when trained on insufficient amounts of data in this study, despite having a larger number of parameters. Furthermore, FPS (Frame per Second) is related to the number of network parameters, so the FPS of the model with ShuffleNet is the best.

To summarize, the model with CSPnet has better performance, therefore, we choose CSPNet as the backbone in the subsequent experiments. The detailed performance of the model with CSPnet is shown in **Fig. 11**, where **Fig. 11(a)** visualizes the multimodal tactile decoupling results of the proposed system for different contact scenarios, demonstrating its ability to accurately perceive different contact poses, positions, force, and object categories. **Fig. 11(b)** shows the box plot of the normal force prediction error with 1 N intervals within the range of 0 to 10 N. The relative absolute error of all samples is 5.4 %. The average prediction errors of the sensing system are within 5 % for the 3 to 10 N range, while the average relative errors for the 0 to 3 N range are greater than 5 %, respectively. Moreover, as shown in **Fig. 11(c)**, the absolute prediction errors in all force ranges are within 0.3 N, and the MAE for all samples is 0.2 N. Here, because the normal force recognition average absolute error of the system is only up to 0.2 N, the MRE (Mean Relative Error) of the system is larger than the loading force in recognizing small loading forces, leading to a relatively large error in the small load range. **Fig. 11(d)** shows the box plot of the estimation error with 30\({}^{\text{o}}\) intervals in different angular ranges. The average prediction error in different pose angles is within 1\({}^{\text{o}}\), and only a few samples have a prediction error of up to 2\({}^{\text{o}}\). The experimental results show that, except for the system error introduced in manual data labeling, the proposed system has relatively high accuracy in pose recognition tasks.

In our experiment, distinct tactile features associated with different contact situations were clearly discernible in the corresponding contact images. In contrast, conventional approaches to multimodal tactile detection often entail the integration of structures based on disparate mechanisms, thereby introducing additional complexity to hardware design and data processing. In light of these observations, our findings demonstrate that the proposed sensing system can effectively exploit image-based features to represent diverse tactile information and achieve high-performance recognition through the integration of machine learning techniques. Thus, our results underscore the potential advantages of utilizing visual representations of tactile information in the context of multimodal tactile sensing.

### _Vision-based tactile sensing intuitively and accurately identified multimodal tactile information_

Multimodal tactile information, including object pose, category, position, and force, plays a crucial role in the precise manipulation of objects. For instance, during grasping, multimodal tactile feedback can provide the grasping force and position feedback, ensuring the accurate grasping of

Fig. 11: (a) Demonstration of the recognition output for normal force and pose angle. (b) Box plot of the relative error of the normal force identification. (c) Box plot of absolute error of normal force identification. (d) Box plot of the absolute error of angle identification.
objects. The proposed sensing system in the previous section enables us to quantitatively perceive the multimodal tactile information of objects during the grasping process by decoupling tactile images. Here, as shown in **Fig. 12**, we simulated four grasping scenarios of a screw (including the head, body, top, and bottom contact points) and collected more than 2,000 samples dividing the train set and test set in a ratio of 9 to 1. The various performances of the proposed sensing system in the grasping simulation for a screw are shown in **Fig. 13**. Here, the AP@50 (Average Precision with 50 % as the threshold) value [50], that is, the harmonic mean of recall and precision when positive samples are considered for the predicted and ground truth samples with _IoU_ greater than 50 %, was chosen as the metric to evaluate the estimation error of contact object location and contact part classes. The error of normal force and pose angle for different contact parts are shown in **Fig. 13(a)**. The MAE of normal force identification for contacting the head, body, top, and bottom are 0.19 N, 0.21 N, 0.15 N, and 0.17 N respectively, and the posture angle identification errors for different contact parts are also all within 1 \({}^{\circ}\). **Fig. 13(b)** shows the AP values of four contact parts, which all reach 100% proving that the proposed system has high accuracy in object location and contact part class prediction tasks.

In addition, to demonstrate the universality and practicality of distinguishing different tactile modes, as shown in **Fig. 14(a) and (b)**, we additionally collected more than 2,000 samples of five common objects (a USB plug, a LEGO brick, an amplifier IC, a potentiometer, and a screwdriver), which then were divided as the train set and test set in a ratio of 9 to 1. **Table III** shows the recall, precision, and AP values of the system for the five common object test sets, where the AP values of the amplifier IC, LEGO brick, and potentiometer all reach 100 %. The AP values of the screwdriver and USB plug are 94.81 % and 97.06 %, respectively, where the recall and precision of the screwdriver are 89.66 % and 92.86 %, and that of the USB plug is 97.14 % and 94.44 %. The violin plots of normal force and pose angle errors for different objects are shown in **Fig. 14(c)**. The mean absolute errors of normal force recognition for the USB plug, LEGO brick, amplifier IC, potentiometer, and screwdriver are 0.26 N, 0.17 N, 0.18 N, 0.17 N, and 0.27 N, respectively. And the overall MAE of the normal force for the test set is 0.2 N. The mean absolute errors of the pose angle recognition of the five different objects are all within 1 \({}^{\circ}\), and the overall MAE of the pose angle for the test set is 0.61 \({}^{\circ}\). In addition, there is no significant difference between the error distributions of normal force recognition and pose angle recognition for the five types of objects.

The system achieves impressive performance in the classification and location of contact objects, while maintaining a decent capability in normal force recognition and pose angle recognition, which is maintained under the same level for different contact cases as in the above experiments. Overall, the results illustrate that the proposed system can perform a perception of contact information for different modalities using only one sensor and without different data processing methods for different contact information while maintaining decent performance in each task. This ability is critical for a wide range of applications

Fig. 14: (a) Schematic diagram of five common objects. From left to right are a USB plug, a Lego brick, an amplifier IC, a potentiometer, and a screwdriver. (b) Tactile images of five common objects. (c) Violin plots of errors in normal force recognition and pose angle recognition for five objects.

Fig. 12: Schematic diagram of the grasping simulation for four cases (the scale in the figure is 10 mm).

Fig. 13: (a) Box line plot of the estimation error of normal force and pose angle for different contact parts. (b) AP values for 4 contact parts.
that require accurate manipulation of objects, such as manufacturing, robotics, and prosthetics. A video demonstration of the system's performance is also available (see **movie S1**).

## V Conclusion

Accurately obtaining multimodal tactile information has been a challenge for most tactile sensing systems. Here, this study proposes a vision-based tactile perception system that achieves multimodal tactile information perception. Compared to existing multi-sensor multimodal tactile perception systems, the proposed system only uses one sensor to achieve multimodal tactile perception providing higher integration. The system uses a soft tactile interface combined with CMOS sensor design to convert the optical response of external tactile stimuli of different modalities into a high-density, high-resolution visual representation. Due to the high density of tactile information represented visually, the proposed system does not need to perform specific decoupling designs on different modalities of tactile information as in traditional methods, thus greatly reducing the difficulty of data processing from different tactile modalities. By using neural networks, the system can decouple the hidden tactile information of different modalities in the optical tactile image, thus achieving multimodal perception. In experiments, the proposed system achieved spatial resolution at the micrometer level comparable to human tactile perception and accurately identified the location, part category, contact posture, and contact load of grasped objects in simulated grasping applications, demonstrating decent tactile feature recognition ability. We believe that the proposed system provides a new solution with high resolution, high integration, and decent information decoupling ability, and is expected to be applied to various fields of multi-modal tactile sensing tasks such as biomedical, biology, and robotics.
