S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

APPLIED SCIENCES AND ENGINEERING Copyright © 2024 The

Authors, some rights
Quantitative softness and texture bimodal haptic reserved; exclusive
licensee American
sensors for robotic clinical feature identification and Association for the
Advancement of
intelligent picking Science. No claim to
original U.S.
Government Works.
Ye Qiu1, Fangnan Wang1, Zhuang Zhang2, Kuanqiang Shi1, Yi Song1, Jiutian Lu1, Minjia Xu1, Distributed under a
Mengyuan Qian3, Wenan Zhang3, Jixuan Wu1, Zheng Zhang1, Hao Chai4, Aiping Liu5, Creative Commons
Hanqing Jiang2*, Huaping Wu1,6* Attribution License 4.0
(CC BY).
Replicating human somatosensory networks in robots is crucial for dexterous manipulation, ensuring the appro-
priate grasping force for objects of varying softness and textures. Despite advances in artificial haptic sensing for
object recognition, accurately quantifying haptic perceptions to discern softness and texture remains challeng-
ing. Here, we report a methodology that uses a bimodal haptic sensor to capture multidimensional static and dy-
namic stimuli, allowing for the simultaneous quantification of softness and texture features. This method
demonstrates synergistic measurements of elastic and frictional coefficients, thereby providing a universal strat-
egy for acquiring the adaptive gripping force necessary for scarless, antislippage interaction with delicate objects.
Equipped with this sensor, a robotic manipulator identifies porcine mucosal features with 98.44% accuracy and
stably grasps visually indistinguishable mature white strawberries, enabling reliable tissue palpation and intelli-

Downloaded from https://www.science.org on July 30, 2024

gent picking. The design concept and comprehensive guidelines presented would provide insights into haptic
sensor development, promising benefits for robotics.

INTRODUCTION Recent efforts behind this unmet need have led to some methods
Somatosensory networks provide precise sensory feedback through for the recognition of softness and texture separately (28–32). To es-
cutaneous mechanoreceptors in which slow adaptive (SA) and fast timate softness, sensory systems (33–37) such as self-­locked stretch-
adaptive (FA) receptors respond sensitively and selectively to static able strain sensors (38) and needle-­based modulus probes (39) were
and dynamic pressure, enabling human hands to realize object rec- developed, relying on static (force-­displacement) and dynamic (am-
ognition (e.g., softness and texture) for dexterous manipulation in plitude, phase, and frequency) principles, respectively. Yet, static
grasping tasks (1–5). The understanding of human haptic percep- measuring techniques risk damaging fragile samples resulting from
tion has promoted advancements in robotics, endowing robots with the same contact force, whereas dynamic methods encounter accu-
sophisticated haptic sensing capabilities for environment aware- racy problems caused by the amplification of mass effects at high fre-
ness, recognition of threats, and fine motor tasks, thus benefiting quencies. Furthermore, their attempts concentrate only on feedback
applications in artificial intelligence, medical diagnostics, agricul- from normal stimuli, ignoring the interference caused by sensor tilt
ture picking, and human-­robot interaction (6–12). Inspired by the on the sample surface due to shear forces in practical application cir-
functionality of SA and FA receptors, several strategies have been cumstances. To recognize surficial textures (40–47), artificial sensory
implemented to integrate flexible haptic sensors, such as piezoresis- systems based on an iontronic sensor (48) and integrated triboelectric-­
tive (13–17), piezoelectric (18–21), triboelectric (22, 23), and ca- piezoresistive sensing modules (40) were created to effectively per-
pacitive (24–27) types, into robotic systems, enabling touch-­based ceive static pressure and dynamic vibration signals. However, the
object recognition through the extraction and analysis of haptic sliding shear force and the interfacial slippage were neglected, lead-
information (e.g., pressure, vibration, and strain). However, achiev- ing to a failure to extract quantitative parameters in the recognition
ing precise execution of adaptive grasping requiring high dexterity process (43, 49). Thus far, the current methods are incapable of si-
in robotic systems remains a daunting challenge due to the diffi- multaneously quantifying softness and texture, primarily due to the
culty in determining the adaptive gripping force for objects with difficulty of synergistically decoding the multidimensional static
distinct softness and texture features. pressotactile and the dynamic vibrotactile encoding of features.
Hence, designing haptic sensors capable of accurately measuring
multidimensional static and dynamic stimuli is of considerable
1
importance toward developing a nondestructive and precise meth-
College of Mechanical Engineering, Key Laboratory of Special Purpose Equipment
and Advanced Processing Technology, Zhejiang University of Technology, Hangzhou, odology for assessing softness and texture features, greatly benefiting
Zhejiang 310023, China. 2School of Engineering, Westlake University, Hangzhou, potential applications in robotic grasping tasks that require adaptive
Zhejiang 310030, China. 3College of Information Engineering, Zhejiang University gripping force for scarless, antislippage interactions.
of Technology, Hangzhou, Zhejiang 310023, China. 4Zhijiang College of Zhejiang
University of Technology, Shaoxing, Zhejiang 312030, China. 5Center for Optoelectronics
Here, we report a quantitative softness and texture bimodal haptic
Materials and Devices, Zhejiang Sci-­Tech University, Hangzhou, Zhejiang 310018, sensor that transcends human haptic perception for implementation
China. 6Collaborative Innovation Center of High-­end Laser Manufacturing Equip- in adaptive grasping tasks of intelligent robots (Fig. 1A). The synergis-
ment (National “2011 Plan”), Zhejiang University of Technology, Hangzhou, tic effect of piezoelectric and piezoresistive modules that mimic the
Zhejiang 310023, China.
*Corresponding author. Email: wuhuaping@​gmail.​com (H.W.); hanqing.​jiang@​ functionality of FA and SA receptors within biological percep-
westlake.​edu.​cn (H.J.) tion (Fig. 1B, I) allows for the detection of both static pressure and

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 1 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

Fig. 1. Human fingertip–inspired softness and texture bimodal haptic sensor. (A) Schematic illustrations of intelligent robots with anthropomorphic perception func- Downloaded from https://www.science.org on July 30, 2024
tions for quantifying the softness and texture of objects involved in the adaptive grasping process. (B) Schematic illustrations of the structure of human skin and the bi-
modal haptic sensor in an exploded view. (I) Human skin consists of slow-­adapting mechanoreceptors [Merkel disc (MD) and Ruffini corpuscle (RC)] for static stimuli and
fast-­adapting mechanoreceptors [Meissner corpuscle (MC) and Pacinian corpuscle (PC)] for dynamic stimuli. When the fingertip touches an object, excitatory potentials
are generated and transmitted along the sensory synapse to the brain, allowing the evaluation of the softness and texture characteristics of the object. The haptic sensor
comprises two flexible sensing layers for piezoelectric and piezoresistive modes: The piezoelectric layer mimics the fast-­adapting mechanoreceptors to sense high-­
frequency dynamic stimuli, while the piezoresistive layers capture static stimuli like the slow-­adapting mechanoreceptors. The artificial haptic sensor is capable of simul-
taneously identifying softness and texture characteristics using multidimensional piezoresistive and piezoelectric feedback mechanisms and with the assistance of the
depth network. (II) The bimodal sensor, designed for dynamic and static three-­dimensional force perception, outperforms traditional sensors lacking softness and texture
feedback, thus enabling stable, damage-­free handling of delicate white strawberries without slippage.

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 2 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

high-­frequency vibrations to further identify softness and texture decreases to 35.8 mV N−1. This decrease is due to the further separa-
characteristics assisted by a deep neural network similar to the central tion between the positive and negative charge carrier centers, which
nervous system. The multidimensional sensing capabilities enable can only orient in a single direction, leading to the upper limit in the
precise elastic coefficient measurements regardless of the interaction pressure range. A uniaxial shear force of 1 Hz applied along the x
orientation while also decoding texture-­induced deformation and in- direction induces compressive strain and a corresponding increase
terfacial adhesion. The robot integrated with this bimodal haptic sen- in piezoelectric voltage at c1 and c3 elements while having negligible
sor is capable of performing challenging tasks such as identifying effects on the strain and voltage output of the c2 and c4 elements
porcine esophageal pathological tissues and adaptively grasping white (inset of Fig. 2D). The piezoelectric module exhibits excellent sensi-
strawberries with an appropriate gripping force without damage or tivity and good linearity toward shear force ranging from 0.1 to 7 N,
slippage (Fig. 1B, II). Our work presents a methodology toward quan- with respective values of 179.1 and 261 mV N−1 (Fig. 2D). Besides,
titative, nondestructive, and precise softness and texture measure- the piezoresistive module relies on the pressure-­induced conductive
ment platforms adaptable to various scenarios, which would open up pathway change, resulting in a sensitivity of 692 mV N−1 below a
many applications including clinical diagnosis, artificial intelligence, normal force of 1 N, whereas sensitivity recorded at the force range
and human-­machine interactions. of 1 to 7 N is 152 mV N−1 (Fig. 2E). The shear force response capa-
bility of the piezoresistive module yields a large resistance change
with a high sensitivity of 839 mV N−1, which becomes saturated at a
RESULTS sensitivity of 85 mV N−1 (Fig. 2F).
Sensing characterization of the bimodal The negligible piezoresistive sensing signals from the c2 and c4
multidimensional sensor elements are due to the specific structural design and assembly
To enable multiple mechanical force sensing functionalities, our method. The piezoresistive layer was connected to the interdigital
haptic sensor is designed with piezoelectric and piezoresistive lay- electrode without adhesive, merely forming a contact. This setup

Downloaded from https://www.science.org on July 30, 2024

ers (Fig. 2A), allowing for the simultaneous measurement of static confines the force within the polyimide (PI) layer and prevents ef-
and dynamic three-­dimensional force information. Specifically, the ficient force transfer to the piezoresistive layer. This observation
piezoelectric layer [poly(vinylidene fluoride-­​co-­trifluoroethylene), aligns well with the FEA results (fig. S4). We measured the sample-­
P(VDF-­TrFE); 3 mm × 3 mm × 28 μm] presents advantages in dy- to-­sample variation in the sensor and found that various sensors
namic high-­frequency stimuli (≤1100 Hz) detection and ultrafast exhibit excellent consistency and negligible measurement discrep-
response time (fig. S1), whereas the piezoresistive layer (composite ancies (the error bars are presented in Fig. 2, C to F). These mea-
of carbon nanotubes and silicone rubber; 3 mm × 3 mm × 102 μm) suring evaluations indicate that the haptic sensor is capable of
is capable of detecting static pressure (fig. S2). To realize multidi- effectively perceiving three-­dimensional forces.
mensional force perception, a top bump-­shaped layer made of soft To further evaluate the dynamic and static multidimensional
polymer is further introduced into one sensory unit (Fig. 2B), effi- force sensing capabilities, the bimodal haptic sensor, integrated with-
ciently transmitting external stimuli to the inner structure. The sen- in a vibration test system, was characterized using a sinusoidal force
sory layers are attached with patterned electrodes, which are used of 1 to 7 N in the 5-­to 300-­Hz range. The measured outputs dem-
to receive the stimuli from the top force-­transmission layer and onstrate its favorable frequency response and efficient differentia-
convert them into electrical signals. In the presented examples here, tion of three-­dimensional forces over a wide range of frequencies
the three-­dimensional force sensing principle of the piezoelectric (fig. S5). Compared to the standard force values obtained from the
sensing module is further studied through finite element analysis testing system, this sensor achieves discrimination accuracies of
(FEA) while applying the normal and shear forces. FEA results 92.8% for measuring normal force and 92.9% for detecting shear
(fig. S3A) demonstrate that the top bump-­shaped layer experiences force (fig. S6). Together, the bimodal multidimensional sensor ex-
uniform compression deformation under a uniaxial normal force hibits high sensitivity (0.52 to 261 mV N−1) for stimuli ranging from
along the z direction, resulting in four distributed piezoelectric 1 to 7 N across a wide frequency range (0 to 300 Hz). Comparative
modules (i.e., c1 to c4) with a similar piezoelectric potential of analysis reveals that our sensor demonstrates an ultrabroad fre-
240 mV. Considering that a shear force is applied along the x direc- quency bandwidth with high sensitivity (Fig. 2G and table S1), out-
tion, it leads to an asymmetrical potential in the piezoelectric layer, performing the tactile sensors reported in the literature (50–62). In
gradually ranging from −300 to 350 mV across the arrays (fig. S3B). addition, it provides excellent pressure resolution (0.01 N) and a
We then develop an analytical decoupling model (text S1) to esti- broad pressure range (0.01 to 35 N; fig. S7). Repetitive compression/
mate the three-­dimensional force applied on the sensor through the release tests over 1000 cycles with normal and shear forces of 1 N
output voltages measured on the four separated sensing units. each were performed, and the sensor exhibits no signal drift or fluc-
Responses of each sensing element in array configurations yield tuation during these cyclic tests (fig. S8). Furthermore, the investi-
separate electrical outputs that allow independent characterization gation into the long-­term stability of the bimodal sensor reveals that
and decoupling of the applied force. When the haptic sensor with it consistently maintained a constant electrical output after 60 days,
integrated bump (upper and bottom side lengths of 13 and 6 mm, suggesting high long-­term stability under pressure (fig. S9). The
respectively; height of 7.5 mm) is subjected to a normal force of aforementioned merits clearly demonstrate the substantial potential
1 Hz, uniform compressive deformations at four distributed piezo- of our sensors for various applications, including health monitoring,
electric elements (i.e., c1 to c4) yield similar strain-­induced changes wearable sensors, and intelligent robotics.
in piezoelectric voltages (inset of Fig. 2C), consistent with FEA re- In actual sensing applications, sensors frequently encounter the
sults. Real-­time voltage output reveals that the piezoelectric module simultaneous coupling of normal and shear forces, necessitating en-
can maintain a high sensitivity of 70.6 mV N−1 up to 1 N (Fig. 2C). hanced decoupling capabilities (63–67). We evaluated the decou-
As the normal force increases from 1 to 7 N, sensitivity considerably pling performance of the sensor under different directions of force

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 3 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

Downloaded from https://www.science.org on July 30, 2024

Fig. 2. Structure, sensing principle, and multidimensional force sensing performance of the bimodal haptic sensor. (A) Design concept of the bimodal haptic sen-
sor. (B) Detailed structure and optical images of the bimodal haptic sensor for dynamic and static multidimensional force sensing, including the top bump-­shaped layer
that transmits the stimuli, the sensory piezoelectric and piezoresistive layers, and electrode layers. The sensitivity of bimodal sensors based on (C and D) piezoelectric and
(E and F) piezoresistive mechanisms varies when the applied normal and shear forces range from 0.1 to 7 N. Insets show the measured voltages from piezoelectric and
piezoresistive modules under a loading force of 2 N. The error bars represent ±SD from the mean value (n = 3). (G) Comparison of the sensitivity of our bimodal haptic
sensor with existing tactile sensors (50–62).

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 4 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

loading (i.e., 10°, 40°, and 70°) and found that the bimodal haptic objects at an arbitrary interaction angle. In the application scenario
sensor exhibits a good decoupling accuracy of 90.5% (fig. S10). of intelligent robots, the practical task of gripping objects of differ-
Meanwhile, it is essential to have the capability to detect and dif- ent shapes and sizes is arbitrary for robots and cannot guarantee
ferentiate multidimensional forces while experiencing variable tem- normal orientation, resulting in complexity when applying pressing
perature conditions and mechanical deformation. The importance force that involves both normal and shear forces. As a proof-­of-­
of this feature originates from the fact that the configuration of in- concept demonstration, when a grape of unknown softness is arbi-
tegrated surfaces and variations in external environmental temper- trarily pressed, the one-­dimensional sensory feedback only classifies
atures often affect the response of sensing devices. Here, an increase the softness of the grape as category B based on a single-­channel
in temperature leads to an enhancement in the signal output (fig. S11), piezoelectric voltage. The FEA results indicate that this phenome-
which is attributed to the increased dipole moments and charge mo- non occurs due to the presence of additional shear strain in addition
bility. The bimodal haptic sensor would, therefore, need to be calibrat- to the normal strain when the sensor comes into oblique contact
ed or temperature compensated to eliminate temperature effects with the grape, resulting in a higher voltage output compared to
when it is used beyond room temperature. In addition, the bimodal when it is pressed normally against the surface (fig. S14). Conse-
haptic sensor maintains a consistent signal and a good decoupling quently, the robotic manipulator applies a gripping force of 2 N ac-
accuracy (i.e., 86.9%) when mounted on varied surfaces (figs. S12 cording to the classification, resulting in the rupture of the fresh
and S13), indicating the capability to adapt to different mechanical grape (Fig. 3F, I). In contrast to the single information–based soft-
deformations. ness analysis strategy, the three-­dimensional force sensor can de-
code the pressing angle (i.e., 15°) and make additional corrections,
Softness measurement strategy combining dynamic and thus accurately classifying the softness and steadily grasping the
static principles fresh grape with a gripping force of 0.5 N (Fig. 3F, III). Meanwhile, a
As the haptic sensor comes into contact with an object in the form of one-­dimensional force sensor, used to detect grapes if these grapes

Downloaded from https://www.science.org on July 30, 2024

active vibration (Fig. 3A), the dynamic response of the piezoelectric have reached a state of deterioration, erroneously identified them as
layer is designed to preliminary evaluate the softness. To further validate good due to recognition errors, enabling stable grasping (Fig. 3F, II).
the ability of the sensing system to discriminate softness, we mounted On the contrary, the three-­dimensional force sensor accurately de-
the sensor on a robotic manipulator and recorded dynamic changes in codes the touch angle and identifies grapes that are rotten, thus
electrical output. The robotic manipulator touches eight standard sam- avoiding their harvest and facilitating the efficient collection of only
ples [sponge, foam, Ecoflex, polydimethylsiloxane (PDMS), ethylene the suitable ones (Fig. 3F, IV). These experiments demonstrate that
vinyl acetate (EVA), polystyrene (PS), rubber, and wood (Fig. 3B)] with the proposed strategy can be regarded as a nondestructive, quantita-
a force of 1 N at a vibration frequency of 10 Hz. The measured results tive, and universal softness measurement approach for robots to
demonstrate how the piezoelectric voltage changes when the sensor in- perform delicate operations.
teracts with various sample materials (Fig. 3C, I). The sponge has a com-
paratively low elastic modulus, which reduces the deformation rate and Texture recognition approach assisted by spectrum analysis
results in a weaker piezoelectric signal of 9 mV originating from the and deep learning
dynamic response characteristics of the loading rate. Real-­time curves of Tactile texture recognition is a complex process mediated by the spa-
the sensor reveal a stable piezoelectric output across seven cycles when tial encoding of the geometrical properties of coarse textures and the
exposed to samples exhibiting different levels of softness. Consequently, vibrotactile encoding of fine textures. The haptic sensor was mount-
the actively vibrating piezoelectric layer provides voltage feedback with ed on the robotic finger to perform sliding action, and the spectrum
varying peaks based on the dynamic response characteristics (Fig. 3C, analysis on the acquired sensing data was first carried out to assist in
II), thus presenting an alternative method to conventional quasistatic identifying coarse textures (Fig. 4A). To evaluate the effectiveness of
measurements for the preliminary classification (e.g., A, B, C, and D) of the haptic sensor in detecting coarse textures, we selected six 3D-­
object softness. printed samples with varying degrees of standard roughness for
The synergistic effect of piezoelectric and piezoresistive modules analysis. Upon observation, the materials’ surfaces exhibit micro-
further allows for the deconvolution of the dynamic and static stim- morphological features characterized by peaks and dips, with two
uli during the complex identifying processes. Once the object’s soft- pitch widths along the sliding direction ranging from 0.5 to 3 mm
ness was preliminarily classified, the robotic manipulator would (inset of Fig. 4C). The acquired sliding data are then subjected to a
apply a corresponding contact force based on the classification re- fast Fourier transform (FFT) to derive the frequency response
sults, allowing the piezoresistive layer to yield different feedback (fig. S15). Results obtained from the FFT spectral analysis (Fig. 4C)
(Fig. 3D). The measuring mechanism is based on contact mechanics reveal a decrease in the main frequency as the sample spacing in-
theory, which takes into account the elastic deformation of the ob- creased, corroborating the theoretical calculation (see text S3 for
ject due to contact forces and evaluates the essential softness perfor- details). This spectrum analysis method effectively detects periodic
mance through the elastic coefficient (text S2). The quantitative information in regular roughness patterns of surface textures, con-
testing results show good agreement with the results obtained from strained by irregular fine textures, enabling comparisons between
experimental observations, with an average relative error controlled observed and theoretically predicted frequencies. Compared to
within 10% (Fig. 3E and table S2). Thus, the designed measuring commercial probe-­based texture analysis strategies, flexible haptic
strategy uses the synergistic effect of piezoelectric and piezoresistive sensing is more convenient and economical and does not damage
modules to decode the softness, providing a nondestructive meth- the sample surface, thus showing potential for applications in future
odology for quantitative identification. intelligent robotics and industrial manufacturing.
To improve the accessibility of this softness recognition ap- Recognizing fine textures presents a notable challenge for
proach, we further validated the measurement ability of sensors and sensors as these surfaces exhibit irregular patterns that include

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 5 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

Downloaded from https://www.science.org on July 30, 2024

Fig. 3. Softness recognition of the haptic sensor based on dynamic classification and static measurement strategies. (A) Schematic illustration of the softness
recognition approach used by a robotic manipulator using active vibrating motion. (B) Photographs depicting a robotic manipulator touching eight different objects with
varying degrees of softness, including sponge, foam, Ecoflex, PDMS, EVA, PS, rubber, and wood. (C) Real-­time piezoelectric outputs and their corresponding voltage peaks
obtained from the piezoelectric mode while touching the eight objects. (D) Relative voltage changes in the piezoresistive mode when the robotic manipulator applies
pressures of 0.5, 2, 5, and 10 N onto the eight objects. (E) Comparison of the elastic coefficients of the measured objects between theoretical predictions and experimen-
tal demonstrations. (F) Comparative experiment involving grape grasping by a robotic manipulator based on (I and III) one-­dimensional force and (II and IV) three-­
dimensional force sensing feedback. It shows that, with the one-­dimensional force feedback, fresh and rotten grapes are damaged and stably grasped by the robotic arm
due to misjudgment of softness. With the decoupled force feedback from the three-­dimensional haptic sensor, the fresh grape is stably grasped without damage, while
the rotten grape is given up for grasping.

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 6 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

Downloaded from https://www.science.org on July 30, 2024

Fig. 4. Texture discrimination of the haptic sensor assisted with spectrum analysis and deep learning. Schematic illustrations depict the identification of (A) coarse
textures through spectral analysis, while (B) fine textures are recognized using neural network models. (C) Comparison of main frequency from six 3D-­printed samples
with spacings ranging from 0.5 to 3 mm between theoretical analysis and experimental demonstrations. (D) Optical images of 15 fabrics with diverse complex patterns.
(E) Schematics of the process and parameters used in constructing the artificial neural networks. Rectified linear unit (Relu) followed by Batch Normalization (BN).
(F) Confusion matrix for machine learning results on 15 fabrics using four channels of feedback. The color bar represents normalized accuracy. (G) Comparison of the
frictional coefficients from touching eight different objects with varying degrees of texture, including sponge, foam, Ecoflex, PDMS, EVA, PS, rubber, and wood.

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 7 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

aperiodic variations in amplitude and spatial intervals. Accurate- transmitted to the computer for machine learning, and the results
ly quantifying textured surfaces requires the identification of were displayed on the computer display interface.
features mediated by the transduction and processing of vi- The surface of the esophageal mucosa exhibits various patterns
brations generated during scanning. To address this challenge, of wrinkling or nonuniformity. The associated elastic modulus can
emerging deep learning algorithms can effectively extract com- be affected by diseases induced by inhomogeneous growth, such as
plex information from elaborated feedback and, although they reflux esophagitis, esophageal hemangioma, Barrett’s esophagus,
rely on a large amount of data, still have the potential to assist in Crohn’s disease of esophagus, and esophageal cancer (68–72). To
identifying fine and irregular textures (Fig. 4B). Fabrics with obtain sufficient esophageal samples for recognition, the simulated
delicate and intricate surface textures were chosen as suitable morphologies of the unhealthy porcine esophagus were used as a
candidates for testing samples (Fig. 4D). A convolutional neural model system because of its ease of availability and similarity to
network (CNN) was developed to effectively extract features the human esophagus. For the purpose of simulating real patho-
from the original complex signals (Fig. 4E). The confusion map logical phenomena with diverse morphologies, we applied me-
demonstrates that the CNN method, using one-­dimensional in- chanical stimulation on the surface of the porcine esophagus to
formation, enabled the manipulator to achieve a texture recogni- create particular characteristics resembling various clinical condi-
tion accuracy above 54.55% (fig. S16). Notably, an improvement tions (Fig. 5B). The recognition process was executed to ensure
in recognition accuracy to 92.17% was achieved by incorporat- controllable and repeatable sliding, and these robotic sliding ac-
ing features learned from three-­dimensional feedback (Fig. 4F). tions were repeated 50 times. The obtained data of each status were
By integrating haptic feedback from different dimensions, a more then used for training (80%) and testing (20%) and labeled accord-
comprehensive set of features related to slip adhesion and geomet- ing to classes. After training by the CNN (fig. S17), the accuracy
ric deformation from the measured objects could be obtained, thus for clinical feature identification reached 98.44% (fig. S18).
effectively enhancing recognition accuracy. The demonstration in- To explore the real-­time clinical feature identification ability of

Downloaded from https://www.science.org on July 30, 2024

dicates that the haptic sensor is capable of detecting various tex- our bimodal sensor, the robotic manipulator continuously touched
tures, including coarse and fine patterns, by simultaneously detecting the porcine esophagi, and the final prediction results with the assis-
static and dynamic stimuli. tance of a pretrained machine learning model were intuitively dis-
To further improve the applicability of this haptic sensor in played on the software display interface (Fig. 5C and movie S1). By
robotic manipulator grasping, the quantitative analysis in the relying on sensing feedback from softness and texture recognition,
texture recognition process is enlightening for acquiring abun- the intelligent robotic manipulator can effectively recognize patho-
dant information related to sliding friction. The constant normal logical features. This demonstration of the real-­time palpation plat-
pressure is performed using the feedback of the piezoresistive form facilitates the precise identification of esophageal characteristics
module, while the dynamic shear force is determined using the of inflammation or other lesions, which are crucial for accurate dis-
piezoelectric module. By decoupling the signal from the flexible ease staging and treatment planning. Real-­time testing on porcine
three-­dimensional force sensor in the piezoelectric and piezore- esophageal tissues has demonstrated the sensors’ relevance in clini-
sistive mechanism, the friction coefficient associated with the cal settings, effectively differentiating between normal and diseased
minimum gripping force required for the measured objects can tissues. Collectively, these findings clearly indicate that sensors ca-
be obtained. To validate the accuracy, these values are compared pable of detecting the physical properties of softness and texture are
to actual friction coefficients measured using a frictional experi- unique tools in the field of gastroenterology, where these devices can
ment regarding various objects (Fig. 4G and table S3). The mea- provide valuable data for diagnosis, treatment tracking, and disease
surement error is found to be within a range of 10%, indicating monitoring.
satisfactory performance of the sensor in measuring friction co-
efficients. This demonstration shows the ability of this sensor to Agricultural application of the bimodal haptic sensors on
accurately determine the surface friction coefficient of objects intelligent picking
through the synergistic effect of the piezoelectric and piezoresis- To address the challenges of steadily picking ripe white strawberries
tive modules, providing an underlying basis for the quantifica- in the absence of visual maturity cues, we propose an innovative ap-
tion of texture perception. proach for in situ robotic sensing that focuses on the simultaneous
sensing of softness and texture for determining the appropriate op-
Medical application of the bimodal haptic sensors on clinical erating force and realizing intelligent picking (Fig. 6A). When pick-
feature identification ing strawberries, it is recommended to manipulate the portion close
The haptic sensor, capable of identifying softness and texture, is ex- to the stem as studies have shown that the best picking point is gen-
pected to enable clinical feature identification. The wrinkling mor- erally 13 to 20 mm above the calyx along the fruit stem (73). Conse-
phology and elastic modulus of the esophageal mucosa are believed quently, the robotic manipulator touches the upper portion of white
to have a crucial physiological role in maintaining the health of bio- strawberries to assess ripeness and ensure reliable grasping. To fur-
logical tissues, which are used to evaluate overall health status and ther accurately evaluate the ripeness of white strawberries, we used
identify diseases. To demonstrate the practical feasibility of the hap- a professional-­grade sweetness meter for quantification. For com-
tic sensor in identifying clinical features based on softness and tex- parison, a grasping task using the robotic manipulator without inte-
ture recognition, a smart robotic manipulator is equipped with grating the haptic sensor was first implemented (movie S2). It shows
integrated bimodal sensing modules, a data acquisition and trans- that the picked strawberries are either underripe (Fig. 6B, II) or
mission module, and a display module (Fig. 5A). The multichannel fully ripe but damaged (Fig. 6B, III), primarily because there is a
acquisition module could accurately record signals containing de- lack of sensory feedback. In contrast, we then mounted a haptic sen-
tailed characteristics of esophageal mucosa. The signals were then sor, equipped solely with softness recognition capabilities, on the

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 8 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

Downloaded from https://www.science.org on July 30, 2024

Fig. 5. Medical application of clinical feature identification by the bimodal haptic sensor and the interactive interface. (A) Schematic illustration of the clinical
feature identification system, containing the signal output information of haptic sensor, multichannel acquisition, signal processing on central processing unit (CPU),
wireless transmission, and real-­time display modules. (B) Optical images of porcine esophagus with diverse textures and softness, associated with different pathological
features (i.e., reflux esophagitis, esophageal hemangioma, Barrett’s esophagus, Crohn’s disease of esophagus, and esophageal cancer). (C) Real-­time presentation of the
computer interface showcasing the results of clinical feature recognition.

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 9 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

Downloaded from https://www.science.org on July 30, 2024

Fig. 6. Agricultural application of the bimodal haptic sensor on intelligent picking white strawberries. (A) Schematic illustration of a robotic manipulator integrated
with the bimodal sensor for grasping white strawberries via the softness and texture perception. (B) Processes of the robotic manipulator grasping white strawberries with
unknown maturity. (I) Optical image of potted white strawberries. Without the sensing feedback, the picked strawberries suffer from the problem of (II) unripe or (III) dam-
age during grasping. An enlarged optical image reveals the validation of a sweetness meter to quantitatively measure the strawberry’s sweetness, represented by Sucrose
Equivalent (SE). Equipped only with feedback functions for softness recognition, the robotic manipulator (IV) bypasses unripe strawberries and (V) carefully picks ripe
ones within maximum force to avoid damage, but occasionally, its gentle grip causes the strawberries to slip. (VI) With softness and texture feedback, the robotic ma-
nipulator executes the prepressing, vibration knocking, and slipping motions to achieve stable grasping. (C) Measurements of piezoelectric and piezoresistive outputs,
providing softness and texture feedback, were obtained from the bimodal sensor during the intelligent picking of white strawberries. (D) Success rate of recognizing and
picking a large number of white strawberries with different shapes and sizes.

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 10 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

robotic manipulator to demonstrate the task of grasping white its compact size, accurately detects three-­dimensional forces with
strawberries of indeterminate ripeness (movie S3). The softness high sensitivity (fig. S20) and excellent stability (fig. S21). Multidi-
feedback from the recognition result prompts the robotic manipula- mensional sensors of various sizes have broad potential applications
tor to bypass this unripe strawberry, and instead, it vibrates and and can be tailored to specific needs to ensure appropriate volumes.
knocks a different one (Fig. 6B, IV). After recognizing a mature We believe that integrating the haptic sensor into moving compo-
strawberry, the robotic manipulator reduces its gripping force for nents and mobile devices, such as prosthetics, robotic fingers,
the task, but the strawberry slips due to insufficient force to over- and minimally invasive surgery instruments, would greatly benefit
come the drag (Fig. 6B, V). many fields.
To further demonstrate the ability of our bimodal sensor with
softness and texture sensing feedback in grasping challenges, the
intelligent robot works by the following steps (Fig. 6B, VI, and DISCUSSION
movie S4): (i) The robotic manipulator performs a preload motion In summary, this paper reports a methodology for simultaneously
over the white strawberry, acquiring data through both piezoelec- quantifying material softness and texture using a bimodal haptic
tric, which generates an instantaneous signal upon contact, and sensor, enabling precise elastic and frictional coefficients measure-
piezoresistive sensors, the latter producing a continuous voltage ments via the synergistic effect of three-­dimensional static and dy-
response (Fig. 6C). (ii) The further knocking action implemented namic stimuli feedback. The proposed softness recognition method,
by the robotic manipulator induces periodic fluctuations in piezo- using piezoelectric predictive classification and piezoresistive quan-
electric (peak value: 147 mV) and piezoresistive sensing signals, titative detection, provides scarless assessment from various touch
leading to the classification of the white strawberry’s softness as cat- angles, overcoming the difficulties of damage and accuracy that are
egory C and determining a maximum gripping force of 5 N. (iii) commonly encountered in standard static and dynamic softness
Upon the softness recognition, the robotic manipulator executes a measurements. The texture identification approach elucidates the

Downloaded from https://www.science.org on July 30, 2024

sliding motion with a slip velocity of 12.76 mm/s to discern its tex- complex evolution of normal and shear stimuli resulting from de-
ture, where decoupling normal force through piezoresistive mod- formation friction and interfacial adhesion during texture recogni-
ules and shear force via piezoelectric modules enables an initial tion, thereby compensating for the common neglect of sliding friction
pressure determination of 2.16 N using a 0.72 friction coefficient force in current measurement methods. Accordingly, quantifying
(i.e., where Finitial/μ ≤ 0.6 Fmax). (iv) During the gripping process, the physical parameters of softness and texture identification dem-
the rotation speed of the index finger on the robotic manipulator is onstrates a nondestructive, precise, and easy-­to-­use measurement
45°/s at the proximal end and 60°/s at the distal end. The robotic technology, which is a universal method that allows for determining
manipulator determines a shear force of 1.3 N by analyzing slip sig- an appropriate grasping force to minimize damage and slippage of
nal fluctuations from the piezoelectric sensor and decoupling from fragile objects. Real-­time visualized recognition platform is further
the piezoresistive sensor, noting that the self-­locking angle falls demonstrated while the robotic manipulator identifies healthy and
outside the friction cone, prompting an adaptive increase in grip- pathological porcine mucosal features with 98.44% classification ac-
ping force until the Fx/Fz ratio is safely within the friction cone. curacy, achieving objective and reliable palpation of living tissues to
Sensing signals from both piezoelectric and piezoresistive modules effectively monitor and diagnose health disorders. The softness and
indicate normal and shear forces of 2.37 and 1.66 N, respectively, texture sensing feedback provided by this haptic sensor endow the
facilitating stable robotic grasping. robotic manipulator with human-­comparable behavior to accom-
Robotic adaptive grasping demonstrates the capability of this bi- plish challenging tasks of adaptively grasping mature white straw-
modal sensor to recognize softness and texture, thereby benefiting berries and achieving intelligent picking, thereby making it one of
intelligent picking. To adapt our method to real-­world scenarios, we the most pragmatic and effective designs to be a promising option
collected and evaluated the recognition and picking processes of for various applications. We anticipate that our haptic sensor design
101 white strawberries, varying in shape and size (Fig. 6D). The has the potential to facilitate the attainment of multimodal haptic
variation in success rates among the groups is primarily due to the perception in various areas of robotics, particularly in tasks involv-
inconsistent number of strawberries, which is influenced by natural ing stable grasping, digitized health monitoring, and haptic inter-
growth conditions of each plant. The overall success rate was 82.18% faces for advanced humanoid robots.
(table S4), which is based on a total of 101 white strawberries with
83 successfully harvested samples, indicating high reliability for
practical applications. These results collectively show that our bi- MATERIALS AND METHODS
modal haptic sensor is well equipped to perform high-­skill tasks, Preparation of the flexible piezoelectric film
which are essential for intelligent picking and help prevent losses The P(VDF-­ TrFE) powder (Piezotech) was mixed with N,N-­
that can occur with manual or indiscriminate mechanized harvest- dimethylformamide in a molar ratio of 70:30. Stirring the mixture
ing in agriculture. On the basis of this proof of concept, the remain- at room temperature for 2 hours resulted in a uniform solution.
ing challenges in practical applications will be addressed through The solution was then cast onto a petri dish and annealed at 120°C
further miniaturizing the bimodal multidimensional haptic sensor. for 2 hours to enhance crystallinity, forming the P(VDF-­TrFE)
We have developed a compact bimodal multidimensional sensor film. After peeling the film off from the petri dish, it was polarized
measuring 2.3 mm in upper side length, 5 mm in bottom side by placing it in a silicone oil bath with an electric field of 70 mega-
length, and 3.1 mm in height (fig. S19). This represents a 94% re- volts m−1 for 30 min. This polarization process led to a high
duction in size compared to the original dimensions (upper and piezoelectric property in the film, such as a piezoelectric strain
bottom side lengths of 13 and 6 mm, respectively; height of 7.5 mm). constant d33 value of 28 pC N−1 along the z-­direction. The d33
The performance investigations demonstrate that the sensor, despite values of piezoelectric films were measured by the quasistatic

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 11 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

d33 tester (ZJ-­6A of the Chinese Academy of Sciences) at room powered by a signal from a high-­precision network data analyzer
temperature. that was amplified by a power amplifier. The force magnitude input
to the sensor was measured by a calibrated piezoelectric force trans-
Preparation of the flexible piezoresistive film ducer with a sensitivity of 10 mV N−1, and the output from this
For the initial synthesis, multiwalled carbon nanotubes (Aladdin transducer was processed through a charge amplifier before record-
Tech. Inc.), sodium dodecylbenzene sulfonate surfactant, SiO2 ing on a high-­precision network data analyzer. The electrical output
nanoparticles, and the silane coupling agent KH-­560 were com- of the piezoresistive sensor was measured through a semiconductor
bined with cyclohexane in a weight ratio of 5:10:3:2:1000. The mix- parameter analyzer (4200-­SCS, Keithley). The analyzer was config-
ture was mechanically stirred at 90°C for 20 min to achieve a ured as a current supply, delivering a bias current to the sensor. This
uniform solution. Subsequently, GD401 silicone rubber was added setup enabled the measurement of voltage changes resulting from
to the cyclohexane solution to form a hybrid mixture, which was the inherent resistance variation in piezoresistive sensors. Compres-
vigorously stirred for 30 min. This mixture was then combined in a sion and tensile tests were conducted using a mechanical testing sys-
beaker and sonicated for 30 min, followed by mechanical stirring tem (Legend2345, Instron).
for 1 hour to ensure uniform dispersion. After the mixed suspen-
sion transformed into a viscous solution, it was transferred to a
vacuum drying oven for 10 min to eliminate air bubbles. Supplementary Materials
The PDF file includes:
Supplementary Text
Fabrication of the bimodal three-­dimensional force sensor
Figs. S1 to S22
Gold electrodes (150 nm in thickness) were evaporated onto a Tables S1 to S4
100-­μm-­thick PI film using an electron beam evaporator (DZS-­500) Legends for movies S1 to S4
to create the distributed electrodes. The piezoelectric layer (thick-

Downloaded from https://www.science.org on July 30, 2024

ness: 28 μm) and piezoresistive layer (thickness: 102 μm) were then Other Supplementary Material for this manuscript includes the following:
Movies S1 to S4
assembled layer by layer to form a dual-­mode sensor. The entire bi-
modal pressure sensor was encapsulated using 50 μm thick polyeth-
REFERENCES AND NOTES
ylene terephthalate (PET) films on both sides. After cleaning the 1. A. Chortos, J. Liu, Z. Bao, Pursuing prosthetic electronic skin. Nat. Mater. 15, 937–950
3D-­printed template with deionized water, a square frustum-­shaped (2016).
PDMS bump (upper side length: 6 mm; bottom side length: 13 mm; 2. K. R. Jinkins, S. Li, H. Arafa, H. Jeong, Y. J. Lee, C. Wu, E. Campisi, X. Ni, D. Cho, Y. Huang,
height: 7 mm) was prepared by casting a mixture of PDMS (Sylgard J. A. Rogers, Thermally switchable, crystallizable oil and silicone composite adhesives for
skin-­interfaced wearable devices. Sci. Adv. 8, eabo0537 (2022).
184, Dow Corning) with a ratio of part A to part B at 10:1 into the
3. L. Tian, B. Zimmerman, A. Akhtar, K. J. Yu, M. Moore, J. Wu, R. J. Larsen, J. W. Lee, J. Li, Y. Liu,
template. The surface treatment was applied to both the PET film B. Metzger, S. Qu, X. Guo, K. E. Mathewson, J. A. Fan, J. Cornman, M. Fatina, Z. Xie, Y. Ma,
and the PDMS bump using a surface treatment agent (Gubaili 707), J. Zhang, Y. Zhang, F. Dolcos, M. Fabiani, G. Gratton, T. Bretl, L. J. Hargrove, P. V. Braun,
followed by a uniform deposition of a commercially available dis- Y. Huang, J. A. Rogers, Large-­area MRI-­compatible epidermal electronic interfaces for
persion (Jule J-­808) onto the PET substrate. The PDMS bump was prosthetic control and cognitive monitoring. Nat. Biomed. Eng. 3, 194–205 (2019).
4. I. You, D. G. Mackanic, N. Matsuhisa, J. Kang, J. Kwon, L. Beker, J. Mun, W. Suh, T. Y. Kim,
then placed at the top and cured at 40°C for 30 min to assemble the J. B. Tok, Z. Bao, U. Jeong, Artificial multimodal receptors based on ion relaxation
bimodal three-­dimensional force sensor. Adhesion tests (fig. S22) dynamics. Science 370, 961–965 (2020).
revealed that three sensing devices consistently exhibited a maxi- 5. Z. Wang, N. Shi, Y. Zhang, N. Zheng, H. Li, Y. Jiao, J. Cheng, Y. Wang, X. Zhang, Y. Chen,
mum adhesive force (e.g., 32.2, 34.6, and 32.1 N), surpassing the Y. Chen, H. Wang, T. Xie, Y. Wang, Y. Ma, X. Gao, X. Feng, Conformal in-­ear bioelectronics
for visual and auditory brain-­computer interfaces. Nat. Commun. 14, 4213 (2023).
force required for their intended application.
6. Z. Zhang, Y. Long, G. Chen, Q. Wu, H. Wang, H. Jiang, Soft and lightweight fabric enables
powerful and high-­range pneumatic actuation. Sci. Adv. 9, eadg1203 (2023).
FEA of the sensors 7. B. Shih, D. Shah, J. Li, T. G. Thuruthel, Y. L. Park, F. Iida, Z. Bao, R. Kramer-­Bottiglio,
FEA was conducted using the commercial software COMSOL (ver- M. T. Tolley, Electronic skins and machine learning for intelligent soft robots. Sci. Robot. 5,
sion 6.2, standard) to study the deformation in sensing layers. The eaaz9239 (2020).
8. Y. Lee, J. Y. Oh, W. Xu, O. Kim, T. R. Kim, J. Kang, Y. Kim, D. Son, J. B. Tok, M. J. Park, Z. Bao,
interface between the PI and PET layers was set with continuity con- T. W. Lee, Stretchable organic optoelectronic sensorimotor synapse. Sci. Adv. 4, eaat7387
ditions to simulate the bonding behavior. The bump structure, sens- (2018).
ing modules, and encapsulation layers were modeled using free 9. C. Wang, K. Sim, J. Chen, H. Kim, Z. Rao, Y. Li, W. Chen, J. Song, R. Verduzco, C. Yu, Soft
tetrahedral meshing solid elements. The mechanical properties of the ultrathin electronics innervated adaptive fully soft robots. Adv. Mater. 30, e1706695
(2018).
materials were set as follows: PI and PET were modeled with linear
10. H. Shim, K. Sim, F. Ershad, P. Yang, A. Thukral, Z. Rao, H. J. Kim, Y. Liu, X. Wang, G. Gu, L. Gao,
elastic behavior, with an elastic modulus of EPI = 7.9 GPa and a Pois- X. Wang, Y. Chai, C. Yu, Stretchable elastic synaptic transistors for neurologically
son’s ratio of νPI = 0.36 for PI and an elastic modulus of EPET = 6.9 GPa integrated soft engineering systems. Sci. Adv. 5, eaax4961 (2019).
and a Poisson’s ratio of νPET = 0.43 for PET. An elastic modulus of 11. Z. Wang, L. Wang, Y. Wu, L. Bian, M. Nagai, R. Jv, L. Xie, H. Ling, Q. Li, H. Bian, M. Yi, N. Shi,
EPDMS = 750 kPa and a Poisson’s ratio of νPDMS = 0.49 were for PDMS, X. Liu, W. Huang, Signal filtering enabled by spike voltage-­dependent plasticity in
metalloporphyrin-­based memristors. Adv. Mater. 33, e2104370 (2021).
and an elastic modulus of EPiezoresistive = 127 MPa and a Poisson’s ratio 12. Y. Lin, C. Zhang, W. Tang, Z. Jiao, J. Wang, W. Wang, Y. Zhong, P. Zhu, Y. Hu, H. Yang, J. Zou,
of νPiezoresistive = 0.24 were for the piezoresistive layer. A bioinspired stress-­response strategy for high-­speed soft grippers. Adv. Sci. 8, 2102539
(2021).
Measurements of the sensing performance of the sensors 13. M. Cai, Z. Jiao, S. Nie, C. Wang, J. Zou, J. Song, A multifunctional electronic skin based on
patterned metal films for tactile sensing with a broad linear response range. Sci. Adv. 7,
The voltage output of the piezoelectric sensor was recorded by a
eabl8313 (2021).
piezoelectric data acquisition system (Beijing Keshang Instrument 14. Z. Zhang, Z. Xu, L. Emu, P. Wei, S. Chen, Z. Zhai, L. Kong, Y. Wang, H. Jiang, Active
Technology Co. Ltd). For high-­frequency loading, the sensor was mechanical haptics with high-­fidelity perceptions for immersive virtual reality. Nat. Mach.
positioned under a cylindrical probe, operated by an exciter, and Intell. 5, 643–655 (2023).

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 12 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

15. S. Li, G. Liu, R. Li, Q. Li, Y. Zhao, M. Huang, M. Zhang, S. Yin, Y. Zhou, H. Tang, L. Wang, 39. X. Yu, H. Wang, X. Ning, R. Sun, H. Albadawi, M. Salomao, A. C. Silva, Y. Yu, L. Tian, A. Koh,
G. Fang, Y. Su, Contact-­resistance-­free stretchable strain sensors with high repeatability C. M. Lee, A. Chempakasseril, P. Tian, M. Pharr, J. Yuan, Y. Huang, R. Oklu, J. A. Rogers,
and linearity. ACS Nano 16, 541–553 (2022). Needle-­shaped ultrathin piezoelectric microsystem for guided tissue targeting via
16. S. M. Won, H. Wang, B. H. Kim, K. Lee, H. Jang, K. Kwon, M. Han, K. E. Crawford, H. Li, Y. Lee, mechanical sensing. Nat. Biomed. Eng. 2, 165–172 (2018).
X. Yuan, S. B. Kim, Y. S. Oh, W. J. Jang, J. Y. Lee, S. Han, J. Kim, X. Wang, Z. Xie, Y. Zhang, 40. S. Chun, W. Son, H. Kim, S. K. Lim, C. Pang, C. Choi, Self-­powered pressure-­and
Y. Huang, J. A. Rogers, Multimodal sensing with a three-­dimensional piezoresistive vibration-­sensitive tactile sensors for learning technique-­based neural finger skin. Nano
structure. ACS Nano 13, 10972–10979 (2019). Lett. 19, 3305–3312 (2019).
17. C. Shi, Z. Zou, Z. Lei, P. Zhu, W. Zhang, J. Xiao, Heterogeneous integration of rigid, soft, 41. H. Yao, W. Yang, W. Cheng, Y. J. Tan, H. H. See, S. Li, H. P. A. Ali, B. Z. H. Lim, Z. Liu, B. C. K. Tee,
and liquid materials for self-­healable, recyclable, and reconfigurable wearable Near-­hysteresis-­free soft tactile electronic skins for wearables and reliable machine
electronics. Sci. Adv. 6, eabd0202 (2020). learning. Proc. Natl. Acad. Sci. U.S.A. 117, 25352–25359 (2020).
18. J. Zhang, H. Yao, J. Mo, S. Chen, Y. Xie, S. Ma, R. Chen, T. Luo, W. Ling, L. Qin, Z. Wang, 42. C. Mu, Y. Song, W. Huang, A. Ran, R. Sun, W. Xie, H. Zhang, Flexible normal-­tangential force
W. Zhou, Finger-­inspired rigid-­soft hybrid tactile sensor with superior sensitivity at high sensor with opposite resistance responding for highly sensitive artificial skin. Adv. Funct.
frequency. Nat. Commun. 13, 5076 (2022). Mater. 28, 1707503 (2018).
19. Y. Qiu, C. Wang, X. Lu, H. Wu, X. Ma, J. Hu, H. Qi, Y. Tian, Z. Zhang, G. Bao, H. Chai, 43. Y. Li, M. Zhao, Y. Yan, L. He, Y. Wang, Z. Xiong, S. Wang, Y. Bai, F. Sun, Q. Lu, Y. Wang, T. Li,
J. Song, A. Liu, A biomimetic drosera capensis with adaptive decision-­predation behavior T. Zhang, Multifunctional biomimetic tactile system via a stick-­slip sensing strategy for
based on multifunctional sensing and fast actuating capability. Adv. Funct. Mater. 32, human-­machine interactions. Npj Flex. Electron. 6, 46 (2022).
2110296 (2021). 44. S. Shu, Z. Wang, P. Chen, J. Zhong, W. Tang, Z. L. Wang, Machine-­learning assisted
20. F. Wang, P. Jin, Y. Feng, J. Fu, P. Wang, X. Liu, Y. Zhang, Y. Ma, Y. Yang, A. Yang, X. Feng, electronic skins capable of proprioception and exteroception in soft robotics. Adv. Mater.
Flexible Doppler ultrasound device for the monitoring of blood flow velocity. Sci. Adv. 7, 35, e2211385 (2023).
eabi9283 (2021). 45. K. Kim, M. Sim, S. H. Lim, D. Kim, D. Lee, K. Shin, C. Moon, J. W. Choi, J. E. Jang, Tactile
21. Y. Qiu, Y. Tian, S. Sun, J. Hu, Y. Wang, Z. Zhang, A. Liu, H. Cheng, W. Gao, W. Zhang, H. Chai, avatar: Tactile sensing system mimicking human tactile cognition. Adv. Sci. 8, 2002362
H. Wu, Bioinspired, multifunctional dual-­mode pressure sensors as electronic skin for (2021).
decoding complex loading processes and human motions. Nano Energy 78, 105337 46. H. Qiao, S. Sun, P. Wu, Non-­equilibrium-­growing aesthetic ionic skin for fingertip-­like
(2020). strain-­undisturbed tactile sensation and texture recognition. Adv. Mater. 35, e2300593
22. X. Zhao, Z. Zhang, L. Xu, F. Gao, B. Zhao, T. Ouyang, Z. Kang, Q. Liao, Y. Zhang, Fingerprint-­ (2023).

Downloaded from https://www.science.org on July 30, 2024

inspired electronic skin based on triboelectric nanogenerator for fine texture 47. X. Qu, Z. Liu, P. Tan, C. Wang, Y. Liu, H. Feng, D. Luo, Z. Li, Z. L. Wang, Artificial tactile
recognition. Nano Energy 85, 106001 (2021). perception smart finger for material identification based on triboelectric sensing. Sci.
23. H. Zhao, K. O'Brien, S. Li, R. F. Shepherd, Optoelectronically innervated soft prosthetic Adv. 8, eabq2521 (2022).
hand via stretchable optical waveguides. Sci. Robot. 1, eaai7529 (2016). 48. N. Bai, Y. Xue, S. Chen, L. Shi, J. Shi, Y. Zhang, X. Hou, Y. Cheng, K. Huang, W. Wang,
24. J. Liu, S. Jiang, W. Xiong, C. Zhu, K. Li, Y. Huang, Self-­healing kirigami assembly strategy for J. Zhang, Y. Liu, C. Guo, A robotic sensory system with high spatiotemporal resolution for
conformal electronics. Adv. Funct. Mater. 32, 2109214 (2021). texture recognition. Nat. Commun. 14, 7121 (2023).
25. N. Bai, L. Wang, Q. Wang, J. Deng, Y. Wang, P. Lu, J. Huang, G. Li, Y. Zhang, J. Yang, K. Xie, 49. G. Lee, J. H. Son, S. Lee, S. W. Kim, D. Kim, N. N. Nguyen, S. G. Lee, K. Cho, Fingerpad-­
X. Zhao, C. F. Guo, Graded intrafillable architecture-­based iontronic pressure sensor with inspired multimodal electronic skin for material discrimination and texture recognition.
ultra-­broad-­range high sensitivity. Nat. Commun. 11, 209 (2020). Adv. Sci. 8, 2002606 (2021).
26. X. Zeng, Y. Liu, F. Liu, W. Wang, X. Liu, X. Wei, Y. Hu, A bioinspired three-­dimensional 50. C. H. Chuang, W. B. Dong, W. B. Lo, Flexible piezoelectric tactile sensor with structural
integrated e-­skin for multiple mechanical stimuli recognition. Nano Energy 92, 106777 electrodes array for shape recognition system, in 2008 3rd International Conference on
(2022). Sensing Technology (IEEE, 2008), pp. 504–507.
27. J. Ye, F. Zhang, Z. Shen, S. Cao, T. Jin, X. Guo, Z. Li, L. Lin, Y. Zhang, Tunable seesaw-­like 3D 51. J. Le, F. Lv, J. Lin, Y. Wu, Z. Ren, Q. Zhang, S. Dong, J. Luo, J. Shi, R. Chen, Z. Hong, Y. Huang,
capacitive sensor for force and acceleration sensing. Npj Flex. Electron. 5, 28 (2021). Novel sandwich-­structured flexible composite films with enhanced piezoelectric
28. C. M. Boutry, M. Negre, M. Jorda, O. Vardoulis, A. Chortos, O. Khatib, Z. Bao, A performance. ACS Appl. Mater. Inter. 16, 1492–1501 (2023).
hierarchically patterned, bioinspired e-­skin able to detect the direction of applied 52. X. Chen, J. Shao, H. Tian, X. Li, C. Wang, Y. Luo, S. Li, Scalable imprinting of flexible
pressure for robotics. Sci. Robot. 3, eaau6914 (2018). multiplexed sensor arrays with distributed piezoelectricity-­enhanced micropillars for
29. S. Sundaram, P. Kellnhofer, Y. Li, J. Y. Zhu, A. Torralba, W. Matusik, Learning the signatures dynamic tactile sensing. Adv. Mater. Technol. 5, 2000046 (2020).
of the human grasp using a scalable tactile glove. Nature 569, 698–702 (2019). 53. K.-­H. Yu, M.-­J. Yoon, T.-­K. Kwon, S.-­C. Lee, Distributed flexible tactile sensor system. Int. J.
30. Q. Shi, Z. Sun, X. Le, J. Xie, C. Lee, Soft robotic perception system with ultrasonic Appl. Electromagn. Mech. 18, 53–65 (2003).
auto-­positioning and multimodal sensory intelligence. ACS Nano 17, 4985–4998 (2023). 54. S. Khan, S. Tinku, L. Lorenzelli, R. S. Dahiya, Flexible tactile sensors using screen-­printed
31. G. Li, S. Liu, L. Wang, R. Zhu, Skin-­inspired quadruple tactile sensors integrated on a robot P(VDF-­TrFE) and MWCNT/PDMS composites. IEEE Sens. J. 15, 3146–3155 (2015).
hand enable object recognition. Sci. Robot. 5, eabc8134 (2020). 55. C. Wei, H. Zhou, B. Zheng, H. Zheng, Q. Shu, H. Du, A. Ma, H. Liu, Fully flexible and
32. L. Beker, N. Matsuhisa, I. You, S. R. A. Ruth, S. Niu, A. Foudeh, J. B. Tok, X. Chen, Z. Bao, A mechanically robust tactile sensors containing core-­shell structured fibrous piezoelectric
bioinspired stretchable membrane-­based compliance sensor. Proc. Natl. Acad. Sci. U.S.A. mat as sensitive layer. Chem. Eng. J. 476, 146654 (2023).
117, 11314–11320 (2020). 56. K. Kumari, R. Reeshma, D. S. ArunKumar, S. Meti, M. R. Rahman, Fabrication of Ag/
33. E. Song, Z. Xie, W. Bai, H. Luan, B. Ji, X. Ning, Y. Xia, J. M. Baek, Y. Lee, R. Avila, H.-­Y. Chen, PDMS-­TiO2 flexible piezoresistive pressure sensor. Physica B. Condens. Matter 597, 412386
J.-­H. Kim, S. Madhvapathy, K. Yao, D. Li, J. Zhou, M. Han, S. M. Won, X. Zhang, D. J. Myers, (2020).
Y. Mei, X. Guo, S. Xu, J.-­K. Chang, X. Yu, Y. Huang, J. A. Rogers, Miniaturized 57. C. Ai, X. Zhao, Y. Bai, Y. Li, D. Wen, Fabrication and characteristic of force sensor based on
electromechanical devices for the characterization of the biomechanics of deep tissue. piezoelectric effect of Li-­doped ZnO thin films. Mod. Phys. Lett. B. 32, 1850208 (2018).
Nat. Biomed. Eng. 5, 759–771 (2021). 58. S. Kärki, J. Lekkala, H. Kuokkanen, J. Halttunen, Development of a piezoelectric polymer
34. S. Chun, J.-­S. Kim, Y. Yoo, Y. Choi, S. J. Jung, D. Jang, G. Lee, K.-­I. Song, K. S. Nam, I. Youn, film sensor for plantar normal and shear stress measurements. Sens. Actuators A Phys.
D. Son, C. Pang, Y. Jeong, H. Jung, Y.-­J. Kim, B.-­D. Choi, J. Kim, S.-­P. Kim, W. Park, S. Park, An 154, 57–64 (2009).
artificial neural tactile sensing system. Nat. Electron. 4, 429–438 (2021). 59. K. L. Xi, Y. C. Wang, D. Q. Mei, G. H. Liang, Z. C. Chen, A flexible tactile sensor array based
35. C. Dhong, R. Miller, N. B. Root, S. Gupta, L. V. Kayser, C. W. Carpenter, K. J. Loh, on pressure conductive rubber for three-­axis force and slip detection, in 2015 IEEE
V. S. Ramachandran, D. J. Lipomi, Role of indentation depth and contact area on human International Conference on Advanced Intelligent Mechatronics (AIM) (IEEE, 2015), pp.
perception of softness for haptic interfaces. Sci. Adv. 5, eaaw8845 (2019). 476–481.
36. Y. Qiu, S. Sun, X. Wang, K. Shi, Z. Wang, X. Ma, W. Zhang, G. Bao, Y. Tian, Z. Zhang, H. Ding, 60. Y. Liu, L. Weng, Z. Li, Y. Chen, B. Hu, K. Liu, Design and characterization of large-­range 3-­D
H. Chai, A. Liu, H. Wu, Nondestructive identification of softness via bioinspired force tactile sensor based on Fe-­Ga alloy. IEEE Sens. J. 23, 30249–30258 (2023).
multisensory electronic skins integrated on a robotic hand. Npj Flex. Electron. 6, 45 (2022). 61. T. Zhang, S. Fan, L. I. Jiang, H. Liu, Development and experiment analysis of
37. Z. Cui, W. Wang, H. Xia, C. Wang, J. Tu, S. Ji, J. M. R. Tan, Z. Liu, F. Zhang, W. Li, Z. Lv, Z. Li, anthropomorphic prosthetic hand with flexible three-­axis tactile sensor. Int. J. HR 10,
W. Guo, N. Y. Koh, K. B. Ng, X. Feng, Y. Zheng, X. Chen, Freestanding and scalable 1350028 (2013).
force-­softness bimodal sensor arrays for haptic body-­feature identification. Adv. Mater. 62. X. Chen, J. Shao, H. Tian, X. Li, Y. Tian, C. Wang, Flexible three-­axial tactile sensors with
34, e2207016 (2022). microstructure-­enhanced piezoelectric effect and specially-­arranged piezoelectric arrays.
38. Z. Cui, W. Wang, L. Guo, Z. Liu, P. Cai, Y. Cui, T. Wang, C. Wang, M. Zhu, Y. Zhou, W. Liu, Smart Mater. Struct. 27, 025018 (2018).
Y. Zheng, G. Deng, C. Xu, X. Chen, Haptically quantifying young’s modulus of soft 63. R. X. Yang, W. Q. Zhang, N. Tiwari, H. Yan, T. J. Li, H. Y. Cheng, Multimodal sensors with
materials using a self-­locked stretchable strain sensor. Adv. Mater. 34, e2104078 (2022). decoupled sensing mechanisms. Adv. Sci. 9, 2202470 (2022).

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 13 of 14

S c i e n c e A d v a n c e s | R e s e ar c h A r t i c l e

64. F. Zhang, P. C. Ma, J. X. Wang, Q. Zhang, W. Feng, Y. W. Zhu, Q. B. Zheng, Anisotropic 72. R. Ishii, K. Ohata, R. Sawada, N. Takeuchi, M. Kurebayashi, R. Inamoto, S. Takayanagi,
conductive networks for multidimensional sensing. Mater. Horiz. 8, 2615–2653 Y. Kimoto, M. Nohara, B. Liu, R. Negishi, Y. Minato, T. Muramoto, Successful esophageal
(2021). endoscopic submucosal dissection with intraoperative release of stenosis due to
65. R. Li, B. Peng, Implementing monocular visual-­tactile sensors for robust manipulation. previous endoscopic submucosal dissection scarring. DEN Open 2, e87 (2022).
Cyborg Bionic Syst. 2022, 9797562 (2022). 73. Y. Yu, K. Zhang, H. Liu, L. Yang, D. Zhang, Real-­time visual localization of the picking points
66. J. Zhu, J. Y. Li, Y. Tong, T. Q. Hu, Z. Q. Chen, Y. Xiao, S. H. Zhang, H. B. Yang, M. Gao, T. S. Pan, for a ridge-­planting strawberry harvesting robot. IEEE Access 8, 116556–116568 (2020).
H. Y. Cheng, Y. Lin, Recent progress in multifunctional, reconfigurable, integrated liquid
metal-­based stretchable sensors and standalone systems. Prog. Mater. Sci. 142, 101228 Acknowledgments: We thank B. Li and G. Xu for participation in the discussions about clinical
(2024). feature identification. Funding: This work was supported by the National Natural Science
67. L. Li, X. Li, B. Ouyang, H. Mo, H. Ren, S. Yang, Three-­dimensional collision avoidance Foundation of China (grant nos. 12372168, 12202388, 32101119, 52205031, and 12350003),
method for robot-­assisted minimally invasive surgery. Cyborg Bionic Syst. 4, 0042 (2023). Zhejiang Provincial Natural Science Foundation of China (grant nos. LQ22A020009 and
68. B. Cha, H. Kim, V. Truong, S. J. Oh, S. Jeong, H. W. Kang, Feasibility study on endoscopic LR20A020002), and the “Pioneer” and “Leading Goose” R&D Program of Zhejiang (grant no.
balloon-­assisted laser treatment (EBLT) of gastroesophageal reflux disease (GERD) in 2023C01051). Author contributions: Conceptualization: Y.Q., K.S., and H.W. Methodology: Y.Q.,
in vivo porcine model. Biomedicines 11, 1656 (2023). F.W., Zhuang Zhang, M.X., H.C., J.W., and M.Q. Investigation: F.W., K.S., J.L., A.L., Y.S., and H.J.
69. J. H. Lee, H. J. Chun, B. Keum, E. S. Kim, Y. T. Jeen, A case of esophageal hemangioma Visualization: Y.Q., Zheng Zhang, K.S., J.L., M.X., and W.Z. Writing–review and editing: Y.Q., H.J.,
treated with endoscopic injection sclerotherapy. Gastrointest. Endosc. 85, 1093–1094 and H.W. Competing interests: The authors declare that they have no competing interests.
(2017). Data and materials availability: All data needed to evaluate the conclusions in the paper are
70. S. Liu, S. Peng, M. Zhang, Z. Wang, L. Li, Multimodal integration for Barrett’s esophagus. present in the paper and/or the Supplementary Materials.
iScience 27, 108437 (2024).
71. R. V. Rodrigues, M. Sladek, K. Katsanos, C. J. van der Woude, J. Wei, S. R. Vavricka, N. Teich, Submitted 4 March 2024
P. Ellul, E. Savarino, M. Chaparro, D. Beaton, A. M. Oliveira, M. Fragaki, A. B. G. Shitrit, Accepted 20 June 2024
L. Ramos, K. Karmiris, E. C. Investigators, Diagnosis and outcome of oesophageal Crohn’s Published 24 July 2024
disease. J. Crohns Colitis 14, 624–629 (2020). 10.1126/sciadv.adp0348

Downloaded from https://www.science.org on July 30, 2024

Qiu et al., Sci. Adv. 10, eadp0348 (2024) 24 July 2024 14 of 14