Zhang Xiaorui (張小瑞)

(*School of Computer and Software, Nanjing University of Information Science & Technology, Nanjing 210044, P.R.China)(**Jiangsu Engineering Center of Network Monitoring, Nanjing University of Information Science and Technology, Nanjing 210044, P.R.China)(***School of Information and Control, Nanjing University of Information Science & Technology, Nanjing 210044, P.R.China)(****School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, 19104, USA)(*****School of Instrument Science and Engineering, Southeast University, Nanjing 210096, P.R.China)(******State key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, P.R.China)

?

Virtual brain surgery simulation system based on haptic interaction①

Zhang Xiaorui (張小瑞)②

(*School of Computer and Software, Nanjing University of Information Science & Technology, Nanjing 210044, P.R.China)(**Jiangsu Engineering Center of Network Monitoring, Nanjing University of Information Science and Technology, Nanjing 210044, P.R.China)(***School of Information and Control, Nanjing University of Information Science & Technology, Nanjing 210044, P.R.China)(****School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, 19104, USA)(*****School of Instrument Science and Engineering, Southeast University, Nanjing 210096, P.R.China)(******State key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, P.R.China)

To improve the accuracy and interactivity of soft tissue deformation simulation, a new plate spring model based on physics is proposed. The model is parameterized and thus can be adapted to simulate different organs. Different soft tissues are modeled by changing the width, number of pieces, thickness, and length of a single plate spring. In this paper, the structural design, calculation of soft tissue deformation and real-time feedback operations of our system are also introduced. To evaluate the feasibility of the system and validate the model, an experimental system of haptic interaction, in which users can use virtual hands to pull virtual brain tissues, is built using PHANTOM OMNI devices. Experimental results show that the proposed system is stable, accurate and promising for modeling instantaneous soft tissue deformation.

haptic feedback, human-computer interaction, surgery simulation, soft tissue deformation

0 Introduction

In virtual reality medical applications, haptic feedback systems can break the isolation between the real world and the virtual world. They provide an augmented environment for doctors to interact with simulated tissues much as they do with real tissues. Haptic feedback systems make it possible to build an accurate, reliable, and flexible training system. Simulating surgical procedures with such systems before performing real surgery increases the safety of the surgery and may reduce the surgery duration. Moreover, through repeated trials in the virtual surgery, an optimal solution could be found to make patients receive minimal trauma.

Brain surgery deals with the most complex and delicate organs of a human body. The complexity and multiplicity of life and cognitive functions of the brain make brain surgery very difficult. A brain surgery simulation system based on virtual reality and haptic interaction could effectively reduce training costs and improve the success rate of brain surgery. This study proposes and describes such a system based on virtual reality and two PHANTOM OMNI haptic interaction devices. The system simulates the deformation in brain tissues caused by pulling forces from virtual operations.

1 System platform

Fig.1 shows the prototype virtual brain surgery simulation system supporting haptic interaction.

1.1 Hardware components

The system is built on a computer with Intel Core2 Duo CPU, 4.00GHz, 4.00GB RAM, and graphics card HD5750. With the joysticks of the PHANTOM OMNI haptic devices, operators are able to control a pair of virtual hands to touch the virtual brain and sense feedback forces. The PHANTOM OMNI working space is 16×12×7cm, which equals the space where a person’s wrist and forearm can move and bend freely. The displacement accuracy is 0.055mm, and the maximum force is 3.3N.

Fig.1 Surgery simulation system

1.2 Software components

The models of the virtual hands and the virtual soft tissue are built using 3DS MAX 2013, OpenGL, and Microsoft Visual C++ 2012.

2 Key technologies of virtual surgery simulation

A simple and accurate model that presents efficiently the deformation and haptic characteristics of a real organ is the key of virtual surgery simulation[1]. The quality of the physical model built directly determines the accuracy and speed of the virtual surgery simulation.

2.1 Modeling the object

Among all popular methods of modeling soft tissues, finite element models[2-5]have the advantages in

high modeling precision and adjustable parameters. However, deformation modeling is complex due to expensive computation. Mass-spring models[6-11]can be quickly computed and easily implemented, and thus can be better adapted to changes in soft tissue structures. However, due to the limitations of mechanical structure, this model presents difficulties in building a system model, determining parameter values and measuring the accuracy of simulation results. Boundary element models[12]can reduce the dimension of the problem and simplify the calculation by focusing on the model boundary, but it is still challenging to make the model stable. The Long element model[13]is a highly simplified model. Although it is easy to compute, its high abstraction leads to a lower level of accuracy. The layered rhombus-chain-connected model[14]is convenient and fast to calculate, and can satisfy real-time requirements due to its simple nature. However, distortions are prone to occur at the boundary of the soft tissue because each layer of the model is composed of an identical rhombus chain structural unit. These conventional modeling methods suffer from either problems of complicated calculations or low simulation accuracy. It is still challenging to make a virtual surgery system satisfy requirements of reality, stability and real-time performance.

2.2 Modeling the interaction between a virtual agent and soft tissues

When the system detects that the virtual hands touch the surface of the virtual soft tissues, the local contact area will be filled with a virtual plate spring model with a given virtual tension F. During the virtual hand and tissue interaction, the output feedback under the given tension is calculated by using the virtual plate spring model, which can then be used for the real-time haptic portrayal of any virtual soft tissue deformation. The structure of the virtual plate spring model is shown in Fig.2.

(a) Original state of the model; (b) Stretched state of the model;

The proposed modeling method is as follows:

(1) Parameters are initialized;

(2) Under the given virtual contact tension F, when the virtual agent hits any point on the surface of the virtual soft tissue, the plate springs in the first layer will be placed under the collision point. Suppose the number of plate springs in the first layer is n1: n1=1. Let the width of the plate spring be b, its thickness be h1, and its length be l1. The plate springs in the second layer are placed under the first layer, and the second layer consists of n2pieces of plate springs, n2=3: their width is b, thickness is h2, h2=h1α, and lengths are l1q, l1q2, and l1q3, respectively. The plate springs in the third layer are placed under the second layer. The third layer has n3pieces of plate springs, n3=5; their width is b, thickness is h3, h3=h1α2, and lengths are l1q4, l1q5, l1q6, l1q7, and l1q8, respectively. The rest of the layers can be calculated analogously. Specifically, the plate springs in the i-th layer are placed under the (i-1)-th layer. The i-th layer has nipieces of plate springs, their width is b, thickness is hi, hi=h1αi-1, and length is l1q(i-1)2, l1q(i-1)2+1, l1q(i-1)2+2,…… and l1qi2-1 respectively. nirepresents the number of the plate springs in the i-th layer, where ni=2i-1, i=1,2,3,……N, N is the number of layers. The thickness of the plate spring in each layer constitutes a geometric sequence whose first term is h1and common ratio is α, where α∈[1, 2]. The length of plate springs in each layer constitutes a geometric sequence whose first term is l1, and common ratio is q, where q∈[1, 1.5]. All the plate springs are clamped together using a spring hoop whose width is B, and b≤B≤b+0.1.

(1)

(2)

The equivalent spring stiffness Piof niplate springs in the i-th layer, without considering the invalid part, is given by

(3)

(4)

(5)

Assuming that the action line of the given virtual contact tension F is the same as the center line of the virtual plate spring model producing the haptic force, and under the effect of the given virtual contact tension F, if M layers of plate springs in the virtual soft tissue model are deformed, then the M-th layer is called deformation cutoff layer (DCL).

Suppose that the given virtual contact tension F can make the deformation X1reach its given deflection value XC1, where X1is generated when n1plate spring is stretched in the first layer. In this case, we assume that the deformation caused by all plate springs in the first M-1 layers is the same as the deflection given in the first layer, and the deformation amount produced by any plate spring is not greater than the deflection XC1given in the first layer when nMplate springs in the M-th layer named DCL are stretched simultaneously.

Force F1is given by Eq.(6) when n1plate spring in the first layer is stretched:

(6)

(7)

(8)

Except for the first layer and the M-th layer named DCL, the tension Fjconsumed is given when njplate springs in the j-th layer are stretched simultaneously:

(9)

where j∈[2, M-1], and j is an integer.

(10)

(11)

The sum XMof the deformation is produced when nMplate springs in the M-th layer named DCL are stretched simultaneously. It is given by

(12)

(13)

(14)

(15)

(16)

Let tiindicate the delay time demanded for the stretch of plate springs in the i-th layer, and Tidenotes the sum of delay time required when all plate springs are stretched in the first i layers. The delay time tiof every layer forms a geometric sequence headed by the delay time t1required when the plate spring in the first layer is stretched with ω as the common ratio:

ti=ωi-1t1

(17)

The calculation begins when the virtual agent contacts the surface of soft tissues, and it is assumed that the total delay time Tishould meet Ti

(18)

The sum of the deformation of all plate springs used in the virtual haptic model can be regarded as the deformation of the virtual soft tissues. Let X denote the sum of the deformation produced when all the plate springs in the first M layers are stretched, Xiindicates the sum of the deformation produced when niplate springs in the i-th layer (i

(19)

2.3 Deformation simulation process

The haptic interactive virtual surgery simulation system is established using the driver GHOST SDK (General Haptics Open Software Toolkit Software Development Kit), which supports PHANTOM OMNI devices. The specific process of real-time simulation is shown in Fig.3.

3 Experiments and Results

In order to evaluate the reality and stability of the proposed haptic system, 35 candidates are invited to participate in the experiment. The age of 35 candidates are ranged from 22 to 28, and the average age is 24.5. Before the experiment, each candidate experiences the sense of touch for 5 minutes in the virtual brain simulation systems to learn the virtual environment.

Fig.3 Flow chart

3.1 Verification of simulation

During the experiment, as shown in Fig.4, the participants are asked to pull the soft tissue along the surface of the brain and give their evaluation of system performance. During the interaction, all the candidates

Fig.4 Simulation progress of virtual hands pulling on virtual brain tissues

can naturally feel the fine haptic information between the virtual hands and the virtual brain tissues.

The experimental results show that the proposed model has some advantages, compared with the classical finite element model[15]and mass-spring model[16]. The contents and a comparison of results of the perception experiments are shown in Table 1. So the established interactive system based on the plate spring model can effectively and naturally simulate the deformation when soft tissues are stretched under the action of the force.

During the actual interaction between haptic device and virtual object, image display is smooth and non-stop, haptic-force sense is also stable, and the simulation result is realistic. From the haptic experimental results, It is verified that the proposed method does not produce any vibration or unstable forces in haptic interaction.

Table 1 Comparison results of perception experiments

3.2 Physical behavior analysis

Generally, the FEM model can provide more excellent performance for object modeling than the other models. Hence, the haptic behavior of our model is compared with that of the FEM[17]. In the study, plotted the feedback force for the objects modeled is plotted with FEM and the plate spring model, respectively.

The haptic rendering cannot satisfy real-time requirement by FEM model because of the computational burden. Fig.5(a) shows the haptic results of the brain tissues modeled with FEM and the plate spring model. As can be seen, the haptic behavior of the plate spring model follows better than that of FEM, so our proposed model can provide stable force to users without vibration.

(a)

(b)

In order to verify the validity of the plate spring model, the discrepancy of the plate spring is calculated on the basis of the FEM value. The differential threshold for the force that the human can reliably discriminate is about 10% over the force range of 0.5～200N, for the force smaller than 0.5N, the threshold increases to 25%. From Fig.5(b), it can be seen that the difference between the feedback force based on the plate spring model and that based on FEM is smaller than the differential threshold for force that the user can reliably discriminate. Hence, our plate spring model can not only simulate the deformation of soft objects in real time but also provide a realistic feeling to users.

4 Conclusions

According to the requirements of real-time human-computer interaction of haptic system for soft tissue simulation, a virtual plate spring model is proposed in this paper. Thus, a virtual brain surgery simulation system is built using PHANTOM OMNI haptic interaction devices, realizing the deformation simulation of virtual hands pulling the brain tissues. Compared with other commonly used haptic models, it is real-time, realistic and quick in computation. Haptic perception experiments indicate that the system can provide users a vivid and efficient operation experience.

[ 1] Zhang X R, Sun W, Song A G, et al. Enhanced haptic model for real-time human-computer interaction. Chinese High Technology Letters, 2012, 22(3): 299-304 (in Chinese)

[ 2] Jog C S, Mokashi I S. A finite element method for the Saint-Venant torsion and bending problems for prismatic beams. Computers and Structures, 2014, 135: 62-72

[ 3] Lee Y C, Basaran C. A multiscale modeling technique for bridging molecular dynamics with finite element method. Journal of Computational Physics, 2013, 253(12): 64-85

[ 4] Feng S Z, Cui X Y, Li G Y. Transient thermal mechanical analyses using a face-based smoothed finite element method. International Journal of Thermal Sciences, 2013, 74(10): 95-103

[ 5] Kwack J, Masud A. A stabilized mixed finite element method for shear-rate dependent non-Newtonian fluids: 3D benchmark problems and application to blood flow in bifurcating arteries. Computational Mechanics, 2014, 53(4): 751-776

[ 6] Kang S, Lee J, Kang H C. Feature-preserving reduction of industrial volume data using gray level co-occurrence matrix texture analysis and mass-spring model. Journal of Electronic Imaging, 2014, 23(1): 1-10

[ 7] Mahmut R, Hervas J R. Robotically controlled sloshing suppression in point-to-point liquid container transfer. Journal of Vibration and Control, 2013, 19(4): 2137-2144

[ 8] Liu T T, Bargteil A W, O'brien J F, et al. Fast simulation of mass-spring systems. ACM Transaction on Graphics, 2013, 32(6): 209-216

[ 9] Fratemali F, Blesgen T, Amendola A, et al. Multiscale mass-spring models of carbon nanotube foams. Journal of the Mechanics and Physics of Solids, 2011, 59(1): 89-102

[10] del R R, Kudryavtsev M, Silva L O. Inverse Problems for Jacobi Operators II: Mass Perturbations of Semi-Infinite Mass-Spring Systems. Journal of Mathematical Physics Analysis Geometry, 2013, 9(2): 165-190

[11] Patete P, Iacono M I, Spadea M F. A multi-tissue mass-spring model for computer assisted breast surgery. Medical Engineering & Physics, 2013, 35(1): 47-53

[12] Wang P, Becker A A, Jones I A, et al. Virtual reality simulation of surgery with haptic feedback based on the boundary element method. Computers and Structures, 2007, 85(7): 331-339

[13] Barone G, Pirrotta A, Santoro R. Comparison among three boundary element methods for torsion problems: CPM, CVBEM, LEM. Engineering Analysis with Boundary Elements, 2011, 35(7): 895-907

[14] Sun W, Zhu J D, Zhang X R, et al. Real-time haptic model for soft tissue deformation in surgery simulation. Chinese High Technology Letters, 2013, 19(3): 233-239 (In Chinese)

[15] Yucesoy C A, Zeynep S F, Peter A, et al. In muscle lengthening surgery multiple aponeurotomy does not improve intended acute effects and may counter-indicate: An assessment by finite element modelling. Computer Methods in Biomechanics and Biomedical Engineering, 2013, 16(1): 12-25

[16] Patete P, Iacono M I, Spadea M F, et al. A multi-tissue mass-spring model for computer assisted breast surgery. Medical Engineering and Physics, 2013, 35(1): 47-53

[17] Sangpradit K, Liu H B, Dasgupta P. Finite-element modeling of soft tissue rolling indentation. IEEE Transactions on Biomedical Engineering, 2011, 58(12): 3319-3327

Zhang Xiaorui, born in 1979. She received her Ph.D degrees in Instrument Science and Technology Department of Southeast University in 2010. Now she is a associate professor in School of Computer and Software, Nanjing University of Information Science & Technology. Her current research interests include human-computer interaction and virtual reality technology.

10.3772/j.issn.1006-6748.2015.02.010

①Supported by the National High Technology Research and Development Programme of China (No. 2013AA010803, 2009AA01Z311, 2009AA01Z314), the National Natural Science Foundation of China (No. 61304205, 61203316, 61272379, 61103086, 41301037), the Natural Science Foundation of Jiangsu Province (BK20141002), the Open Funding Project of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Jiangsu Ordinary University Science Research Project (No. 13KJB120007), Innovation and Entrepreneurship Training Project of College Students (No. 201410300153, 201410300165), and the Excellent Undergraduate Paper (design) Supporting Project of NUIST.

②To whom correspondence should be addressed. E-mail: zxr365@126.com Received on Jan. 25， 2014***, Zhu Jiandong*, Sun Wei***, Norman I. Badler****Song Aiguo*****, Niu Jianwei******, Liu Jia***