Haptic Suturing Surgical Simulator

A Prototype Haptic Suturing Simulator

Randy Haluck, M.D.¹, Roger Webster, Ph.D.²,
Dean Zimmerman², Betty Mohler², Alan Synder, Ph.D.¹, Mike Melkonian, M.D.¹

¹Department of Minimally Invasive Surgery
Penn State University College of Medicine
Milton S. Hershey Medical Center
Hershey, PA USA 17033
rhaluck@psu.edu

²Department of Computer Science
School of Science and Mathematics
Millersville University
Millersville, PA. USA 17551
webster@cs.millersville.edu

Abstract. A new haptic simulation designed to teach basic suturing for simple wound closure is described. The simulator incorporates several interesting components such as real-time modeling of deformable skin, tissue and suture material and real-time recording of state of activity during the task using a finite state model. Initial work and results with the simulator are encouraging. Actual needle holders are attached to the haptic device as the graphics of the hemostats, needle, sutures, and virtual skin are displayed and updated in real-time. Although somewhat realistic tactile force feedback is provided to the learner, we are currently experimenting with the haptic programming issues of trying to simulate 6DOF twist constraints with a 3DOF device, and angular collision force feedback of a curved needle with the tissue model. Ongoing development is focused upon providing users with more accurate force feedback, a scoring mechanism, and added instructional functions. The goal is to provide an efficient method to learn the suturing procedure by measuring surgical skills in a simulator.

1. Introduction.

Suturing is a common surgical procedure that requires good hand-eye coordination and dexterous skill on the part of the administering surgeon or nurse. The ability to suture properly is a necessary and fundamental skill for many surgical procedures. Incorrect techniques resulting from inadequate training can result in sub optimal outcomes. The disadvantages for the patient are obvious. The patient is forced to absorb the errors of the medical student. The possibility of incorrectly suturing can be stressful for the student, and may lead to increased anxiety. With our system the student can practice the virtual procedure at any time without the fear and anxiety of performing suture stitching on a real patient.

Researchers (in [1],[2],[3],[4],[5],[6]), have attempted to solve some of the problems that arise in the development of a generalized force feedback surgical simulator, though no one has developed a readily available system for widespread use. Some researchers have limited the training application (see [7], [8], [9],[10]), although few commercial haptic surgical simulators are in production. The intent of this project is to develop a haptic suturing simulator that is effective, not cost prohibitive, relatively simple to maintain, limited to suturing, and available for widespread use. This would also provide a platform to perform an analysis of surgical skill.

Figure 1. Hemostats are firmly secured to the Phantom end effector by a specially built bracket.

2. Hardware.

The haptic device used is the Sensable Technologies Phantom 1.5 Desktop unit (see Figure 1). The Phantom provides force feedback so students can actually feel the soft tissue surrounding a wound and the appropriate forces when suturing soft tissue (pulling and pushing a needle through skin and pulling the suture material). The development computer is a Windows workstation with dual Pentium processors and the OpenGL graphics accelerator. A "Reachin Display" unit is used to provide the user with the ergonomic feel of actual surgery. The Crystal Eyes stereo glasses are used to provide 3D stereo graphic imagery, enhancing the effect of the simulation experience. The medical student feels he/she can "reach" into the 3D display (see Figure 2). The Phantom is mounted upside down inside the Reachin frame unit. This hardware setup can produce a realistic haptic and ergonomic interaction.

Figure 2. The "Reachin Display" unit is used so that the user has the ergonomic and haptic feel of actual surgery.
The phantom end effector is held under the mirror that reflects the stereo imagery of the mounted monitor.

3. Modeling, Software and Haptics.

The three dimensional (3D) models of the virtual needle and hemostat tool (needle holders) were built in 3D Studio Max and are stored as 3ds files. These models are loaded into the simulation (both in graphics and in haptics). The hemostat model is attached in the scene graph to the Phantoms position and orientation. A pair of actual hemostats (needle holders) is physically secured to the Phantom end effector.

The skin or soft tissue is modeled as a mesh of mass-springs (dynamic vertices) with a wound texture map wrapped onto the geometry. This consists of a set of point masses (nodes) connected to each other with a network of springs and dampers. Mass points are extracted from the vertices of the mesh object. Each mass point is linked to its neighbor's mass point and to every other mass point by massless springs and dampers. Internal and external forces act upon the springs. Each spring has a length > 0, and its original length is calculated at initialization time. The dynamics of the M x N masses are calculated as such: each mass is positioned at time t on the point P_i,j (t), , where i = 1,...M and j= 1,...,N. The morphology of the system is controlled by Newton's fundamental law of dynamics: F_i,j = ma_i,j, where m is the mass of each point P_i,j, and a_i,j is the acceleration caused by force F_i,j. Mass points (vertices) around the edge of the mesh are nailed down to prevent dynamic movement of the mesh geometry as a whole. In the case of arbitrary 3D geometry we nail down those vertices that intersect a polygonal object that the 3D geometry is resting on, for example, a stomach resting on the abdomen wall.

Figure 3. A mass-springs grid of dynamic vertices is used to model tissue displacement.

The simulation software calculates contact forces and generates tissue displacement (see figure 4). The resistant force calculations vary depending upon the depth of insertion, and the insertion angle of the curved needle. The forces also change when the needle is initially inserted or punctures the skin. As the user performs the stitching procedure the software pulls the stitches together utilizing the mass-springs deformation. The user is constrained (force effect) when the needle has penetrated the skin. The forces ease off if the user pulls the needle out in the same path as it was inserted. If the user attempts to move an inserted needle along the plane of the skin, a dynamic line constraint provides force feedback. The user is constrained from sliding the needle along the plane once it has penetrated the skin. The user can, of course, continue penetration of the needle into the soft tissue. Forces are exerted if the user deviates from the natural penetration of the projected needle path. The Phantom device, however, has only 3 degrees of freedom (3 DOF)[13]. To generate true 6 DOF force feedback (with twist constraints), a 6 DOF force feedback device must be used. We are attempting to simulate the twist constraint by using a point shell model around the needle tip within the confines of the projected needle path.

Figure 4. Screen shot of vertex displacements using mass-spring model.

Figure 5 shows Mr. John Chance's actual wound texture map picture. A mass-springs model of the deformation of soft tissue allows the user to probe the skin, push and pull the needle, as well as pull the stitches together (see Figure 6).

Figure 5. Wound image of cut..

Figure 6. Screen shot of haptic virtual suturing.

The user also feels the forces of pulling on the suture when the suture string is drawn tight. A Ghost manipulator is used to provide the forces of the suture pulling. At each step in the suturing process, the student feels the haptic forces. The software also manages the real time 3D computer graphics of the deformation of virtual skin, and the movement of the virtual needle, sutures, and hemostats. Another software module draws the 3D virtual suture stitches as the user creates them. As the user pulls the needle out of the skin, the 3D suture material is drawn from the puncture in the skin to the needle. The haptics software modules make calls to the General Haptic Software Toolkit (GHOST) development kit from Sensable Technologies. The graphics modules make calls to OpenGL. The haptics loop must run at 1KHz to keep the haptic state persistent and stable. The graphics simulation loop will run at 30Hz. The software loads all of the 3D models into the graphics loop and some into the haptics application loop (only those that need force feedback).

Another software module records the motions of the user. This is accomplished by recording the positions and orientations of the Phantom encoders and all 3D graphics objects. Thus, the 3D graphics (virtual needle, sutures, hemostats) are used to replay the suturing technique, showing the medical student what he/she did during the training session.

Software to analyze the surgical motions for scoring purposes is in development. We utilize the Rosen, et. al. method (with modifications) of a finite state machine model of states and their transitions to show motions of surgical procedures [14]. States include: Idle, Grasping the needle with the hemostats, Puncturing the skin, Pushing the needle through the skin, Opening the hemostats, Closing the hemostats, Pulling the needle through the skin, etc. The original idea for a finite state machine model of surgical motions comes from the groundbreaking work of Jacob Rosen, Mark MacFarlane, Christina Richards, Blake Hannaford, Mika Sinanan in [14]. Figure 8 shows a simple Finite State machine with a subset of the states. An analysis of the data extracted by the software will measure the transitions from state to state, and the amount of time the user spent at each state.

Figure 8. Screen Shot of Finite State Machine Showing a Subset of States and Transitions.

4. Conclusion.

Initial work and results with the haptic suturing simulator are encouraging. Actual hemostats (needle holders) are attached to the haptic device as the graphics of the hemostats, needle, sutures, and virtual skin are displayed and updated in real-time. Although somewhat realistic tactile force feedback is provided to the learner, we are currently experimenting with the haptic programming issues of trying to simulate 6DOF twist constraints with a 3DOF device, accurate compliance, and angular collision force feedback of a curved needle with the tissue model. The scoring mechanism to measure and track the user's performance is currently being developed. This mechanism will also record the amount of time it took to complete the procedure. A numeric score will be displayed for the user upon completion of the simulation. Ongoing development is focused upon providing users with more accurate force feedback and added instructional functions. The goal is to provide an efficient method to learn the suturing procedure by measuring surgical skills in a simulator.

5. Future Work.

Future work includes using a true 6 DOF haptic device to provide twist constraints and torques. The long-term goal of this project is to measure skills in both haptic virtual surgery and in physical surgery. The end result is to provide a data analysis of skill scoring and to show that virtual surgery scores do or do not correlate with physical skill scores with respect to expert surgeons versus novice surgeons.

Acknowledgments.

This project was funded, in part, by the National Science Foundation under grant numbers EIA-00116616, DUE-9950742 and DUE-9651237, and a Penn State University College of Medicine Department of Surgery Feasibility Grant, the Eberly Virtual Hospital Project, the Millersville University Neimeyer-Hodgson Grants Program and by the Faculty Grants Committee of Millersville University.

References/Literature.

[1] Morten Bro-Nielsen, D. Helfrick, B. Glass, X. Zeng, and H. Connacher, "VR Simulation of Abdominal Trauma Surgery", Medicine Meets Virtual Reality 6 (MMVR-6), San Diego, California, 1998, IOS Press, pps. 117-123.

[2] Mark Ottensmeyer, E. Ben-Ur, K. Salisbury, "Input and Output for Surgical Simulation: Devices to Measure Tissue Properties in vivo and a Haptic Interface for Laparoscopic Simulators", Proceedings of Medicine Meets Virtual Reality (MMVR '2000), Newport Beach, CA. January 27-30, 2000, IOS Press, pps. 236-242.

[3] Morten Bro-Nielsen, "Fast Finite Elements For Surgery Simulation", Proceedings of Medicine Meets Virtual Reality 5 (MMVR '97), 1997, pps. 395-400.

[4] Jeffrey Berkley, P. Oppenheimer, S. Weghorst, D. Berg, G. Raugi, D. Haynor, M. Ganter, C. Brooking, G. Turkiyyah, "Creating Finite Element Models from Medical Images", Proceedings of Medicine Meets Virtual Reality (MMVR '2000), Newport Beach, CA. January 27-30, 2000, IOS Press, pps. 26-32.

[5] Karl Renig, V. Spitzer, H. Pelster, T. Johnson, T. Mahalik, "More Real-Time Visual and Haptic Interaction with Anatomical Data", Proceedings of Medicine Meets Virtual Reality 5 (MMVR '97), 1997, pps. 155-158.

[6] Suvranu De, M. Srinivasan, "Thin Walled Models for Haptic and Graphical Rendering of Soft Tissues in Surgical Simulations", Proceedings of Medicine Meets Virtual Reality (MMVR '99), IOS Press, San Francisco, CA. January 1999, pps. 94-99.

[7] Paul Gorman, T. Krummel, R. Webster, M. Smith, D. Hutchens,, "A Prototype Haptic Lumbar Puncture Simulator", Proceedings of Medicine Meets Virtual Reality (MMVR '2000), Newport Beach, CA. January 27-30, 2000, IOS Press, pps. 106-109.

[8] Tobias Salb, S. Ghanai, O. Burgert, R. Dillmann,, "Interactive Simulation of Tooth Cleaning with an Interdental Brush", Proceedings of Medicine Meets Virtual Reality (MMVR '2000), Newport Beach, CA. January 27-30, 2000, IOS Press, pps. 295-301.

[9] Morten Bro-Nielsen, J. Tasto, R. Cunningham, G. Merril, "PREOP Endoscopic Simulator: A PC-Based Immersive Training System for Bronchoscopy", Proceedings of Medicine Meets Virtual Reality (MMVR '99), San Fransisco, CA. January 1999, IOS Press, pps. 76-89.

[10] O'Toole RV, Playter RR, Krummel TM, Blank WC, Cornelius NH, Roberts WR, Bell WJ, Raibert M: Measuring and developing suturing technique with a virtual reality surgical simulator. J Am Coll Surg 1999;189:114-27.

[11] Baraff, D. and A. Witkin, "Large Steps in Cloth Animation", in Proceedings of the Annual ACM SIGGRAPH '98 Conference, Vol. 33, pps. 43-54, July 1998.

[12] Mathieu Desbrun, Peter Schröder, Alan Barr, " Interactive Animation of Structured Deformable Objects", Computer Graphics, Proceedings of the Annual Graphics Interface '99, Kingston, Canada, June 1999.

[12] Gilles Debunne, Mathieu Desbrun, Alan Barr, and Marie-Paule Cani, " Interactive Multiresolution Animation of Deformable Models", Eurographics Workshop on Computer Animation and Simulation '99, Sept 99.

[13] "General Haptic Open Software Toolkit GHOST Programmer's Guide", Sensable Technologies Inc., June 1999.

[14] Jacob Rosen, M. MacFarlane, C. Richards, B. Hannaford, M. Sinanan, "Surgeon-Tool Force/Torque Signatures - Evaluation of Surgical Skills in Minimally Invasive Surgery", Proceedings of Medicine Meets Virtual Reality (MMVR '99), San Francisco, CA, January 1999, IOS Press, pps. 290-296.

Suturing video 1 (Betty Mohler). (8 MB)

Suture video 2 (close up screen video). (8 MB)

Suture video 3 (tissue displacements I video). (7 MB)

Suture video 4 (tissue displacements II video). (4 MB)

MMVR 2001 Poster Paper in PDF format

**NOTE: Verefi Technologies Inc now owns this software and its Intellectual Property.

Additional Information can be found at the Verefi website: http://www.verefi.com/

Back to MU Haptics page