3D Surface Reconstruction and Feature Tracking from Endoscopic Images

Background: The images of inter-cavity tissue from the views of the endoscopes and laparoscope during minimally invasive procedures have an incredible amount of information present. Tissues can be differentiated from one another based on their photometric properties, and certain regions of interests such as vasculature can be located and tracked with the naked eye. If we can visually track the in-vivo tissue as it moves and deforms in the body, then why can't computer algorithms do the same?

Summary: The goal of this project, and ultimately, my thesis, is to use computer vision techniques in order to track the translation and deformation of tissue. This information can be used to track the location of tumors that can be seen in the camera; in addition, it can be used to track the locations of subcutaneous lesions if its position were known and determined in the past, i.e, using intra-operative ultrasound.
Video

Stero Images

Feature Detector (SIFT)

Surface Reconstruction

Registration beteween Laparoscopic Cameras and 3D Ultrasound

Background: Augmented reality in the operating room is hot topic of research; to provide surgeons with medical image overlays while they perform surgery offers the image guidance they need to improve their error margins and improve surgical outcome. Current techniques for augmented reality involve registering medical images intra-operatively to the surgical camera view. The most accessible images are ultrasound images, since they can be acquired in realtime and are general safe for any procedure. Conventional registration is performed by tracking the endoscopic and laparoscopic cameras and the ultrasound probe using external equipment, and obtaining ther spatial locations such that a registration between the two images can be found. This is unideal for the operating room since it takes up significant space, requires calibration, and adds new sources of error.

Summary: This work aims to perform registration with simply the un-altered laparoscopic camera and ultrasound probes. By pressing ultrasound fiducials (steel beads) against the tissue being imaged, their locations in both the ultrasound image and the camera images provide a registration path. We demonstrated this principle using a miniature tool designed for laparoscopic surgery, showing that initial experiments had an overall registration error of 1.69mm.

[1]   Michael C. Yip, Troy Adebar, Robert N. Rohling, Septimiu E. Salcudean and Christopher Y. Nguan, "3D Ultrasound to Stereoscopic Camera Registration through an Air-Tissue Boundary," Medical Image Computing and Computer-Assisted Intervention - MICCAI 2010. Lecture Notes in Computer Science, vol. 6362 (2010), pp. 626-634. [Link]
[2]   Septimiu E. Salcudean, Michael C. Yip , Troy Adebar, Robert N. Rohling, and Christopher Y. Nguan. A Method for 3D Ultrasound to Stereoscopic Camera Registration through an Air-Tissue Boundary. US Provisional Patent Application Submitted on Sept. 17, 2010.

Transformation Set for Registration

Kidney Ultrasound Fiducial at air/tissue boundary

A Low-cost Dextrous Laparoscopic Robot for Image-guided Telesurgery

Summary: This was my senior thesis at the University of Waterloo. With two other outstanding classmates, we built a dextrous surgical robot for laparoscopic surgery. It featured:
   • 5 degrees of freedom
   • a gripper that could grasp conventional laparoscopic tools
   • Provided image-guidance through 3DSlicer
   • Controlled by a haptics-enabled controller (Novint Falcon)
   • Costed a total of $1800 USD
With the only FDA approved laparoscopic robot costing $1.5 Million USD (DaVinci, Intuitive Surgical), robots like ours that offer similar dexterity and features can actually be pretty useful for surgical training, trying new surgical approaches, and for research development.

3D CAD Image

Wrist Design

Sensing and Controlling Forces during Beating Heart Surgery

Background: Beating Heart Surgery is a technique where the surgeon repairs areas of the heart without using bypass techniques and forcing the heart to stop beating. Instruments passed into the heart wall must be secured by sutures in order to prevent the leakage of blood. Due to the considerable movement of the heart and the constraints of the suture, the surgeon effectively has very little force or visual feedback. Real-time 3D Ultrasound is often used to guide the surgeon.

Summary: I designed a miniature force sensor for beating heart surgery that could be used during beating heart surgery. To the best of our knowledge, It is the first to be used successfully within the beating heart for animal study cases. It has given insight into how much force is required for mitral valve annuloplasty, where staple tools are used to secure a rigid ring in order to improve the structural integrity of the valve (it turns out 3-5N is a conservative goal [1]). We also used the force measurements, in conjunction with ultrasound imaging, to predict the beating heart motion and move the instruments with the heartbeat [2].

[1]   Michael C. Yip, Shelten G. Yuen and Robert D. Howe. "A Robust Uniaxial Force Sensor for Minimally Invasive Surgery," IEEE Trans. Biomedical Engineering. vol. 57, no. 5, pp. 1008-1011, May 2010. [Link]
[2]   Shelten G. Yuen, Michael C. Yip, Nikolay V. Vasilyev, Douglas P. Perrin, Pedro J. del Nido and Robert D. Howe, "Robotic Force Stabilization for Beating Heart Intracardiac Surgery," Proceedings of Medical Image Computing and Computer-Assisted Intervention - MICCAI 2009. Lecture Notes in Computer Science, vol. 5761 (2009), pp. 26-33. [Link]

Beating Heart Surgery: Mitral Valve Annuloplasty

Force Sensor Design

Force Sensor In-vivo Testing

Teleoperation under Significant Time Delay

Background: There is ample research on the effect of haptic teleoperation under delayed communication channels in terms of stability and system performance. Little attention, however, has been paid to the effect of delayed force feedback on users' task performance, and whether force feedback is beneficial under significant communication delays. Our goal was to investigate whether force feedback improves user's task performance in delayed teleoperation.

Summary: A user study involving unilateral (without force feedback), bilateral (with force feedback), and graphical-feedback teleoperation under various communication delays was set up. We observed that:
   • Task completion times increase as delay increases.
   • Haptics helps reduce contact forces and the occurrence of large reaction forces.
   • Graphical feedback results in the lowest forces at the cost of higher completion times.
   • Haptics/Graphical feedback causes the task to take more time than unilateral control.
Therefore, when short completion times are crucial given a tolerance for larger forces, force feedback only serves to increase the time required to perform the task; thus, unilateral control may be sufficient.
Video (200msec delay teleoperation)

[1]   Michael C. Yip, Mahdi Tavakoli, Robert D. Howe. "Performance Analysis of a Haptic Telemanipulation Task under Time Delay," Advanced Robotics, Special Issue on Physical Human-Robot Interaction Through Force Interface. vol. 25(5), 20 pages. March 2011. In Press.
[2]   Michael C. Yip, Mahdi Tavakoli, Robert D. Howe. "Performance Analysis of a Manipulation Task in Time-Delayed Teleoperation," Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems - IROS 2010. 18-22 October 2010, pp. 5270-5275. In Press.

Passivity-based Delay-Compensated Teleoperation

Force Histograms

---

Low-Intensity Ultrasound for Stimulating Cellular Development

Massachusetts Institute of Technology

Background: Low-intensity ultrasound (LIUS) treatment has been shown to increase mass transport, which could benefit tissue grafts during the immediate postimplant period, when blood supply to the implanted tissue is suboptimal. In this in-vitro study, we investigated effects of LIUS stimulation on dye diffusion, proliferation, metabolism, and tropomyosin expression of muscle cells (C2C12) and on tissue viability and gene expression of human adipose tissue organoids.

Summary: Cellular studies showed that LIUS significantly increased proliferation,metabolic activity, and expression of tropomyosin. Adipose tissue treated with LIUS showed significantly increased metabolic activity and the cells had similar morphology to normal unilocular adipocytes. Gene analysis showed that tumor necrosis factor-alpha expression (a marker for tissue damage) was signi?cantly lower for stimulated organoids than for control groups. Our data suggests that LIUS could be a useful modality for improving graft survival in vivo.

[1] Hyoungshin Park, Michael C. Yip, Joseph Kost, James Kobler, Robert S. Langer, Steven M. Zeitels and Beata Chertok. "Indirect low-intensity ultrasonic stimulation for tissue engineering," Journal of Tissue Engineering and Regenerative Medicine, vol. 2010, Article ID 973530, 9 pages, 2010. doi:10.4061/2010/973530 [Link]

Control (day 6) 10 min LIUS (day 6)

Cell Number (day 6) Lactate Levels (day 6)

Controls and Robotics Group
University of British Columbia
2332 Main Mall, Vancouver, BC Canada | V6T 1Z4
Dept. of Electrical and Computer Engineering
University of British Columbia