Emerging Challenges in Technology-based Support for Surgical Training



This paper stipulates several technological research and development thrusts that can assist in modern day approaches to simulated training of minimally invasive laparoscopic and robot surgery. Basic tenets of such training are explained, and specific areas of research are enumerated. Specifically, augmented and mixed reality are proposed as a means of improving perceptual and clinical decision-making skills, haptics are proposed as mechanism not only to provide force feedback and guidance, but also as a means of reflecting a tactile feel of surgery in simulated training scenarios. Learning optimization is discussed to fine tune the difficulty levels of various exercises. All the above elements can serve as the foundation for building computer-based virtual coaching environments that can reduce the training costs and provide a broader access to learning highly complex, technology driven surgical techniques.

Author Biographies

Minsik Hong, University of Arizona

Department of Electrical and Computer Engineering

Jerzy Rozenblit, University of Arizona

Department of Electrical and Computer Engineering

Department of Surgery



W. Lau, C. Leow, and A. Li, “History of endoscopic and laparoscopic surgery,” World journal of surgery, vol. 21, no. 4, p. 444, 1997.

I. Nisky, F. Huang, A. Milstein, C. M. Pugh, F. A. Mussa-Ivaldi, and A. Karniel, “Perception of stiffness in laparoscopy–the fulcrum effect,” Studies in health technology and informatics, vol. 173, p. 313, 2012.

T. Leal Ghezzi and O. Campos Corleta, “30 years of robotic surgery,” World journal of surgery, vol. 40, pp. 2550–2557, 2016.

C. Freschi, V. Ferrari, F. Melfi, M. Ferrari, F. Mosca, and A. Cuschieri, “Technical review of the da vinci surgical telemanipulator,” The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 9, no. 4, pp. 396–406, 2013.

N. J. Soper and G. M. Fried, “The fundamentals of laparoscopic surgery: its time has come,” Bull Am Coll Surg, vol. 93, no. 9, pp. 30–32, 2008.

N. Stylopoulos, S. Cotin, S. Maithel, M. Ottensmeyer, P. Jackson, R. Bardsley, P. Neumann, D. Rattner, and S. Dawson, “Computer-enhanced laparoscopic training system (celts): bridging the gap,” Surgical Endoscopy and Other Interventional Techniques, vol. 18, pp. 782–789, 2004.

A. S. Soyinka, T. Schollmeyer, I. Meinhold-Heerlein, D. V. Gopalghare, H. Hasson, and L. Mettler, “Enhancing laparoscopic performance with the lts3e: a computerized hybrid physical reality simulator,” Fertility and sterility, vol. 90, no. 5, pp. 1988–1994, 2008.

S. Cotin, N. Stylopoulos, M. Ottensmeyer, P. Neumann, D. Rattner, and S. Dawson, “Metrics for laparoscopic skills trainers: The weakest link!” in Medical Image Computing and Computer-Assisted Intervention—MICCAI 2002: 5th International Conference Tokyo, Japan, September 25–28, 2002 Proceedings, Part I 5. Springer, 2002, pp. 35–43.

P. B. Andreatta, D. T. Woodrum, J. D. Birkmeyer, R. K. Yellamanchilli, G. M. Doherty, P. G. Gauger, and R. M. Minter, “Laparoscopic skills are improved with lapmentor™ training: results of a randomized, double-blinded study,” Annals of surgery, vol. 243, no. 6, p. 854, 2006.

K. Kawaguchi, H. Egi, M. Hattori, H. Sawada, T. Suzuki, and H. Ohdan, “Validation of a novel basic virtual reality simulator, the lap-x, for training basic laparoscopic skills,” Minimally Invasive Therapy & Allied Technologies, vol. 23, no. 5, pp. 287–293, 2014.

C. Basdogan, S. De, J. Kim, M. Muniyandi, H. Kim, and M. A. Srinivasan, “Haptics in minimally invasive surgical simulation and training,” IEEE computer graphics and applications, vol. 24, no. 2, pp. 56–64, 2004.

R.-J. Chen, H.-W. Lin, Y.-H. Chang, C.-T. Wu, and S.-T. Lee, “Development of an augmented reality force feedback virtual surgery training platform,” International Journal of Automation and Smart Technology, vol. 1, no. 1, pp. 41–51, 2011.

R. Smith, V. Patel, and R. Satava, “Fundamentals of robotic surgery: a course of basic robotic surgery skills based upon a 14-society consensus template of outcomes measures and curriculum development,” The international journal of medical robotics and computer assisted surgery, vol. 10, no. 3, pp. 379–384, 2014.

A. J. Hung, I. S. Jayaratna, K. Teruya, M. M. Desai, I. S. Gill, and A. C. Goh, “Comparative assessment of three standardized robotic surgery training methods,” BJU international, vol. 112, no. 6, pp. 864–871, 2013.

J. D. Bric, D. C. Lumbard, M. J. Frelich, and J. C. Gould, “Current state of virtual reality simulation in robotic surgery training: a review,” Surgical endoscopy, vol. 30, pp. 2169–2178, 2016.

S. Puliatti, E. Mazzone, and P. Dell’Oglio, “Training in robot-assisted surgery,” Current Opinion in Urology, vol. 30, no. 1, pp. 65–72, 2020.

M. Hong, J. W. Rozenblit, and A. J. Hamilton, “Simulation-based surgical training systems in laparoscopic surgery: a current review,” Virtual Reality, vol. 25, pp. 491–510, 2021.

L. St-Martin, P. Patel, J. Gallinger, and C.-a. Moulton, “Teaching the slowing-down moments of operative judgment,” Surgical Clinics, vol. 92, no. 1, pp. 125–135, 2012.

C.-a. Moulton, G. Regehr, L. Lingard, C. Merritt, and H. MacRae, “‘slowing down when you should’: initiators and influences of the transition from the routine to the effortful,” Journal of Gastrointestinal Surgery, vol. 14, pp. 1019–1026, 2010.

M. Speicher, B. D. Hall, and M. Nebeling, “What is mixed reality?” in Proceedings of the 2019 CHI conference on human factors in computing systems, 2019, pp. 1–15.

Q. Feng, H. P. Shum, and S. Morishima, “Resolving hand-object occlusion for mixed reality with joint deep learning and model optimization,” Computer Animation and Virtual Worlds, vol. 31, no. 4-5, p. e1956,2020.

S. Rokhsaritalemi, A. Sadeghi-Niaraki, and S.-M. Choi, “A review on mixed reality: Current trends, challenges and prospects,” Applied Sciences, vol. 10, no. 2, p. 636, 2020.

X. Zhao, G. Wang, Z. He, and H. Jiang, “A survey of moving object detection methods: A practical perspective,” Neurocomputing, vol. 503, pp. 28–48, 2022.

A. M. Schreiber, M. Hong, and J. W. Rozenblit, “Monocular depth estimation using synthetic data for an augmented reality training system in laparoscopic surgery,” in 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2021, pp. 2121–2126.

A. Hernansanz, D. Zerbato, L. Gasperotti, M. Scandola, P. Fiorini, and A. Casals, “Improving the development of surgical skills with virtual fixtures in simulation,” in Information Processing in Computer-Assisted Interventions: Third International Conference, IPCAI 2012, Pisa, Italy, June 27, 2012. Proceedings 3. Springer, 2012, pp. 157–166.

K. Tagawa, H. T. Tanaka, Y. Kurumi, M. Komori, and S. Morikawa, “Laparoscopic surgery simulator using first person view and guidance force,” in Medicine Meets Virtual Reality 20. IOS Press, 2013, pp. 431–435.

A. Bettini, P. Marayong, S. Lang, A. M. Okamura, and G. D. Hager, “Vision-assisted control for manipulation using virtual fixtures,” IEEE Transactions on Robotics, vol. 20, no. 6, pp. 953–966, 2004.

J. van Oosterhout, J. G. Wildenbeest, H. Boessenkool, C. J. Heemskerk, M. R. de Baar, F. C. van der Helm, and D. A. Abbink, “Haptic shared control in tele-manipulation: Effects of inaccuracies in guidance on task execution,” IEEE transactions on haptics, vol. 8, no. 2, pp. 164–175, 2015.

M. Hong and J. W. Rozenblit, “An adaptive force guidance system for computer-guided laparoscopy training,” IEEE Transactions on Cybernetics, vol. 52, no. 8, pp. 8019–8031, 2021.

S. A. Bowyer, B. L. Davies, and F. R. y Baena, “Active constraints/virtual fixtures: A survey,” IEEE Transactions on Robotics, vol. 30, no. 1, pp. 138–157, 2013.

I. A. Sucan, M. Moll, and L. E. Kavraki, “The open motion planning library,” IEEE Robotics & Automation Magazine, vol. 19, no. 4, pp. 72–82, 2012.

A. Paraschos, E. Rueckert, J. Peters, and G. Neumann, “Probabilistic movement primitives under unknown system dynamics,” Advanced Robotics, vol. 32, no. 6, pp. 297–310, 2018.

A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann, “Probabilistic movement primitives,” Advances in neural information processing systems, vol. 26, 2013.

F. C. Fernandez and W. Caarls, “Deep reinforcement learning for haptic shared control in unknown tasks,” arXiv preprint arXiv:2101.06227, 2021.

V. Havard, B. Jeanne, M. Lacomblez, and D. Baudry, “Digital twin and virtual reality: a co-simulation environment for design and assessment of industrial workstations,” Production & Manufacturing Research, vol. 7, no. 1, pp. 472–489, 2019.

K. Rangarajan, H. Davis, and P. H. Pucher, “Systematic review of virtual haptics in surgical simulation: a valid educational tool?” Journal of surgical education, vol. 77, no. 2, pp. 337–347, 2020.

K. Salisbury, F. Conti, and F. Barbagli, “Haptic rendering: introductory concepts,” IEEE computer graphics and applications, vol. 24, no. 2, pp. 24–32, 2004.

S. Sadeghnejad, F. Khadivar, E. Abdollahi, H. Moradi, F. Farahmand, S. M. Sadr Hosseini, and G. Vossoughi, “A validation study of a virtual-based haptic system for endoscopic sinus surgery training,” The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 15, no. 6, p. e2039, 2019.

R. L. Williams, M. Srivastava, R. Conaster, and J. N. Howell, “Implementation and evaluation of a haptic playback system,” Haptics-e, vol. 3, no. 3, 2004.

R. Varshney, S. Frenkiel, L. H. Nguyen, M. Young, R. Del Maestro, A. Zeitouni, E. Saad, W. R. J. Funnell, and M. A. Tewfik, “The mcgill simulator for endoscopic sinus surgery (msess): a validation study,” Journal of Otolaryngology-Head & Neck Surgery, vol. 43, no. 1, pp. 1–10, 2014.

W. C. Agboh, M. Yalcin, and V. Patoglu, “A six degrees of freedom haptic interface for laparoscopic training,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016, pp. 1107–1112.

H. Cakmak, H. Maass, and U. K ̈uhnapfel, “Vs one, a virtual reality simulator for laparoscopic surgery,” Minimally Invasive Therapy & Allied Technologies, vol. 14, no. 3, pp. 134–144, 2005.

C. V ̊apenstad, E. F. Hofstad, L. E. Bø, M. K. Chmarra, E. Kuhry, G. Johnsen, R. M ̊arvik, and T. Langø, “Limitations of haptic feedback devices on construct validity of the lapsim® virtual reality simulator,” Surgical endoscopy, vol. 27, pp. 1386–1396, 2013.

S. Misra, K. Ramesh, and A. M. Okamura, “Modelling of non-linear elastic tissues for surgical simulation,” Computer methods in biomechanics and biomedical engineering, vol. 13, no. 6, pp. 811–818, 2010.

J. Zhang, Y. Zhong, and C. Gu, “Deformable models for surgical simulation: a survey,” IEEE reviews in biomedical engineering, vol. 11, pp. 143–164, 2017.

S. Sadeghnejad, F. Farahmand, G. Vossoughi, H. Moradi, and S. M. S. Hosseini, “Phenomenological tissue fracture modeling for an endoscopic sinus and skull base surgery training system based on experimental data,” Medical engineering & physics, vol. 68, pp. 85–93, 2019.

A. Talasaz, A. L. Trejos, and R. V. Patel, “The role of direct and visual force feedback in suturing using a 7-dof dual-arm teleoperated system,” IEEE transactions on haptics, vol. 10, no. 2, pp. 276–287, 2016.

A. Abiri, S. J. Askari, A. Tao, Y.-Y. Juo, Y. Dai, J. Pensa, R. Candler, E. P. Dutson, and W. S. Grundfest, “Suture breakage warning system for robotic surgery,” IEEE Transactions on Biomedical Engineering, vol. 66, no. 4, pp. 1165–1171, 2018.

N. Haouchine, W. Kuang, S. Cotin, and M. Yip, “Vision-based force feedback estimation for robot-assisted surgery using instrumentconstrained biomechanical three-dimensional maps,” IEEE Robotics and Automation Letters, vol. 3, no. 3, pp. 2160–2165, 2018.

A. Saracino, A. Deguet, F. Staderini, M. N. Boushaki, F. Cianchi, A. Menciassi, and E. Sinibaldi, “Haptic feedback in the da vinci research kit (dvrk): A user study based on grasping, palpation, and incision tasks,” The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 15, no. 4, p. e1999, 2019.

C. Pacchierotti, D. Prattichizzo, and K. J. Kuchenbecker, “Cutaneous feedback of fingertip deformation and vibration for palpation in robotic surgery,” IEEE Transactions on Biomedical Engineering, vol. 63, no. 2, pp. 278–287, 2015.

J. K. Koehn and K. J. Kuchenbecker, “Surgeons and non-surgeons prefer haptic feedback of instrument vibrations during robotic surgery,” Surgical endoscopy, vol. 29, pp. 2970–2983, 2015.

A. J. Spiers, H. J. Thompson, and A. G. Pipe, “Investigating remote sensor placement for practical haptic sensing with endowrist surgical tools,” in 2015 IEEE World Haptics Conference (WHC). IEEE, 2015, pp. 152–157.

E. S. Toberer, L. L. Baranowski, and C. Dames, “Advances in thermal conductivity,” Annual Review of Materials Research, vol. 42, pp. 179–209, 2012.

A. G. Gallagher and G. C. O’Sullivan, Fundamentals of surgical simulation: principles and practice. Springer Science & Business Media, 2011.

E. N. Spruit, G. P. Band, J. F. Hamming, and K. R. Ridderinkhof, “Optimal training design for procedural motor skills: a review and application to laparoscopic surgery,” Psychological research, vol. 78, pp. 878–891, 2014.

E. M. Ritter and D. J. Scott, “Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery,” Surgical innovation, vol. 14, no. 2, pp. 107–112, 2007.

R. C. Wilson, A. Shenhav, M. Straccia, and J. D. Cohen, “The eighty five percent rule for optimal learning,” Nature communications, vol. 10, no. 1, p. 4646, 2019.

S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” in Advances in psychology. Elsevier, 1988, vol. 52, pp. 139–183.

G. Fenza, F. Orciuoli, and D. G. Sampson, “Building adaptive tutoring model using artificial neural networks and reinforcement learning,” in 2017 IEEE 17th international conference on advanced learning technologies (ICALT). IEEE, 2017, pp. 460–462.

T. Minematsu, A. Shimada, and R.-i. Taniguchi, “Simple background subtraction constraint for weakly supervised background subtraction network,” in 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, 2019, pp. 1–8.

M. Hong, K. Meisner, S. Lee, A. M. Schreiber, and J. W. Rozenblit, “A fuzzy reasoning system for computer-guided laparoscopy training,” in 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2020, pp. 1712–1717.

Additional Files





Biomedical Engineering