AI That Watches Surgery: The Next Revolution in Surgical Training

What if every operation could become a learning system?

Surgery has always been one of medicine’s most demanding forms of training.

A trainee watches. A senior surgeon demonstrates. The trainee assists. Then, slowly, under supervision, the trainee performs more of the operation. This apprenticeship model has produced generations of excellent surgeons. But it also has limits.

Feedback can be subjective. It can arrive late. It can depend on who is watching. It can be influenced by workload, memory, personality, time pressure, and operating-room culture.

Now surgery is generating a new kind of educational data, a surgical video.

Every laparoscopic, robotic, endoscopic, arthroscopic, and microscope-assisted operation can create a detailed visual record of technique. These videos do not just show that an operation happened. They show how it happened. They show the surgical steps, the instruments, the tissue handling, the camera movement, the sequence of decisions, the difficult moments, the errors, the recovery, and the flow of the procedure.

Artificial intelligence may turn this visual record into something far more powerful: A personalized surgical learning system. The goal is not to replace surgical mentors. The goal is to give every trainee more objective, consistent, and useful feedback, and to give every operation the chance to become a lesson.

What Is AI Surgical Video Analysis?

AI surgical video analysis means using artificial intelligence to study surgical video and extract useful information from it.

The system may be trained to recognise:

• what stage of the operation is happening
• which instruments are being used
• what action the surgeon is performing
• how efficiently instruments are moving
• whether the sequence of steps is correct
• whether anatomy is clearly exposed
• whether tissue is being handled safely
• whether there are signs of technical difficulty
• where a trainee may need targeted practice

In simple terms, the AI is not just “watching” surgery like a camera. It is trying to understand the operation.

This is the shift that makes the field futuristic. Traditional video recording captures the event. AI video analysis tries to interpret the event. That difference could change surgical education.

Surgical Video as the New Textbook

Traditional surgical textbooks explain anatomy, surgical indications, and procedural steps. They are essential, but they are static.

Surgery is not static. Surgery is movement, timing, exposure, tension, depth, decision-making, and adaptation. A textbook can show the ideal step. A surgical video can show the messy reality. It can show what happens when anatomy is difficult, tissue is inflamed, the camera view is imperfect, bleeding obscures the field, or the trainee struggles with a step. This is why surgical video may become one of the most important educational resources in modern surgery.

But there is a problem.

Surgical video libraries can become enormous. Thousands of hours of footage are difficult to search, label, review, and teach from manually. AI could help convert surgical videos into structured learning material.

Instead of only storing videos by procedure name, AI could help organise them by:

• surgical phase
• anatomy shown
• instrument used
• technical difficulty
• critical step
• error pattern
• expert example
• trainee learning need
• complication-related moment
• teaching value

The future surgical textbook may not be a book. It may be a searchable, AI-annotated library of real surgical performance.

Step Recognition: Knowing Where the Surgeon Is

One of the most important tasks in surgical video AI is step recognition or phase recognition.

This means the system identifies which part of the operation is happening. For example, in a laparoscopic procedure, the AI may learn to recognise whether the surgeon is exposing anatomy, dissecting tissue, clipping, cutting, removing tissue, irrigating, suturing, or closing.

This matters because AI cannot assess surgery intelligently unless it first understands the workflow.

A movement that is appropriate in one step may be dangerous in another.

An instrument that is expected at one moment may be unusual at another.

A delay may be normal in a complex step but inefficient in a routine step.

Step recognition creates the timeline of the operation.

Once the AI understands the timeline, it can begin to ask deeper questions:

Was the sequence logical?

Was a critical step delayed?

Was a safety step missed?

Was the trainee struggling repeatedly at the same phase?

Did the operation follow expert patterns?

This is where surgical video becomes more than a recording. It becomes a map.

Instrument Recognition: Knowing What Tools Are Being Used

Another major task is instrument recognition.

AI can be trained to identify surgical tools in the video field. In minimally invasive and camera-based surgery, this may include graspers, scissors, energy devices, clip appliers, sutures, needles, scopes, drills, shavers, forceps, retractors, and other procedure-specific tools. Instrument recognition helps the AI understand surgical context.

A tool is not just an object. It is a clue.

A grasper suggests traction.

A needle suggests suturing.

A clip applier suggests vessel or duct control.

A shaver suggests arthroscopic tissue work.

A drill suggests bone preparation.

An energy device suggests cutting, sealing, or cauterising.

When AI knows which tool is present, it can begin to infer what action may be happening. But the tool alone is not enough. A surgical instrument is only meaningful when we understand how it interacts with tissue.

Action Recognition: Understanding What the Tool Is Doing

Action recognition moves beyond identifying the tool. It asks: What is the surgeon doing with it?

The AI may analyse actions such as:

• grasping
• cutting
• dissecting
• retracting
• suturing
• tying
• clipping
• cauterising
• drilling
• shaving
• irrigating
• suctioning
• exposing anatomy
• controlling bleeding
• manipulating tissue

This is important because surgical skill is not just about choosing the correct instrument. It is about how the instrument is used. Two surgeons may use the same tool in the same step, but with very different levels of control, economy, gentleness, and safety.

AI action recognition could help separate:

• what instrument was present
• what movement was performed
• what tissue was contacted
• whether the action matched the expected step
• whether the action looked efficient or risky

This is where surgical video analysis starts to move closer to true surgical understanding.

Skill Assessment: From Opinion to Objective Feedback

Surgical skill is difficult to measure.

A senior surgeon can often tell when a trainee is improving, but formal assessment may still depend on subjective observation. AI could help make surgical assessment more objective.

A video-analysis system may study:

• instrument path
• economy of motion
• hand stability
• camera control
• unnecessary movements
• time spent on each step
• smoothness of movement
• tissue handling
• respect for tissue planes
• error frequency
• depth control
• consistency compared with expert patterns

This does not mean a score should replace a mentor’s judgement. It means the mentor may receive better information.

Instead of saying, “Your suturing needs improvement,” the system might show:

“You spent longer than expected in needle positioning.”

“Your instrument path had repeated corrections.”

“Camera instability increased during knot tying.”

“Tissue traction was inconsistent during exposure.”

“This step improved compared with your previous five cases.”

That kind of feedback is more specific. And specific feedback is easier to train.

Error Detection: Supporting Safety, Not Creating Fear

One of the most powerful future uses of surgical AI is error detection.

The system may eventually help identify:

• wrong tissue plane
• unsafe traction
• poor visualisation
• excessive instrument movement
• delayed anatomy recognition
• accidental tissue contact
• missed step
• deviation from expected workflow
• instrument collision
• bleeding event
• unsafe proximity to critical anatomy

This is exciting, but it must be handled carefully. Surgical video AI should not become a blame machine. If hospitals use AI video analysis only to punish surgeons, the technology will fail culturally before it succeeds technically. The better vision is a learning culture.

The question should not be:

“Who made a mistake?”

The question should be:

“What happened, why did it happen, and how can training or systems improve?”

AI error detection should support surgical safety, coaching, and reflection. It should not create fear inside the operating room.

Real-Time Feedback: The Safest First Use May Be Simulation

The operating room is a high-stakes environment. Real-time AI feedback during live surgery must be extremely accurate, validated, explainable, and carefully integrated.

But simulation is different. A surgical simulator is one of the safest places for AI to begin. In simulation, AI can watch a trainee perform a task and immediately provide feedback on motion, accuracy, timing, error patterns, and improvement areas.

For example, during a simulated suturing task, the AI might detect:

• unstable needle angle
• repeated hand repositioning
• excessive path length
• poor knot tension
• inefficient instrument transfer
• delay between steps
• tissue handling errors

This could make simulation more personalised.

Instead of every trainee receiving the same generic training module, the system could identify what each trainee actually needs.

One trainee may need camera-control practice.

Another may need knot-tying practice.

Another may need depth-perception training.

Another may need help with tissue handling.

AI could make surgical simulation less like a checklist and more like a personal coach.

Postoperative Video Coaching

AI may become useful after surgery before it becomes useful during surgery. Postoperative video coaching is a powerful near-term direction.

After a recorded procedure, AI could generate:

• a timeline of key steps
• time spent on each phase
• instrument-use patterns
• video clips needing review
• moments of technical difficulty
• comparison with expert benchmarks
• objective performance metrics
• personalised practice recommendations

This could save time for mentors.

A senior surgeon may not have time to watch an entire two-hour operation with every trainee. But if AI highlights the most important moments, the mentor can focus on the clips that matter.

This creates a better feedback loop.

The trainee performs.

The video is analysed.

The system highlights learning moments.

The mentor reviews the key clips.

The trainee practises targeted skills.

Progress is tracked over time.

This is not replacing mentorship.

It is making mentorship more focused.

From “How Many Cases?” to “How Well Did You Perform?”

Traditional surgical training often tracks numbers.

How many cases did the trainee observe?

How many did they assist?

How many did they perform?

How many were logged?

Case volume matters. Repetition matters. But case volume alone does not prove competence. A trainee may perform many cases but continue to struggle with a critical step. Another trainee may perform fewer cases but demonstrate consistent precision, safety, and decision-making.

AI surgical video analysis could help shift training from quantity to quality.

The question becomes:

Not only, “How many cases did you do?”

But also, “How well did you perform the critical steps?”

This is a major educational shift.

It supports competency-based surgical education, where progression depends not only on time served or procedures logged, but on demonstrated ability.

Personalised Surgical Learning

Every trainee is different.

One trainee may be technically smooth but slow.

Another may be fast but rough with tissue.

Another may understand anatomy well but struggle with depth perception.

Another may control instruments well but lose orientation with the camera.

Another may be efficient in simple cases but struggle when anatomy is distorted.

AI could create a personalised surgical learning profile.

That profile may track:

• strengths
• recurring weaknesses
• step-specific difficulties
• progress over time
• comparison with expert benchmarks
• performance in simulation
• performance in real procedures
• areas needing deliberate practice

This could make surgical education more precise. Instead of telling every trainee to “do more cases,” programs could say:

“You need more practice with safe exposure.”

“You need targeted simulation for suturing.”

“You need to review anatomy recognition in this step.”

“You are improving in motion economy but still struggle with camera control.”

“Your performance is strong in routine cases, but complex anatomy increases your error rate.”

That is the beginning of personalised surgical training.

AI as a Surgical Coach

A future surgical coach may combine multiple data streams:

• surgical video
• instrument motion data
• robotic system data
• simulation performance
• operative notes
• structured feedback
• complication data
• patient outcomes
• mentor review
• trainee learning history

The most advanced version of this system would not simply give a score. It would explain performance.

A useful surgical coach should answer:

What happened?

Why did it matter?

Where did the trainee struggle?

Which clip should be reviewed?

What should be practised next?

How has the trainee changed over time?

How does performance compare with expert benchmarks?

The best feedback is not a number.

The best feedback is specific, actionable, and clinically meaningful.

Robotic Surgery: A Natural Home for Surgical AI

Robotic surgery is one of the most natural environments for AI-based training tools because it can generate rich digital data.

In addition to video, robotic systems may provide information about:

• instrument movement
• camera angle
• motion patterns
• clutching
• energy use
• timing
• instrument coordination
• tremor-related signals
• workspace efficiency
• force-related patterns where available

This allows AI to assess performance more deeply than video alone. For example, the system may detect whether the surgeon is using smooth movements, whether the camera is positioned efficiently, whether the instrument path is unnecessarily long, or whether the trainee is repeating the same correction.

Robotic surgery also makes it easier to compare performance across similar tasks.

That makes it a strong platform for objective surgical training.

Laparoscopy, Arthroscopy, Endoscopy, and Microsurgery

AI surgical video analysis is especially relevant in fields where the procedure is already camera-based.

Laparoscopy

Laparoscopic surgery is viewed through a camera, making it ideal for video analysis. AI can study operative phases, instrument use, tissue handling, camera control, bleeding events, safe dissection, anatomy exposure, and performance patterns.

Arthroscopy and Sports Surgery

Arthroscopy is also highly visual. In the future, AI could support training in shoulder arthroscopy, knee arthroscopy, ACL reconstruction, meniscus surgery, rotator cuff repair, cartilage procedures, and hip arthroscopy.

It may help recognise portals, instruments, anatomical landmarks, repair steps, and technical errors.

Endoscopy

Endoscopy already depends on video as the primary operating environment. AI can support recognition of anatomy, procedural steps, lesions, tool use, and operator performance.

This is one of the most advanced areas for video-based medical AI.

Microsurgery

Microsurgery depends on precision, hand stability, instrument handling, and tissue respect.

AI video analysis could support training in vessel anastomosis, nerve repair, reconstructive microsurgery, ophthalmic surgery, neurosurgery, and plastic surgery.

In microsurgery, useful metrics may include tremor, needle angle, bite symmetry, instrument tip movement, tissue interaction, and procedural flow.

Surgical Anatomy Recognition

One of the most cutting-edge goals is helping AI recognise anatomy during surgery.

This could include:

• vessels
• ducts
• nerves
• organs
• tendons
• cartilage
• bone landmarks
• tumour boundaries
• safe zones
• tissue planes

This is difficult.

Anatomy is not always clean. Tissue can be inflamed, scarred, bleeding, distorted, hidden, or changed by previous surgery. Lighting, camera angle, smoke, fluid, movement, and instrument obstruction can all affect what the AI sees.

This is why surgical AI is much harder than ordinary image recognition. A system that performs well on clean training videos may struggle in real-world surgery.

True surgical intelligence requires the AI to handle variation, uncertainty, and complexity.

Safe-Zone Recognition

A future AI system may help identify whether the surgeon is operating within a safe zone.

This could be useful in complex anatomy where critical structures are close together.

But it is also one of the highest-risk applications.

If AI incorrectly labels a dangerous area as safe, the consequences could be serious. If it produces too many false alarms, surgeons may ignore it.

Safe-zone recognition must be validated carefully before clinical use.

In training, however, it could become very valuable. A trainee reviewing a video may learn:

“This was the correct tissue plane.”

“This movement came close to a critical structure.”

“This exposure was incomplete before the next step.” “This was the safer angle of approach.”

“This clip shows why anatomy recognition matters.”

Used correctly, AI could help trainees understand not only what to do, but where not to go.

Surgical Workflow Optimisation

AI can also analyse how an operation flows.

It may measure:

• time per step
• delays
• repeated movements
• instrument changes
• interruptions
• variation between surgeons
• variation between hospitals
• equipment-related delays
• inefficient sequencing
• operating-room bottlenecks

This matters because surgery is not only individual technical skill.

It is also a system.

A skilled surgeon can still be slowed by poor instrument readiness, equipment problems, communication gaps, or inefficient workflow.

AI could help identify where the process breaks down.

This could improve training, operating-room efficiency, and patient safety.

Operating-Room Team Training

Surgery is not only the surgeon.

A safe operation depends on the coordination of the surgeon, assistant, scrub team, anaesthesia team, nurses, technicians, and operating-room staff.

In the future, AI video and sensor analysis may help study:

• timing of instrument readiness
• handoffs
• safety checks
• team communication
• delays
• equipment placement
• coordination between team members
• response to unexpected events

This expands AI surgical analysis from individual skill to team performance.

That is important because many surgical risks are not purely technical. They are also organisational.

The future of surgical training may include not only better surgeons, but better surgical teams.

Surgical Video Foundation Models

The cutting edge is moving from small, task-specific AI models toward larger surgical video models.

A task-specific model might be trained only to identify instruments in one procedure.

A broader surgical video model aims to understand surgical scenes more generally.

It may learn patterns across:

• phases
• instruments
• actions
• anatomy
• tissue interaction
• motion
• workflow
• surgical context

These models could become more flexible across procedures.

Instead of building a separate system for every operation, researchers are exploring models that can learn transferable surgical video knowledge.

This is exciting, but it also raises major questions.

Does the model work across hospitals?

Does it work with different equipment?

Does it work in complex cases?

Does it recognise rare events?

Does it understand surgical context or only visual patterns?

Does it fail silently?

Can surgeons trust its explanation?

In surgery, a model that looks impressive in research is not enough. It must be safe, validated, transparent, and clinically useful.

Vision-Language Surgical AI

Another future direction is combining video understanding with language.

This means the AI does not only detect what is happening. It can also describe it.

A vision-language surgical AI system could generate:

• surgical summaries
• teaching points
• step-by-step feedback
• trainee progress reports
• annotated video clips
• error explanations
• simulation guidance
• case review notes

This matters because surgeons do not learn from numbers alone.

They learn from explanation.

A useful system should not only say:

“Performance score: 72.”

It should say:

“You had repeated instrument corrections during the dissection step. The camera angle reduced depth perception. Review clips 03:12 to 04:05 and practise maintaining exposure before cutting.”

That is the type of feedback that could change training.

The Danger of Black-Box Assessment

Surgeons should not be judged by unexplained AI scores. If an AI system says performance was poor, it must show why.

Was the motion inefficient?

Was the tissue plane incorrect?

Was the camera unstable?

Was there excessive traction?

Was the step sequence delayed?

Was there a safety risk?

Was the model uncertain?

Explainability is essential. This is especially important if AI is used in formal assessment, credentialing, or progression decisions.

A black-box score can create mistrust. A transparent system can create learning.

The best AI surgical coach should show the video segment, explain the concern, and allow human expert review.

Data Privacy, Consent, and Ownership

Surgical video is sensitive.

It may contain patient information, surgeon behaviour, staff interactions, audio, workflow details, and hospital-specific practices.

Before surgical video AI becomes routine, hospitals must address:

• patient consent
• surgeon consent
• team consent
• video ownership
• data storage
• access control
• anonymisation
• de-identification
• use for education
• use for research
• use for quality improvement
• use for performance review
• protection from misuse

A surgical video is not just a file.

It is a record of a patient, a surgeon, and a team inside one of the most sensitive environments in medicine.

The technology must be built with trust from the beginning.

The Culture Problem

The biggest barrier may not be technical. It may be cultural.

Surgeons may fear that AI video analysis will be used against them. Trainees may worry that every mistake will become permanent. Hospitals may be tempted to use performance data for punishment rather than improvement.

That would be the wrong direction.

The best use of AI surgical video analysis is not surveillance. It is coaching.

The message should be clear: AI should be used to improve training and safety, not to create fear. A healthy surgical AI culture should protect learning, encourage reflection, and involve surgeons in how the technology is used. Without trust, the system will not work.

Bias, Fairness, and Generalisation

AI systems can fail when they are used outside the environment where they were trained.

A model may perform well in one hospital, one procedure, one camera system, one patient group, or one surgical style, but fail elsewhere.

Surgical AI must be tested across:

• different surgeons
• different skill levels
• different hospitals
• different patient anatomy
• different equipment
• different procedures
• different surgical styles
• different levels of case complexity

Without this, AI feedback may be unfair or inaccurate.

Generalisation is one of the biggest challenges in surgical AI.

A model trained on clean, well-lit, carefully selected videos may struggle with:

• bleeding
• smoke
• poor lighting
• adhesions
• distorted anatomy
• emergency situations
• rare complications
• unusual instruments
• different camera angles
• incomplete views

Real-world surgery is not always clean. AI must be ready for that reality.

Fact Base

AI surgical video analysis is already a real research field. Current systems can analyse surgical video for phases, instruments, actions, workflow patterns, and some skill-related features.

Surgical skill assessment using AI is promising, especially in minimally invasive surgery, simulation, and robotic environments, but many models still need stronger external validation and real-world testing.

Simulation is one of the safest early uses because it allows immediate feedback without risking patient safety.

Postoperative video coaching may be more realistic in the near term than universal real-time intraoperative guidance.

Surgical video data requires careful governance because it contains sensitive information about patients, surgeons, teams, and hospitals.

Large surgical video models and vision-language systems represent a cutting-edge direction, but they must prove accuracy, fairness, explainability, and clinical value before broad deployment.

What is possible today:

• AI can analyse surgical videos for phases, tools, actions, and some skill-related features.
• AI-assisted surgical training is growing.
• Simulation-based AI feedback is a practical early use.
• Surgical video can support more objective coaching.
• Automated assessment may help standardise parts of training.

What is not fully possible yet:

• universal real-time surgical coaching
• fully reliable error detection across all procedures
• AI replacing expert surgical judgement
• AI systems that understand every surgical context
• AI scores used alone for credentialing or certification

What Readers Should Understand

AI that watches surgery is not about replacing the surgeon, the mentor, or the educator.

It is about making surgical training more objective, personalised, and scalable.

The most powerful idea is not that AI can record surgery.

It is that AI can help surgeons learn from surgery.

A future training system may look like this:

A trainee performs a simulated or real procedure.

The video is analysed by AI.

The system recognises each step.

It identifies instruments and tissue interaction.

It detects inefficient movements or unsafe patterns.

It compares performance with expert benchmarks.

It generates personalised feedback.

The trainee practises targeted skills.

Progress is tracked over time.

The human mentor reviews the most important clips.

This creates a continuous loop: Operate. Analyse. Learn. Practise. Improve.

That loop could change surgical education.

Final takeaway

AI that watches surgery is not about creating a surveillance system.

It is about creating a learning system.

The next revolution in surgical training may not happen in the classroom.

It may happen inside the surgical video itself.

References

1. King A, Fowler GE, Hill R, et al. Use of artificial intelligence in the analysis of digital videos of invasive surgical procedures: scoping review. BJS Open. 2025;9(4):zraf073. DOI: 10.1093/bjsopen/zraf073
2. Pedrett R, Mascagni P, Beldi G, Padoy N, Lavanchy JL, et al. Technical skill assessment in minimally invasive surgery using artificial intelligence: a systematic review. Surgical Endoscopy. 2023;37:7412–7424. DOI: 10.1007/s00464-023-10335-z
3. Escobar-Castillejos D, Barrera-Animas AY, Noguez J, Magana AJ, Benes B. Transforming Surgical Training With AI Techniques for Training, Assessment, and Evaluation: Scoping Review. Journal of Medical Internet Research. 2025;27:e58966. DOI: 10.2196/58966
4. Eckhoff JA, Rosman G, Altieri MS, et al. SAGES consensus recommendations on surgical video data use, structure, and exploration for research in artificial intelligence, clinical quality improvement, and surgical education. Surgical Endoscopy. 2023;37:8690–8707. DOI: 10.1007/s00464-023-10288-3
5. Walsh R, Kearns EC, Moynihan A, et al. Ethical perspectives on surgical video recording for patients, surgeons and society: systematic review. BJS Open. 2023;7(3):zrad063. DOI: 10.1093/bjsopen/zrad063
6. Yang S, Zhou F, Mayer L, et al. Large-scale self-supervised video foundation model for intelligent surgery. npj Digital Medicine. 2026;9:220. DOI: 10.1038/s41746-026-02403-0