Video Summarization of

Key EventsStage I - The Critical View

Michael A. Grasso, MD, PhDUniversity of Maryland School of

MedicineUMBC Computer Science

MichaelGrasso.com

AbstractLaparoscopic surgery is a minimally invasive technique with unique training requirements. Video-assisted evaluation is one method that surgical residents can use to demonstrate competence. Automated video summarization can increase the efficiency of evaluations by directing the senior surgeon to key portions of a surgical procedure. We are using image classification techniques to segment videos of laparoscopic cholecystectomies to assist with surgical training and evaluation.

Overview Background

Laparoscopic Surgery Image Classification

Methods Discussion

Laparoscopic Surgery Minimally Invasive Surgery. First performed in 1987. Used in many surgical procedures.

Gall bladder removal (cholecystectomy). Esophageal surgery (fundoplication). Colon surgery (colectomy). Others.

Laparoscopic Approach Narrow tubes (trocars)

are inserted into the abdomen through small incisions.

www.fda.gov

Laparoscopic Procedure Camera is passed

through trocar. Procedure is often

videotaped. Carbon dioxide is

infused through trocar.

Instruments are passed through the trocars to cut, manipulate, and sew.

Laparoscopic Aftercare Compared with an

open procedure. Smaller scars. Reduced pain. Quicker recovery.

http://www.nlm.nih.gov/medlineplus/ency/presentations/100166_1.htm

Technical Challenges Access limited to small incisions. Long instruments with only the tips

visible. Two-dimensional video. Limited tactile feedback.

British Journal of Surgery. 2004 Dec;91(12):1549-1558

Laparoscopic Training Traditional apprenticeship model.

Acquire skills during actual procedures. Not sufficient for laparoscopic skills.

Other methods. Box trainer with animal or synthetic

models. Virtual reality simulator. Video-based assessment.

Assessment of Skills Trainee must demonstrate

competency. Evaluation by a senior surgeon.

Direct observation of the trainee. Video-based assessment.

Question: Can we organize video in order to assist in video-based assessment?

American Journal of Surgery. 1991 Mar;161(3):399-403

Objective Identity key portions of surgical

procedure to aid in video-based assessment.

Stage I is to identify the "critical view".

Video

Segments

Frames

Overview Background

Laparoscopic Surgery Image Classification

Methods Discussion

Summary: Organize surgical video to make it easier for expert to review.

The Critical View Helps ensure that the anatomy has

been properly identified. Occurs after dissecting anatomy. Occurs before clipping the cystic

artery and cystic duct.

The Critical View

Cystic artery

Liver

Cystic duct

Fundus

Netter's Atlas of Human Anatomy

The Critical View

Image Classification - Human Features a person might use. Spectral features.

Tonal variations. Textural features.

Spatial distribution of tonal variations. Contextual features.

Features from surrounding areas.

Image Classification - Computed Features extracted from image. Spectral features.

Distribution, size, width. Textural features.

Homogeneity, contrast, correlation. Similarity/distance metrics.

Jaccard coefficient, Jeffrey divergence.Journal of WSCG. 2003; 11(1):269-273

IEEE Transaction on Systems, Man, and Cybernetics. 1973 Nov; 3(6):610-621

Color Histogram Red, green, blue, or gray.

Count number of pixels for each tone. One 28 set for an 8-bit image for each color. Does not vary with translation and rotation. Ignores shape and texture.

4x4 image. 4 gray tones. H = {5, 4, 5, 2}

0 0 1 10 0 1 10 2 2 22 2 3 3

Binary Histogram Quantize values for each tone to 0

or 1. Background color given less weight. Subtle changes given more weight.

HB = {1, 0, 1, 1}

0 0 0 00 0 0 00 2 2 22 2 3 3

3D Histogram Distribution within a 3D color-space.

3D color space (red, green, blue). Used in object recognition & image retrieval. n3 entries, where n = number of tones.

Example. Quantized to 3 tones

for each color.

Spatial-Dependency Matrix Co-occurrence

matrix. Co-occurring values

(0o, 45o, 90o, 135o). Four 28 x 28

matrices for 8-bit image.

Co-occurring Bits

0 1 2 3Reference Bits

0 4 2 1 01 2 4 0 02 1 0 6 13 0 0 1 2

0 0 1 10 0 1 10 2 2 22 2 3 3

135o 90o 45o

0o Ref 0o

45o 90o 135o

M0 =

Additional Spectral Features Location of the distribution.

Mean = Σ (bin*freq) / Σ (freq). Mode = bin of the max freq.

Size of the distribution. Standard deviation.

Width of the distribution. Max(bin) - Min(bin).

Additional Textural Features Homogeneity.

Number of tone transitions. Contrast.

Amount of local variation. Correlation.

Measure of linear dependencies.

Similarity/Distance Metrics Jaccard Coefficient.

Similarity of two sample sets.|A B| / |A B|

Two binary sets.M11 / (M01 + M10 + M11)

Jeffrey Divergence. Distance between two vector spaces.Σ (xi log(xi/avgi) + yi log(yi/avgi))n

i=1

Other Distance Metrics City Block or Manhattan Distance. Euclidean Distance. Chi-Square. Canberra Distance.

Proceedings ACM SAC. 2008;:1225-1230

Related Efforts - Hysteroscopy Use Jeffrey divergence on color

histogram to identify segments. Relevant segments based on image

redundancy. No understanding

of the content of each segment.

Proceedings 27th IEEE-EMBS. 2005;:5680-5683

Mayo Clinic

Related Efforts - Echocardiogram Use cosine similarity and edge

change ratio to identify video segments.

State-based modeling. Identify states in each

video segment. Diastole (resting). Systole (contracting).

IEEE Transaction on Information Technology in Biomedicine. 2008 May;12(3):366-376

Medline Plus

Overview Background

Laparoscopic Surgery Image Classification

Methods Discussion

Summary: Spectral and textural features compared with similarity metrics.

Methods Our objective.

Identity key portions of surgical procedure to aid in video-based assessment.

Stage I is to identify the "critical view".Video

Segments

Frames

Tools FFmpeg

http://ffmpeg.mplayerhq.hu/ Extract JPEG images.

ImageJ http://rsbweb.nih.gov/ij/ Macros and Java plugins.

Work Plan Identify videos for analysis. Convert videos to JPG. Evaluate ability to identify critical view.

Color histogram. Binary histogram. 3D histogram. Spatial-dependency matrix. Jaccard coefficient, Jeffrey divergence.

Algorithm

Feature ExtractionImageJ Color Histograms

Binary Histograms3D Histograms

Spatial-Dependency Matrices

Similarity Metric

Critical View?

Critical View

Random ImageImage Extraction

FFmpeg

Overview Background

Laparoscopic Surgery Image Classification

Methods Discussion

Summary: Attempt to identify the critical view by comparing image features with similarity metrics.

Discussion Color and binary histograms do not

correlate with the critical view. They do, however, predict when we are

in the abdomen. Currently working on 3D histograms

and spatial-dependency matrices. NIH grant application under

development.

Challenges Live tissue (vs. solid objects).

Deformable. Normal variation. Disease states.

May need to consider. Temporal information. Relevant clinical data of the patient. Critical view "rectangle" (contextual).

Summary We are comparing image features

with similarity metrics to identify the critical view.

This is a first step in automated video summarization, to help with video-assisted evaluation of laparoscopic surgery.

Thank You