Scalable Streaming of JPEG2000 Images using Hypertext

Transfer ProtocolSachin Deshpande, Wenjun ZengSharp Laboratories of America, Inc.

ACM Multimedia 2001

Outline

Introduction HTTP Streaming Architecture for JPEG2000 An Interactive, Scable JPEG2000 Image Browsing S

ystem A Sample Session JPEG2000 Codestream Structure Streaming Selected portions of the Codestream Structure of the Index File Conclusion

Introduction

JPEG2000 is a new image compression standard Support scalability Support large image (ex. test image, aerial1.raw, 14565*14780)

JPEG2000 defines a scalable bitstream, but it doesn’t define a client-server architecture

The term “streaming” here is that the client requests only selected portions of JPEG2000 bitstreams form the web server

Motivation for using “HTTP” Ease of deployment HTTP support reliable

Introduction (cont’)

Vfile “Vfile – a virtual file media access mechanism and its application

in JPEG2000 images for browsing over internet,” ISO/IECJTC1/SC29/WG1 N1473, Nov. 1999.

proposes a set of API for random access of segments of JPEG20

00 file, and an architecture for cache management, network packetization, packet loss recovery for JPEG2000 files

It needs a proprietary server

These concepts are already built-in when we use HTTP Use byte-range feature of HTTP to handle scalability

HTTP Streaming architecture for JPEG2000

An interactive, scalable JPEG2000 image browsing system HTTP/1.1 compliant Apache server (version 1.3) HTTP client helper application

HTTP streaming part JPEG2000 partial bitstream decoder

“Zoom” menu options request any of the higher resolution image

Quality scalability bitstreamis organized into several quality layers

Mark any region on this lowest resolution image Advantage

quickly browse large image databases choose viewing experience based on available bandwidth and th

e capability of the display device

A sample session (1)

1. Get the index file :

Client (Web Browser) to Server Request :

GET /index_files/index1.idx HTTP/1.1Host: www.host1.comCRLF

Server to Client (Web Browser) Response :

HTTP/1.1 200 OKDate: Fri, 07 Jul 2000 22:45:48 GMTServer: Apache/1.3.12 (Win32)Last-Modified: Fri, 07 Jul 2000 22:45:28 GMTETag: "0-56-39665d88"Accept-Ranges: bytesContent-Length: 86Content-Type: text/jpeg2000_index_fileCRLFhttp://www.imageshost.com/image_files/image1_jpeg2000.jp2[main_header_data]..[tile-part_header_data]..[packet_header_data]..

2. Get the lowest resolution JPEG2000 image:

Client (Helper application) to Server Request :

GET /image_files/image1_jpeg2000.jp2 HTTP/1.1Host: www.imageshost.comRange: bytes=120-1068CRLF

Server to Client (Helper application) Response:

HTTP/1.1 206 Partial ContentDate: Fri, 07 Jul 2000 22:38:58 GMTServer: Apache/1.3.12 (Win32)Last-Modified: Wed, 05 Jul 2000 19:04:01 GMTETag: "0-fffb-396386a1"Accept-Ranges: bytesContent-Length: 949Content-Range: bytes 120-1068/65531Content-Type: image/jpeg2000CRLF[image data bytes 120-1068]

A sample session (2)

3. Client requests to get a rectangular region-of-interest at a particular resolution with top left corner at (100, 180) and width and height of 200 and 500 respectively

Client (Helper application) to Server Request:

GET /image_files/image1_jpeg2000.jp2 HTTP/1.1Host: www.imageshost.comRange: bytes=120-168,175-200,205-300,345-346,400-500,555-666,667-800,900-1000,2500-2567,2890-3056,5678-9000,10000-12004,12050-12060,15600-15605,17000-17001,17005-17010,17050-17060,17800-17905,20000-20005CRLF

Server to Client (Helper application) Response:

HTTP/1.1 206 Partial ContentDate: Tue, 01 Aug 2000 16:38:35 GMTServer: Apache/1.3.12 (Win32)Last-Modified: Wed, 05 Jul 2000 19:04:01 GMTETag: "0-fffb-396386a1"Accept-Ranges: bytes

Content-Length: 8350Content-Type: multipart/byteranges;boundary=3986fd0b22dCRLFCRLF--3986fd0b22dContent-type: image/jpeg2000Content-range: bytes 120-168/65531CRLF[image data bytes 120-168]--3986fd0b22dContent-type: image/jpeg2000Content-range: bytes 175-200/65531CRLF[image data bytes 175-200]--3986fd0b22d……--3986fd0b22dContent-type: image/jpeg2000Content-range: bytes 20000-20005/65531CRLF[image data bytes 20000-20005]--3986fd0b22d--

Tiles, component, precincts, code-block Tiles (spatial domain)

Tile-component Component Transform

ICT (Irreversible Component Transform) For lossy compression

RCT (Reversible Component Transform) For lossy and lossless compression

Precincts (wavelet domain) Code-block

JPEG2000 codestream structure

Main header Width, height, width of a tile, height of a tile, number of

components, bit-depth of each component Tile-part header Tile-part data

packets

Progressive order

The interleaving of the packets can progress along 4 axes:

Component e.g. Y, Cb, Cr

Resolution e.g. LL0, HL1,LH1,HH1, HL2,LH2,HH2, …

Position Layer

1. Layer-resolution-component-position progressive 2. Resolution-layer-component-position progressive 3. Resolution-position-component-layer progressive 4. Position-component-resolution-layer progressive 5. Component-position-resolution-layer progressive

Layer-resolution-component-position Progressive (1)

layer0 layer1

higher sub-bitplanes lower sub-bitplanes

LL0HL1 LH1 HH1HL2 LH2 HH2 …

Y Cb Cr

position0posotion1posotion2

…

…

Resolution-layer-component-position Progressive (2)

layer0 layer1

LL0HL1 LH1 HH1HL2 LH2 HH2 …

Y Cb Cr

position0position1position2

…

…

Resolution-position-component-layer Progressive (3)

layer0 layer1

LL0HL1 LH1 HH1HL2 LH2 HH2 …

Y Cb Cr

position0position1position2

…

…

Position-component-resolution-layer Progressive (4)

layer0 layer1

LL0HL1 LH1 HH1HL2 LH2 HH2 …

Y Cb Cr

position0position1position2

…

…

Component-position-resolution-layer Progressive (5)

layer0 layer1

LL0HL1 LH1 HH1HL2 LH2 HH2 …

Y Cb Cr

position0position1position2

…

…

Streaming selected portions of the codestream (1) Resolution Scalability

2. Resolution-layer-component-position progressive 3. Resolution-position-component-layer progressive

Streaming selected portions of the codestream (2) SNR Scalability

1. Layer-resolution-component-position progressive

Streaming selected portions of the codestream (3) Region of Interest (ROI) Streaming

For an arbitrary region in the spatial domain, trace how each coefficient and pixel value is reconstructed in the inverse wavelet transform, and find the corresponding region in each subband that contributes to the reconstruction of the ROI

Reference

JPEG2000-Based Scalable Reconstruction of Image Local Regions Zhong MinSheng, Ma ZhengMing Proceedings of 2001 International Symposium on

Intelligent Multimedia, Video and Speech Processing May 2-4 2001 Hong Kong

JPEG 2000 Part I Final Committee Draft Version 1.0

Reconstruction of image local region

High Pass: Y(2n+1) = X(2n+1) - floor( (X(2n) + X(2n+2)) / 2 )

Example of DWT:Lifting-based (5, 3) filter

171172 164165142 99 63 55 69171164 55 630 1 2 3 4 5 6 7 8

3

174

12 -3 -113 -11

168 144 60 64

Low Pass: Y(2n) = X(2n) + floor( (Y(2n-1) + Y(2n+1) + 2) / 4 )

Extension

Run…

-12

-12

14

14

1-12

-12

-18

14

34

14

-18

IDWT

X(2n) = Y(2n) - floor( (Y(2n-1) + Y(2n+1) + 2) / 4 )

X(2n+1) = Y(2n+1) + floor( (X(2n) + X(2n+2)) / 2 )

High pass: ceil((x-f_p-1)/2) ~ floor((x+f_n-1)/2)

Low pass: ceil((x-f_p)/2) ~ floor((x+f_n)/2)

Contribute x are those in the range:

f_p and f_n denote the positive and negative extents of the relevant synthesis filter

f_n f_p f_n f_p

Coefficients in 2-D case

(x0, y0) denotes upper-left corner and (x1, y1) denotes lower-right corner

(x0’, y0’) and (x1’, y1’) For LL band

x0’= ceil((x0-f_p)/2) , y0’= ceil((y0-f_p)/2), x1’= floor((x1+f_n)/2), y1’= floor((y1+f_n)/2)

For LH band x0’= ceil((x0-f_p)/2), y0’= ceil((y0-f_p-1)/2), x1’= floor((x1+f_n)/2), y1’= floor((y1+f_n-1)/2)

For HL band x0’= ceil((x0-f_p-1)/2) , y0’= ceil((y0-f_p)/2), x1’= floor((x1+f_n-1)/2), y1’= floor((y1+f_n)/2)

For HH band x0’= ceil((x0-f_p-1)/2), y0’= ceil((y0-f_p-1)/2), x1’= floor((x1+f_n-1)/2), y1’= floor((y1+f_n-1)/2)

Mapping from code-block boundaries to lines in the spatial domain

if the left boundary of a code-block is at a position x, then the corresponding line in its parent subband will be at a position x’ where:

If the code-block isin LLband or LHbandx’= x*2+f_p-1

If the code-block is in HL or HH bandx’= x*2+f_p

Partition of the spatial image for code-blocks in a particular subband

Structure of the index file

Trade off ease of computation at the client side storage requirement at the server side transmission time for the index file

Simply as a collection of the header information of the codestream main header, tile-part headers, all packet headers, and packet length and tile-part leng

th information size of the index file will be around 1% of the original codestream

Including header information and progressive partition information Resolution-layer-component-position progressive order

LRi (for the i’th resolution) LRiLj (for the j’th layer within the ith resolution) LRiLjCk (for the k’th component within j’th layer and i’th resolution) LRiLjCkPl (for each packet)

Including headers, progressive partition information and spatial domain partition information precincts of each resolution code-blocks of each subband mapping from each spatial segment to a precinct or a code-block

Conclusion

Proposed a scalable architecture for streaming JPEG2000 images

Proposed architecture uses standard web server to host the JPEG2000 image content and streams the images to the client using HTTP protocol

Proposed approach can handle resolution, quality and region of interest scalability features of JPEG2000 images

Using standard web server to achieve the scalable streaming will ease the deployment of JPEG2000 images

Enable some novel and versatile applications