remote graphical rendering
DESCRIPTION
We have developed technology that allows a wide range of graphical interfaces to be streamed efficiently over wide area networks. This is an enabling technology that enables remote graphics akin to the way MPEG compression enables video streaming.TRANSCRIPT
![Page 1: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/1.jpg)
Remote Graphical Rendering
Joel Isaacson
Ascender Technologies Ltd.
Copyright 2012 Joel Isaacson
Ascender Technology Ltd Remote Rendering
![Page 2: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/2.jpg)
Cellphone App
Ascender Technology Ltd Remote Rendering
![Page 3: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/3.jpg)
Application Level (1)
Ascender Technology Ltd
This is the normal way Linux/Android/Iphone runs apps.
The application itself is exported and run locally. Normally applications either come installed with
the system or are added later (e.g. Android Market).
Applications can run locally without using any network bandwidth.
Remote Rendering
![Page 4: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/4.jpg)
Bitmap Level (4)
Ascender Technology Ltd
Common approach to exporting graphics (VNC, Chromoting).
Works well for any mix of graphics applications. Simple to implement. Heavy user of network bandwidth. Amenable to compression. Computationally intensive.
Remote Rendering
![Page 5: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/5.jpg)
Toolkit Level (2)
Ascender Technology Ltd
It is technically very complex. Android, to date, has 15 different toolkit API variants.
Every application can extend the toolkit with custom widgets (subclasses of android.view.View).
Clearly impossible.
Remote Rendering
![Page 6: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/6.jpg)
Android Rendering Level
Ascender Technology Ltd
This work exports graphics at the rendering level. In Android there are a number of rendering
interfaces that can be used:
1. Skia graphics
2. OpenGL ES 1.1 or OpenGL ES 2.0
3. Android.view.View We shall see that in effect toolkit and application
level graphics can be exported.
Remote Rendering
![Page 7: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/7.jpg)
The Rendering Layer Expanded
Ascender Technology Ltd Remote Rendering
![Page 8: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/8.jpg)
Software Rendering Level (Skia)
Ascender Technology Ltd
The rendering level is the graphics layer that actually “colors” the pixels in the bitmap.
The Skia Graphics Engine is a compact open source graphics library written in C++.
It was originally developed by Skia Inc., which was subsequently acquired by Google, who released the software as open source.
Now known as Skia, it is currently used in Google Chrome, Chrome OS, Chromium OS, and Android.
Remote Rendering
![Page 9: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/9.jpg)
Rendering Level (Skia)
Ascender Technology Ltd
Skia is a complete 2D graphic library for drawing Text, Geometries, and Images. Features include:
3x3 matrices w/ perspective Antialiasing, transparency, filters Shaders, xfermodes, maskfilters,
patheffects
Remote Rendering
![Page 10: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/10.jpg)
Rendering Level (Skia)
Ascender Technology Ltd
Each Skia call has two components: the primitive being drawing (SkRect, SkPath, etc.) and color/style attributes (SkPaint).
For example:canvas.save();
canvas.rotate(45);
canvas.drawRect(rect, paint);
canvas.drawText(“abc”, 3, x, y, paint);
canvas.restore();
Remote Rendering
![Page 11: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/11.jpg)
NinePatch Widget
Ascender Technology Ltd Remote Rendering
![Page 12: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/12.jpg)
Ascender Technology Ltd Remote Rendering
GUI WidgetNinePatch Code
DrawbitmapRect(); // 1DrawbitmapRect(); // 2DrawbitmapRect(); // 3DrawbitmapRect(); // 4DrawRect()DrawbitmapRect(); // 5DrawbitmapRect(); // 6DrawbitmapRect(); // 7DrawbitmapRect(); // 8DrawbitmapRect(); // 9
![Page 13: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/13.jpg)
Android Remote Rendering
Ascender Technology Ltd Remote Rendering
![Page 14: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/14.jpg)
Android Remote Rendering
Ascender Technology Ltd
We can simply modify the Skia library to export graphics by adding a small stub to Android to serialize and send a RPC-like rendering stream.
The local client can be on any system (Linux, MS Windows, Mac OS).
The prototype developed was on Linux-X11. Only 17 Skia routines had to be modified to
provide remote Android graphics.
Remote Rendering
![Page 15: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/15.jpg)
Round Trip Latencies
Ascender Technology Ltd
The US east-west coast round trip is about 70 ms. Just one round trip delay per frame would reduce
the possible frame rate to 14 FPS. Potentially every Skia call returns a result. Some
results are critical for “Measure Passes”. In order to obtain measurements, we run a Skia
renderer in parallel on the remote machine. We can then send the rendering stream at full
network bandwidth, via a simplex link.Remote Rendering
![Page 16: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/16.jpg)
Android Remote RenderingSimplex Network Connection
Ascender Technology Ltd Remote Rendering
![Page 17: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/17.jpg)
Compression
Ascender Technology Ltd
Android apps have a frame rate of 30-60 FPS. For every frame the complete rendering sequence
is sent. Compression is a good idea.
Remote Rendering
![Page 18: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/18.jpg)
MPEG Compression
Ascender Technology Ltd
Video streams, photographically taken or photorealistically synthesized, can be compressed with MPEG standard codecs.
Even though no specific technique used in MPEG compression is applicable for the compression of our problem domain (rendering streams), some of the assumptions about how the data sets are generated are similar and the compression model design is similar.
Remote Rendering
![Page 19: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/19.jpg)
MPEG Compression Techniques
Ascender Technology Ltd
The conversion of RGB images to YUV and sub-sampling is motivated by the color physiological opponency theory model of human vision.
Within each frame, the accuracy of spacial changes with shorter wavelengths are less important than the accuracy of longer wavelengths.
Inter-frame compression is based on finding motion vectors of the current frame based on previous and subsequent frames.
Remote Rendering
![Page 20: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/20.jpg)
Rendering Stream Assumptions
Ascender Technology Ltd
The rendered material consists of a large number of sequential images (frames).
The target of the GUI images is the human visual system.
The subject matter while not being a product of our everyday visual world is modeled on the visual contexts of this world.
The GUI images are generated frame after frame by repeated invocation of GUI procedures.
Remote Rendering
![Page 21: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/21.jpg)
Repeated Invocation of GUI Procedures
Ascender Technology Ltd
In practice, the number of unique sequences of rendering functions in execution paths taken within the code are bounded. This is because the rendering commands are generated by a fixed number of GUI functions and an application running a bounded amount of code.
The execution paths can be incrementally learned and entered into a procedure dictionary as the rendering commands are streamed.
Remote Rendering
![Page 22: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/22.jpg)
Rendering Compression Techniques
Ascender Technology Ltd
Even if the sequences of rendering functions themselves are in the dictionary, the data arguments associated with these functions might be quite different from one another. Therefore, we keep a dictionary of the data arguments previously encountered for this particular sequence.
As a rendering procedural sequence with associated data is encountered, the data sequence dictionary for this procedural sequence is searched for the closest match to the current data sequence.
Remote Rendering
![Page 23: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/23.jpg)
Android Contact List(Froyo)
Ascender Technology Ltd Remote Rendering
![Page 24: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/24.jpg)
Android Contact
Ascender Technology Ltd Remote Rendering
![Page 25: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/25.jpg)
Structured Procedural Compression
Ascender Technology Ltd
A careful examination of the rendering stream generated by the Android GUI reveals additional structural information that can be used to improve the data model.
The rendering stream has balanced save()-restore() pairs within each frame of the rendering stream.
Each save() is found at the beginning of a GUI function and a restore() is found at the end of each GUI function.
Remote Rendering
![Page 26: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/26.jpg)
Structured Procedural Compression
Ascender Technology Ltd
This information can be used to reverse engineer individual GUI and application procedures.
It will also reveal the call-graph of these procedures. Using this information, the rendering code
dictionary becomes a rendering procedure dictionary.
The call-graph data is best embedded in the per-procedure data dictionary.
DEMORemote Rendering
![Page 27: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/27.jpg)
Structured Procedural Compression
Ascender Technology Ltd
The server (encoder) constructs a “procedure dictionary” that is identical to the dictionary that is constructed by the client (decoder). Similar of LZW compression.
The procedures in this dictionary, surprisingly, are reversed engineered toolkit and application level routines.
We in effect have ended up exporting graphics at the toolkit-application level.
Remote Rendering
![Page 28: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/28.jpg)
Structured Procedural CompressionStatistics
Ascender Technology Ltd
Our compression algorithm was tested on the rendering trace of a 60 frame sequence.
There were 13702 rendering commands for an average of 228 rendering commands per frame.
Of the 13702 rendering commands, there were 2691 functions (save/restore pairs).
Of these, only 47 were unique. This gives a compression rate of 1.75% (about 1:57).
Remote Rendering
![Page 29: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/29.jpg)
Structured Procedural CompressionStatistics
Ascender Technology Ltd
Of the 13702 rendering commands only 354 had completely unique data parameter sets and 203 had data sets which are partially different.
Only these 557 data sets must be transmitted, thereby giving data compression of 4.06% (about 1:25).
If the partially different data sets are differentially transmitted, a data compression of 3.3% (about 1:30) is obtained.
Remote Rendering
![Page 30: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/30.jpg)
Structured Procedural CompressionEntropy Encoding
Ascender Technology Ltd
For the last stage of compression, general techniques such as run length encoding (RLE) and entropy encoding (Huffman or Arithmetic) are used to produce a minimum bit count representation of the compressed material.
This additional phase should be expected to reduce the number of bits by a factor of 2-3
Remote Rendering
![Page 31: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/31.jpg)
ICS GUI Rendering
Ascender Technology Ltd
Under ICS remote rendering can be done in any of three different interfaces:
1) Software rendering – Skia
2) Hardware rendering – OpenGL ES 2.0
3) Abstract rendering – Canvas class All three interfaces can use procedural
compression. The OpenGL rendering stream is much more
verbose (5x) than the equivalent Skia stream.
Remote Rendering
![Page 32: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/32.jpg)
Cloud computing
Ascender Technology Ltd
Any Android app can be run on the server. It is not necessary that the remote server and the
local device have the same architecture, i.e. an ARM server can provide services for an Intel device.
For example: You might use Remote Rendering to provide Android apps to Chrome OS.
Remote Rendering
![Page 33: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/33.jpg)
App Library / Subscription Model
Ascender Technology Ltd
Currently, apps are loaded into the local device - either installed at the time of purchase or added subsequently.
If efficient remote execution of apps can be supported, then instead of software purchases, a subscription model becomes possible.
A fixed monthly fee would entitle the subscriber to access a large library of applications.
Remote Rendering
![Page 34: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/34.jpg)
App Demos
Ascender Technology Ltd
In a purchase/rental model, apps can be demoed remotely, prior to purchase.
If the user of the device finds the app to his/her liking, it can then be bought.
Remote Rendering
![Page 35: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/35.jpg)
Remote Enterprise Applications
Ascender Technology Ltd
Applications may benefit from running within the enterprise's data centers, with the obvious benefits of scalability, security and maintainability.
Enterprise workers typically migrates through various computing environments (mobile-fixed-mobile) during a typical day.
Having the application running on a server will allow the running applications to migrate to various clients seamlessly.
Remote Rendering
![Page 36: Remote Graphical Rendering](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4d86e4a79593d368b46cb/html5/thumbnails/36.jpg)
Set-top Boxes
Ascender Technology Ltd
Many set-top boxes cannot practically run Android since the set-top box might not have sufficient resources or might not contain an ARM processor.
A local client application that only performs remote rendering takes limited resources and needs no long term persistent state.
Set-top boxes typically don't have complex local installations. Installing Android apps locally would present a support challenge to the operator.
Remote Rendering