introduction to 2d/3d graphics
DESCRIPTION
Takes the reader through the various components of windowing systems, and how to develop and benchmark various Graphics applications using OpenGL and other toolsets. Also includes a Cheatsheet that covers various terminologies used in the Graphics world.TRANSCRIPT
TIITC2009 : Customer Centric Strategies
1
Engaging the consumer with visual effects –
Architectures and implementations for graphics
Prabindh Sundareson ([email protected])
Texas Instruments India
DSPS
TIITC2009 : Customer Centric Strategies
2
Outline
• 2D Graphics
• 3D Graphics
• The OpenGL API
• Internals of SGX, benchmarking
• Future roadmap
• Conclusion
• [Graphics Cheatsheets]
TIITC2009 : Customer Centric Strategies
3
What can Graphics do for me ?
http://cg.cs.tsinghua.edu.cn:8080/cmm/?page_id=155
TIITC2009 : Customer Centric Strategies
4
Graphics Frameworks Cheatsheet
FB, V4l2FB, V4l2
CairoCairo
Clutter, QT, OperaClutter, QT, Opera
GDI
DirectFBDirectFB DDraw
Surfaces
XX Win32/ Windowing SystemWindow System/Mgrs
Application Frameworks
Buffers / blits
Win32, .NET, Silverlight
Note: Cheatsheet table in appendix
TIITC2009 : Customer Centric Strategies
5
2D Graphics
• Almost all existing Consumer Graphical User Interfaces are still built around 2D, and not around OpenGL– Exception of course, iPhone, other Mobile phones, Games…
– Specific operations can still be accelerated with 3D accelerators. Like blits of large sizes
• What are the typical 2D applications ?– All Windowing Systems (X, Win32, …)
– Application Frameworks like QT, Flash
– Browsers
– All 3D operations ultimately end in 2D blitting (ex, EGL uses PVR2D)
• An example, a Browser application…
TIITC2009 : Customer Centric Strategies
6
Case study: Webkit’s Call Sequence
FILL RECTANGLE
SHOW GLYPHS
SET CLIP REGION
ACQUIRE FONT CACHE
ALLOCATE FONT CACHE
SURFACE COMPOSITE
PREPARE/FINISHCOMPOSITE
FillRectangles()
BatchBlit()
Blit(), mem_copy()
Clear()Blit()TileBlit()
TIITC2009 : Customer Centric Strategies
7
Summary of 2D – Browser Rendering Processes
• To render one page of Youtube, for example
– A large blit of size = page width x block size ~ typical (500 x 100 pix)
– Followed by multiple blits of a glyph cache line of one line of text (page width x text height ) ~ ex (50 x 8), this renders blocks line by line
– Followed by multiple compositing operations for each block to render the whole page ~ ex (16 x 16 to page size)
TIITC2009 : Customer Centric Strategies
8
2D Features - Anti-aliasing
Coverage proportional to (1/x)
xx
Many advances – Supersampling is common now in all GPUs
TIITC2009 : Customer Centric Strategies
9
2D Features - Compositing
SRC
1 -asrc
DST
1-adst adst
SRC & DST
SRC & NO-DST
NO-DST & NO-SRC
Compositing
TIITC2009 : Customer Centric Strategies
10
Porter-Duff OperationsComposition modes were originally proposed by Thomas Porter and Tom Duff in
the article Compositing Digital Images in 1984Describes a model for combining the pixels in a source image with the pixels in a
destination, there are 12 basic composition modes
Formula for combining source and destination pixels
cdst and csrc represent color components of destination and source
adst and asrc represent alpha components of destination and source
Cdst and Csrc are the color components pre-multiplied by the alpha components
The parameters X, Y and Z are either 0 or 1.
Permutations of X, Y, Z, and f produce various composition modes Most Common = SRC, SRC_OVER
TIITC2009 : Customer Centric Strategies
11
A typical 2D Graphics HW Feature Set
• Bit blit feature supporting all P-D combinations and ROPs and blending
• Data expansion (Gray scale to full colour)
• Line/curve drawing based on paths
• Area/pattern filling
• Anti-aliased drawing with enable/disable
• Alpha
• Color-keys
• Hardware cursor support
• SW implementations also have to take care of Glyphs/cache
TIITC2009 : Customer Centric Strategies
12
3D vs 2D
• Windowing systems already comprehend z-order
• 3D != 2D with depth
• So what is 3D ?
TIITC2009 : Customer Centric Strategies
13
Pixology – – Video & Graphics
• Video Compression– Analytic approach to describe a scene
• Describes a scene using its pixel values. Quality is dependent on the source content/ encoder.
• Encoder can be highly complex, decoder is constrained by spec• Residual compressor
• OpenGL– Object based
• Describes a scene using its components and properties. Output quality is dependent on the renderer, not source
• GL authoring tools are complex, but quality depends on renderer– GL Spec is intentionally non-pixel compliant !
• Each frame rendering is independent of any other– Complexity now dependent on shaders !!
Basics
TIITC2009 : Customer Centric Strategies
14
3D Concepts
This is hidden by another object !
Light Source
Shadows,Realism
Not visible in the currentViewport !
1
2
3
4
TIITC2009 : Customer Centric Strategies
15
3D Graphics
• 3D Graphics APIs allow recreating a natural scene, by using– One or more light sources
– Replicating natural phenomena like reflections, shadows, surface textures, and cloth surface properties like bending or water properties like waves
– A specified viewport from where viewer is looking
• This emphasis on recreating realistic “Physics”, has resulted in driving computational complexity higher and higher, in GPUs
• In order to accelerate this in HW, several standard blocks are defined• What is OpenGL anyway ?
TIITC2009 : Customer Centric Strategies
16
Key OpenGL Terminologies
• OpenGL– An API from Khronos (from SGI), for constructing a 3D object, doing operations on
the object, and displaying it
• Primitives– Triangles, Lines, Points, that can be specified through vertices to define an
arbitrary shape
• Texture– Small (!) bitmap to make objects more realistic
• EGL– The EGL API defines a portable mechanism for creating GL contexts and windows
for rendering into, which may be used in conjunction with different native platform window systems using the WSEGL layer
• Shader Language– A C like language, for writing vertex and fragment shaders. For GL ES2.0.
TIITC2009 : Customer Centric Strategies
17
The OpenGL BazaarThe OpenGL Bazaar
OPENGL Full versionOPENGL Full version
ES versionES version
CommonCommon
Common-LiteCommon-Lite
GLSL companionGLSL companion
GLSL-ES companionGLSL-ES companion
What we miss in ES compared to desktop version:Polygons, Display lists, Accumulation buffers,…
Currently in 3.0
Currently in 2.0Currently in 1.0.16
Currently in 1.20
EGLEGLCurrently in 1.3
Core GL SpecCore GL Spec Vertex/FragshadersVertex/Fragshaders
Platform interface and Configuration setupPlatform interface and Configuration setup
EGLEGLCurrently in 1.3
2.0 vs 1.1
TIITC2009 : Customer Centric Strategies
18
ES2.0,ES1.1 and Full FunctionES2.0,ES1.1 and Full Function
• ES2.0 offers a Programmable pipeline vs Fixed function in ES1.1– Ex, Lighting, Fog functions in ES1.1 not present in ES2.0, so need to write
code to generate these effects.– More flexibility, little more work, but lot of available code– Android uses ES 1.1
• ES2.0 NOT backward compatible with ES1.1
• See “Difference Specification” documents in Khronos OpenGL website, for summary of changes compared to openGL “full” function specification
• Ok, so what exactly is the OpenGL pipeline ?
TIITC2009 : Customer Centric Strategies
20
Sample OpenGL ES 2.0 program
Handle = get_platform_specific_window_handle();eglGetDisplay(handle);eglInitialize();
eglBindAPI(EGL_OPENGL_ES_API);eglChooseConfig();eglCreateWindowSurface();eglCreateContext();eglMakeCurrent();
//Compile and Load the shaders before this step …float afVertices[] = {G,H,F, F,H,E, E,A,F, F,A,B, …};glEnableVertexAttribArray(0);glVertexAttribPointer(VERTEX_ARRAY, 3, GL_FLOAT, GL_FALSE, 0, (const void*)afVertices);glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
eglSwapBuffers(eglDisplay, eglSurface);
Setup
Actors
Show
Keep showing the actors, in a loop, change view/position/texture .. How does the shader code look like ?
TIITC2009 : Customer Centric Strategies
21
Simple ShaderSimple Shader
• char* pszFragShader = "\
• uniform sampler2D sTexture; \
• varying mediump vec2 TexCoord; \
• void main (void)\
• {\
• gl_FragColor = texture2D(sTexture, TexCoord);\
• }";
• char* pszVertShader = "\
• attribute highp vec4 inVertex;\
• attribute mediump vec2 inTexCoord;\
• uniform mediump mat4 MVPMatrix;\
• varying mediump vec2 TexCoord;\
• void main()\
• {\
• gl_Position = MVPMatrix * inVertex;\
• TexCoord = inTexCoord;\
• }";
GL ES 2.0 permits shader code to be precompiled and loaded, or dynamically compiled and loaded
TIITC2009 : Customer Centric Strategies
22
Alpha Blending
• Many options for blending –
– Multi-textures, Blending with Framebuffer, Can be looped…
• Blending with framebuffer
– glEnable(GL_BLEND); //disabled by default– glBlendEquation(GL_FUNC_ADD); //many other equations
available– glBlendFunc(GL_SRC_ALPHA, GL_DST_ALPHA); //many other
modes available
TIITC2009 : Customer Centric Strategies
23
Texturing using OpenGL
• Pass Texture coordinates– glEnableVertexAttribArray();
– glVertexAttribPointer();
• Use glTexImage2D() to load image buffer for texture
• Apply texture in the fragment shader program– gl_FragColor = texture2D();
• OMAP3 supports ARGB 8888 textures– Also YUV textures using extensions (IMG_STREAM)
When we do texturing, scaling is automatic
TIITC2009 : Customer Centric Strategies
24
Rotation with OpenGL
• Rotation is accomplished by modifying the viewing angle
• Update the Model View Projection Matrix before drawing
• Update the vertex position in the vertex shader
– gl_Position = MVPMatrix * inVertex; //
• Along with texturing, can be used to create “wet floor” effects
– Apply texture in reverse way
TIITC2009 : Customer Centric Strategies
25
Texturing with SGX - Example
An exampleWindow managerusing texturing and SGX only
Efficient Rendering
TIITC2009 : Customer Centric Strategies
26
Efficient Rendering Paths
Discussion:How many ways can we implement this ?
..SGX Core
Dis
play
Draw
TIITC2009 : Customer Centric Strategies
27
Overview of the SGX core
SGX uses “deferred” rendering
REF: POWERVR SGX.OpenGL ES 2.0 Application Development Recommendations.1.8f.External.pdf (from AnandTech)
TIITC2009 : Customer Centric Strategies
28
Measuring Performance
• Performance of Graphics is not a single number, but a collection of parameters
• Almost all operations other than texturing, take about 10-15% of 500 MHz of Cortex-A8. 2D Texturing can take 100% CPU depending on size of texture
• Optimised 2D Texturing methods using SGXSink bring CPU utilisation to minimum
Raw Throughput Qua
lity
Raw Throughput Qua
lity
CPU Utilisation
DDR Bandwidth
TIITC2009 : Customer Centric Strategies
29
Benchmarking Tools
• Throughput measured under 2 cases,– Vertex Throughput– Pixel Throughput
• Vertex throughput is very important for gaming scenerios (Millions of vertices)
• UI/ Blit use cases have more dependency on Pixel throughput
• SGXPERF – tool allows benchmarking of various texturing methods (2D Pixel throughput)
– https://gforge.ti.com/gf/project/gleslayer/
• How do we compare against ARM-NEON, and all APIs available on SGX ?
TIITC2009 : Customer Centric Strategies
30
SGX Debugging Tools
• For debugging, PDUMP tools are provided that is similar to a stack trace – useful for debugging crash scenarios during power management transitions
• In addition, Imagination Tech provides PVRTUNE for tuning the performance– Shows loading of each shader, number of tasks
GPU futures…
TIITC2009 : Customer Centric Strategies
31
Texture as data
• Fragment Shader is primarily SIMD, processing all pixels as a parallel stream
• Each computation is independent of the other
• Vertex Shader can do MIMD processing, but General Purpose processing is done with the more efficient Fragment Shader
• Right way to represent data arrays is as a Texture Stream
TIITC2009 : Customer Centric Strategies
32
The OpenCL standard
• OpenCL/DirectCompute take the concept of multi-core programming to a common model across the industry !
• Each core is considered a “Compute Unit”
• The host can “submit” “kernels” (like FFT or DCT or any C function) to any Compute Unit
• Event interactions are defined, so that results of computation can be retrieved
• Textures as Arrays– N elements of 1D array, as AxB 2D
Texture, where AxB = N– 1:1 mapping of buffers to be used
• GPU Computing != GP.GPU which is shader based
• Refer OpenCL Cheatsheet Possibilities
Possibilities of GPU moving forward
• General Purpose programming - C/C++ (+) OpenGL (OpenCL)– More complicated because, Games just have to look good, Computations have to be accurate !
– Will make more processing power available to general applications !
• Integrated Web with OpenGL (WebGL)
• Medical Imaging applications
• Usage of higher computing power - Raytracing, Physics processing
Conclusion
TIITC2009 : Customer Centric Strategies
34
Conclusion
• Core clocks are not going to get much faster, but multi-core is here to stay
• Requirements for 2D Graphics vastly different from 3D Graphics
• Continued focus needed on enabling UI frameworks like QT, Browsers using 2D accelerators
• Full usage of 3D in commercial UI is soon approaching - Video texturing, Coverflow are examples that show that it is possible
• Video+Graphics+display integration is key to having a high texturing performance
TIITC2009 : Customer Centric Strategies
35
References
• Porter & Duff Operations
• SGXPERF
• The SGX architecture used in IPhone– http://www.anandtech.com/printarticle.aspx?i=3579
• The OpenGL Redbook– http://www.glprogramming.com/red/chapter01.html
• OpenGL ES2.0 programming handbook– OpenGL® ES 2.0 Programming Guide, Aaftab Munshi; Dan Ginsburg; Dave Shreiner
• Khronos organisation (OpenGL, OpenMAX, OpenVG ..)– http://Khronos.org
TIITC2009 : Customer Centric Strategies
37
Graphics CheatsheetName Author Description
Cairo Cairo-graphics API that can take care of interfacing to multiple backends like X, DirectFB etc
Flash Adobe Application framework for games, animation, video and others
Clutter Intel UI development framework, based completely on OpenGL, with different backends
Gst-Gl Gstreamer Plugin for allowing video to be played as OpenGL ES textures
XGL,WGL,EGL Multiple Interfaces for native platform support for interfacing to the windowing system
OpenGL/ES Khronos An API for 3D Graphics programming
OpenCL Khronos An API for multi core programming
QT Nokia/Trolltech An application framework to develop native applications
Silverlight Microsoft An API primarily for Web Application programming, but also for native
.NET Microsoft An API for application programming, with C# language
Surfaceflinger Android/Google Framework for surface creation and managing drawing
DirectFB DirectFB Framework for surface creation and managing drawing, mostly using HW support
X11 X org A windowing system/Manager very common on Linux system
DirectX Microsoft A collective name for interfaces like DirectDraw etc. Currently in DirectX11 version. Handles 3D and 2D
GDI Microsoft Native API for Win32 programming
Opera Opera Optimised Browser for embedded platforms
Mesa3D/Gallium Mesa An Opensource implementation of OpenGL API on Linux, can use HW if present. Moving to Gallium3D
Vincent3D Vincent An Opensource pure software implementation of OpenGL API on WinCE/Mo
SDL SDL A UI API that is cross platform, primarily meant for Games but also used as backend for rendering
GLEW - Linux library that makes it easier for an OpenGL application to use GL extensions
GLUT - Cross platform library, having window system and GL utility funcs on different platforms