2014, 13th Kandroid minmax

GPU, Graphics and Networking

OpenGL and EGL

SK플래닛/모바일 플랫폼 개발팀

남정수 (yegam400@gmail.com)

http://www.linkedin.com/in/yegam400

목 차

OpenGL

• What is OpenGL

• OpenGL ES 2.0 Rendering Pipeline

• GLSL(OpenGL Shading Language)

• Android GLSL Usages

EGL

• OpenGL Utility Libraries

• EGL Overview

• EGL Operations

• EGL Usages

• Android GLSurfaceView

3 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : What is OpenGL

• OpenGL

– OpenGL (Open Graphics Library)[2] is a cross-language, multi-platform application programming interface (

API) for rendering 2D and 3D vector graphics. The API is typically used to interact with a Graphics processin

g unit (GPU), to achieve hardware-accelerated rendering.

– OpenGL was developed by Silicon Graphics Inc. (SGI) from 1991 and released in January 1992[3] and is wid

ely used in CAD, virtual reality, scientific visualization, information visualization, flight simulation, and video g

ames. OpenGL is managed by the non-profit technology consortium Khronos Group.

• OpenGL ES

– OpenGL for Embedded Systems (OpenGL ES or GLES) is a subset of the OpenGL

– It is designed for embedded systems like smartphones, computer tablets, video game consoles and PDAs.

• OpenGL development

– In addition to the features required by the core API, GPU vendors may provide additional functionality in the f

orm of extensions. Extensions may introduce new functions and new constants, and may relax or remove re

strictions on existing OpenGL functions. Vendors can use extensions to expose custom APIs without needin

g support from other vendors or the Khronos Group as a whole, which greatly increases the flexibility of Ope

nGL. All extensions are collected in, and defined by, the OpenGL Registry.

4 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : OpenGL ES 2.0 Rendering Pipeline

• Fixed Function Pipeline removed at ES2.0

5 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : Shading and Shaders

• Shading

– Shading refers to depicting depth perception in 3D models or illustrations by varying levels of

darkness.

– In computer graphics, shading refers to the process of altering the color of an object/surface/

polygon in the 3D scene, based on its angle to lights and its distance from lights to create a p

hotorealistic effect. Shading is performed during the rendering process by a program called a

shader.

6 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : Transform and Lighting

surfaceColor = emissive + ambient + diffuse + specular

emissive = Ke

ambient = Ka x globalAmbient

diffuse = Kd x lightColor x max(N · L, 0)

specular = Ks x lightColor x facing x (max(N · H, 0)) shininess

7 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : Rasterization & Raster Operations(ROP)

8 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : What is GLSL

• GLSL

– OpenGL Shading Language (abbreviated: GLSL or GLslang), is a high-level shading langu

age based on the syntax of the C programming language. It was created by the OpenGL AR

B (OpenGL Architecture Review Board) to give developers more direct control of the graphic

s pipeline without having to use ARB assembly language or hardware-specific languages.

– Supports OpenGL ES(Android, iOS, and etc.)

9 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : GLSL – Simple Example

ftransform() is used for fixed function pipeline.

10 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : GLSL keywords – attribute, uniform, varying

• (Vertex) Attribute

– Vertex attributes are used to communicate from outside to the vertex shader.

• Unlike uniform variables, values are provided per vertex (and not globally for all vertices).

• There are built-in vertex attributes like the normal or the position, or you can specify your own vertex at

tribute like a tangent or another custom value.

• Attributes can't be defined in the fragment shader.

• Uniform

– Uniform variables are used to communicate with your vertex or fragment shader from "outsid

e". In your shader you use the uniform qualifier to declare the variable

• Uniform variables are read-only and have the same value among all processed vertices. You can only

change them within your C++ program.

• Varying

– Varying variables provide an interface between Vertex and Fragment Shader.

• Vertex Shaders compute values per vertex and fragment shaders compute values per fragment.

• If you define a varying variable in a vertex shader, its value will be interpolated (perspective-correct) ov

er the primitive being rendered and you can access the interpolated value in the fragment shader.

11 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : GLSL - Build-in Variables(1/2)

12 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : GLSL - Build-in Variables(2/2)

13 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : GLSL APIs

• GLSL flow APIs

– Loading Shader

• glCreateShaderObject()

• glShaderSource()

– Compiling Shader

• glCompileShader()

– Linking

• glCreateProgramObject()

• glAttachObject()

• glLinkProgram()

– Using Shaders

• glUseProgramObject()

• GLSL communication APIs

– Attribute

• glGetAttribLocation()

• glEnableVertexAttribArray()

• glVertexAttribPointer()

– Uniform

• glGetUniformLocation()

• glUniform()

14 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : GLSL Vertex Shader Example – Directional Lighting

15 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : GLSL Fragment Shader Example – Per Pixel Directional Lighting

http://www.lighthouse3d.com/tutorials/glsl-tutorial/directional-light-per-pixel/

16 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : GLSL Fragment Shader Example – YUV to RGB Conversion

http://helloworld.naver.com/helloworld/1207

17 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : GLSL Fragment Shader Example – Image Filter(Box Blur)

18 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : Android GLSL Usages(1/4) - Initializing

19 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : Android GLSL Usages(2/4) - Shaders

20 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : Android GLSL Usages(3/4) – Rendering environment setup

21 2014, 13th Kandroid minmax - www.kandroid.org

OpenGL : Android GLSL Usages(4/4) – Rendering a mesh

목 차

OpenGL

• What is OpenGL

• OpenGL ES 2.0 Rendering Pipeline

• GLSL(OpenGL Shading Language)

• Android GLSL Usages

EGL

• OpenGL Utility Libraries

• EGL Overview

• EGL Operations

• EGL Usages

• Android GLSurfaceView

23 2014, 13th Kandroid minmax - www.kandroid.org

EGL : OpenGL ARB and Khronos Group

• OpenGL ES

– OpenGL ES is a lightweight graphics API which is designed for Embedded System from Ope

nGL which is designed for Work Station.

– OpenGL is maintained by OpenGL ARB(Architecture Review Board), OpenGL ES is

maintained Khronos Group.

24 2014, 13th Kandroid minmax - www.kandroid.org

EGL : OpenGL Utility Libraries

Prefix Name Functions

gl OpenGL API glClear,glDrawArrays,…

glu OpenGL Utility Library gluPerspective,gluLookAt

glut OpenGL Utility Toolkit glutInitDisplayMode,glutSwapBuffers

aux OpenGL Auxiliary Library auxDIBImageLoad,auxInitWindow

glew OpenGL Extension Wrangler Library glewInit, glewIsSupported

wgl/agl/cgl/glx/egl Native Interface for OpenGL wglCreateContext,wglMakeCurrent,e

glCreateContext,eglMakeCurrent

25 2014, 13th Kandroid minmax - www.kandroid.org

EGL : WGL

• OpenGL Native Interface for Windows

– WGL or Wiggle is an API between OpenGL and the windowing system interface of Microsoft

Windows.

• http://nehe.gamedev.net/tutorial/creating_an_opengl_window_(win32)/13001/

ChoosePixelFormat()

SetPixelFormat()

wglCreateContext()

wglMakeCurrent()

26 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL Overview

• Native Platform Interface

– EGL™ is an interface between Khronos rendering APIs such as OpenGL ES or OpenVG and

the underlying native platform window system. It handles graphics context management, surf

ace/buffer binding, and rendering synchronization and enables high-performance, accelerate

d, mixed-mode 2D and 3D rendering using other Khronos APIs. EGL also provides interop ca

pability between Khronos to enable efficient transfer of data between APIs – for example bet

ween a video subsystem running OpenMAX AL and a GPU running OpenGL ES.

• Portable Layer for Graphics Resource Management

– EGL can be implemented on multiple operating systems (such as Android and Linux) and

native window systems (such as X and Microsoft Windows). Implementations may also

choose to allow rendering into specific types of EGL surfaces via other supported native

rendering APIs, such as Xlib or GDI. EGL provides:

• Mechanisms for creating rendering surfaces (windows, pbuffers, pixmaps) onto which client APIs can d

raw and share

• Methods to create and manage graphics contexts for client APIs

• Ways to synchronize drawing by client APIs as well as native platform rendering APIs

27 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL Features

• Specifically EGL is a wrapper over the following subsystems;

– WGL: Windows GL-the Windows-OpenGL interface (pronounced wiggle)

– CGL: the Mac OS X-OpenGL interface (the AGL layer sits on top of CGL)

– GLX: the equivalent X11-OpenGL interface

• EGL not only provides a convenient binding between the operating system resources

and the OpenGL subsystem, but also provides the hooks to the operating system to i

nform it when you require something, such as;

1. Iterating, selecting, and initializing an OpenGL context.

2. This can be the OGL API level, software vs. hardware rendering, etc.

3. Requesting a surface or memory resource.

4. The OS services requests for system or video memory.

5. Iterating through the available surface formats (to pick an optimal one)

6. You can find out properties of the video card(s) from the OS – the surfaces presented will re

sides on the video card(s) or software renderer interface.

7. Selecting the desired surface format.

8. Informing the OS you are done rendering and it’s time to show the scene.

9. Informing the OS to use a different OpenGL context.

10. Informing the OS you are done with the resources.

28 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGLWindow, EGLDisplay, EGLSurface

• EGLDisplay

– Abstract display on which graphics are drawn

– Single physical screen

– Initialize by querying a default display

– All EGL objects are associated with an EGLDisplay

• EGLContext(Rendering Contexts)

– EGLContext is a state machine defined by a client API

– EGLContext is associated with EGLSurfaces

• EGLSurface(Drawing Surfaces)

– Types: windows, pbuffers, pixmaps

• Windows, pixmaps: tied to native window system

– Created with EGLConfig

• Describes depth of color buffer component and types, quantities, and sizes of the ancillary

buffers(depth, multisample, stencil buffers)

– Ancillary buffers are associated with an EGLSurface

• Not EGLContext

29 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL Operation(1/2)

• Interaction with native rendering

– Native rendering will always be supported by pixmap surfaces

• Pixmap surfaces have restricted capabilities and performance relative to window and pbuffer surfaces

– Native rendering will not be supported by pbuffer surfaces, since the color buffers of pbuffers are allocated internally

by EGL and are not accessible through any other means.

– Native rendering may be supported by window surfaces, but only if the native window system has a compatible

rendering model allowing it to share the back color buffer, or if single buffered rendering to the window surface is

being done.

– When both native rendering APIs and client APIs are drawing into the same underlying surface, no guarantees are

placed on the relative order of completion of operations in the different rendering streams other than those provided by

the synchronization primitives discussed in section 3.8

• Shared State

– OpenGL and OpenGL ES Texture Objects

• OpenGL and OpenGL ES makes no attempt to synchronize access to texture objects. If a texture object is bound to

more than one context, then it is up to the programmer to ensure that the contents of the object are not being

changed via one context while another context is using the texture object for rendering. The results of changing a

texture object while another context is using it are undefined.

– OpenGL and OpenGL ES Buffer Objects

• hen it is up to the programmer to ensure that the contents of the object are not being changed via one context while

another context is using the buffer object for rendering. The results of changing a buffer object while another context

is using it are undefined.

• Multiple Threads

– EGL guarantees sequentiality within a command stream for each of its client APIs , but not between these APIs and

native APIs which may also be rendering into the same surface.

– Client API commands are not guaranteed to be atomic.

• Synchronization is in the hands of the client.

• It can be maintained at moderate cost with the judicious use of commands such as glFinish, vgFinish, eglWait-Client,

and eglWaitNative, as well as (if they exist) synchronization commands present in native rendering APIs.

30 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL Operation(2/2)

• Power Management

– Power management events can occur synchronously while an application is running. When

the system returns from the power management event the EGLContext will be invalidated,

and all subsequent client API calls will have no effect (as if no context is bound).

– Following a power management event, calls to eglSwapBuffers, eglCopy-Buffers, or

eglMakeCurrent will indicate failure by returning EGL_FALSE. The error EGL_CONTEXT_LOST

will be returned if a power management event has occurred.

• On detection of this error, the application must destroy all contexts (by calling eglDestroyContext for each

context). To continue rendering the application must recreate any contexts it requires, and subsequently restore

any client API state and objects it wishes to use.

• Any EGLSurfaces that the application has created need not be destroyed following a power management event,

but their contents will be invalid.

31 2014, 13th Kandroid minmax - www.kandroid.org

EGL : OpenGL Framebuffer

• Framebuffer

– A Framebuffer is a collection of buffers that can be used as the destination for rendering.

Name Description

Color Buffer Double buffering, stereo buffering

glDrawBuffer,glReadBuffer,glClearColor

Depth Buffer Shadow map, internal rendering

glDepthFunc,glClearDepth

Stencil Buffer Shadow volume, color masking, reflection

glStencilFunc,glStencilOp,glClearStencil

Accum Buffer Motion blur, anti-aliasing

glAccum,glClearAccum

Functions Parameters

glClear GL_COLOR_BUFFER_BIT,GL_DEPTH_BUFFER_BIT,GL_STENCIL_B

UFFER_BIT,GL_ACCUM_BUFFER_BIT

glEnable GL_DEPTH_TEST, GL_STENCIL_TEST

32 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL Basic Usage

• The basic usage of EGL and similar API are the following;

1. (Android) Obtain the EGL interface.

• So you can make EGL calls

2. Obtain a display that’s associated with an app or physical display

3. Initialize the display

4. Configure the display

5. Create surfaces

• Front, back, offscreen buffers, etc.

6. Create a context associated with the display

• This holds the “state” for the OpenGL calls

7. Make the context “current”

• This selects the active state

8. Render with OpenGL (OpenGL not EGL calls, the OpenGL state is held by EGL context)

9. Flush or swap the buffers so EGL tells the OS to display the rendered scene. Repeat rend

ering till done.

10. Make the context “not current”

11. Clean up the EGL resources

33 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL API(1/3)

Initializing Funtion Description

EGLDisplay eglGetDisplay(EGLNativeDisplayType

display_id);

EGL_DEFAULT_DISPLAY

EGLBoolean eglInitialize(EGLDisplay dpy, EGLint

*major, EGLint *minor);

the values of *major and *minor would be 1 and 2, respectively).

major and minor are not updated if they are specified as NULL.

EGLBoolean eglTerminate(EGLDisplay dpy);

const char *eglQueryString(EGLDisplay dpy, EGLint

name);

EGL_CLIENT_APIS, EGL_EXTENSIONS, EGL_VENDOR, or EGL_

VERSION.

Rendering to Textures Funtion Description

EGLBoolean eglBindTexImage(EGLDisplay dpy,

EGLSurface surface, EGLint buffer);

Bind OpenGL ES Texture

EGLBoolean eglReleaseTexImage(EGLDisplay dpy,

EGLSurface surface, EGLint buffer);

Release OpenGL ES Texture

Posting the color buffer Funtion Description

EGLBoolean eglSwapBuffers(EGLDisplay dpy,

EGLSurface surface);

Posting to a Window

When native window resizing, called automatically

EGLBoolean eglCopyBuffers(EGLDisplay dpy,

EGLSurface surface, EGLNativePixmapType

target);

Copying to a Native Pixmap

EGLBoolean eglSwapInterval(EGLDisplay dpy, EGLint

interval);

Control swap buffer interval

EGL Extension Funtion Description

void (*eglGetProcAddress(const char *procname))(void); eglQueryString(dpy, EGL_EXTENSIONS)

34 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL API(2/3)

Configuration Funtion Description

EGLBoolean eglGetConfigs(EGLDisplay dpy,

EGLConfig *configs, EGLint config_size,

EGLint *num_config);

Get configs

EGLBoolean eglChooseConfig(EGLDisplay

dpy, const

EGLint *attrib_list, EGLConfig *configs,

EGLint config_size, EGLint *num_config);

Choose configs by attrib_list

EGLBoolean eglGetConfigAttrib(EGLDisplay

dpy,

EGLConfig config, EGLint attribute, EGLint

*value);

Get config attribs

35 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL API(3/3)

Surface Funtion Description

EGLSurface eglCreateWindowSurface(EGLDisplay dpy,

EGLConfig config, EGLNativeWindowType win,

const EGLint *attrib_list);

Create surface for window rendering

EGLSurface eglCreatePbufferSurface(EGLDisplay dpy,

EGLConfig config, const EGLint

*attrib_list);

Create surface for offline rendering

EGLSurface eglCreatePbufferFromClientBuffer(EGLDisplay dpy, EGLenum buftype, EGLClientBuffer

buffer, EGLConfig config, const EGLint *attrib_list);

Create surface for offline rendering

from client buffer

EGLSurface eglCreatePixmapSurface(EGLDisplay dpy,

EGLConfig config, EGLNativePixmapType

pixmap, const EGLint *attrib_list);

Create surface for pixmap rendering

EGLBoolean eglDestroySurface(EGLDisplay dpy,

EGLSurface surface);

Destroy a surface

EGLBoolean eglSurfaceAttrib(EGLDisplay dpy,

EGLSurface surface, EGLint attribute,

EGLint value);

Get surface attributes

EGLBoolean eglQuerySurface(EGLDisplay dpy,

EGLSurface surface, EGLint attribute,

EGLint *value);

Get surface attributes

36 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL API Usage(1/2)

37 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL API Usage(2/2)

38 2014, 13th Kandroid minmax - www.kandroid.org

EGL : GLSurfaceView

• Features

– Inherited from SurfaceView

– EGL encapsulation

• Functions

– setRenderer()

– setEGLConfigChooser()

• Default egl config chooser included

– setRenderMode()

• RENDERMODE_WHEN_DIRTY

• requestRender()

– setDebugFlags()

• DEBUG_CHECK_GL_ERROR

• DEBUG_LOG_GL_CALLS

39 2014, 13th Kandroid minmax - www.kandroid.org

EGL : GLSurfaceView.Renderer

• Functions

– onSurfaceCreate()

• Create window surface

– onSurfaceChanged()

• Surface size changed

– onDrawFrame()

• Automatically eglSwapBuffers when non-dirty rendering

40 2014, 13th Kandroid minmax - www.kandroid.org

EGL : EGL PBuffer

2014, 13th Kandroid minmax

GPU, Graphics and Networking

Q & A

SK플래닛/모바일 플랫폼 개발팀

남정수 (yegam400@gmail.com)

http://www.linkedin.com/in/yegam400