i.mx 6 and i.mx 8 graphics sdk and gpu toolkit

PUBLIC USE

HUGO OSORNIO / CESAR MONTERO

GTEC ENGINEER / GTEC VISUAL DIRECTOR

SESSION #FTF-DES-N2011

MAY 17, 2016

FTF-DES-N2011

i.MX 6 AND i.MX 8 GRAPHICS

SDK and GPU Toolkit

PUBLIC USE1 FTF-DES-N2011 PUBLIC USE1 FTF-DES-N2011

AGENDA• What is the graphics demo framework

• How to get it

• How to install it in your PC

• Creating your first application

• FslGraphics vertex helper classes

• GLVertexBuffer, GLIndexBuffer

• Shader class, program class

• Texture class. Importing textures

• Content loading to the demo framework and content monitor, fsl assimp

• Fonts on the demo framework

• Math and diverse helpers

• Handling inputs on the demo framework.

• Available Input Parameters on the Demo Framework

• Real time content updates.

• Rapid prototyping using Native Batch

• UI capabilities

• NXP Memory profiling tools.

• VTK vProfiler and vAnalyzer

• VTK vTracer and vPlayer.

PUBLIC USE2 FTF-DES-N2011

WHAT IS THE i.MX

GRAPHICS SDK?


What Is the i.MX Graphics SDK

• It is a multi-platform framework that enables demos to run on various platforms

without any changes.

• The framework abstracts away all the boilerplate & OS specific code of allocating

surfaces, creating the context, model loading, texture loading, shader compilation,

render loop, animation ticks, benchmarking graph overlays etc. This allows the

demo/benchmark developer to focus on writing rendering code.

• It also enables them to develop demos on PC or Android where the tool chain and

debug facilities allows for faster turnaround time and then take the working code

and deploy without code changes to the supported platforms.


What Is the i.MX Graphics SDK – Supported Platforms

• The platforms we currently support are Windows (for development via emulated

backends), Android NDK and Linux with various windowing systems. The

framework allows us to provide ‘real’ comparative benchmarks between the

different OS and windowing systems we support, since we can run the exact same

demo/benchmark code on them all.


What Is the i.MX Graphics SDK – Supported Hardware Platforms

• The Demo Framework is currently supported in both i.MX6 (all its derivatives) and

initial tests have been successful on the i.MX8 emulation platforms!

• Looking at the i.MX6Q Graphics Architecture you will notice how well integrated to

the HW platform the SDK is, providing you with samples that enable you to

exercise the different GPUs that the i.MX6 family uses via useful samples, and also

by providing a reference platform for you to use them.


HOW TO GET THE

GRAPHICS SDK?


How to get the Graphics SDK

• Go to www.nxp.com, not bragging here, but our site was recently called the 2nd

best semiconductor page in the solar system!!!

• Select PRODUCTS -> i.MX Applications Processors -> i.MX6 Quad -> Software &

Tools->i.MX6DQP/DQ/DL/SL/SX GPU Demo Framework SDK

http://www.nxp.com/


How to get the Graphics SDK

• The latest 2.3 Version of the Demo Framework consists on a .zip file.

• Within the zip file you will find 2 files, fsl-gpu-sdk-2.3.bin is the one that contains all the files; if you are under Linux execute the following steps on a terminal:

• chmod a+x fsl-gpu-sdk-2.3.bin

• ./fsl-gpu-sdk-2.3.bin

• If you are under windows then you are in for some pain, just kidding, install Cygwin and do the same as in the windows terminal.

• $ ./fsl-gpu-sdk-2.3.bin

• Welcome to NXP fsl-gpu-sdk-2.3.bin

• Bla blah


Ok I installed it – Now what?

• You will now have a nice directory structure. The only folders we will focus on on

this training are:

• DemoApps

• DemoFramework

• The DemoApps folder contains all the sample applications for all 3 types of GPUs

supported on the i.MX6

• The DemoFramework folder contains the Demo Framework that those samples

use and that you will also use when creating your applications.


VERI SILICON

GRAPHICS TOOLS


Veri Silicon Graphics Tools

• In order to download the Veri Silicon:

• Select PRODUCTS -> i.MX Applications Processors -> i.MX6 Quad -> Software &

Tools->i.MX6_L4.1.15_GPU_TOOL

•


Veri Silicon Graphics Tools

• The Veri Silicon Tool Kit is a set of tools that enable the user to:

• Develop embedded graphics and compute applications (OpenCL 1.1 and OpenGL

ES 1.1/2.0/3.0/3.1) on your PC via the vEmulator.

• Trace performance of the HW and SW Driver via the vProfiler.

• Create and test Shaders on a PC platform via the vShader.

• Compile shader code into binary executables vCompiler.

• Compress, transfer and decompress images with the vTexture.

• The Demo Framework uses the binaries for the vEmulator to succesfuly emulate

the OpenGL ES drivers from Veri Silicon.


CREATING YOUR

FIRST APPLICATION


Creating your first application

• For this training we will use the Windows Version of the Demo Framework. Césarwill use the Embedded Linux Target for you to see the artist friendly Framework capabilities!

• Within the Doc folder you will find details on how to set the different platforms, Android, Linux, Embedded Linux or Windows.

• For the Demo Framework to work properly on Windows you need:

• VS2013 Express (Or newer).

• Python 2.7.x.

• Veri Silicon vEmulator.


Creating your first application – Setting up your environment

• Start a windows console (cmd.exe) in the DemoFramework folder

• Run the 'prepare.bat' file located in the root of the framework folder to configure the

necessary environment variables and paths.


Creating your first application

To create a new GLES2 demo project named 'CoolNewDemo'

0. Make sure that you performed the simple setup

1. Change directory to the GLES2 sample directory: cd DemoApps/GLES2

2. Create the project template using the FslNewDemoProject.py script FslNewDemoProject.py all -t GLES2 CoolNewDemo

3. Change directory to the newly created project folder 'CoolNewDemo' cd CoolNewDemo

4. Generate build files for Android, Ubuntu and Yocto (this step will be simplified soon) FslBuildGen.py When you add the generated build.sh to git on windows then please remember to set the executable bit using: git update-index --chmod=+x build.sh

5. Launch visual studio using an emulator: .StartProject.bat arm

6. Compile and run the project (The default is to press F5) and start creating your new demo.


INTRODUCTION


Introduction

• A common flow on the Graphics Application development consists on:

• Designing the assets and the overall look and feel of the application.

• Design the assets that the application will use.

• Program the application.

• Assess performance.

• Optimize.

• Assess performance.

• Deploy.

•

• The Demo Framework and the Veri Silicon Toolkit provide tools and samples to simplify your life on each stage of the development, no matter the API or GPU you decide to use for your application.

• Cesar will show you how easy an artist can see his designs come to life on target in real time!

• I will provide an overview on all the technical details that will help you to program your application, assess its performance, debug and deploy it.

• For some samples we will provide a comparison on how would that be done using a bare graphics API vs the Demo Framework.


VERTEX HELPER

CLASSES


Vertices

• Vertices are one of the most important input data to the 3D GPU Pipeline. The

vertices will be passed as inputs to the GPU so it can transform them and generate

rendering primitives and ultimately render them to the display.

• The Demo Framework provides some interesting ways to construct vertices that

can either rearrange vertex format, change index format or even create your Vertex

data.

• In the next slide we will find the available vertex manipulation methods.


Vertices

IndexConverter Simple utility class to convert between index formats. It

might not be efficient but it gets the job done.

VertexConverter Simple utility class to convert between vertex formats. It might

not be efficient but it gets the job done.

VertexDeclaration Defines how a vertex is constructed in an API independent

way.

VertexElementEx Defines a vertex element

VertexPositionColor A vertex comprised of

position

color.

VertexPositionColorNormalTe

xture

A vertex comprised of

position

color

normal

texture coordinates

VertexPositionColorTexture A vertex comprised of

position

color

texture coordinates

VertexPositionNormalTexture A vertex comprised of

position

normal

texture coordinates

VertexPositionTexture A vertex comprised of

position

texture coordinates


Vertices

• In the following examples we will cover:

• Example 1: Vertex Buffer Objects: Creation of Vertex Data with Positions and

Colors.

• Example 2: Indexed Drawing: Creation of Vertex Data that supports Indexed

Drawing.

• Example 3: Vertex Array Objects.


EXAMPLE 1.0 – The Vertex Helper Classes

• Open the Example1_Vertex folder within DemoApps\GLES3\ folder, Open the file

Example1_Vertex.cpp file.

• Ignore everything for the time being, focus on the Step 1:

• //Step 1 Declare the vertex positions

• You will notice that we declare a VertexPosition Array consisting of 3, each

element corresponds to a vertex.• VertexPosition(-0.5f, 0.5f, 0.0f),

• VertexPosition(-1.0f, -0.5f, 0.0f),

• VertexPosition(-0.0f, -0.5f, 0.0f),

• Each vertex requires of a position to be defined, X, Y and Z.



• Now look on Step 2, Here is where we populate the Vertex Buffer Object with the

data from our vertex.• //Step 2 Populate the Vertex Buffer

• m_vertexBuffer0.Reset(verticesSimple, sizeof(verticesSimple) / sizeof(VertexPosition), GL_STATIC_DRAW);

• This line of code, copies the data in verticesSimple to a Vertex Buffer, (QUESTION

1: WHAT IS THE ADVANTAGE OF USING VERTEX BUFFER OBJECTS?)

• The same code on OpenGL ES 3.0 would look like:• glGenBuffers(1, &sampleVertBuffer);

• glBindBuffer(GL_ARRAY_BUFFER, sampleVertBuffer);

• glBufferData(GL_ARRAY_BUFFER, sizeof(float) * 3 * numberOfVertices, vertexData, GL_STATIC_DRAW);



• Now Jump to the Step 3, this is the Draw Method that will render something to screen.

• //Step 3 Bind the Vertex Buffer

• glBindBuffer(m_vertexBuffer0.GetTarget(), m_vertexBuffer0.GetHandle());

• //Step 4 Enable the Vertex Attribute Array.

• m_vertexBuffer0.EnableAttribArrays();

• //Step 5 Draw the triangle

• glDrawArrays(GL_TRIANGLES, 0, 3);

• //Step 6 Disable the attribs arrays that we enabled above (using the corresponding disable method)

• m_vertexBuffer0.DisableAttribArrays();

• //Step 7 Unbind the buffer

• glBindBuffer(m_vertexBuffer0.GetTarget(), 0);

• This would look pretty much the same using only the OpenGL ES 3.0 API, the only difference would be the Step 4, which would look like:

• glEnableVertexAttribArray(vertexLoc);

• glVertexAttribPointer(vertexLoc, 3, GL_FLOAT, 0, 0, 0);



• Press F5, this will compile and launch the code. You will see a window like the one

below:



• The simple triangle on the previous slide is the result of the GPU rendering pipeline

after providing 3 vertex positions (and some shader code) to it.

• Now on Step 1 uncomment the remaining 3 VertexPosition lines and substitute 3

by 6 in the verticesSimple declaration.

• On Step 5 substitute also the 3 by a value of 6, this will feed the pipeline 6 vertices

getting the following result.


Example 1.1 – Adding Color to the Vertex Data

• Modify the conditional compile statement located above Step 1.1:• #if(0)

• //Step 1.1, declare the vertices data

• VertexPositionColor verticesColor[3] =

• {

• Change the #if(0) by #if(1)

• Modify the Step 5 line replacing the 6 by a 3, compile and render.


Example 1.1 – Adding Color to the Vertex Data

• The Triangle will now have some colors in it. That is because we no longer provide

only position data to our vertices, we are also providing color information to it by

using the VertexPositionColor class.• VertexPositionColor verticesColor[3] =

• {

• VertexPositionColor(Vector3(-0.5f, 0.5f, 0.0f), Vector4(1.0f, 0.0f, 0.0f, 1.0f)), // v0, c0

• VertexPositionColor(Vector3(-1.0f, -0.5f, 0.0f), Vector4(0.0f, 1.0f, 0.0f, 1.0f)), // v1, c1

• VertexPositionColor(Vector3(0.0f, -0.5f, 0.0f), Vector4(0.0f, 0.0f, 1.0f, 1.0f)), // v2, c2

• Each vertex will have a position and a respective color, colors consist of 4 floating

point values. The first one is RED, second is BLUE, Third is GREEN and Fourth is

ALPHA.

• The Vertex Buffer reset function detects the type of input that you provide and will

create an optimized buffer that OpenGL ES 3.0 will use to render.


Example 2.0 – Indexed Drawing

• Indexed drawing is an efficient way to render 3D objects with multiple vertices without the need to repeat information on those vertices. Let’s use the a simple example for reference:

• Let’s assume that we plan to render a cube, consisting of 6 square faces.

• Given that OpenGL ES 3.0 can only render triangles, rendering a cube would involve rendering 12 triangles. Each triangle consists of 3 vertices, that would be 36 vertices, and each vertex requires 3 positions to be defined, so in order to render a cube we would be feeding the pipeline 108 floating point values!

• Using an indexed technique, we would only need to define 8 vertices, and then a list of indexes that the pipeline would use to assemble those triangles, the list can be made of smaller data types and also, having less vertex data would allow for a better vertex cache use.



• To render the following square without using Indexed Drawing we would require:

• 6 – vertices.

• 18 – Floating Point Values

• Using Indexed Drawing we would require:

• 4 – Vertices.

• 12 – Floating Point Values

• 6 – Short Type Values



• The code is quite similar to Example1. On Step 1 we will define 4 Vertices to draw

2 triangles:• // Step1 Define 4 vertices with (x,y,z)

• VertexPosition vertices[4] =

• {

• VertexPosition(Vector3(-1.0f, -1.0f, 0.0f)), // v0

• VertexPosition(Vector3(1.0f, -1.0f, 0.0f)), // v1

• VertexPosition(Vector3(1.0f, 1.0f, 0.0f)), // v2

• VertexPosition(Vector3(-1.0f, 1.0f, 0.0f)), // v3

• };

• On Step 2 we will define the indexes that we need to use for drawing the triangles:• // Step2 Define the Indices for the 2 triangles that we will draw

• GLushort indices[6] = { 1, 0, 3, 1, 3, 2 };



• On Step 3 we will prepare the index buffer and vertex buffer, a special kind of buffers that allows us to store index and vertex data in GPU memory.

• // Step 3 Prepare the vertex and index buffer

• m_vertexBuffer.Reset(vertices, sizeof(vertices) / sizeof(VertexPosition), GL_STATIC_DRAW);

• m_indexBuffer.Reset(indices, sizeof(indices) / sizeof(GLushort), GL_STATIC_DRAW);

• On the Draw Method, you first need to bind both buffers from step 3• // Step 4 Bind the Buffers

• glBindBuffer(m_vertexBuffer.GetTarget(), m_vertexBuffer.GetHandle());

• glBindBuffer(m_indexBuffer.GetTarget(), m_indexBuffer.GetHandle());

• Then enable the Vertex Attribute Array• // Step 5 Enable the Vertex Attribute Arrays

• m_vertexBuffer.EnableAttribArrays();

• And finally draw• // Step 6 Draw the 2 triangles!!!

• glDrawElements(GL_TRIANGLES, 6, m_indexBuffer.GetType(), nullptr);



• Using only OpenGL ES 3.0 Step 3 would be coded like this:

• // Create a Buffer for Indexes

• glGenBuffers(1, &sampleIndexBuffer);

• glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, sampleIndexBuffer);

• glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(unsigned int) * 6, indexData, GL_STATIC_DRAW);

• Now as a simple exercise, change a common vertex between these 2 used

triangles.

• I changed Vertex 1, compile and render, you will see that BOTH triangles will be

affected, and this is a huge advantage. Imagine that you want to several faces

using non indexed draw. You would need to modify that vertex on EVERY instance

it appears.


Example 3.0 – Vertex Array Objects

• Every time you want to use a Vertex Buffer Object to draw, you would need to set

some OpenGL ES states using the following functions:

• glBindBuffer

• glVertexAttribPointer

• glEnableVertexAttribArray

• These states will be used to define which buffer will be used and which entrance to

the Vertex Shader will be related to that buffer data. Using Vertex Array Objects,

you can save these configurations and use only a bind call to that state to save

code and CPU overhead.

• The Demo Framework provides support for these kind of buffers also.



• Open the DemoApps\GLES3\Example3_VAOs\Example3_VAOs.cpp file.

• The Vertex data definition from Steps 1, 2 and 3 are similar to the previous 2

Examples.

• On Step 4, we will create and bind a Vertex Array Object

• All vertex state changes after this step and before Step 6 will be recorded.• //Step 5 Set up VBO Vertex Attribute information

• GL_CHECK(glBindBuffer(m_vertexBuffer.GetTarget(), m_vertexBuffer.GetHandle()));

• GL_CHECK(glBindBuffer(m_indexBuffer.GetTarget(), m_indexBuffer.GetHandle()));

• m_vertexBuffer.EnableAttribArrays();

• So every time the Vertex Array Object is bound again, they will be restored.



• In the Draw Method, before drawing you only need to bind the Vertex Array Object.• //Step 7 Bind the TRIANGLE VAO

• m_vertexArray.Bind();

• And then Draw.• //Step 8 Draw with the VAO settings

• glDrawElements(GL_TRIANGLES, 3, m_indexBuffer.GetType(), nullptr);

• In order to exemplify how the buffers are swapped. Comment the 3 lines below Step 5 and uncomment the 3 lines below Step 5.1 like this:

• //Step 5 Set up VBO Vertex Attribute information

• //GL_CHECK(glBindBuffer(m_vertexBuffer.GetTarget(), m_vertexBuffer.GetHandle()));

• //GL_CHECK(glBindBuffer(m_indexBuffer.GetTarget(), m_indexBuffer.GetHandle()));

• //m_vertexBuffer.EnableAttribArrays();

•

• //Step 5.1 Set up VBO Vertex Attribute information for other buffers

• GL_CHECK(glBindBuffer(m_vertexBuffer2.GetTarget(), m_vertexBuffer2.GetHandle()));

• GL_CHECK(glBindBuffer(m_indexBuffer2.GetTarget(), m_indexBuffer2.GetHandle()));

• m_vertexBuffer2.EnableAttribArrays();



• This will cause the Vertex Array Object to store within its context other vertex states bound to a different set of Vertex and Index Buffers.

• Go to the Draw Method and change the Step 8 from a value of 3 to 6. Compile and Render.

• As you can see, the rendered element is going to change from a Triangle to a Square.

• Now, to exemplify its use, uncomment the code below the FINAL STEP comment on both the Constructor and Draw Methods

• With this you will have 2 Vertex Array Objects, each one of them bound to a different Vertex Buffer Object, In order to render each object you will only need to:

• Bind a VAO.

• Draw.


SHADERS: SHADER

AND PROGRAM

CLASSES


Shader Class – Shader Intro

• A shader is a program designed to run in the programmable cores of a GPU, in the Vertex

Processing and Fragment Processing stages, and, usually, implements algorithms related

to the lighting and shading effects of an image.

• Vertex Processor:

• Its a programmable unit that operates on the incoming Vertices and their associated data.

Programs written in GLSL that run in this processor are called VERTEX SHADERS.

•

• Fragment Processor:

• Its a programmable unit that operates on covered pixel values (or fragments) and their

associated data. Programs written in GLSL that run in this processor are

called FRAGMENT SHADERS.



• Your PC and your i.MX6 or i.MX8 board are examples of Heterogeneous

Processing System, that is, a system that has a CPU and one or more GPUs

coexisting.

• If you have some previous experience developing Software, you will know that,

ultimately, the code you programmed runs in the CPU of your device/pc.

• Shaders are simply that, software that will be ultimately executed on the GPU of

your device. All HW accelerated OpenGL applications involve CPU and GPU

interaction, as the main application runs in CPU and takes care of providing the

GPU all the data it needs to generate an output via the OpenGL ES API.



• In order to Create a usable program object you need to:

1. Create the Vertex Shader Object.

2. Assign the source code to the vertex shader object

3. Compile the Vertex Shader.

4. Create the Fragment Shader Object

5. Assign the Source code to the fragment shader.

6. Compile the Fragment Shader.

7. Create a Shader Program Object.

8. Attach Shader Objects to the Program Object.

9. Link Shader objects into the Program Object.

10. Validate the Program



• Those are the steps to generate a Shader Program object WITHOUT checking for any errors. If you want to check for errors your code would end up looking like this:

• // Compile the shaders

• GLuint hVertexShader = glCreateShader(GL_VERTEX_SHADER);

• glShaderSource(hVertexShader, 1, &g_strVertexShader, NULL);

• glCompileShader(hVertexShader);

• // Check for compile success

• GLint nCompileResult = 0;

• glGetShaderiv(hVertexShader, GL_COMPILE_STATUS, &nCompileResult);

• if (0 == nCompileResult)

• {

• char strLog[1024];

• GLint nLength;

• glGetShaderInfoLog(hVertexShader, 1024, &nLength, strLog);

• return FALSE;

• }

• GLuint hFragmentShader = glCreateShader(GL_FRAGMENT_SHADER);

• glShaderSource(hFragmentShader, 1, &g_strFragmentShader, NULL);

• glCompileShader(hFragmentShader);

• // Check for compile success

• glGetShaderiv(hFragmentShader, GL_COMPILE_STATUS, &nCompileResult);

• if (0 == nCompileResult)

• {

• char strLog[1024];

• GLint nLength;

• glGetShaderInfoLog(hFragmentShader, 1024, &nLength, strLog);

• return FALSE;

• }

• // Attach the individual shaders to the common shader program


Shader Class – Shader Class on DemoFW

• The Demo Framework offers 2 special classes to help you with this task:

• GLShader class provides a way to compile shaders while checking for errors, if an error is encountered, the program will stop execution and state the exact problem on the shader that it failed.

• You have a constructor, that based on the parameters you provide, can either create an empty shader with an invalid handle or compile a specific type of shader:

• const GLShader vertexShader(GL_VERTEX_SHADER, strVertexShader);

• const GLShader fragmentShader(GL_FRAGMENT_SHADER, strFragmentShader);

• A destructor that will eliminate the shaders.

• And a GetHandle option to get the OpenGL ES Handle for that shader object.



• Open the DemoApps\GLES3\Example4_Shaders\Example4_Shaders.cpp and

.hpp file. In the hpp file you will see the declaration for each one of the shaders.

• GLES3::GLShader m_vertexShader;

• GLES3::GLShader m_fragmentShader;

• In the cpp file on the main demoapp constructor, you will find the 2 shader compile

statements.• m_vertexShader.Reset(GL_VERTEX_SHADER, content->ReadAllText("Shader.vert"));

• m_fragmentShader.Reset(GL_FRAGMENT_SHADER, content->ReadAllText("Shader.frag"));

• These 2 steps will execute all the required steps to compile, while error checking your 2 shaders.



• In order to check the error checking features, modify the shader, Open the

Shader.vert file under the Content folder, on line 9 modify the variable name

a_position, change it by a_positionFTF2016. execute the program by steps. You

will notice that in the Console you will get the exact location of where the shader

failed.


Shader Class – Program Class on DemoFW

• The GLProgram class provides a way to attach and link the shaders while error checking the complete process, if an error is encountered, the demo fw will issue an error and stop while stating where the error happened.

• You have plenty of constructors to choose, you can even register a callback that will be executed when the process ends.

• My preferred constructor is:

• m_program.Reset(VertexSource, FragmentSource, g_pszShaderAttributeArray);

• This will get the source for a Vertex and Fragment shader, as well as the vertex attributes and generate a program object binding those attributes to specific attribute location defined by the user.


Shader Class – Program Class on DemoFW

• Using the same file that you already opened for the Shader Class, comment the 2

shader compilation lines and then uncomment the line below the FINAL STEP

comment.• //FINAL STEP

• m_program.Reset(content->ReadAllText("Shader.vert"), content->ReadAllText("Shader.frag"), g_pszShaderAttributeArray);

• This will compile the shaders and link them in a program object, as well as to bind

the shader attributes into their specific location. And avoiding you calls to

glGetAttribLoc.

• Compile and execute the code, you will get the same error as before. Return the

shader to its original value and you will now have a compiled program.


CONTENT LOADER


Content Loader

• The framework supports loading files from the Content folder.

• Given a content folder like this:

• Content/Texture1.bmp

• Content/Stuff/Readme.txt

• You can load the files via the IContentManager service like this:

• Bitmap bitmap;

• contentManager->Read(bitmap, "Texture1.bmp", PixelFormat::RGB888);

• const std::string content = contentManager->ReadAllText("Stuff/Readme.txt");


Content Loader

• If you prefer to control the loading yourself you can retrieve the path to the

• files like this:

• IO::Path contentPath = contentManager->GetContentPath();

• IO::Path texture1Path = IO::Path::Combine(contentPath, "Texture1.bmp");

• IO::Path readmePath = IO::Path::Combine(contentPath, "Stuff/Readme.txt");


TEXTURES: LOADING

AND HANDLING

THEM


Texture Class – Textures Intro

• A huge topic in 3D APIs are Textures, textures allow the Graphics Programmer to add further detail to the geometry without increasing its complexity, that detail can modify the fragment color, but also other can embed more data as a normal map, bump map, etc.

• On an OpenGL ES Application the regular flow to create a Texture is:

1.- Create the Texture.

2.- Bind the Texture.

3.- Generate the texture and assign its content.

4.- Set the texture parameters like the filtering.



• Your end code would end up looking like this:

• /* Create The Texture */

• glGenTextures(1, texture);

• /* Typical Texture Generation Using Data From The Bitmap */

• glBindTexture(GL_TEXTURE_2D, *texture);

• /* Generate The Texture */

• glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB565, image1->sizeX, image1->sizeY, 0, GL_RGB, GL_UNSIGNED_BYTE, image1->data);

• /* Linear Filtering */

• glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);

• glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);



• Using the Demo Framework, your end code would look like this:

• GLTextureParameters params(GL_LINEAR, GL_LINEAR, GL_REPEAT, GL_REPEAT);

• m_texture.SetData(bitmap, params);

• The bitmap variable is a pointer to the pixel data of an image, The Demo Framework also provides tools to import a huge variety of texture formats, this will be covered in a later chapter of this presentation.

• Params is a class that stores both the filtering parameters as well as the wrap parameters on a texture, the Texture class will use them to specify how the texture is created.

• By the end of the SetData function, your texture will be populated and ready to use, you can get the handle of the texture using the method:

• GetHandle().

•

• You can also use the constructor to create and populate your target, using:

• GLTexture m_texture(bitmap, parameters);


Texture Class – Texture Class

• Open the sample under DemoApps\GLES3\S06_Texturing\S06_Texturing.cpp

• In the Demo App Constructor within its own scope you will see the following code:• { // Load the texture (we use a scope here, so the bitmap objects is thrown away as soon as we dont need it)

• Bitmap bitmap;

• content->Read(bitmap, "Texturing.png", PixelFormat::RGB888);

• //content->Read(bitmap, "Badger.jpg", PixelFormat::RGB888);

• //content->Read(bitmap, "PolarBear.bmp", PixelFormat::RGB888);

• GLTextureParameters params(GL_LINEAR, GL_LINEAR, GL_REPEAT, GL_REPEAT);

• m_texture.SetData(bitmap, params);

• }

• The Read function uses the Demo Framework Content Loader to load the Texture

image (Explained at the end of this section).

• .SetData will copy the pixel data into an OpenGL ES 3.0 usable structure.


Texture Class – Texture Class

• Once you have set up your geometry and textures, the only additional step to

render is to bind the texture to the current rendering pipeline.• // Select Our Texture

• glActiveTexture(GL_TEXTURE0);

• glBindTexture(GL_TEXTURE_2D, m_texture.GetHandle());

• You will, of course, need to use a Sampler2D within the fragment shader and pass

the UV coordinates to the shader. You can do so with the VertexPositionTexture

helper, pretty similarly as what we covered on Examples 1 and 2


Texture Class – Retrieving the Texture Data - DEVIL

• The Demo Framework uses DevIL as its default image service, it also provides a fall back

loader supporting BMP, in case you have a different Image Loader service it can be easily

integrated into the Demo Framework.

• Developer's Image Library (DevIL) is a programmer's library to develop applications with

very powerful image loading capabilities, yet is easy for a developer to learn and use.

Ultimate control of images is left to the developer, so unnecessary conversions, etc. are

not performed. DevIL utilizes a simple, yet powerful, syntax. DevIL can load, save, convert,

manipulate, filter and display a wide variety of image formats.

Currently, DevIL can load .bmp, .cut, .dds, .doom, .exr, .hdr, .gif, .ico, .jp2, .jpg,.lbm, .mdl,

.mng, .pal, .pbm, .pcd, .pcx, .pgm, .pic, .png, .ppm, .psd, .psp, .raw, .sgi, .tga and .tif files.


Texture Class – Retrieving the Texture Data - DEVIL

• The read function that we saw in previous slides in this chapter:• content->Read(bitmap, "Texturing.png", PixelFormat::RGB888);

• //content->Read(bitmap, "Badger.jpg", PixelFormat::RGB888);

• //content->Read(bitmap, "PolarBear.bmp", PixelFormat::RGB888);

• Uses the Image Service (DevIL) to populate the bitmap and extract the size, bpp

and other information. It will also use it to convert the pixel format to the desired

one you want to use!

• Go ahead and try to load any of the 3 different picture formats!!


Texture Class – Cube Map Textures

• A cubemap texture is generally used for environmental mapping to achieve

reflections and refractions in your application.

• The concept is simple, you surround your mesh with a cube, each face of the cube

will have a texture mapped to it. Based on the surface normal and the camera

vector, you will calculate the pixel from that box for a specific fragment, so the end

value of the fragment will have a mix of the surface color and the pixel within that

box were the reflection or refraction vector intersects with the box.



• The usual code to do this would require:

• 1.- Generate a texture

• 2.- Bind it as a cubemap texture.

• 3.- Pre allocate its storage.

• 4.- Set its filtering and wrapping options.

• 5.- Populate each one of its 6 targets or sides of the cube (POSX, NEGX, POSY,

NEGY, POSZ, NEGZ) with image Data.



• glGenTextures(1, &cubeHandle);

• glBindTexture(GL_TEXTURE_CUBE_MAP, cubeHandle);

• glTexStorage2D(GL_TEXTURE_CUBE_MAP, 1, GL_RGBA8, 400, 400);

• glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

• glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_MIN_FILTER, GL_LINEAR);

• glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);

• glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

• glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_WRAP_R, GL_CLAMP_TO_EDGE);

• for (int i = 0; i<6; i++)

• {

• success = LoadImageCode(i);

• if (success)

• {

• glTexSubImage2D(cubeTargets[i], 0, 0, 0, IMAGE_WIDTH,

• IMAGE_HEIGHT, GL_RGBA, GL_UNSIGNED_BYTE,

• ImageData);

• }

• }



• The Demo Framework provides an easy way not only to create a Cubemap texture but also provides a way to do so based on a SINGLE dds image from which the Demo Framework will extract each one of the sides.

• std::vector<uint8_t>rawData;

• contentManager->ReadAllBytes(rawData, filename);

• RawCubeBitmap rawCubeBitmap = BasicDDSLoader::Extract(rawData.data(), int32_t(rawData.size()), BitmapOrigin::LowerLeft);

• GLTextureParameters params(GL_LINEAR, GL_LINEAR, GL_CLAMP_TO_EDGE, GL_CLAMP_TO_EDGE);

• rTexture.SetData(rawCubeBitmap, params, TextureFlags::GenerateMipMaps);

• After this code, your texture object will be ready to be used in your rendering pipeline.



• The Demo Framework provides an easy way not only to create a Cubemap texture but also provides a way to do so based on a SINGLE dds image from which the Demo Framework will extract each one of the sides.

• std::vector<uint8_t>rawData;

• contentManager->ReadAllBytes(rawData, filename);

• RawCubeBitmap rawCubeBitmap = BasicDDSLoader::Extract(rawData.data(), int32_t(rawData.size()), BitmapOrigin::LowerLeft);


• rTexture.SetData(rawCubeBitmap, params, TextureFlags::GenerateMipMaps);

• After this code, your texture object will be ready to be used in your rendering pipeline.



• Open the file at

DemoApps\GLES3\S07_EnvironmentMapping\S07_EnvironmentMapping.cpp

• Look in the constructor and you will find the following code:• //DDS TEXTURE LOADING!

• LoadCubeDDS(m_cubeTexture, content, "stpeters_cross_mipmap_256.dds");

• //LoadCubeDDS(m_cubeTexture, content, "NightSky1.dds");

• //LoadCubeDDS(m_cubeTexture, content, "FullskiesOvercast0004_2_M.dds");

• This code will execute the DDS texture loading into a GLES3 Cubemap texture,

using the code from the previous slide. Compile and Run the code. Then comment

the first line and uncomment the second and see how the texture changes.



• If you don’t like DDS textures, the SetData method within the Texture class supports Cube Map Textures out of the box! You only need to pass 6 Bitmap Data Structures to it and it will create you automatically the cube map Texture!

• Notice how most of the code is only to extract the data from the images and only the last line of code does all the magig of setting your texture.

• contentManager->Read(posX, IO::Path::Combine(cubeDirName, "PosX.jpg"), PixelFormat::RGB888);

• contentManager->Read(negX, IO::Path::Combine(cubeDirName, "NegX.jpg"), PixelFormat::RGB888);

• contentManager->Read(negY, IO::Path::Combine(cubeDirName, "PosY.jpg"), PixelFormat::RGB888);

• contentManager->Read(posY, IO::Path::Combine(cubeDirName, "NegY.jpg"), PixelFormat::RGB888);

• contentManager->Read(posZ, IO::Path::Combine(cubeDirName, "PosZ.jpg"), PixelFormat::RGB888);

• contentManager->Read(negZ, IO::Path::Combine(cubeDirName, "NegZ.jpg"), PixelFormat::RGB888);


• rTexture.SetData(posX, negX, posY, negY, posZ, negZ, params, TextureFlags::GenerateMipMaps);



• Comment all the LoadCubeDDS lines and uncomment the line below FINAL STEP• //FINAL STEP

• CreateCubemapTexture(m_cubeTexture, content, strPath);


3D MODELS

CONTENT LOADER


3D Meshes

• Vertex data are a fundamental part of any 3D application as they define the shape

and structure of what is going to be shown on the screen.

• From previous examples you can see that even defining a cube using Data Arrays

might prove to be a daunting task, now imagine how challenging and time spending

would be to model a 3d Shape of thousands of faces.

• This is why this tasks are done by a 3D modelling software. And it is up to the

Graphics application to know how to process the results from those software suites

and use them.


3D Meshes – Importing with Assimp

• Open Asset Import Library (short name: Assimp) is a portable Open Source library

to import various well-known 3D model formats in a uniform manner.

• Assimp aims to provide a full asset conversion pipeline for use in game engines /

realtime rendering systems of any kind, but it is not limited to this purpose. In the

past, it has been used in a wide range of applications.

• The Demo Framework includes support for Assimp through the SceneImporter

class.

• Its usage boils down to using the Load method of the scene importer class to load

the 3D mesh. • SceneImporter sceneImporter;

• const std::shared_ptr<TestScene> scene = sceneImporter.Load<TestScene>(modelPath, DEFAULT_MODEL_SCALE, true);



• Afterwards you get the root node of the mesh (The hierarchically higher mesh) with

the instruction:• //Step 2 Get the most important mesh hierarchically

• m_rootNode = scene->GetRootNode();

• if (! m_rootNode)

• throw NotSupportedException("Scene did not contain a root node");

• Then for each mesh, you will get the Index and Vertex buffers, which contain all the

needed geometry information needed to render a mesh.• m_indexBuffers.Reset(i, mesh->GetIndexArray(), GL_STATIC_DRAW);

• m_vertexBuffers.Reset(i, mesh->GetVertexArray(), GL_STATIC_DRAW);



• In order to draw the Mesh, you only need to bind those Vertex and Index buffers to the shader attributes via:

• // Bind and enable the vertex buffer

• glBindBuffer(m_vertexBuffers.GetTarget(), vertexBuffer.GetHandle());

• glBindBuffer(m_indexBuffers.GetTarget(), indexBuffer.GetHandle());

• // Since all our meshes use the same attrib pointers we dont have to enable/disable them all the time

• m_vertexBuffers.SetVertexAttribPointers(m_attribLink.data(), m_attribLink.size());

• glDrawElements(GL_TRIANGLES, indexBuffer.GetCapacity(), indexBufferType, (void*)0);



• Open the DemoApps\GLES3\ModelLoaderBasics\ModelLoaderBasics.cpp file

• Search for the Step 1 Comment and comment the first line below the comment,

and uncomment the following code, the end code should look like this:• //Step 1 Load the .obj file

• //auto modelPath = IO::Path::Combine(contentPath, "FinalGuardian/test.obj");

• auto modelPath = IO::Path::Combine(contentPath, "BlueFalcon/BlueFalcon.obj");

• Then Search for Step 4 and comment the first line below it and uncomment the

following line, the end code should look like this:• //Step 4 Load the Texture

• //contentManger->Read(bitmap, "FinalGuardian/test.jpg", PixelFormat::RGB888);

• contentManger->Read(bitmap, "BlueFalcon/BlueFalcon.JPG", PixelFormat::RGB888);

• Compile and run.


REAL TIME CONTENT

UPDATE FOR

ARTISTS


Real Time Content Update

• The Demo Framework offers the capability to update the asset files that your demo

uses on real time.

• No matter if it is a new 3D mesh, a new texture or even a new Shader.


FONTS


Fonts on the Demo Framework

• One common way to handle written messages is by creating bitmap objects of the

whole message, this, however, would demand the user to create a new bitmap if

the message is to be changed.

• Fonts provide the graphics programmer the means to show dynamic written

messages into the display. Dynamic, on this case means that they can be changed

on runtime without needing to generate new assets.



• The DemoFramework provides the chance to the user to print messages pretty easily. The following example will cover how to add font support to your newly created Demo Application.

• Open the .hpp file of the new demo and add the following headers:

• #include <FslSimpleUIApp/UIDemoAppExtension.hpp>

• #include <FslSimpleUI/Control/Label.hpp>

• If you are using Visual Studio, you will also need to add 2 projects to your solution, FslSimpleUI.sln and FslSImpleUIApp.sln as well as 2 include directories to your Project:• $(FSL_GRAPHICS_SDK)\DemoFramework\FslSimpleUIApp\include

• $(FSL_GRAPHICS_SDK)\DemoFramework\FslSimpleUI\include

• Under VS you might also want to update the References on the Common Properties of your project.



• Add an UI event listener to your Demo Application class constructor declaration.• namespace Fsl

• {

• class Example5_Fonts : public DemoAppGLES3

• , public UI::EventListener

• Then, within the Class, declare the following members:• // The UI event listener is responsible for forwarding events to this classes implementation of the

UI::EventListener (while its still alive).

• UI::CallbackEventListenerScope m_uiEventListener;

• // The UIDemoAppExtension is a simple extension that sets up the basic UI framework and listens for the events it needs.

• std::shared_ptr<UIDemoAppExtension> m_uiExtension;

• std::shared_ptr<UI::Label> m_label;

• With them you will be able to listen and send events to the UI framework.



• Now open the .cpp file, in the main constructor add the lines below the //ADD

THESE LINES COMMENT:• Example5_Fonts::Example5_Fonts(const DemoAppConfig& config)

• : DemoAppGLES3(config)

• , m_program()

• //ADD THESE LINES FOR FONT SUPPORT

• , m_uiEventListener(this) // The UI listener forwards call to 'this' object

• , m_uiExtension(std::make_shared<UIDemoAppExtension>(config, m_uiEventListener.GetListener(), "MainAtlas")) // Prepare the extension

• “MainAtlas” is the name of the main texture atlas that your application will use to

fetch the fonts (And all the picture information you will use on the UI)



• Now open the .cpp file, in the main constructor add the lines below the //ADD THESE LINES COMMENT:

• Example5_Fonts::Example5_Fonts(const DemoAppConfig& config)

• : DemoAppGLES3(config)

• , m_program()

• //Step 1 ADD THESE LINES FOR FONT SUPPORT

• , m_uiEventListener(this) // The UI listener forwards call to 'this' object

• , m_uiExtension(std::make_shared<UIDemoAppExtension>(config, m_uiEventListener.GetListener(), "MainAtlas")) // Prepare the extension

• “MainAtlas” is the name of the main texture atlas that your application will use to fetch the fonts (And all the picture information you will use on the UI).

• Open the Content folder: you will find 3 sets of files: a bta, fbk and png file. These 3 files represent a Texture Atlas. The bta file stores exact pixel location for each one of the characters, fbk contains kerning information for those characters and the png file stores the actual pixel information of the characters.



• Next, register the extension to catch UI events and create the UI using the

following lines:• // Step 2 Give the UI a chance to intercept the various DemoApp events.

• RegisterExtension(m_uiExtension);

• // Step 3 Next up we prepare the actual UI

• auto context = m_uiExtension->GetContext();

• You will also need to use the UI namespace, so you can create context, FillLayouts

and labels.• //Step 4 Add the UI namespace so you can use the Item Alignment, FillLayout and Label functions

• using namespace UI;



• Once you have the context, you will create a label (the object that will have the string you need to display)

• // Step 5 Create a label

• m_label = std::make_shared<Label>(context);

• m_label->SetAlignmentX(ItemAlignment::Center);

• m_label->SetAlignmentY(ItemAlignment::Center);

• // Step 6 Create a 'root' layout we use the recommended fill layout as it will utilize all available space on the screen

• // We then add the label to it

• auto fillLayout = std::make_shared<FillLayout>(context);

• fillLayout->Children.Add(m_label);

• // Step 7 Finally add everything to the window manager (to ensure its seen)

• context->WindowManager->Add(fillLayout);

• // Step 8 Set Content to your Label

• m_label->SetContent("FTF2016 FONTS!!!");



• Finally on the Draw Method, you can call the UI extension method to Draw, and it

will render the Label:• // Calling this last allows the UI to draw on top of everything.

• // Beware that the UI drawing methods might alter the OpenGL state!

• m_uiExtension->Draw();

• Try changing the Content or the Alignment to experiment!


MATH AND MATRIX

HELPERS


Math and Matrix Helpers

• The Demo Framework includes math helpers useful for working with graphics. It

focuses on ease of use, rather than raw performance.

• The most used helper within these classes are the Matrix ones:

• Matrices are hugely used to transform vertices to create animations, perspective,

lookats and complex lightning schemes.

• You can find the supported matrix functions on the file:

\DemoFramework\FslBase\source\FslBase\Math\Matrix.cpp


Math and Matrix Helpers – Look At

• CreateLookAt: Creates a Matrix that can be used to transform the vertices from world

space to camera space.

• Matrix Matrix::CreateLookAt(const Vector3& cameraPosition, const Vector3&

cameraTarget, const Vector3& cameraUpVector)

• By using this function you can effectively create a camera position and look vector for your

3D applications.

• The EnvironmentMapping application we saw when rendering a cube map uses this

function to move the camera around the sphere. It does so using the following function:

• Matrix::CreateLookAt(ViewMatrix, EyeVector, LookAt, UpVec);


Math and Matrix Helpers – Perspective Projection

• CreatePerspectiveFieldOfVIew: This function creates a projection matrix that

projects a 3D plane into a 2D plane while keeping the depth appearance, properly

imitating the way human eye perceives objects.



• Open the DemoApps\S04_Projection.cpp

• The perspective projection matrix is built using the CreatePerspectiveFieldOfView

method within the Matrix helpers.

• m_matProj = Matrix::CreatePerspectiveFieldOfView(MathHelper::ToRadians(57.0f), aspectRatio, 1.0f, 1000.0f);

• After this function m_matProj will have a proper perspective matrix that you can

use to modify your vertices.



• Compile and excute the the sample, you will have a nice screen like the one below.


Math and Matrix Helpers – Ortho Projection

• Orthographic is one of the 2 most widely used projection matrices on graphics, its

main characteristic is that parallel lines remain parallel regardless of the depth

value. I use this kind of projection to render 2 dimensional objects on a scene.

• m_matProj = Matrix::CreateOrthographic(currentSize.X, currentSize.Y, near, far);

• Just for laughs, on the S04_Projection.cpp file uncomment the lines directly below

the //Step 2 Comment, the code comment will tell you exactly which lines to

uncomment.


Math and Matrix Helpers – Ortho Projection

• The most important line to comment is:

• //Step 2 Now use a Orthographic Projecion UNCOMMENT 1 LINE BELOW

• m_matProj = Matrix::CreateOrthographic(res.X, res.Y, 1.0f, 1000.0f);

• This will replace the perspective projection by an Ortho one, in the image below

you can see how the perspective is completely removed.


Math and Matrix Helpers - Translating

• Translation matrices displace a point, vertex or vector by a given amount on a

given axis, this is the most straightforward way to move elements on screen.

• The Matrix Helpers support the creation of these kind of matrices.

• Open the Sample and demo at: DemoApps\GLES3\Example6_Transforms

• The translation Matrix is created using the function:

• m_matTranslate1 = Matrix::CreateTranslation(transX, transY, transZ);

• Uncomment the code below //Step 1 and see what happens


Math and Matrix Helpers - Scaling

• Scale matrices scale an object by a given ratio in a given axis.



• The Scale Matrix is created using the function:

• m_matTranslate1 = Matrix::CreateTranslation(transX, transY, transZ);

• Uncomment the code below //Step 2 and see what happens


Math and Matrix Helpers - Rotate

• A rotation matrix rotates a point or vector around a given axis based on an input

angle passing through the origin.



• The Scale Matrix is created using the function:

• m_matRotate1 = Matrix::CreateRotationZ(m_angle);

• The Demo Framework supports creation of rotation Matrices on every axis, have

some fun and change the axis, also uncomment the code below the //Step 3

comment


HANDLING INPUTS


Handling Inputs

• The Demo Framework, on all its available platforms, provides means for the user to

listen to input device events.

• Currently we support:

• Windows – Keyboard and Pointer Devices.

• Ubuntu – Keyboard and Pointer Devices.

• Linux Frame Buffer Backend – Keyboard.

• Linux X11 Backend – Keyboard and Pointer Devices.

• Linux Wayland Backend – Keyboard and Pointer Devices.


Handling Inputs

• The input events are always broadcasted from the low level window

implementation. It is up to your application to listen and react to those events.

• In order to listen to a Keyboard event you need to add the following function to your

Demo Class:

• void YourClass::OnKeyEvent(const KeyEvent& event)

• {

• //Process the event

• }


Handling Inputs

• The mouse provides 3 types of events: button press, movement and wheel. You

can listen to them adding the following functions in your class:• void YourClass::OnMouseButtonEvent(const MouseButtonEvent& event)

• {

• }

• void YourClass::OnMouseMoveEvent(const MouseMoveEvent& event)

• {

• }

• void YourClass::OnMouseWheelEvent(const MouseWheelEvent& event)

• {

• }


Handling Inputs

• Open the sample DemoApps/GLES2/InputEvents compile and execute, this

sample reads the input events from the devices connected to your application and

will print information related to each event it receives


INPUT PARAMETERS


Input Parameters

• Every application you write using the Demo Framework will support the following list of parameters:

•

• -h, --help = Display options

• --ghelp <arg> = Display option groups: all, demo or host

• -v, --verbose = Enable verbose output

• --ExitAfterFrame <arg> = Exit after the given number of frames has been rendered

• --ScreenshotFrequency <arg> = Create a screenshot at the given frame frequency

• --LogStats = Log basic rendering stats

• --Stats = Display basic frame profiling stats

• --AppFirewall = Enable the app firewall, reporting crashes on-screen instead of exiting

• --ContentMonitor = Monitor the Content directory for changes and restart the app on changes. WARNING: Might not work on all platforms and it might impact app performance (experimental)


Input Parameters

• --ForceUpdateTime <arg> = Force the update time to be the given value in microseconds (can be useful when taking a lot of screen-shots). If 0 this option is disabled

• --Window <arg> = Window mode [left,top,width,height]

• --DisplayId <arg> = DisplayId <number>

• --EGLLogConfig = Output the EGL config to the log

• --EGLRedSize <arg> = Force EGL_RED_SIZE to the given value

• --EGLGreenSize <arg> = Force EGL_GREEN_SIZE to the given value

• --EGLBlueSize <arg> = Force EGL_BLUE_SIZE to the given value

• --EGLAlphaSize <arg> = Force EGL_ALPHA_SIZE to the given value

• --EGLDepthSize <arg> = Force EGL_DEPTH_SIZE to the given value


Input Parameters

• --mmdc = Enable basic MMDC profiling using the most appropriate

profiler for the app type (if mmdcUsage is specified it will determine what we

profile)

• --mmdcUsage <arg> = Which DDR master will be profiled

• --mmdcDuration <arg> = Select the profile duration in milliseconds (can not

be lower than the rendering time of one frame)

• --mmdcLog = Print the mmdc profiling stats to the log


Input Parameters - mmdc

• The mmdc infrastructure is a good tool to profile how much memory is the GPU application consuming , you can also profile the whole system.

• --mmdc will enable the basic profiling for the GPU that is currently running.

• --mmdcUsage will be used to select any of the DDR masters: The default one will be the core that the demo API is exercising directly.

• "DEFAULT", "CUSTOM", "ARM", "DSP1", "DSP2", "GPU2D", "GPU2D1", "GPU2D2", "GPU3D", "GPUVG", "VPU", "M4", "PXP", "USB", "SUM“

• --mmdcDuration controls the amount of time that the profiling windows will last. The recommended duration is 1 second (1000 miliseconds).

•

• --mmdcLog will print the results of the log to the console.



• Now let's take a look at the values that the mmdc profiler will spit out when you execute it.

• ./S07_EnvironmentMapping_FB --mmdc --mmdcUsage DEFAULT --mmdcDuration 1000 –mmdcLog

•

• MMDC GPU3D

•

• MMDC new Profiling results:

• ***********************

• Measure time: 1004ms – Time that the profiler will be enabled measuring the bus transactions

• Total cycles count: 530331432 – Total cycle count that the profiler is enabled.

• Busy cycles count: 522489189 – Number of cycles where the MMC read and write state machines are busy during the profiling period.

• Read accesses count: 17539003 – Number of read accesses that the selected AXI master had on the MMDC

• Write accesses count: 36965953 – Number of write accesses that the selected AXI master had on the MMDC

• Read bytes count: 964144176

• Write bytes count: 1381279872



• Avg. Read burst size: 54 – is the result of dividing the total read bytes / the read accesses.

• Avg. Write burst size: 37 – is the result of dividing the total write bytes / the write accesses.

• Read: 915.82 MB/s / Write: 1312.04 MB/s Total: 2227.86 MB/s

• Utilization: 28% – total number of bytes read and written / number of busy cycles

• Overall Bus Load: 98% – Busy cycles / Total Cycles

• Bytes Access: 43 – total number of bytes read and written / number of accesses.

• If you would like to measure the complete system you can pass SUM as the mmdcUsage

parameter.

• ./S07_EnvironmentMapping_FB --mmdc --mmdcUsage SUM --mmdcDuration 1000 –

mmdcLog


Input Parameters -- ExitAfterFrame

• --ExitAfterFrame will execute the program for a given amount of frames, and then it

will quit. This is useful when you want to include your application in an automated

environment. We use it in our automatic test systems.

•

• The following command would execute the program for 500 frames.

• ./S07_EnvironmentMapping_FB --ExitAfterFrame 500


Input Parameters -- ScreenshotFrequency

• --ScreenshotFrequency will define, in frames, the frequency of screenshots that will

be generated.

• The command below will generate a screenshot every 500 frames.

• ./S07_EnvironmentMapping_FB --ScreenshotFrequency 500

•

• This will generate a file named: test-frame-0000000500.bmp. The picture below

was taken using this functionality

•


Input Parameters -- LogStats

• --LogStats will print some rendering stats in the following format.

• ./S07_EnvironmentMapping_FB --LogStats

• All: 5013 FPS: 199.481 Updates: 13 Draw: 439

•

• All is the time difference between the previous frame and the present frame.

• FPS: is the division of 1 second / the delta time between the past frame and the

present frame.

• Updates is the time that update take.

• Draw the time that the Draw call took.


Input Parameters -- Stats

• --Stats will print on screen the FPS number and the milliseconds that the frame

took to be rendered. (Lower Left Corner of the screen)

• And a nice graph with 3 colors, Green will represent the Total time for the frame.

Cyan the time it took to update. Yellow the time it took to Draw. (Lower Right

Corner of the Screen)


Input Parameters – AppFirewall and Content Monitor

• --AppFirewall will catch any fatal error or crash in the application, and instead of

exiting as it currently does, it will pause it.

•

• --ContentMonitor is the parameter that enables the automatic update of the

Content folder that enables the features that Cesar showed in his demonstration.


Input Parameters – ForceUpdateTime and Window Size

• --ForceUpdateTime will control the value of the update time in microseconds.

• --Window [x, y, width, height] will control the size of the EGL Window created.

•

• ./S07_EnvironmentMapping_FB --Window [0,0,800,400]


Input Parameters – EGLLogConfig

• --EGLLogConfig will print the EGL configuration that was used to render the demo.

• ./S07_EnvironmentMapping_FB --EGLLogConfig

• Requested EGL config

• EGL_RED_SIZE=8

• EGL_GREEN_SIZE=8

• EGL_BLUE_SIZE=8

• EGL_RENDERABLE_TYPE=64

• EGL_DEPTH_SIZE=24

• EGL_SURFACE_TYPE=4

• EGL_SAMPLES=0

• EGL_ALPHA_SIZE=0

• Final vs requested EGL config

• EGL_RENDERABLE_TYPE is 69 requested 64

• EGL_SURFACE_TYPE is 1287 requested 4


RAPID PROTOTYPING

USING NATIVE

BATCH


Native Batch 2D

• NativeBatch2D allows you to quickly display a texture on the screen without doing

a lot of work. NativeBatch comes with full texture atlas and bitmap font support.

• Using this class you will not need to define any kind of geometry nor UV

coordinates to display your textures. You only need to Create and Populate your

texture and then use the NativeBatch draw method to display it.

• It is also important to mention that this functionality is API independent.

• Open the DemoApps\GLES3\DFNativeBatch2D\ Sample and the

DFNativeBatch2D.cpp file


Native Batch 2D

• On the constructor you will see the creation and population of the 2 textures used on this sample, one of the textures uses the Texture class that we already reviewed and the other sets an API independent 2D texture known as Texture2D

• const std::shared_ptr<IContentManager> contentManager = GetContentManager();

• {

• Fsl::Bitmap bitmap;

• contentManager->Read(bitmap, "Test.png", PixelFormat::RGBA8888);

• m_texture2D.Reset(m_graphics->GetNativeGraphics(), bitmap, Texture2DFilterHint::Nearest);

• }

• {

• GLES3::GLTextureParameters params(GL_LINEAR, GL_LINEAR, GL_REPEAT, GL_REPEAT);

• Fsl::Bitmap bitmap;

• contentManager->Read(bitmap, "Test2.png", PixelFormat::RGB888);

• m_nativeTexture.SetData(bitmap, params);

• }


Native Batch 2D

• On the Draw Call we show you how to use the Native batch to draw both kinds of

textures:

• The m_nativeBatch->Draw function supports input using both types of textures,

Native Texture (GLES3, GLES2 and OpenVG) or an API independent texture

named Texture 2D.

• It also supports a way for you to increase or decrease the size of the texture by

providing a Rectangle structure that will define Location and Size!


Native Batch 2D

• // GLES3 native texture handle

• m_nativeBatch->Draw(GLES3::GLTextureInfo(m_nativeTexture.GetHandle(), m_nativeTexture.GetSize()), Rectangle(res.X - 256, offsetY+256, 256, 256), Color::White());

• // GLES3 native texture

• m_nativeBatch->Draw(m_nativeTexture, Vector2(res.X - nativeTexSize.X - 128 * 2, offsetY), Color::White());

• m_nativeBatch->Draw(m_nativeTexture, Rectangle(res.X - nativeTexSize.X - 128*2, offsetY + 256 * 0 + 128, nativeTexSize.X / 2, nativeTexSize.Y / 2), Color::White());

• // API independent texture

• const Point2 texSize = m_texture2D.GetSize();

• m_nativeBatch->Draw(m_texture2D, Vector2(res.X - texSize.X, offsetY), Color::White());

• m_nativeBatch->Draw(m_texture2D, Rectangle(res.X - texSize.X - texSize.X / 2, offsetY + texSize.Y, texSize.X / 2, texSize.Y / 2), Color::White());


UI CAPABILITIES


UI Capabilities

• FslSimpleUI is a backend independent way to render a simple UI inspired by how WPF works (when used from code).

• It supports

• A UI Tree.

• Layout management (caching layout engine).

• UI tree events (bubble, tunnel, direct)

• Input click support (easily extendable).

• Layout controls: Stack, Fill

• Custom controls.

• Works on multiple backends.


UI Capabilities

• The UI capabilities contemplate screen resolution changes, and, as WFP, allows

you to create platform independent UIs that adapt to the current screen resolution.

• Open the Sample DemoApps\GLES3\DFSimpleUI101, compile and execute it, you

will get an output like this one:


UI Capabilities

• Feel free to play with the provided controls. We provide buttons, slides,

checkboxes, image support, labels and fonts.

• Now, execute your application but pass the following parameter: --Window

[0,0,800,800], this will create a 800x800 window and see what happens.

• That’s right! your controls and layout will adapt to the new resolution!!! The assets

and images will keep their form factor, but the location and functionality will be

effectively moved!!!


NXP MEMORY

PROFILING TOOLS


NXP Memory Profiling Tools – gmem_info

• gmem_info is a fast tracer to check the overall memory utilization of the different memory pools in the system.

•

• root@imx6qsabresd:~# gmem_info

• Pid Total Reserved Contiguous Virtual Nonpaged Name

• 669 16,537,028 16,012,740 0 524,288 0 ./Example1_FB

• 660 10,095,260 9,308,828 0 786,432 0 ./S07_EnvironmentMapping_FB

• ------------------------------------------------------------------------------

• 2 26,632,288 25,321,568 0 1,310,720 0 Summary

− - 108,896,160 - - - Available

• GPU Idle time: 0.000000 ms

•

• The overall memory used is reported in the Total field.

• The Reserved field refers to the CMA memory pre-allocated for the GPU. This memory is the fastest to allocate and free and should be your primaty pool.

•

• The Contigous memory is used when the Reserved memory is depleted, it will try to use first CMA to allocate and then jump to System memory.

•

• Virtual memory is where the command buffers live.


NXP Memory Profiling Tools – gpuinfo.sh

• gpuinfo is a script to gather GPU runtime status through debugfs interface. It exports below information:

• GPU hardware information.

• GPU total memory usage.

• GPU memory usage of certain process or all processes (user space only).

• GPU idle percentage.

•

• GPU Info

• gpu : 0

• model : 2000

• revision : 5108

•

• gpu : 1

• model : 320

• revision : 5007

•

• gpu : 2

• model : 355

• revision : 1215



• VIDEO MEMORY:

• gcvPOOL_SYSTEM:

• Free : 124908900 B

• Used : 9308828 B

• Total : 134217728 B

• gcvPOOL_CONTIGUOUS:

• Used : 0 B

• gcvPOOL_VIRTUAL:

• Used : 0 B



• It will provide a complete report of how much memory is being used, and where!

•

• You will see memory used by Vertex Data. Indices, Textures, Render Target

Buffer, Depth Buffer, Bitmap Board, Tile Status, vgImages, vgMasks, vgScissor,

Hierarchical Zdepth buffer.


VTK VPROFILER AND

VANALYZER


NXP Memory Profiling Tools – vProfiler and vAnalyzer

• vProfiler is a run-time environment for collecting performance statistics of an

application and the graphics pipeline.

• vAnalyzer is a utility for graphically displaying the data gathered by vProfiler and

aiding in visual analysis of graphics performance.

• Used together, these tools can assist software developers in optimizing application

performance on Vivante enabled platforms. The GPU includes performance

counters that track a variety of GPU functions.


vProfiler and vAnalyzer


vProfiler and vAnalyzer tools

• vProfiler collects information on GPU usage and on calls to Vivante functions within

the graphics pipeline. As such it provides an excellent view into what is happening

on the GCCORE graphics processor at any point in time, down to the individual

frame.

• When the application performance is GPU-bound, vProfiler and vAnalyzer are the

right tools to help determine why.


Enabling vProfiler on NXP boards.

• On our Yocto based BSPs you already have everything you need to start profiling.

• You only need to add the command parameter on u-boot:

• Setenv galcore.gpuProfiler=1

• Add to mmcargs galcore.gpuProfiler=1

• Boot the board and now set the VIV_PROFILE env variable to 1 (you can set it to 2, but on this training we will only cover the 1 value)

• export VIV_PROFILE=1

• Now run your application, in our case we will use the ./S07_EnvironmentMapping_FB application.

• Leave the application executing, the vProfiler will profile it, now close your application and you will have a file named similarly to this: vprofiler_510_1.vpd


Using the vAnalyzer

• Install the vAnalyzer on your Windows machine, open a console on its installation

path and run the vAnalyzer.exe and open the vpd file you just generated.


Using the vAnalyzer

• You will get tons of per frame information as well as overall system information.

• On the main screen you will get a list of the top 10 slowest frames. For each frame

you will get a list of the GL functions called, the time they took and intimate GPU

detail of primitive data. Triangle Count, Amount of pixels processed and shader

information.

• You will be able to check where your bottleneck is on your application, sometimes

you have too many vertices, too many state changes, you lose too much time

filtering a texture, etc.


VTK VTRACER AND

VPLAYER


vTracer and vPlayer

• The Vivante vTracer package includes two components, vTracer and vPlayer.

vTracer is a tool which can record an application’s EGL 1.4 and OpenGL ES

1.1/2.0/3.0 API traces on the Linux OS or Android platform.

• vTracer supports two trace modes for Linux OS: Intercept Trace Mode and Direct

Trace Mode. Intercept Trace Mode is the mode used with the Android platform.

• The package also includes vPlayer, a tool for playing back the recorded API trace

files.


vTracer and vPlayer


vTracer – Setting Up

• In this training we will cover how to use the vTracer on Intercept mode.

• First locate the vTracer_vPlayer.zip file, this is included in the VTK that you downloaded in the beginning!

• Unzip it and locate the bin\fb\ folder. This will hold the libraries that you need to intercept the API commands and log them.

• Copy the libGLES_vtracer19.so to your target board to the /usr/lib/ folder.

• You need to rename the file to: libGLES_vtracer.so

• Copy the vtracer.conf file to the same directory where the application you plan to profile is located.


vTracer – Configuring the Target

• Now, on the target board, execute the following command:

• $ export LD_PRELOAD=/usr/lib/libGLES_vtracer.so

• Execute your application.

• Your application will generate a special kind of file named:

• tracer_<PROCESS_NAME>_<PID>.bin


vPlayer – Configuring the Target

• Copy the vPlayer binary to the target board.

• Clear the LD_PRELOAD env variable so the player is not traced!

• Execute the player providing the .bin file as an input. ./vPlayer tracer_<PROCESS_NAME>_<PID>.bin

• vPlayer playback can be controlled by hot keys on the keyboard when using Linux and Windows operating systems.

• By default, vPlayer reads key strokes from the console terminal on the host device. Using the option --targetkey

• enables vPlayer to read key strokes from the target device.

• The following Hot Keys are available for control of playback in vPlayer:

• Esc Exit application

• M Pause / Resume the playback

• Q Exit application

• SPACE Render a single frame and then pause


ATTRIBUTION STATEMENT

NXP, the NXP logo, NXP SECURE CONNECTIONS FOR A SMARTER WORLD, CoolFlux, EMBRACE, GREENCHIP, HITAG, I2C BUS, ICODE, JCOP, LIFE VIBES, MIFARE, MIFARE Classic, MIFARE

DESFire, MIFARE Plus, MIFARE FleX, MANTIS, MIFARE ULTRALIGHT, MIFARE4MOBILE, MIGLO, NTAG, ROADLINK, SMARTLX, SMARTMX, STARPLUG, TOPFET, TrenchMOS, UCODE, Freescale,

the Freescale logo, AltiVec, C 5, CodeTEST, CodeWarrior, ColdFire, ColdFire+, C Ware, the Energy Efficient Solutions logo, Kinetis, Layerscape, MagniV, mobileGT, PEG, PowerQUICC, Processor Expert,

QorIQ, QorIQ Qonverge, Ready Play, SafeAssure, the SafeAssure logo, StarCore, Symphony, VortiQa, Vybrid, Airfast, BeeKit, BeeStack, CoreNet, Flexis, MXC, Platform in a Package, QUICC Engine,

SMARTMOS, Tower, TurboLink, and UMEMS are trademarks of NXP B.V. All other product or service names are the property of their respective owners. ARM, AMBA, ARM Powered, Artisan, Cortex,

Jazelle, Keil, SecurCore, Thumb, TrustZone, and μVision are registered trademarks of ARM Limited (or its subsidiaries) in the EU and/or elsewhere. ARM7, ARM9, ARM11, big.LITTLE, CoreLink,

CoreSight, DesignStart, Mali, mbed, NEON, POP, Sensinode, Socrates, ULINK and Versatile are trademarks of ARM Limited (or its subsidiaries) in the EU and/or elsewhere. All rights reserved. Oracle and

Java are registered trademarks of Oracle and/or its affiliates. The Power Architecture and Power.org word marks and the Power and Power.org logos and related marks are trademarks and service marks

licensed by Power.org. © 2015–2016 NXP B.V.

i.mx 6 and i.mx 8 graphics sdk and gpu toolkit

Documents