LCU14 BURLINGAME

Mathieu Poirier, LCU14

LCU14-101: Coresight Overview

● What is CoreSight● Why Use CoreSight?● Coresight Components and HW tracing● Presentation of the Coresight Framework● Minimal steps to enable tracing● Upstreaming Strategy and Current Status● What is Coming Next● Question and Comments

About this Presentation

Coresight in One Image

Source: ARM Ltd.

● A set of IP blocks added to ARM SoCs at synthesis time● IP blocks provide HW assisted system tracing using an out of band

bus (ATB), i.e non-intrusive debug capabilities● Very useful for system profiling and debugging without costly

external tools● CoreSight blocks are enabled/disabled at will● Synchronisation of many CoreSight sources allow for debugging of

very complex scenarios.**Also includes JTAG specification, not covered in this

presentation

What is CoreSight

● Enhanced tracing capabilities over common solutions:● Decoupling of CPU power state and tracing engine● Tuning and configuration of scenarios simply impossible to achieve with a JTAG● Synchronisation of traces from different source → multi-dimensional debugging

● Two different trace format for enhance granularity (STM vs ETM)● Some debugging scenarios can’t afford to stop the target for

debugging when a condition occurs:● Assembly chain in production ● A vehicle in motion

● Some Systems don’t have a JTAG port or the port is not accessible● Server with 256 processors● A probe in an oil well, somewhere in the field

Why Using CoreSight?

● JTAGs require to have a connection with a core, making it difficult to trace power management intensive scenarios.

● Enhance tuning and capabilities:● Address range comparators● Single address comparator● Context ID and Virtual ID tracing● Start and Stop address matching● Security states ex, secure, non-secure or both● External input for global system awareness● Three-state sequencer and counters

● Tunables can be mixed to make very complex cases:● Start tracing only if in “secure mode”, between addresses 0xABC and 0xDEX, when

the ID of the running process is 456.

Enhanced Tracing Capabilities

Two Different Trace Format

CPU

kernel

user space

DSP

TPIU

ETB

STM

● STM: transform SW writes into STPv2● ETM/PTM: represent program flows using PFT

SW writesPFTv1.1STPv2

● Coresight IP blocks are divided in 4 categories:● Source, link, sink and “global” (not official term)

● Source:● Generate trace data (PFTv1.x and STPv2)● ETM (Embedded Trace Macrocell), PTM (Program Trace Macrocell)● STM (System Trace Macrocell)

● Links: ● Replicators - duplicate trace data, usually in 2 streams● Funnel - aggregate trace data, usually 8 to 1

● Global:● CTI - Cross Trigger Interface, for communication between entities

** The above list is not exclusive, there are other components **

Coresight Components

A Typical Coresight System

Source: ARM Ltd

● Trace output from ETM/PTMs are compressed to reduce the amount of space taken in the trace buffer (ETB).

● The format is called Program Flow Trace (PFT)● Only waypoints are recorded:

● A waypoint is a point where instruction execution by the processor might involve a change in the program flow

● Can be an instruction (ex. indirect branches) or an event (ex. all exceptions)● Each waypoint means that instructions between the current

waypoint address and the previous one have been executed.● Using waypoints and the original program image, the program

flow can be reconstructed.● Atoms are used to indicate waypoint instruction execution.

ETM/PTM Trace Output

● Decoding trace output is not to be taken lightly.● Program Linaro has experimented with:

● ptm2human [1]: ● Not bad for an experimental exercise● List received packet, i.e, sync, atoms, timestamps● Does not interpret the content of the packet, so no waypoint decoding● completely free, thumbs up to “hwangcc23”

● ARM DS-5: ● Yields very detailed traces● Program execution can be reconstructed perfectly - see [2] for more information

● Other trace decoding programs:● Texas Instrument’s Code Composer Studio● Lauderbach Trace32

● Trace decoding is currently a big problem. ● Linaro is weighing options on how to proceed in this area.

ETM/PTM Trace Output Decoding

● STM produces a trace stream conformant to the MIPI’s STPv2.● For more information see:

● http://mipi.org/specifications/debug

● Tools to decode STM traces:● ARM’s DS-5● ST-Microelectronic’s STM-Probe● Liewenthal Electronic’s Fido box

STM Trace Decoding

● Linaro Started working on CoreSight in March after LCA2014● Started from the initial framework submitted by Pratik Patel [3]● The framework provides support for:

● source: ETMv3.3 to ETMv3.5 along with PTMv1.0 and PTMv1.1● link: 8 port funnel and non-configurable replicator● sink: ETBv1.0, TPIU and TMC (Trace Memory Controller)

● Support for STM and CTI will be submitted when the base framework is accepted upstream.

● With the framework it is possible to aggregate, configure and collect trace data seamlessly on a platform.

Linaro CoreSight Framework

● Coresight branch on git.linaro.og:● https://git.linaro.org/kernel/coresight.git/● Always look at the “master” branch for the latest code● The code is rebased on the latest release● The Coresight framework and drivers are under drivers/coresight/● Menuconfig options can be found under “Kernel Hacking/Coresight Tracing Support”

● Earlier submission and initial RFC are also present.● Upstream submission if you want to see what we’ve done:

● V1: http://thread.gmane.org/gmane.linux.kernel/1714785● V2: http://thread.gmane.org/gmane.linux.kernel/1734361● V3: https://lkml.org/lkml/2014/8/7/439● V4: https://lkml.org/lkml/2014/8/20/618

Where to get the code

● Provides an easy integration via DT for any platforms with generic coreSight components

● Plenty of flexibility for addition of, new non-generic devices● A front-end/back-end configuration for common and architecture specific parts are very

likely to happen.● Device specific method to access DT configuration can also be done easily

● Access via debugfs of all ETM/PTM configuration registers● Processor state (hibernation) not coupled with ETM/PTM

configuration● Multiple configuration of source and sink is supported.● Provides interface for reporting the status of various components

and the gathering of meta data.

CoreSight Framework Highlights

● When the framework boots coresight devices are discovered● The individual drivers will then register each newly found device

with the framework:

coresight-tpiu 20030000.tpiu: TPIU initialized

coresight-etb10 20010000.etb: ETB initialized

coresight-funnel 20040000.funnel: FUNNEL initialized

coresight-replicator replicator: REPLICATOR initialized ← non-configurable replicator don’t have an address map

coresight-etm3x 2201c000.ptm: ETM initialized

coresight-etm3x 2201d000.ptm: ETM initialized

coresight-etm3x 2203c000.etm: ETM initialized

coresight-etm3x 2203d000.etm: ETM initialized

coresight-etm3x 2203e000.etm: ETM initialized

CoreSight Framework

● When the kernel has booted, coresight devices and attributes are found under /sys/kerne/debug/coresight:

root@linaro-developer:~# cd /sys/kernel/debug/coresight/

root@linaro-developer:/sys/kernel/debug/coresight# ls00000000.replicator 20030000.tpiu 2201c000.ptm 2203c000.etm 2203e000.etm20010000.etb 20040000.funnel 2201d000.ptm 2203d000.etm

root@linaro-developer:/sys/kernel/debug/coresight# ls 20010000.etb/enable status trigger_cntr

root@linaro-developer:/sys/kernel/debug/coresight# ls 2201c000.ptm/addr_acctype cntr_idx enable nr_ctxid_cmp seq_32_eventaddr_idx cntr_rld_event enable_event reset seq_curr_stateaddr_range cntr_rld_val etmsr seq_12_event statusaddr_single cntr_val fifofull_level seq_13_event sync_freqaddr_start ctxid_idx mode seq_21_event timestamp_eventaddr_stop ctxid_mask nr_addr_cmp seq_23_event traceidcntr_event ctxid_val nr_cntr seq_31_event trigger_eventroot@linaro-developer:/sys/kernel/debug/coresight#

CoreSight Framework

● By default the first address comparator is configured with “_stext” and “_etext” → good health check of coresight implementation

root@linaro-developer:/sys/kernel/debug/coresight# ls00000000.replicator 20030000.tpiu 2201c000.ptm 2203c000.etm 2203e000.etm20010000.etb 20040000.funnel 2201d000.ptm 2203d000.etmroot@linaro-developer:/sys/kernel/debug/coresight# ls 20010000.etbenable status trigger_cntr root@linaro-developer:/sys/kernel/debug/coresight# cat 2201c000.ptm/addr_range 0x80008280 0x805d34d8 ← Addresses match with _stext and _etext in “System.map”root@linaro-developer:/sys/kernel/debug/coresight# ls 20010000.etb/enable status trigger_cntrroot@linaro-developer:/sys/kernel/debug/coresight# echo 1 > 20010000.etb/enable ← Enable at least one sourceroot@linaro-developer:/sys/kernel/debug/coresight# echo 1 > 2201c000.ptm/enable ← System already configure, simply enable tracingroot@linaro-developer:/sys/kernel/debug/coresight# cat 20010000.etb/statusDepth: 0x2000Status: 0x1RAM read ptr: 0x0RAM wrt ptr: 0x1aeb ← RAM write pointer is incrementing, i.e, traces are being collectedTrigger cnt: 0x0Control: 0x1Flush status: 0x0Flush ctrl: 0x2001root@linaro-developer:/sys/kernel/debug/coresight#

CoreSight Framework

Enabling a “Path” using the Framework

Source: ARM Ltd

● When satisfied with trace collection, simply harvest the content of the ETB buffer:

root@linaro-developer:/sys/kernel/debug/coresight# echo 0 > 2201c000.ptm/enableroot@linaro-developer:/sys/kernel/debug/coresight# ls /dev/Display all 166 possibilities? (y or n).initramfs/ snd/ tty55.udev/ stderr tty5620010000.etb stdin tty57audio stdout tty58block/ tty tty59bus/ tty0 tty6char/ tty1 tty60root@linaro-developer:/sys/kernel/debug/coresight# dd if=/dev/20010000.etb of=~/cstrace.bin64+0 records in64+0 records out32768 bytes (33 kB) copied, 0.00233771 s, 14.0 MB/sroot@linaro-developer:/sys/kernel/debug/coresight#

root@linaro-developer:/sys/kernel/debug/coresight# ll ~/cstrace.bin -rw-r--r-- 1 root root 32768 Aug 22 16:00 /root/cstrace.bin ← File with the trace data, in PFT formatroot@linaro-developer:/sys/kernel/debug/coresight#

CoreSight Framework

● CodeAurora’s submission created a lot of talk on STM and CTI implementation

● Linaro’s goal was to establish a foundation without being impaired by STM and CTI debates

● Concentrating on basic tracing would concentrate efforts and discussions on the framework implementation

● STM and CTI will be coming as soon as the framework gets accepted upstream.

● At this time we have been through 4 submission, each round yielding very fruitful comments and ideas

Upstreaming Stargegy and Status

● Currently supported platforms:● ARM Vexpress-TC2● TI beagle/beagleXM● Huawei’s D01 board (upstreaming currently under way)

● New platforms we are currently working on:● Qualcomm APQ8074● TI UEVM5432 (OMAP5)

● Support for STM/STM500● Need to find a way to manage channel allocation between user and kernel space● How to model external input in a flexible and generic way?● Userspace tool and libraries…

● CTI (Cross Trigger Interface)● John Hunter (formely?) at TI published a good driver on github● codeAurora also has one, need to look at both

What is next

● Support for ARMv8 ● The framework has been developed on ARMv7, need to see how it will work on V8● Devices, like STM500 ant ETMv4, are tailored for V8● STM500 may turn out to be like STM● ETMv4 is a very different implementation than ETMv3.x

● How to support the tracing of application code?● Integration with other kernel subsystem, i.e Perf and Ftrace● Follow what other architectures are doing and not re-invent the

wheel:● Intel’s PT architecture [6][7]● MIPS ?

What is Next (Cont’d)

● How data is represented in the trace stream depends on how the trace source has been configure (cycle accurate, timestamp)● Unlike ftrace, trace streams can’t simply be exported off platform and decoded● There has to be knowledge about the configuration and the image in memory● Commonly called the “metadata” problem

● Identification of trace stream decoder that is open and free● ptm2human is a very good start.● etm-objdump[4]]5] also did a very promising decoder● Both need to be improved (and maintained)

● How do we control traces from the kernel? ● ftrace has the capability to start and stop traces using a kernel API● Do we need the same mechanism for coresight?

● STM channel allocation and management

Challenges (Preview of the Next Session)

[1]. https://github.com/hwangcc23/ptm2human[2]. https://wiki.linaro.org/WorklingGroups/Kernel/Coresight/traceDecodingWithDS5[3]. http://lists.infradead.org/pipermail/linux-arm-kernel/2012-December/138646.html[4]. http://lists.linaro.org/pipermail/linaro-dev/2012-November/014439.html[5]. http://lists.linaro.org/pipermail/linaro-dev/2012-November/014476.html[6]. http://thread.gmane.org/gmane.linux.kernel/1771197[7]. https://software.intel.com/en-us/blogs/2013/09/18/processor-tracing

Question and Comment

More about Linaro Connect: connect.linaro.org Linaro members: www.linaro.org/membersMore about Linaro: www.linaro.org/about/