After knowing about the challenges involved in validating multicore systems and domains of system and application level tracing as explained by Don Dingee in his article “Tracing methods to multicore gladness” which is based on the first part of Mentor Embedded multicore whitepaper series, it’s time to take a deeper insight into what all has to be considered and done for an effective tracing of a multicore system and application software.
Software tracing can be based on static instrumentation or dynamic breakpoints and hardware based tracing use probing technology needing extra hardware. It’s not necessarily technical merit that need to be considered for the approach to be used in tracing, rather strategic decisions need to be made with a combined approach depending on the target system and its architecture. The parameters to be considered for trade-off include intrusiveness, performance, capacity, granularity and availability of hardware and software resources for the trace infrastructure.
The tracing cycle consists of certain pertinent steps that need to be performed with the trace event data; the steps include collection of data, import of data into an analysis and visualization tool, analysis, exploration and post-processing of the collected data, and deficiency identification and improvement. The event data collection includes trace instrumentation, configuring tracing options, start and stop of running applications on the target system. The collected data is loaded into a trace analysis/visualization tool on a host system, where it generally is transformed and optimized into manageable visual representation, according to the tool’s preference.
Mentor’s Sourcery Analyzer supports trace event data generated by the LTTng framework. It also extracts symbolic names from the binary image of the user application that has been traced and maps these to trace event data, thus enabling the look-up of source code from trace data. Sourcery Analyzer supports the import of custom trace event data into its native event format by text importer facilities and JavaScript scripted data imports.
The analysis tools abstract data from the raw trace event data in user understandable form such as function hit count, heat map, and top time-in function chart as shown above. In order to extract the resource usage per CPU in a multicore system, an analysis is required that is able to deliver such meta-information from the raw trace event data.
The visualization tools can represent the data in appropriate forms such as logs, graphs, charts, and tables as per the need; a log view can provide details about event sequences and associated information; a graph can provide trends, patterns, min/max information and so on; and a chart can provide statistical information about main factors such as producers, consumers, polluters, etc.
Then comes the most important part of exploration of the data where correlation and synchronization of trace data from different domains, such as kernel and user space (via a common time-axis) takes place. The data section corresponding to any anomaly or peculiarity needs to be located and its initiator identified, may be through user space traces.
The accuracy and success of analysis of a complex multicore system depends highly on the availability of solutions to correlate trace data from different domains that have been recorded during the execution of the examined application code. Generally, system and application time sources are not synchronized, and time is counted with different resolutions by different methods such as time-stamp count registers (in x86 architectures) and memory mapped external timers (in embedded architectures), which vary in their handling of numeric value overflow, linearity and resolution. The above illustration shows a time-correlated kernel and user-space traces scroll-synchronized through a selection cursor, the red vertical bar.
Sourcery Analyzer supports a palette of facilities to correlate time synchronized (scroll-synchronized by placing dedicated synchronization cursors at desired time-stamps) as well as time offset traces (that can be rebased to align their time scales).
The abstracted data from analysis and visualization tools provide a good ground for the trace tools to do further measurements and computations to determine the system behavior, for example event blocks spending most execution time.
Measurement and calculation tools for post-processing the data are seamlessly integrated into Sourcery Analyzer with easy and useful user interface. The above illustration shows frame events of different durations with annotations applied by the measurement tool. The ‘PulseWidth’ graph indicates the absolute frame duration values across the x-axis.
From the visualization graphs, the calculation tool can extract and compute derived data such as load curves for individual CPUs and load trend over all CPUs obtained from moving averages of individual CPU load graphs.
Finally, the user should be able to look up the source code (through accurate and comprehensive source code information display) from the trace event data in case of any encountered problem or concern and apply appropriate improvement.
Trace data formats and customization options are other criteria to look at when choosing a particular tracing tool and technology. Mentor Embedded Sourcery Analyzer provides a versatile platform for trace data analysis, visualization, extraction, correlation, measurement and computation. It supports, but is not limited to, LTTng instrumented Linux kernels and software trace instrumented user applications. Read the second part of Mentor Embedded multicore whitepaper series – “Software Tracing Tools and Techniques for Advanced Multicore Development” to know more. Manfred Kreutzer provides a great detailed description about all the procedures required for tracing.
More Articles by PawanFangaria…..
Share this post via:
Next Generation of Systems Design at Siemens