Trace Analysis

Stack: torch-spyre (new, Inductor-based).

Traces written by torch.profiler (see PyTorch Profiler) are Chrome-trace JSON files. They open in any of three viewers:

Quick start

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.PrivateUse1],
    record_shapes=True,
    on_trace_ready=torch.profiler.tensorboard_trace_handler("./logs/my_run"),
) as prof:
    output = model(inputs)

tensorboard_trace_handler writes the trace to ./logs/my_run/ for TensorBoard. For Perfetto / Chrome, call prof.export_chrome_trace("trace.json") and load the file directly.

aiu-trace-analyzer

aiu-trace-analyzer is an open-source post-processing tool for traces from the PyTorch profiler. The repository README is the authoritative guide; minimum setup follows.

Install from source:

git clone https://github.com/IBM/aiu-trace-analyzer.git
cd aiu-trace-analyzer
pip install --editable .

Run the workload with profiling enabled (see PyTorch Profiler) and the runtime env vars that expose compiler exports:

export DTCOMPILER_KEEP_EXPORT=true
export DEEPRT_EXPORT_DIR=<workload-directory>
export DTCOMPILER_EXPORT_DIR=<workload-directory>
export DT_DEEPRT_VERBOSE=0

python3 workload.py > logs.txt

Post-process the trace:

acelyzer -i <trace_file_json> -c logs.txt

Known issues (from torch-spyre-docs)

  • On the new stack, logs.txt can end up empty, in which case the processed output files are created but contain no additional information beyond the input trace.

See also