Debugging

Scope: correctness — why is the result wrong? For performance questions (why is it slow?) see Profiling.

This guide describes a systematic approach to debugging incorrect or unexpected behaviour in Torch-Spyre. The workflow applies whether you are investigating a wrong numerical result, a compilation failure, or a runtime error.

Overview

Debugging a Torch-Spyre issue typically follows these layers, working from the outside in:

  1. Isolate — reduce the problem to a minimal, self-contained script

  2. Observe data transfers — verify tensors arrive on device correctly

  3. Inspect compiler artifacts — trace the issue through the compilation pipeline (FX Graph → Loop IR → sdsc_<index>.json)

  4. Bisect frontend vs. backend — use the sendnn backend to determine whether the bug is in Torch-Spyre’s front-end or in the DeepTools back-end compiler

Step 1 — Create a Minimal Reproducer

Before investigating, reduce the failing model or script to the smallest possible program that still shows the wrong behaviour. This makes the compiler artifacts much easier to read.

import torch

# Minimal reproducer — replace with the failing op
x = torch.arange(65, dtype=torch.float16)
result = x.clone().to("cpu")
print(result)

Step 2 — Enable Debug Environment Variables

The following environment variables control the level of diagnostic output:

Variable

Effect

TORCHINDUCTOR_FORCE_DISABLE_CACHES=1

Forces full recompilation on every run; ensures you see fresh artifacts, not cached ones

TORCH_SPYRE_DEBUG=1

Logs all CPU↔Spyre data transfers, including tensor shapes, layouts, and raw values

TORCH_COMPILE_DEBUG=1

Writes intermediate compiler artifacts to a local directory for offline inspection

SPYRE_INDUCTOR_LOG=1

Deprecated. Use TORCH_LOGS="spyre.inductor:INFO" instead

SPYRE_INDUCTOR_LOG_LEVEL=DEBUG

Deprecated. Use TORCH_LOGS="spyre.inductor:DEBUG" instead

SPYRE_LOG_FILE=path/to/file.log

Redirect Spyre Inductor log output to a file

TORCH_SPYRE_DOWNCAST_WARN=0

Suppress int64→int32 warnings

TORCH_LOGS="+inductor"

PyTorch provided tool to selectively enable Inductor or other parts of the torch.compile to the log

Run your reproducer with all three enabled:

TORCHINDUCTOR_FORCE_DISABLE_CACHES=1 \
TORCH_SPYRE_DEBUG=1 \
TORCH_COMPILE_DEBUG=1 \
python my_reproducer.py

TORCH_COMPILE_DEBUG writes artifacts to a subdirectory under /tmp/torchinductor_<user>/ (or torch_compile_debug/ in the current directory, depending on your PyTorch version).

Step 3 — Examine Compiler Artifacts

TORCH_COMPILE_DEBUG preserves one subdirectory per compiled function. Inside you will find the intermediate representation at each stage of the pipeline:

torch_compile_debug/
└── run_<timestamp>-pid_<pid>/
    ├── torchdynamo/
    │   └── debug.log
    └── torchinductor/
        ├── aot_model___0_debug.log
        └── model__0_inference_0.0/
            ├── fx_graph_readable.py                              ← traced FX Graph (ATen ops)
            ├── fx_graph_runnable.py                              ← self-contained runnable graph
            ├── fx_graph_transformed.py                           ← FX Graph after Inductor passes
            ├── inductor_provenance_tracking_node_mappings.json   ← IR-to-source mapping
            ├── ir_pre_fusion.txt                                 ← LoopLevelIR before kernel fusion
            ├── ir_post_fusion.txt                                ← LoopLevelIR after kernel fusion
            └── output_code.py                                    ← generated host code

What to look for at each layer

FX Graph (fx_graph_readable.py) Verify the traced operation matches what you expect. Check that the operation is present, that input shapes are correct, and that no unexpected decompositions have changed the semantics.

LoopLevelIR (ir_pre_fusion.txt, ir_post_fusion.txt) Check that loop ranges and buffer shapes reflect the correct tensor sizes including padding. Mismatches here indicate a problem in the Inductor lowering or stickification pass.

sdsc_<index>.json The final specifications fed to the DeepTools back-end compiler — one file per compiled kernel (sdsc_0.json, sdsc_1.json, …), indexed in lowering order. Each file encodes the op name, input/output tensor layouts (device_size, stride_map, device_dtype), work division, and scratchpad allocations. Bugs that appear only in the final output often trace back to one of these files.

Example: debugging an incorrect clone result

Consider a float16 tensor of size [65]. The default Spyre layout pads the stick dimension to 128 bytes (64 elements per stick), so the tensor is laid out on device as shape [2, 64] — two sticks.

TORCH_SPYRE_DEBUG=1 output confirming the CPU→Spyre transfer is correct:

[TORCH_SPYRE_DEBUG] copy_to_device: shape=[65] dtype=float16
  device_layout: SpyreTensorLayout(device_size=[2, 64], stride_map=[64, 1], ...)
  transfer OK

Inspecting the sdsc_<index>.json for the clone kernel (locate it via output_code.py, which maps each launched kernel to its index) then revealed:

{
  "op": "clone",
  "input_layout": { "device_size": [2, 64], ... },
  "copy_range": [1, 64]   // ← only copying the first stick, should be [2, 64]
}

The bug: the codegen only emitted a copy for the first stick ([1, 64]) instead of both sticks ([2, 64]). The second stick — which holds element index 64 — was never written, leaving it at zero. (See issue #524 for the full investigation.)

Step 4 — Bisect Frontend vs. Backend with sendnn

If the sdsc_<index>.json files look correct, the bug is likely in the DeepTools back-end compiler. To confirm, re-run the same script using the sendnn backend instead of spyre:

import torch

def test(a, b):
    return torch.eq(a, b).to(dtype=torch.float16)

x = torch.tensor([-0.0, -0.0], dtype=torch.float16)
y = torch.tensor([0.0, 0.0], dtype=torch.float16)

# Test with sendnn backend
compiled = torch.compile(test, backend="sendnn")
result = compiled(x, y).to(dtype=torch.bool)
print(f"sendnn: {result}")

# Compare with spyre backend
compiled_spyre = torch.compile(test, backend="spyre")
result_spyre = compiled_spyre(x.to("spyre"), y.to("spyre")).to("cpu").to(dtype=torch.bool)
print(f"spyre:  {result_spyre}")

Outcome

Interpretation

sendnn wrong, spyre wrong

Bug is in the back-end compiler (DeepTools); file issue against the backend

sendnn correct, spyre wrong

Bug is in Torch-Spyre’s front-end (codegen, stickification, or host code)

Both correct

The issue may be environment-specific or in data transfer

(See issue #628 for an example where both backends returned the same incorrect result, confirming a back-end compiler bug.)

Checklist for Filing a Bug Report

When opening an issue, include:

  • [ ] Minimal reproducer script

  • [ ] Full error output or incorrect value observed

  • [ ] PyTorch version (python -c "import torch; print(torch.__version__)")

  • [ ] Torch-Spyre version or commit SHA

  • [ ] Output of TORCH_SPYRE_DEBUG=1 showing the data transfer log

  • [ ] Relevant excerpts from fx_graph_readable.py and the affected sdsc_<index>.json

  • [ ] Result of the sendnn comparison (frontend vs. backend bisect)

Quick Reference

# Full debug run
TORCHINDUCTOR_FORCE_DISABLE_CACHES=1 \
TORCH_SPYRE_DEBUG=1 \
TORCH_COMPILE_DEBUG=1 \
python my_reproducer.py

# Find the generated artifacts
find . -name "sdsc_*.json" 2>/dev/null
find /tmp -name "fx_graph_readable.py" 2>/dev/null