Results
A TorchCTS result turns a backend run into engineering actions. It records what passed, what needs implementation work, what crashed, and what did not execute.
Find The Next Fix
Correct the import, device name, dtype configuration, capability configuration, or runtime environment before interpreting operator results.
Inspect values, shape, dtype, layout, aliasing, mutation, error behavior, or property checks for the failing case.
Implement the configured path or update the manifest if that support is not currently part of the backend.
Use isolation, the runlog, and subprocess records to retain evidence while allowing the rest of the run to continue.
Check whether the case is intentionally outside the current support surface or represents support the team plans to add.
Determine whether the missing evidence requires target hardware, a matching build family, a backend runner, or a new test strategy.
Run Artifacts
Save the full results directory.
During an active run, the latest result file is a recovery snapshot. After a completed run, TorchCTS writes the canonical result once in the run history and the latest file can become a small relative reference to it. TorchCTS report and loading commands resolve that reference automatically.
The important files are the latest result reference, canonical history artifact, markdown report, runlog, coverage artifacts, probe and reference diagnostics, and any subprocess crash records.
Sample Results
Sample TorchCTS result sets live in the TorchCTS repository. GitHub renders the directory README, and the artifacts are versioned with the source that produced them.
Use the samples to see how a focused development run, a broader regression run, and their compact result artifacts, reports, runlogs, filtered accounting, and crash evidence fit together.
Artifact Anatomy
results/*_latest.jsonActive-run recovery snapshot or completed-run reference to the canonical history artifact. TorchCTS tools resolve it automatically.
results/*_history/Canonical completed result artifacts. Large results use compact JSON storage when it reduces artifact size; loading restores the normal metadata, results, skips, and diagnostics.
results/*.mdHuman report generated from the resolved result artifact.
results/*_runlog*Rolling execution trace with the latest 32 test starts used for crash and hang diagnosis.
results/coverage/Coverage audit files, pending-review files, generated case files, and coverage summaries.
results/*_harness_probe_failures_*.jsonlFailure-only diagnostic probe records when harness probes find import, dtype, capability, or compiler issues.
results/*_opinfo_oracle_failures_*.jsonlDiagnostic records created when TorchCTS cannot build a valid CPU-based expected result for a PyTorch OpInfo-derived sample. They explain reference construction problems; they do not become backend passes, skips, or oracle QA results.
Crash, timeout, hang, signal, and isolation evidence when subprocess execution is used.
Create A Submission Archive
Submit an archive of the real run artifacts. Do not send an empty template archive or only screenshots.
zip -r torchcts-results-$(date +%Y%m%d-%H%M%S).zip results manifest.pyCompress-Archive -Path results, manifest.py -DestinationPath torchcts-results.zip -ForceReading A Result
The markdown report is for humans. The resolved result artifact is the source of truth.
The backend matched the expected behavior for that test.
The test ran but the backend result was wrong.
The test did not complete normally.
A configured in-contract path raised unsupported or not implemented.
TorchCTS did not run the case and recorded why.
Pytest marked the case skipped for a runtime condition.
The process exited by signal, timeout, hang, or subprocess failure.
What The Backend Result Was Compared Against
A passing result means the backend satisfied the expected behavior selected for that test. Depending on the operator contract, that may be:
- PyTorch CPU reference behavior
- a reviewed TorchCTS reference
- a legal-output check for an operator with more than one valid representation
- a direct property check for behavior such as aliasing, mutation, storage, RNG, or allocation
Development-only oracle QA records are not added to backend results. See Expected Result Sources for details.
Scorecard
The scorecard is the human report generated from the same resolved result artifact.
Treat the scorecard as a summary, not as a replacement for the resolved result artifact. Use it to find failure clusters, filtered counts, crash records, and next actions.
Diagnostic Probe Evidence
Probe failures are diagnostic. Normal backend runs print a structured probe table with kind, probe, status, and note columns. Result artifacts keep failure-only harness probe records plus pass and total metadata. Probe output does not change whether configured tests run.
Known Crash Isolation
TorchCTS currently ships 13 known crash isolation rules. All current rules are for MPS, including generated dispatcher cases and a path-shape scaled dot-product attention case with bool masks and causal attention.
Known crash rules are subprocess isolation policy only. They do not skip, xfail, forgive, hide, or downgrade crashes.
Submit Validation Results
Include:
- backend name
- hardware key
- OS
- Python version
- PyTorch version
- TorchCTS version
- manifest
- exact command
- submission archive when available
- latest result reference
- markdown report
- runlog
- coverage audit if relevant
- whether the run completed normally
Submit Backend Evidence
Use backend evidence when a maintainer cannot run the target backend or build family directly.
The preferred form is a pull request or patch that updates the canonical store under evidence/backends/. Include the device, backend gate, PyTorch version, selected dispatcher surfaces, path-free runtime modification labels when relevant, oracle or property result, command intent, and run status.
Do not include host identity, absolute local paths, full audit snapshots, or transport archive metadata in canonical backend evidence. Send the result artifacts separately when a maintainer needs raw debugging context.
Submit Known Segfault Candidates
Users can submit known-segfault candidates to Kris/the maintainer for review.
Include:
- backend
- hardware key
- OS
- PyTorch version
- TorchCTS version
- exact command
- pytest node id or dispatcher surface
- signal or exit code
- runlog tail
- latest result artifact if one was produced
- minimal repro if available
Accepted entries can be added to torchcts/known_segfaults.json.
Inclusion means TorchCTS can isolate the matching test in a subprocess. It does not skip, xfail, forgive, hide, or downgrade the crash.
Where To Send It
Open a GitHub issue or send the evidence directly to Kris/the maintainer. Do not send only a screenshot of terminal output. The resolved result artifact and runlog are the evidence.