Run The Suite - TorchCTS

Install

If you are working on a backend, install TorchCTS in the same virtual environment you use to build and run that backend. That keeps TorchCTS, PyTorch, and the backend package in the environment you are actually validating.

python -m pip install torchcts

Use this path for backend development, CI inside a backend repository, and release validation against a backend package you already build.

Not A Backend Engineer?

If you just want to try TorchCTS on a machine without wiring it into a backend development environment, use the online installer. It creates a separate TorchCTS install and chooses a PyTorch wheel family for you.

macOS / Linux

curl -fsSL https://torchcts.ai/scripts/install.sh | sh

Windows PowerShell

irm https://torchcts.ai/scripts/install.ps1 | iex

The website installer chooses the PyTorch wheel family. The run command still chooses the backend under test. If an existing PyTorch install is outside 2.7.0-2.12.1, the installer warns and points to the validated range. Use TORCHCTS_TORCH_VARIANT when you need a specific wheel family, TORCHCTS_UPGRADE_TORCH=1 when you want the installer to install the validated range, and TORCHCTS_NON_INTERACTIVE=1 in scripts.

Backend Development Loop

TorchCTS can bootstrap test-driven development for a backend. Use an existing TorchCTS case as the failing test for the behavior being implemented, keep the run focused while working, and retain the broader coverage after the fix is complete.

1Choose The Behavior

Pick the operator, dtype, capability, semantic level, or path-shape case you are implementing.

2Configure The Current Surface

Set the manifest to the behavior the backend is intended to support at this stage.

3Run A Focused Selection

Limit the run so the implementation problem is quick to reproduce and easy to inspect.

4Read The Result

Use the result category, markdown report, runlog, probe records, and crash record to identify the next engineering action.

5Fix And Rerun

Rerun the same selection until the behavior matches the PyTorch expectation.

6Expand One Axis

Add one dtype, semantic level, suite, operator family, or resource tier at a time.

7Keep It In Regression Coverage

Add the appropriate focused or broader command to the backend's normal CI after the implementation is reliable.

torchcts run --device my_backend --level 2 --dtype torch.float32 --report-skips -k matmul

This example narrows the run to float32 cases through semantic level 2 whose pytest selection matches matmul. Replace the selection with the operator or behavior currently under development.

First Validation

torchcts init --template smoke --non-interactive
torchcts check-manifest
torchcts run --device cuda --level 1 --report-skips

The first run checks the smallest useful backend surface and separates environment setup, manifest configuration, and implementation work.

First Backend Validation Walkthrough

Use this sequence when you are validating a backend for the first time. It is a tutorial path, not the release gate.

1Install in the backend venv

Install TorchCTS where your backend package and PyTorch wheel already live. That makes the run test the environment you actually ship.

2Create a starter manifest

Run torchcts init --template smoke --non-interactive, then edit device_name, backend_import, dtypes, and capabilities.

3Validate the manifest

torchcts check-manifest must pass before backend results are meaningful.

4Run level 1

Start with the core primitive surface. If the run does not complete cleanly, use the result to separate environment setup, manifest configuration, and backend implementation work.

5Read the result

Open the markdown report, latest result reference, canonical history artifact, and runlog. Compare the artifact shape with the sample results in the TorchCTS repo before interpreting a failure cluster.

6Widen one thing

After the first slice is reliable and understandable, expand one axis at a time: one dtype, one semantic level, one suite, or one operator selection.

For the relationship between TorchCTS and PyTorch's testing foundations, see How TorchCTS Fits With PyTorch.

Backend Bring-Up

torchcts run --device my_backend --level 2 --dtype torch.float32 --report-skips
torchcts run --device my_backend --level 4 --dtype torch.float32 --suite generated --report-skips
torchcts show-skips --device my_backend --level 4 --dtype torch.float32

Keep bring-up narrow while implementing a feature. Make the focused slice reliable, then widen the test surface one axis at a time.

The manifest is both a development configuration and a release-facing support declaration. During bring-up it should describe the support the current implementation is intended to exercise, not an aspirational final state.

Filtered does not mean passed. It means TorchCTS did not run the case and recorded why.

Path-Shape Runs

Path-shape runs use the normal pytest harness and result format. They are a focused way to run tracked shape coverage by family, runner, resource tier, cost class, model role, dtype group, or exact case id.

torchcts path-shapes validate
torchcts path-shapes list --family matmul
torchcts path-shapes run --device cuda --family matmul --level 8 -q
torchcts path-shapes run --device mps --level 8 --subprocess-per-shape -q
torchcts path-shapes run --device mps --all-resource-tiers --level 8 --subprocess-per-shape -q

When --subprocess-per-shapeRuns each path-shape case in its own subprocess so a crash in one shape doesn't prevent testing the rest. The crash still counts as a result. is used, TorchCTS forwards the selected path-shape filters to each child process so isolated crash runs execute the same cases.

Do not use --validation --device mpsValidation mode forces CPU execution to verify the harness itself. It does not test any accelerator backend, so it cannot be used as MPS or CUDA evidence. as MPS backend evidence. Validation mode forces CPU. Real MPS evidence must come from non-validation runs with --device mps.

Scope Controls

--level

Runs all cases with semantic_level less than or equal to the requested level. See Semantic Levels.

--level-exact

Runs one semantic level.

--level-range

Runs an inclusive level range.

--dtype

Narrows the dtype support included in the run.

--suite

Limits collection to one suite.

Manifest False

Leaves unsupported paths outside the configured test surface.

show-skips

Prints TorchCTS run accounting without executing the full suite.

Backend Release Validation

torchcts check-manifest
torchcts coverage audit
torchcts coverage check --fail-on-unknown
torchcts run --device "$TORCHCTS_DEVICE" --level 8 --report-skips --results-dir results
torchcts report

A release run needs the whole results directory, not just terminal output. Keep the latest result reference, canonical history artifact, markdown report, runlog, coverage files, probe diagnostics, reference diagnostics, and crash evidence.

The release gate reuses the development test system at greater depth. It confirms that focused fixes still work in the broader suite and preserves unsupported paths, regressions, crashes, and coverage gaps as engineering evidence before release.

TorchCTS project release QA

TorchCTS maintainers separately validate TorchCTS-owned references, routing, provenance, and evidence manifests from the source repository. Those checks are not installed backend tests and do not contribute to backend results.

Read how oracle QA works or review the TorchCTS release checklist.

Interpret The Run Correctly

Filtered is not passed

Filtered means TorchCTS did not execute the case. The recorded reason explains whether it was outside the manifest, operator contract, semantic level, suite selection, or another policy boundary.

Runtime unsupported is an implementation result

When a configured in-contract path reports unsupported or not implemented at runtime, the manifest and implementation are out of alignment. The result remains a failure or error.

Artifacts are part of the run

Keep the result, report, runlog, coverage files, probe diagnostics, reference diagnostics, and crash evidence. Terminal output alone is not enough for reliable debugging or review.

Isolation contains crashes

Subprocess isolation protects the parent run and allows later tests to continue. It does not change the crash result.