CLI Reference - TorchCTS

Commands

The CLI has a small command set. Most backend work happens in run, show-skips, check-manifest, and the coverage subcommands.

`torchcts init`

Create manifest.py from a packaged template.

--template smokeSmallest baseline manifest. Uses semantic level 1.

--template minimalBasic correctness manifest. Uses semantic level 2.

--template inferenceInference-oriented manifest. Uses semantic level 4.

--template trainingTraining-oriented manifest. Uses semantic level 4.

--template completeRelease-depth manifest. Uses semantic level 8 and the broadest capability surface.

--non-interactiveBoolean flag. Fail instead of prompting.

`torchcts run`

Run the conformance suite. Unknown arguments are forwarded to pytest, so pytest selectors still work.

`torchcts show-skips`

Print TorchCTS filtered accounting and pytest skip-marked accounting without running the full suite.

`torchcts check-manifest`

Validate manifest syntax and schema.

--manifest PATHOpen-ended path. Defaults to manifest.py in the current directory.

`torchcts report`

Regenerate reports from an existing result artifact.

--from-file PATHOpen-ended path to a TorchCTS result artifact.

`torchcts sync-opinfo`

Refresh the PyTorch OpInfo registry cache used for PyTorch OpInfo-derived operator tests.

--discover-ieee754-undefinedBoolean flag. Rebuild the local list of CPU operations with undefined NaN or Inf behavior.

`torchcts path-shapes`

Inspect, validate, and run the tracked targeted path-shape corpus through the normal pytest harness.

validateValidate corpus schema, budgets, and release targets.

summaryPrint corpus counts and budget usage.

budgetCheck count budgets and collection baseline metadata.

listList tracked cases selected by path-shape filters.

runRun selected cases through pytest and TorchCTS result reporting.

Run And Show-Skips Flags

Focused development runs use the same torchcts run command as broader validation. Combine backend, dtype, suite, semantic-level, path-shape, and forwarded pytest selection controls to reproduce the behavior currently under development.

`--device`

Selects the backend under test.

autoDefault. Detects or prompts for a backend when possible.

cpucudampsxpuCommon in-tree runtime device names.

any device nameOpen-ended string. Use this for PrivateUse1 backends such as npu, ocl, flagos, or a project-specific name.

`--dtype`

Narrows the run to one or more configured dtypes.

torch.float32Fully qualified dtype name.

float32The torch. prefix is optional for recognized PyTorch dtype names.

repeatablePass the flag more than once to test more than one dtype.

`--level`, `--level-exact`, `--level-range`

These flags are mutually exclusive.

--level NRuns levels 1 through N. N must be 1 through 8.

--level-exact NRuns only level N. N must be 1 through 8.

--level-range MIN:MAXRuns an inclusive range. MIN-MAX is also accepted. Both numbers must be 1 through 8.

`--suite`

Limits collection to one suite. This is a finite enum.

opinfoPyTorch OpInfo-derived operator tests.

operatorsHand-authored operator behavior.

generatedGenerated dispatcher and variant cases.

autogradAutograd behavior.

dtypesDtype-specific behavior.

trainingTraining and training-adjacent behavior.

compilerCompile integration.

device_apiDevice module and device API behavior.

memoryAllocator and memory behavior.

stridesStride and memory-format behavior.

workloadsModel-shaped workload coverage.

rngRandom number generation behavior.

serializationSave/load behavior.

errorsExpected error behavior.

stressRelease-depth stress cases.

multi_deviceMulti-device behavior.

adversarialAdversarial correctness cases.

customProject-provided custom test directories.

`--memory-mode`

Controls cleanup cadence during a run.

conservativeClean up more often. Use when memory headroom is limited.

balancedDefault. Normal cleanup cadence.

performanceClean up less often. Use only when memory headroom is known.

Isolation Flags

--subprocess-per-testBoolean flag. Runs every test in its own subprocess.

--subprocess-timeout SECONDSPositive number. Defaults to 120 seconds.

--known-segfault-policy isolateDefault. Isolates matching known crash candidates in subprocesses.

--known-segfault-policy offDisables known crash isolation policy.

--known-segfault-auditBoolean flag. Collects tests, validates known crash rules, prints matches, and exits.

--adaptive-isolation autoDefault. Isolates tests that match prior crash or hang evidence.

--adaptive-isolation offDisables adaptive isolation.

Output And Selection Flags

--show-skipsBoolean flag. Prints filtered and pytest skip-marked accounting and exits.

--report-skipsBoolean flag. Includes filtered and pytest skip-marked accounting in the report.

--show-tracebackBoolean flag. Keeps pytest tracebacks visible for the run. The manifest field show_traceback can set the same behavior by default.

--results-dir PATHOpen-ended directory path. Defaults to ./results.

--max-device-memory MBPositive integer. Caps device memory for the run.

--max-tensor-size MBPositive integer. Caps single tensor size for the run.

--non-interactiveBoolean flag. Fails instead of prompting during auto device selection.

--validationBoolean flag. Runs harness and CPU-compatible validation without probing an accelerator.

Coverage Commands

`torchcts coverage`

The first argument after coverage is a finite subcommand.

inventoryBuild dispatcher inventory using default paths.

auditBuild coverage audit using default paths.

reportRender coverage report from the default audit.

materializeWrite deterministic generated coverage cases using default paths.

checkValidate coverage audit consistency.

collect-backend-evidenceCollect backend evidence directly into the canonical tracked store.

`coverage check`

--strict-unknownsBoolean flag. Return nonzero if unknown surfaces remain.

--fail-on-unknownAlias for --strict-unknowns.

`coverage collect-backend-evidence`

--store PATHRequired canonical backend evidence store directory. Standard repository path is evidence/backends.

--device DEVICEOpen-ended device name. Defaults to cuda.

--backend-gate GATEOpen-ended gate selector. May be repeated. Common forms include cuda+rocm and cpu+fbgemm+cpu_build.

--surface SURFACEOpen-ended dispatcher surface. May be repeated.

--no-run-oraclesBoolean flag. Collect environment and schema evidence without executing oracle runners.

--runtime-modification LABELPath-free semantic label for a runtime modification used during collection. May be repeated.

--run-pending-candidatesBoolean flag. Execute pending backend-pack specs that have real oracle runners.

--require-oracle-resultsBoolean flag. Return nonzero unless every selected surface produces a passing oracle result.

--fail-on-oracle-failureBoolean flag. Return nonzero when any executed oracle fails.

Path-Shape Commands

Path-shape commands work with a curated corpus. Selectors narrow existing cases. They do not create new shape combinations.

`path-shapes validate`, `summary`, and `budget`

--jsonBoolean flag. Emits machine-readable output.

--strict-budgetBoolean flag. Fails baseline-ratio budget violations.

--enforce-targetsBoolean flag. Fails target gaps as release blockers.

budget --checkChecks current budget metadata.

budget --no-strict-budgetSkips baseline-ratio budget checks for the budget command.

Path-Shape Selectors

These selectors apply to path-shapes list and path-shapes run. Each selector can be repeated. Comma and plus syntax are also accepted.

--familyFinite set: matmul, attention, convolution, reduction, indexing, spatial, model_patterns, normalization, fft, sorting, broadcasting, linear_algebra.

--resource-tierFinite set: smoke, standard, heavy, stress. Default selection is smoke plus standard.

--cost-classFinite set: tiny, small, medium, large.

--dtype-groupFinite set: float, integer, bool, complex.

--runnerFinite set from the corpus, such as matmul.mm, attention.sdpa, convolution.conv2d, reduction.sum, indexing.gather, fft.rfft, and normalization.layer_norm.

--model-roleFinite set from the corpus, such as attention_kernel_selection, vision_convolution, transformer_norm, batch_norm, and decode_or_tile_boundary.

--categoryFinite set from the corpus, such as path_shape, algorithmic_shape, resource_boundary, layout_path_shape, and the family-specific path-shape categories.

--caseExact case id from the corpus.

--all-resource-tiersSelects every tracked resource tier instead of the default smoke plus standard tiers.

`path-shapes list`

--jsonBoolean flag. Emits selected cases as JSON.

--require-casesReturns nonzero when selectors match no cases.

`path-shapes run`

--device DEVICEOpen-ended backend device name, such as cpu, cuda, mps, xpu, npu, ocl, flagos, or webgpu.

--level NRuns path-shape cases at or below level N. Defaults to 8.

--level-exact NRuns only path-shape cases at level N.

--level-range MIN:MAXRuns path-shape cases in an inclusive semantic-level range.

--validationRuns validation mode. Do not use it as MPS backend evidence because validation mode forces CPU.

--subprocess-per-shapeRuns each materialized shape in its own subprocess and forwards the active path-shape selectors to each child process.

--output-dir PATHForwarded pytest results directory.

Triage Command

`torchcts triage mps`

The only current triage subcommand is mps.

--from-file PATHOpen-ended path to a seed result artifact.

--nodes-file PATHOpen-ended path to a newline-delimited pytest node list.

--include-crashersBoolean flag. Includes known and runlog crash candidates.

--output-dir PATHOpen-ended directory path. Defaults to results/mps_triage.

--timeout SECONDSPositive number. Defaults to 120 seconds per subprocess run.

--level NSemantic level from 1 through 8. Defaults to 8.

--no-runBoolean flag. Classifies existing artifacts without executing queued pytest nodes.

--repros-onlyBoolean flag. Runs standalone MPS repro scripts without executing queued pytest nodes.

Examples

torchcts run --device cuda --level 1 --report-skips
torchcts run --device my_backend --level 2 --dtype torch.float32 --report-skips -k matmul
torchcts run --device mps --level 4 --dtype torch.float32 --suite generated
torchcts show-skips --device xpu --level-range 3:5 --dtype torch.bfloat16
torchcts coverage check --fail-on-unknown
torchcts coverage collect-backend-evidence --store evidence/backends --device cuda --backend-gate cuda+rocm
torchcts path-shapes validate --strict-budget --enforce-targets
torchcts path-shapes run --device cuda --family matmul --level 8 -q
torchcts path-shapes run --device mps --resource-tier heavy --level 8 --subprocess-per-shape -q
torchcts triage mps --include-crashers

Commands

torchcts init

torchcts run

torchcts show-skips

torchcts check-manifest

torchcts report

torchcts sync-opinfo

torchcts path-shapes

Run And Show-Skips Flags

--device

--dtype

--level, --level-exact, --level-range

--suite

--memory-mode