Skip to content

Enhance detailed profiling runner to support any runner like ml_vart - #74

Merged
jvillarre merged 1 commit into
Xilinx:mainfrom
IshitaGhosh:vartRunner
Nov 19, 2026
Merged

jvillarre merged 1 commit into
Xilinx:mainfrom
IshitaGhosh:vartRunner

Conversation

@IshitaGhosh

Copy link
Copy Markdown

Enhance detailed profiling runner for generic runner

Updates src/mldebug/scripts/vaiprofile_detailed_report_runner.py so it can drive any runner command (for example an ml_vart application), finds the model from the test's compiled cache, and judges runs by exit code and FlexMLRT errors in the log.

Changes

--runner-cmd replaces --runner

  • --runner-cmd "CMD" takes the whole command line as one quoted argument. It is run as sh -c "CMD" from each test folder, for the Base Run and Runs A to D, with nothing added (no --input_model, --analyzer_profiling or --vaiml_custom_ops).
  • vitisai_config.json is still rewritten before each run, so the command must read it from its current folder.
  • With --runner-cmd, no model is looked for; a test only needs vitisai_config.json.
  • An empty --runner-cmd is an error. --runner is removed, and argparse prefix matching is turned off, so an old --runner is rejected instead of being taken as --runner-cmd.

Default ve2.py runner: model from the cache folder

  • Without --runner-cmd, the nearest ve2.py is used as before: python3 ve2.py --input_model <model> --analyzer_profiling True [--vaiml_custom_ops ...].
  • The model is found from the cache folder X, an immediate subfolder of the test holding X.rai: onnx_model/X.onnx, else X/X.onnx. This matches ve2.py's cacheKey = Path(model).stem. If there are several cache folders, the first in sorted order is used, with a warning.
  • With no cache folder, or neither .onnx file, the test is an ERROR and the next test runs.
  • If no test can run, setup is skipped, but the Summary and the CSV are still written, and the exit code is 1.

One PASS rule for both runners

  • A run passes if its exit code is 0 and its board_run.log contains none of [error] [FLEXMLRT], ERT_CMD_STATE_TIMEOUT and FlexMLRT Exception:.
  • A match gives FAIL (<text> in board_run.log), even if the exit code is not 0. The script's own Command: ... header line is not scanned.
  • ofm_0.npy is no longer required.

fpgautil check in setup

  • Setup checks whether fpgautil is on PATH, and reports whether /sys/class/fpga_manager has an entry (for information only). Both are printed and written to setup.log.
  • If fpgautil is missing: one warning, no overlay reload for any test, and the tests still run. A test with overlay/ gets the note fpgautil not available; fpgautil skipped; a test without it gets overlay/ not found; fpgautil skipped.

Other

  • The discovery line now counts errors: Found N ...: X to run, Y skipped, Z error.
  • The --help and --help-all text describes the new behaviour, including how to quote --runner-cmd.

Compatibility

  • --runner PATH is no longer accepted; use --runner-cmd "python3 PATH ...".
  • Shortened options (for example --test for --test-path) no longer work; spell options out in full.
  • With the default runner, tests without an X/X.rai cache folder are now ERRORs; before, any onnx_model/*.onnx was used.
  • A ve2.py run that exits 0 without writing ofm_0.npy now passes.

Testing

Tested on stub tools (pip3, fpgautil, vaiprofile-collect-data, a stub ve2.py), not on a board:

  • default runner: cache folder with onnx_model/X.onnx, with only X/X.onnx, several cache folders, no .rai, .rai without .onnx, and every test in error;
  • --runner-cmd: dry run, a real run without any model, the python3 ve2.py --input_model onnx_model/resnet50.onnx example, and an empty value;
  • PASS rule with both runners: each error text with exit code 0, exit code 1, and an error text in Run B only;
  • fpgautil missing (real run and dry run), and present with no /sys/class/fpga_manager.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jvillarre
jvillarre self-requested a review November 19, 2026 17:19
@jvillarre
jvillarre merged commit fac739f into Xilinx:main Nov 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants