Repository navigation
Enhance detailed profiling runner to support any runner like ml_vart - #74
Merged
Merged
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
jvillarre
self-requested a review
November 19, 2026 17:19
jvillarre
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enhance detailed profiling runner for generic runner
Updates
src/mldebug/scripts/vaiprofile_detailed_report_runner.pyso it can drive any runner command (for example an ml_vart application), finds the model from the test's compiled cache, and judges runs by exit code and FlexMLRT errors in the log.Changes
--runner-cmdreplaces--runner--runner-cmd "CMD"takes the whole command line as one quoted argument. It is run assh -c "CMD"from each test folder, for the Base Run and Runs A to D, with nothing added (no--input_model,--analyzer_profilingor--vaiml_custom_ops).vitisai_config.jsonis still rewritten before each run, so the command must read it from its current folder.--runner-cmd, no model is looked for; a test only needsvitisai_config.json.--runner-cmdis an error.--runneris removed, and argparse prefix matching is turned off, so an old--runneris rejected instead of being taken as--runner-cmd.Default
ve2.pyrunner: model from the cache folder--runner-cmd, the nearestve2.pyis used as before:python3 ve2.py --input_model <model> --analyzer_profiling True [--vaiml_custom_ops ...].X, an immediate subfolder of the test holdingX.rai:onnx_model/X.onnx, elseX/X.onnx. This matchesve2.py'scacheKey = Path(model).stem. If there are several cache folders, the first in sorted order is used, with a warning..onnxfile, the test is an ERROR and the next test runs.One PASS rule for both runners
board_run.logcontains none of[error] [FLEXMLRT],ERT_CMD_STATE_TIMEOUTandFlexMLRT Exception:.FAIL (<text> in board_run.log), even if the exit code is not 0. The script's ownCommand: ...header line is not scanned.ofm_0.npyis no longer required.fpgautil check in setup
fpgautilis onPATH, and reports whether/sys/class/fpga_managerhas an entry (for information only). Both are printed and written tosetup.log.fpgautilis missing: one warning, no overlay reload for any test, and the tests still run. A test withoverlay/gets the notefpgautil not available; fpgautil skipped; a test without it getsoverlay/ not found; fpgautil skipped.Other
Found N ...: X to run, Y skipped, Z error.--helpand--help-alltext describes the new behaviour, including how to quote--runner-cmd.Compatibility
--runner PATHis no longer accepted; use--runner-cmd "python3 PATH ...".--testfor--test-path) no longer work; spell options out in full.X/X.raicache folder are now ERRORs; before, anyonnx_model/*.onnxwas used.ve2.pyrun that exits 0 without writingofm_0.npynow passes.Testing
Tested on stub tools (
pip3,fpgautil,vaiprofile-collect-data, a stubve2.py), not on a board:onnx_model/X.onnx, with onlyX/X.onnx, several cache folders, no.rai,.raiwithout.onnx, and every test in error;--runner-cmd: dry run, a real run without any model, thepython3 ve2.py --input_model onnx_model/resnet50.onnxexample, and an empty value;fpgautilmissing (real run and dry run), and present with no/sys/class/fpga_manager.