Skip to content

Automatic dumping for multi-partition designs - #75

Merged
hackwa merged 3 commits into
Xilinx:mainfrom
richard-byron:multi-partition
Nov 20, 2026
Merged

hackwa merged 3 commits into
Xilinx:mainfrom
richard-byron:multi-partition

Conversation

@richard-byron

Copy link
Copy Markdown
Collaborator

Allows automatic dumping of every partition in a design.

This should do nothing if the design only has one partition, include_subgraphs is set (limits the design to one NPU partition), or the halt plugin has signaled that the design will only halt on the first partition. In any of those cases, MLDebugger should behave exactly as it does now.

When the elf being debugged does have multiple partitions and the design is correctly being debug halted at the start of each partition, MLDebugger can now iterate through every partition that is run and dump out data for each.

@jvillarre
jvillarre requested a review from hackwa November 19, 2026 18:05
@hackwa

hackwa commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

High severity

  1. t20/t10 designs now run with telluride geometry from the very first partition. This is a definite regression.
    set_device() treats any args.device that is already set as if the user passed it. After one create_run_flags() call, args.device holds telluride (the base device), so the next call resolves sub_device = "telluride". Telluride has core_row_start 3; t20/t10 have 2.
    Before, only the second and later partitions were hit (a gotcha the memory already records). Now select_halted_partition() calls create_run_flags() on runs[0], and so does the Windows branch at mldebug_cli.py:529-533. Both run before launch() calls it again.
    So on the identify path every partition, including the first, is debugged with the wrong row offset. From the second halt on, the counters are read at the wrong rows (3..6 instead of 2..5), and that read will likely fail.
    Fix: keep the user's original -d value separately and restore it before each create_run_flags() call.

  2. The polling assumes every partition has the same footprint and the same hardware context.
    wait_for_next_partition() polls self.impls, which are the finished partition's tiles bound to its ctx_id.
    If the next partition uses fewer columns, the unused cores never halt and the wait times out.
    If each vaiml_par_N gets its own hardware context, the poll fails with the "hardware context no longer exists" error.
    _candidate_core_tiles() reads the counters from the union of all partitions' tiles and requires every tile to agree (_uniform_counter). Tiles outside the halted partition will hold stale values or be unreadable, which causes a spurious exit.
    Please confirm with the flexmlRT/driver owners whether subgraphs share one hardware context, and which cores the plugin halts and writes to. Then poll and read only the tiles the plugin actually covers.

Medium severity

  1. Interactive mode with -l3/-auto will hang at the partition boundary.
    In batch mode _handle_fsp closes the DebugServer before waiting, and its docstring says why: so the runtime can load the next partition.Interactive continue_execution() calls wait_for_next_partition() but never closes the server. FlexML keeps waiting for the terminate message, so the next partition never loads, and the wait times out after 5 minutes.

  2. The halt counters share hardware with skip_iter and the profilers.
    init_skip_iterations() and skip_iterations() take over perf counter 1 and PERF_CNTR_0_CTRL (set to count PC_0_CORE) and never restore them. That configuration stays live into the next partition, with a stale PC_EVENT0. Any counting between the plugin's write and the halt corrupts the halt count.

  3. A timeout loses the run summary for a partition that succeeded. _handle_fsp() now runs before _write_run_summary("SUCCESS"), and a timeout calls cleanup_and_exit(1). So run_summary.json is never written for a partition that finished fine, and automated_debug, which reads that file, sees stale state.

  4. Standalone mode: -s and --dump-aie-status with an unpinned multi-subgraph model on xrt go through identification. On a hung app without the plugin they exit, or silently use runs[0]'s metadata, when they could just show status. Exclude aie_only from identify.

@richard-byron

Copy link
Copy Markdown
Collaborator Author
  1. Users -d value is now kept and restored
  2. This is a non-issue. Currently MLDebugger does not support multiple-contexts, and the halt plugin writes to all cores in a context.
  3. Server is now closed down and restarted between partitions
  4. Non-issue. MLDebugger can't write to the halt counter between the breakpoint being set and the design resuming. If anything did, catching, it would lead to big problems and catching that with the halt counter is a feature, not a bug.
  5. I believe a timeout in this situation is actually a failure, even though some partitions may have been dumped correctly (that dumps for those partitions survive anyway). It did highlight an issue where the context going away between partitions was treated as a failure though, when this is actually the expected behavior. That is fixed.
  6. Standalone and dump-aie-status should behave exactly as they did before now, and not expect/need the new halt plugin

Some other things Cursor caught:

  • If include_subgraphs pointed at one graph that contained multiple parts, it would use the multi-partition flow, whereas the intention is that if include_subgraphs is set or halt_on_every_run in xrt.ini is not set, MLDebugger should behave exactly as it does now. That is fixed too.
  • dump_layers now reports all partitions unless include_subgraphs limits the design to a single partition
  • claude.md is updated to reflect all the changes

@hackwa

hackwa commented Nov 20, 2026

Copy link
Copy Markdown
Member

Claude found a bunch of moderate issues but nothing that's super high severity. I am merging it for now but please test on stx, ve2 etc.

@hackwa
hackwa merged commit 82a19be into Xilinx:main Nov 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants