You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Automatic dumping for multi-partition designs - #75
Allows automatic dumping of every partition in a design.
This should do nothing if the design only has one partition, include_subgraphs is set (limits the design to one NPU partition), or the halt plugin has signaled that the design will only halt on the first partition. In any of those cases, MLDebugger should behave exactly as it does now.
When the elf being debugged does have multiple partitions and the design is correctly being debug halted at the start of each partition, MLDebugger can now iterate through every partition that is run and dump out data for each.
t20/t10 designs now run with telluride geometry from the very first partition. This is a definite regression.
set_device() treats any args.device that is already set as if the user passed it. After one create_run_flags() call, args.device holds telluride (the base device), so the next call resolves sub_device = "telluride". Telluride has core_row_start 3; t20/t10 have 2.
Before, only the second and later partitions were hit (a gotcha the memory already records). Now select_halted_partition() calls create_run_flags() on runs[0], and so does the Windows branch at mldebug_cli.py:529-533. Both run before launch() calls it again.
So on the identify path every partition, including the first, is debugged with the wrong row offset. From the second halt on, the counters are read at the wrong rows (3..6 instead of 2..5), and that read will likely fail.
Fix: keep the user's original -d value separately and restore it before each create_run_flags() call.
The polling assumes every partition has the same footprint and the same hardware context.
wait_for_next_partition() polls self.impls, which are the finished partition's tiles bound to its ctx_id.
If the next partition uses fewer columns, the unused cores never halt and the wait times out.
If each vaiml_par_N gets its own hardware context, the poll fails with the "hardware context no longer exists" error.
_candidate_core_tiles() reads the counters from the union of all partitions' tiles and requires every tile to agree (_uniform_counter). Tiles outside the halted partition will hold stale values or be unreadable, which causes a spurious exit.
Please confirm with the flexmlRT/driver owners whether subgraphs share one hardware context, and which cores the plugin halts and writes to. Then poll and read only the tiles the plugin actually covers.
Medium severity
Interactive mode with -l3/-auto will hang at the partition boundary.
In batch mode _handle_fsp closes the DebugServer before waiting, and its docstring says why: so the runtime can load the next partition.Interactive continue_execution() calls wait_for_next_partition() but never closes the server. FlexML keeps waiting for the terminate message, so the next partition never loads, and the wait times out after 5 minutes.
The halt counters share hardware with skip_iter and the profilers.
init_skip_iterations() and skip_iterations() take over perf counter 1 and PERF_CNTR_0_CTRL (set to count PC_0_CORE) and never restore them. That configuration stays live into the next partition, with a stale PC_EVENT0. Any counting between the plugin's write and the halt corrupts the halt count.
A timeout loses the run summary for a partition that succeeded. _handle_fsp() now runs before _write_run_summary("SUCCESS"), and a timeout calls cleanup_and_exit(1). So run_summary.json is never written for a partition that finished fine, and automated_debug, which reads that file, sees stale state.
Standalone mode: -s and --dump-aie-status with an unpinned multi-subgraph model on xrt go through identification. On a hung app without the plugin they exit, or silently use runs[0]'s metadata, when they could just show status. Exclude aie_only from identify.
This is a non-issue. Currently MLDebugger does not support multiple-contexts, and the halt plugin writes to all cores in a context.
Server is now closed down and restarted between partitions
Non-issue. MLDebugger can't write to the halt counter between the breakpoint being set and the design resuming. If anything did, catching, it would lead to big problems and catching that with the halt counter is a feature, not a bug.
I believe a timeout in this situation is actually a failure, even though some partitions may have been dumped correctly (that dumps for those partitions survive anyway). It did highlight an issue where the context going away between partitions was treated as a failure though, when this is actually the expected behavior. That is fixed.
Standalone and dump-aie-status should behave exactly as they did before now, and not expect/need the new halt plugin
Some other things Cursor caught:
If include_subgraphs pointed at one graph that contained multiple parts, it would use the multi-partition flow, whereas the intention is that if include_subgraphs is set or halt_on_every_run in xrt.ini is not set, MLDebugger should behave exactly as it does now. That is fixed too.
dump_layers now reports all partitions unless include_subgraphs limits the design to a single partition
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allows automatic dumping of every partition in a design.
This should do nothing if the design only has one partition, include_subgraphs is set (limits the design to one NPU partition), or the halt plugin has signaled that the design will only halt on the first partition. In any of those cases, MLDebugger should behave exactly as it does now.
When the elf being debugged does have multiple partitions and the design is correctly being debug halted at the start of each partition, MLDebugger can now iterate through every partition that is run and dump out data for each.