Skip to content

ci: upload the behavioral report and transcripts as an artifact - #44

Open
johnl-amd wants to merge 1 commit into
mainfrom
upload-behavioral-report
Open

johnl-amd wants to merge 1 commit into
mainfrom
upload-behavioral-report

Conversation

@johnl-amd

@johnl-amd johnl-amd commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

The routing job already uploads its report. The behavioral jobs don't, so when a behavioral case fails the only record that outlives the run is the summary table: a case id, an expectation, and a judge sentence truncated at 160 characters.

That isn't enough to tell a skill problem from an environment one. A case that fails in CI and passes everywhere else leaves nothing to compare against, because the transcript of what the agent actually did is written to .skillscope/ on the runner and thrown away with it.

Concretely, chasing one such case this week meant re-running CI repeatedly to re-read a summary line, while the same case passed 29 times locally. The transcripts that would have settled it were produced on every one of those runs and discarded each time.

skillscope already writes both halves there by default, the JSON report under runs/ and the inspect transcript under logs/. So this adds no arguments, changes no behaviour, and keeps what is already being produced.

  • behavior and behavior-scoped each gain an Upload behavioral report step, matching the one routing already has.
  • if: always(), because a failed leg is exactly when the transcript is wanted.
  • Artifact names carry the matrix keys, since one job uploads per skill and OS, and per environment on the scoped leg.
  • path is the .skillscope/ directory rather than named files. Routing can name its files because it passes --output routing-report.json --keep-logs routing-logs; the behavioral calls pass neither, so their output lands in the defaults, .skillscope/runs/ and .skillscope/logs/. Those two are the only paths anything writes under that directory, so naming them and taking the directory cover the same files.
  • if-no-files-found: warn, same as routing, so a leg that produced nothing doesn't fail the run.

ROCm/rocm-cli's docs/skill-quality.md lists this missing artifact as a known gap under "What is not available yet". That bullet can go once this lands.

@johnl-amd johnl-amd left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (a single-file, 16-line addition to .github/workflows/skill-evals.yml). No ticket is named; the PR description is the only stated intent.

Blocking

  • The description cites a doc file that does not exist — PR description, final paragraph: docs/skill-quality.md.
    The PR justifies not touching any documentation by saying "docs/skill-quality.md already lists the missing artifact as a known gap under 'What is not available yet'." There is no docs/skill-quality.md in this repository (the docs/ tree on this branch contains only authoring-evals.md, harnesses.md, and usage.md), and a repo-wide code search for both skill-quality and the phrase What is not available yet returns zero hits. The claim that this gap is "already" documented elsewhere is false, which means the stated reason for leaving docs untouched doesn't hold.
    Confidence 85 · documentation · Fix: correct the description — either point at wherever this gap actually is tracked, or drop the claim and add a line to README.md/docs/usage.md noting the artifact now exists (and that it didn't before).

Non-blocking

None

Decisions for the author

  • Uploading the whole .skillscope/ directory vs. named paths — tradeoff.
    The existing routing job's artifact step lists exact paths (routing-report.json, routing-logs/). The new steps instead glob the entire .skillscope/ directory. That's more future-proof against whatever skillscope writes there next, but less precise than the routing job's approach right next to it, and would silently sweep in anything else ever written under that path. Worth confirming this breadth is intentional rather than an inconsistency with the sibling job.
  • Behavioral transcripts are richer and uncapped compared to routing's — tradeoff, see Deployment notes.

Positive signals

  • The new steps are a faithful, minimal mirror of the existing routing job's upload step — same if: always() reasoning, same if-no-files-found: warn, and artifact names correctly extended with the matrix keys (skill/os/environment) so two legs never collide on one name, which is required since upload-artifact@v4 fails on a duplicate name within a run.
  • The claim that this "changes no behaviour" holds up: traced cmd_behavioral → engine/behavioral.py::run(), which already writes the JSON report to .skillscope/runs/ and the inspect transcript to .skillscope/logs/ by default (log_dir=log_dir or str(Path(".skillscope") / "logs")) with no new CLI argument introduced by this change.

Deployment notes

  • This repo is public, so these artifacts become world-downloadable the moment a run completes. That risk is not new — the routing job already publishes full inspect transcripts the same way — but it is worth noting that behavioral runs have no tool-call cap (routing caps at --max-tool-calls/--max-inspection-calls), run to a much higher message limit, and are not scrubbed or redacted anywhere in the codebase before being written or uploaded. Credentials themselves are not at risk (resolved only into environment variables and GitHub's own masked $GITHUB_ENV, never into the report or transcript, confirmed by reading credentials.py and the harness env-wiring), but whatever a test case's tool calls touch (file contents, shell output, etc.) lands in a now-public artifact exactly as routing's already does. Flagging for awareness, not as a defect — this PR is consistent with precedent already accepted in this repo.

What this covered

Read the full diff and the complete current .github/workflows/skill-evals.yml on commit 34f2e41, plus README.md, docs/usage.md, .gitignore, skillscope/cli.py, skillscope/engine/behavioral.py, and skillscope/engine/models.py to verify the PR's factual claims about default output paths and credential handling. A separate pass, run by an independent worker with no exposure to the PR description or this report, specifically investigated whether secrets or other sensitive content could end up in .skillscope/ and whether a redaction mechanism exists; its findings are folded into Deployment notes above.

Did not run: history/prior-changes passes beyond git log on the touched file (no earlier PR or review thread touches this artifact-upload pattern); agent-instruction-adherence (no CLAUDE.md/AGENTS.md in this repository). No CI check in this PR's run validates the workflow YAML itself (the file is a reusable workflow invoked by callers, not self-triggering), so YAML correctness here rests on manual reading rather than a check.

One caveat on independence: the driver (this review) read the PR's title and description together with the diff in its first lookup, before splitting into separate passes, so the direct read of the workflow diff itself was not blind to stated intent — only the delegated security-impact investigation was. Given how small and mechanical this diff is (a straight copy of an existing, adjacent block), the risk from that is low, but it is disclosed here rather than left implicit. Not reconciled against any other discussion of this PR.

@johnl-amd johnl-amd added the agent-hub-reviewed agent-hub has reviewed this label Nov 13, 2026
@johnl-amd

Copy link
Copy Markdown
Collaborator Author

On the blocking finding: the doc does exist, just not in this repo. It's docs/skill-quality.md in ROCm/rocm-cli, on main, under What is not available yet:

A kept series. The harness writes its JSON report on the runner, but the behavioral run did not upload it as an artifact, so the figure lives in the job summary and the run's log retention, not in a stored history.

So the claim holds, the description just never said which repo, and a repo-wide search here was always going to come up empty. I've qualified it. The follow-up belongs on the rocm-cli side, that bullet goes once this merges.

On .skillscope/ versus routing's named paths: deliberate, and now spelled out in the description. Routing only gets to name its files because it passes --output routing-report.json --keep-logs routing-logs. The behavioral calls pass neither, so output goes to the defaults, .skillscope/runs/ and .skillscope/logs/, which are the only paths anything writes under there. So the directory and an explicit list cover the same files, and the directory keeps covering them if a default moves.

The public-artifact note is fair and matches what routing already does, so nothing changes here on that.

The routing job already uploads its report. The behavioral jobs do not,
so when a behavioral case fails, the only record that outlives the run
is the summary table: a case id, an expectation, and a truncated judge
sentence.

That is not enough to tell a skill bug from an environment one. A case
that fails in CI and passes everywhere else leaves nothing to compare,
because the transcript of what the agent actually did is written to
.skillscope/ on the runner and then discarded with it.

skillscope already writes both halves there by default, the JSON report
under runs/ and the inspect transcript under logs/, so this adds no
arguments and changes no behaviour. It keeps what is already produced.

Artifact names carry the matrix keys, since one job uploads per skill
and OS, and per environment on the scoped leg.

Signed-off-by: John Lybeck <john.lybeck@amd.com>
@johnl-amd
johnl-amd force-pushed the upload-behavioral-report branch from 34f2e41 to d628914 Compare November 16, 2026 08:24

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · d628914

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (1 file, 16 added lines, one commit).
No ticket named. The description says both behavioral jobs now upload their report and transcripts and "adds no arguments, changes no behaviour".

Blocking

  • Both new upload steps will silently upload nothing, because .skillscope/ is a hidden directory (new) — .github/workflows/skill-evals.yml:642, .github/workflows/skill-evals.yml:707
    • actions/upload-artifact@v4 resolves to v4.4.0 or later. Since v4.4.0 it excludes hidden files by default: anything starting with ., or inside a directory starting with ..
    • Naming the hidden path explicitly in path does not bypass the exclusion; the fix is include-hidden-files: true.
    • if-no-files-found: warn keeps the leg green with only a warning, so the PR's goal (keeping transcripts after a failed leg) is not met and nothing visibly fails.
    • Basis: upload-artifact v4.4.0 release notes and issues #602 / #610. Not run on a runner.
    • The existing routing upload avoids the hidden directory: it passes --output routing-report.json --keep-logs routing-logs and uploads those visible paths (skill-evals.yml:566-581); reusable.yml:340-352 and reusable.yml:380-406 follow the same pattern. The description's argument that naming the paths and taking the directory cover the same files holds for the files but misses the hidden-directory filter.
      Confidence 85 · mechanical · Fix: add include-hidden-files: true under with: in both steps, or follow the routing pattern (add --output behavioral-report.json --keep-logs behavioral-logs to the behavioral args and upload those visible paths). Consider if-no-files-found: error so a leg that uploads nothing is not silent.

Non-blocking

None

Decisions for the author

  • Uploading raw agent transcripts as artifacts — tradeoff
    For: the transcripts are what tells a skill bug from an environment bug. Against: the .eval logs can carry whatever the agent saw or printed, including, on the scoped legs, environments with their own secrets; artifacts are readable by anyone with read access to the repo. Confirm that is acceptable, or consider a shorter retention-days.
  • Behavioral outputs now have two conventions — non-blocking-improvement
    Routing uses explicit visible paths; the new steps rely on tool defaults under a hidden directory. Explicit flags in both would be uniform and also resolve the blocking finding.

Positive signals

  • if: always() on the upload, so failed legs still keep their transcripts.
  • Artifact names include matrix.os and matrix.environment, so matrix legs do not collide under v4's unique-name rule.

Deployment notes

None

What this covered

  • Every line of the diff (prw-base...HEAD, head d628914), plus the surrounding job definitions in skill-evals.yml, the reusable.yml routing and behavioral jobs, and the --output / --keep-logs options in skillscope/cli.py.
  • The PR description was read only after the findings were settled; no comments or reviews were read.
  • Sign-off check ran: checked: 1, missing: []. No tests in scope (workflow-only change).
  • DEGRADED REVIEW: no fan-out; the change is one 16-line workflow file, reviewed in a single pass.
  • Did not run: CI rollup read; design-question workers (considered inline); history and prior-changes passes (nothing earlier touches the added lines).
  • Not reconciled against any discussion here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-hub-reviewed agent-hub has reviewed this

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants