Repository navigation
Conversation
|
Commit: Regression test reportSelected 5 changed tests (limit=20, 0 not selected) 5 tests: regression=1 unreleased=0 coverage=0 failing=4
TestAccept/cmd/aitools/auto-install/enable-disable: branch ✅ | main (4c8b7a1) ❌ | latest (v1.26.0) ❌main (4c8b7a1): latest (v1.26.0): TestAccept/cmd/aitools/auto-install/first-run: branch ❌ | main (4c8b7a1) ❌ | latest (v1.26.0) ❌branch: main (4c8b7a1): latest (v1.26.0): TestAccept/cmd/aitools/auto-install/retries: branch ❌ | main (4c8b7a1) ❌ | latest (v1.26.0) ❌branch: main (4c8b7a1): latest (v1.26.0): 2 more failure details omitted, see the full report in the run. |
✅ Integration test reportCommit: 15cf94d
Top 8 slowest tests (at least 2 minutes):
|
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 019d40c. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 2 blocking issue(s), tagged [blocking]; 1 open question about a change users could be hurt by.
This PR installs or updates Databricks AI tools after the first successful command of each new CLI version. It is gated off behind cli-compat and has opt-out, CI, holdout, retry and lock handling. The decision logic, locking and tests are careful, and CI is green. Two regressions ship even with the gate off: apps init drops the DATABRICKS_CLI_SELF_EXEC marker on its children, and the new process group stops Ctrl-C from reaching agent CLIs in the explicit aitools install/update. Several gaps should be closed before the gate is turned on: the skip list (auth docker token, CI providers, DBR/non-interactive runs), the gate-failure settle, a stuck partial first install, plugin migration hiding edits, and telemetry outcomes.
Questions
- [blocking] Auto-install skips only CI (
CI=true), but the version-check notice andMaybeHintalso skip Databricks Runtime, non-interactive runs and-o json. Scripts, IDE extensions and ephemeral homes (containers, DBR clusters) pay the gate lookup now: up to 3 s, once per version per fresh home. Once the gate is on, they also pay up to about 20 s, plus retries, after their command's output. Should auto-install reuse those skips, or is running there intended? If it is intended, a comment saying why would help.
Accepted answers
- Changelog fragment for this change
| cmd.Stdout = os.Stdout | ||
| cmd.Stderr = os.Stderr | ||
| cmd.Stdin = os.Stdin | ||
| cmd.Env = autoinstall.ChildEnv() |
There was a problem hiding this comment.
[blocking] cmd.Env = autoinstall.ChildEnv() replaces the cmd.Env = agents.SelfExecEnv(os.Environ()) set four lines above (line 1968). The same happens in runPostCreateDev (lines 1998 and 2002). The apps deploy / apps dev-remote children therefore lose DATABRICKS_CLI_SELF_EXEC=1, and MaybeHint (libs/aitools/agents/hint.go:196) no longer suppresses the coding-agent hint in them. Under Claude Code and other agents, the child prints the hint again inside the parent's output and logs a second hint event. This ships even while the gate is off. Let ChildEnv take the environ slice and compose the two, e.g. cmd.Env = agents.SelfExecEnv(autoinstall.ChildEnv(os.Environ())), at both call sites. That also stops it from reading os.Environ() in library code. A test that checks both markers would cover these call sites, which currently have none.
There was a problem hiding this comment.
Fixed in 879027f. ChildEnv now takes the environ, and both call sites use a new selfExecEnv(environ) in cmd/apps/init.go = agents.SelfExecEnv(autoinstall.ChildEnv(environ)), so the children keep DATABRICKS_CLI_SELF_EXEC=1. TestSelfExecEnvCarriesBothMarkers checks both markers, before and after a "no".
| out, err := process.Background(cctx, argv) | ||
| // A timed-out agent CLI is stopped with its descendants: one left running | ||
| // would hold the output pipes open and block past the timeout. | ||
| out, err := process.Background(cctx, argv, process.WithProcessGroup()) |
There was a problem hiding this comment.
[blocking] WithProcessGroup moves the agent CLI out of the terminal's foreground process group (here and in libs/aitools/agents/detect.go:115), so Ctrl-C no longer reaches it. Nothing in cmd/root or cmd/aitools catches SIGINT. Ctrl-C kills the CLI, the context is never cancelled, and claude plugin install / marketplace add keep running as orphans that modify ~/.claude. Before this PR, they were interrupted too. This changes the explicit aitools install / update today, for a feature that is off. cmd/environments/sync.go:92-101 documents the same effect for uv and adds a signal handler for that reason. With the gate on, it also releases the run lock while the orphan is still installing, which is the hazard used to exclude Windows. Either cancel the context on SIGINT/SIGTERM while agent CLIs run (see cmd/air/run.go:350 for restoring the default disposition), or use the process group only on the auto-install path.
There was a problem hiding this comment.
Fixed in 879027f, with the signal route so the explicit commands and auto-install both get it. New process.BackgroundInGroup (libs/process/interrupt_unix.go) runs the child with WithProcessGroup and, while it runs, catches the first SIGINT/SIGTERM. It calls signal.Stop first, so a second signal kills at once (the same order as cmd/air/run.go), then cancels, which stops the group, and then raises the signal again in the CLI. The CLI therefore ends as it did before the group, and nothing is orphaned or keeps running past the run lock. runAgentCmd and ProbePlugin use it. On Windows the child stays in the console group, so nothing is forwarded there. TestBackgroundInGroupForwardsInterrupt interrupts a helper process mid-run, then checks that it died by SIGINT and that its grandchild was reaped.
| return true | ||
| } | ||
| } | ||
| return slices.Equal(path, []string{"auth", "token"}) |
There was a problem hiding this comment.
[nice to have] isSkippedCommand exempts only the exact path auth token. docker-credential-databricks runs databricks auth docker token (libs/dockercredentials/shim.go:68) on pulls and pushes. So auto-install runs inside Docker's credential lookup: up to 3 s for the gate now, and an install plus a stderr line once the gate is on. agents.isCredentialHelperCommand (hint.go:268-279) already covers both paths, and versioncheck.isExemptCommand covers the rest of this list. Reuse them instead of a parallel list. That also replaces the hard-coded "__complete" strings with cobra's constants.
There was a problem hiding this comment.
Fixed in 879027f. The command list is gone from Decide; it takes a SkippedCommand bool. skipsAutoInstall(cmd) in cmd/aitools is versioncheck.IsExemptCommand || agents.IsCredentialHelperCommand || in an aitools tree. Both helpers are now exported, so auth docker token is covered and the cobra constants are used. Test: TestSkipsAutoInstall.
| Command: path, | ||
| CacheDisabled: autoinstall.CacheDisabled(ctx), | ||
| Version: version, | ||
| CI: versioncheck.InCI(ctx), |
There was a problem hiding this comment.
[nice to have] versioncheck.InCI is env.GetBool(ctx, "CI"). It misses CI=<provider name>, which GetBool maps to false. It also misses providers that don't set CI, such as Azure Pipelines (TF_BUILD), Jenkins and CodeBuild, which useragent.CiCdProvider() detects. libs/aitools/agents/hint.go:253-266 (inCI) handles both cases, and its comment explains why GetBool can't be used. Such build agents are treated as user machines: the gate lookup today, and installs once the gate is on. Reuse inCI.
There was a problem hiding this comment.
Fixed in 879027f. It now uses agents.InCI (exported), so CI=<provider> and SDK-detected providers such as TF_BUILD count. versioncheck.InCI is reverted, so notice.go only has the IsExemptCommand export now.
| } | ||
| // Skills without install state have no checksums to tell the user's edits | ||
| // from ours, and an install would replace them. | ||
| if installer.HasUntrackedSkills(ctx, installer.ScopeGlobal) { |
There was a problem hiding this comment.
[nice to have] InstallSkillsForAgents writes skill directories one by one and saves .state.json only after the loop (installer.go:478). A first attempt that hits the 10 s installTimeout partway leaves skills on disk with no state, and counts as FAILED. On the next run, HasUntrackedSkills is true, so planTools takes the keep path and the run settles INSTALLED. Every later version takes the same path. The machine keeps a partial skill set forever and reports success. Tell the CLI's own interrupted install apart from a legacy one (for example, save state incrementally, or write a marker before the first skill), and retry it.
There was a problem hiding this comment.
Fixed in 879027f with a marker. Before a fresh install, the state gets installing: true, which only settling clears. planTools takes the keep path for untracked skills only when that marker is not set, so an install cut short by the timeout is finished (and retried) on a later run, while hand-made or legacy skills are still left alone. Tests: TestPlanToolsKeepsSkillsWithoutInstallState (second half) and TestRecordClearsInstallingWhenSettled; the retries golden shows the marker.
| return installResult{}, err | ||
| } | ||
| result := installResult{updated: true} | ||
| if after != nil && toolsChanged(before, after) { |
There was a problem hiding this comment.
[nit] With KeepModified, UpdateSkills still advances state.Release to the new tag (update.go:225) even when every outdated skill was kept. toolsChanged then sees a different Release, and the notice says the tools were updated when no file changed. It also lists every agent with a global install, not just the ones that changed. Compare skills and plugin versions only, or have UpdateSkills report whether anything changed.
There was a problem hiding this comment.
Fixed in 879027f. toolsChanged became changedAgents, which ignores Release and names an agent only if it has skills and a skill version changed, or if its own plugin version changed. With no changed agent, the run reports UP_TO_DATE and prints nothing. Test: TestChangedAgents.
| // explicit choice matters more than a broken record. | ||
| func SetOptedOut(ctx context.Context, optedOut bool) error { | ||
| s, err := Load(ctx) | ||
| unlock, err := lock(ctx, stateLockFileName, stateLockWait) |
There was a problem hiding this comment.
[nit] SetOptedOut now requires the state lock. On a home directory whose filesystem rejects flock (some NFS/FUSE mounts), lock fails at once, so databricks aitools auto-install disable exits 1. In the base it just wrote the file. Auto-install can't run there either, because Load fails. Consider letting the explicit opt-out write without the lock when locking is unsupported (not merely held).
There was a problem hiding this comment.
Done in 879027f. SetOptedOut still returns errLockHeld when another process holds the lock. On any other lock error it writes without the lock (logged at debug), since auto-install cannot run there anyway. Test: TestSetOptedOutWithoutLocks.
| } | ||
| // Commands and builds that never auto-install skip before the state is | ||
| // read, and without a log line: they are most runs. | ||
| if reason := autoinstall.Decide(in); reason == autoinstall.ReasonSkippedCommand || reason == autoinstall.ReasonDevelopmentBuild { |
There was a problem hiding this comment.
[nit] The early return covers only skipped commands and development builds. With the cache disabled, the run still calls Load, which creates ~/.databricks/aitools/ and auto-install.json.lock. Add ReasonCacheDisabled to the early return.
There was a problem hiding this comment.
Fixed in 879027f. The early return now covers skipped command, Databricks Runtime, cache disabled and development build (stateFreeSkips), so none of them touches ~/.databricks/aitools/.
| state, err := autoinstall.Load(ctx) | ||
| require.NoError(t, err) | ||
| assert.True(t, state.OptedOut) | ||
| assert.True(t, autoinstall.Declined()) |
There was a problem hiding this comment.
[nit] This test leaves the package-global autoinstall.declined flag set, and the agents package can't reset it. Separately, TestChildEnvCarriesTheRefusal (libs/aitools/autoinstall/run_test.go:102) fails when the developer has DATABRICKS_AITOOLS_AUTO_INSTALL=false set, because ChildEnv reads os.Environ(). Either export a test reset or pass the environ in, and clear EnvVar in that test.
There was a problem hiding this comment.
Fixed in 879027f. Added autoinstall.ResetDeclined(), which TestRecommendSkillsInstallInteractiveDecline and the new apps test pass to t.Cleanup. ChildEnv takes the environ, so TestChildEnvCarriesTheRefusal no longer reads the developer's environment.
| $CLI completion bash > /dev/null 2>&1 | ||
| $CLI __complete jobs > /dev/null 2>&1 | ||
| $CLI help jobs > /dev/null 2>&1 |
There was a problem hiding this comment.
[nit] .agents/rules/testing.md asks to route noisy output to LOG.<name> rather than /dev/null, and to capture a tolerated non-zero exit with errcode rather than || true. cursor_skills in script.prepare:63 uses || true; a [ -d ... ] check, as auto_install_state already does, avoids it.
There was a problem hiding this comment.
Fixed in 879027f. cursor_skills checks [ -d ] instead of using || true, and the completion/__complete/help runs in skipped/script go to LOG.* instead of /dev/null.
First of three changes that install Databricks AI tools automatically on the first run of a new CLI version. This one stores the state and the opt-out and adds the remote gate; nothing installs yet. - ~/.databricks/aitools/auto-install.json holds last_version, attempts and opted_out. It is a dedicated file, not a libs/cache entry, so an opt-out survives upgrades and `databricks cache clear`. - `databricks aitools auto-install disable|enable` toggles the opt-out; enable warns when DATABRICKS_AITOOLS_AUTO_INSTALL=false is set. - A global-scope `databricks aitools uninstall` (including --skills and --agents) opts out. - cli-compat.json entries get an optional auto_install field, resolved by clicompat.ResolveAutoInstall. No entry sets it, and TestManifestLeavesAutoInstallOff keeps it off. Co-authored-by: Isaac <no-reply@databricks.com>
258418b to
c5eef9c
Compare
After a command succeeds, the root PersistentPostRunE installs (or updates) Databricks AI tools for the coding agents on the machine, once per new CLI version. It ships off: no cli-compat entry sets the gate. - autoinstall.Decide is a pure decision over the command, version, state, CI, opt-out, gate and holdout; skipped commands and development builds return before any state read. - A fresh install runs as `aitools install --scope=global` without prompts for agents whose config dir exists; an existing global install runs `aitools update --scope=global`, keeping edited skills. The run has a 10s budget, discards installer output and prints one notice line. - State is updated under OS file locks that are only tried, never waited on beyond 1s. Attempts are retried up to 10 times per version. - Agent CLIs run in their own process group so a timeout stops their descendants. Windows cannot do that, so auto-install skips plugin installs there. - Answering "no" at the `apps init` skills prompt opts out and is passed to the CLI children `apps init` starts. - Telemetry: aitools_install_event.trigger = AUTO_FIRST_RUN and a new aitools_auto_install_event. - Refactor: `aitools update`'s RunE body moves to runUpdate, versioncheck.InCI is exported, and the print_telemetry acceptance helper moves up a directory. Co-authored-by: Isaac <no-reply@databricks.com>
019d40c to
e5662ad
Compare
- A global uninstall opts out only when it removed something, so a mistyped --skills or an empty scope no longer turns auto-install off. - `aitools auto-install <typo>` now fails instead of printing help and exiting 0. - Document auto_install in internal/build/README.md (format, the copy-forward step for new entries, the trust model). - Say that any non-true DATABRICKS_AITOOLS_AUTO_INSTALL value opts out, and move EnvVar next to the other constant. - Add unit tests for the state file; pin the env var in the acceptance test and log the reset steps instead of discarding them. Co-authored-by: Isaac <no-reply@databricks.com>
- apps init children keep DATABRICKS_CLI_SELF_EXEC: ChildEnv takes the environ and composes with agents.SelfExecEnv. - Agent CLIs run through process.BackgroundInGroup, which forwards the first SIGINT/SIGTERM to the child's process group and then re-raises it, so Ctrl-C stops `claude plugin install` again in aitools install/update. - Skips reuse versioncheck.IsExemptCommand, agents.IsCredentialHelperCommand (covers `auth docker token`) and agents.InCI (CI=<provider>, TF_BUILD...), and add Databricks Runtime. Cache-disabled runs no longer read the state. - A gate lookup that only reaches the embedded manifest fails instead of reading as off; it uses up an attempt and does not settle the version. - An unfinished first install is marked in the state, so its skills are finished on a later run instead of being kept as untracked forever. - Auto-install updates no longer migrate or remove raw skills. - New outcomes UP_TO_DATE, KEPT and SKIPPED_PLATFORM; the update notice names only the agents whose skills or plugin changed. - Saying "no" in apps init says that auto-install is off too. - The explicit opt-out writes without the lock where locks are unsupported. - Tests: ResetDeclined, environ passed in; acceptance helpers use LOG.* and no `|| true`. Co-authored-by: Isaac <no-reply@databricks.com>
|
Thanks for the review. All 14 threads are addressed in 879027f; there's a reply on each one. The PR description is updated too. Re question 1 [blocking], the skips:
Once the gate is on, the cost after a script's first command of a version is still there: up to about 20 s, plus retries. I've left that as is, because a script can't be told apart from an agent here. Also: the lumberjack proto needs the new outcomes |
…nstall-first-run Co-authored-by: Isaac <no-reply@databricks.com>
…ests agents.InCI also detects providers such as GITHUB_ACTIONS, so on CI runners every case reported SKIPPED_CI. Co-authored-by: Isaac <no-reply@databricks.com>
Stacked on #7701, the PR that stores the auto-install state and adds the opt-out commands and the cli-compat gate.
After a command succeeds, the CLI installs (or updates) Databricks AI tools for the coding agents on the machine, once per new CLI version. Ships off: nothing sets the gate yet.
Summary
last_version = current, attempts = 0and clearsinstalling: each machine reports at most one outcome per version, plus one per retry. A new version resets the retry budget.flock/LockFileEx) on files that are never removed. They are only tried, never waited on. The state lock is retried for up to 1 s while another process holds it; any other lock error returns at once. A busy run lock skips the attempt.process.BackgroundInGroup), so a timeout stops their descendants too. That takes them out of the terminal's foreground group, so while one runs, the first SIGINT/SIGTERM is caught, stops the group, and is raised again in the CLI: Ctrl-C endsaitools install/updateas before, without orphans. Windows has no such stop, so there auto-install runs no agent CLI: a fresh install drops its plugin installs (skills are still installed), and an existing install whose update would touch plugins is left alone. A Windows machine with only plugin-capable agents settles asSKIPPED_NO_AGENTS.apps initskills prompt opts out, and says so with theauto-install enablecommand. It is stored, honored in the process even if the write fails, and passed asDATABRICKS_AITOOLS_AUTO_INSTALL=falseto theapps deploy/apps dev-remotechildren thatapps initstarts, next to theirDATABRICKS_CLI_SELF_EXEC=1.version,help, completion), the credential helpers (auth token,auth docker token) and theaitoolstrees. Non-interactive and-o jsonruns are not skipped, unlike for the notice: coding agents run the CLI that way, and auto-install writes nothing to stdout. Databricks Runtime is skipped (ephemeral home, platform-managed CLI).clicompat.ResolveAutoInstallfails): the version is not settled, and the lookup uses up one of the version's 10 attempts, so an offline machine stops paying for it.HOLDOUT/TREATMENT) is reported.DATABRICKS_AITOOLS_AUTO_INSTALL_CLI_VERSIONto simulate releases and upgrades.aitools_install_event.trigger = AUTO_FIRST_RUN(unset for the command's own installs, so existing goldens do not change) and a newaitools_auto_install_event {outcome, holdout_arm, interactive, error_category}, documented inlibs/telemetry/protos/README.md. Outcomes:INSTALLED,UP_TO_DATE,KEPT,SKIPPED_CI,SKIPPED_OPTOUT,SKIPPED_HOLDOUT,SKIPPED_NO_AGENTS,SKIPPED_PLATFORM,FAILED,RETRIES_EXHAUSTED; the lumberjack proto needs the same values.aitools update's RunE body moves torunUpdate;agents.InCI,agents.IsCredentialHelperCommandandversioncheck.IsExemptCommandare exported for reuse; the acceptance helperprint_telemetrymoves up a directory.Known limitations
Both must be fixed before the gate is turned on:
aitools uninstallthat runs while an automatic install is in flight can see that attempt recreate the tools. Running uninstall again removes them.INSTALLED, and later updates never add the missing agent. The same applies to the agents the Windows rule above leaves out. One fix is to record the targeted agents inauto-install.json. (An install cut short by the timeout is different: itsinstallingmarker makes the next run finish it.)Review order
libs/aitools/autoinstall/decide.go: the decision, pure.cmd/aitools/autoinstall_run.go: the hook,planTools, reporting.libs/aitools/autoinstall/run.go,lock_unix.go,lock_windows.go,state.go: locks, attempts,Decline/ChildEnv.cmd/cmd.go,cmd/apps/init.go,libs/aitools/agents/recommend.go: the wiring.cmd/aitools/update.go(refactor),libs/aitools/installer/update.go(KeepModified,HasUntrackedSkills),libs/aitools/installer/plugin.go(process group),libs/telemetry/protos/.Evidence
New acceptance tests in
acceptance/cmd/aitools/auto-install/rundatabricks jobs listagainst the fake server and a fake skills source, with a temporary HOME and agent dirs:first-run:upgrade: 1.30.0 installs skills release 0.3.0; 1.31.0 with release 0.4.0 printsUpdated Databricks AI tools for Cursor., updates one skill and keeps a skill the user edited; 1.32.0 while the latest release is 0.2.0 refuses the downgrade (files unchanged, nothing printed, settled asKEPT, not retried); a 1.31.0 CLI run afterwards does nothing.skipped:CI=true(reported once per version),DATABRICKS_AITOOLS_AUTO_INSTALL=false,auto-install disable,version/aitools/completion/__complete/help,DATABRICKS_CACHE_ENABLED=false, gate off (nothing reported, version settled), holdout machine, no agent config dir.retries: nineFAILEDattempts,RETRIES_EXHAUSTEDon the tenth, nothing on the eleventh, a fresh install on the next version.TestDecide(table-driven over every input);TestSkipsAutoInstall;TestPlanToolsRunsNoAgentCLIWhereTimeoutsCannotStopThem,TestPlanToolsReportsWhyNothingIsInstalledandTestPlanToolsKeepsSkillsWithoutInstallState(also: an unfinished install of ours is finished);TestChangedAgents;TestTryLockExcludesOtherRuns,TestLoadWaitsForStateLock,TestLoadFailsWhenStateLockStaysHeld,TestLockReturnsOtherErrorsAtOnce,TestRecordKeepsConcurrentOptOut,TestSetOptedOutWithoutLocks,TestRecordClearsInstallingWhenSettled;TestChildEnvCarriesTheRefusal,TestSelfExecEnvCarriesBothMarkers(theapps initchildren's env);TestResolveAutoInstallFailsOnTheEmbeddedFallback;TestBackgroundInGroupForwardsInterrupt(a helper process interrupted mid-run dies by SIGINT and its grandchild is reaped);TestRunAgentCmdTimeoutStopsDescendants;TestUpdateKeepModifiedKeepsEditedSkill.cmd/...andlibs/..., the aitools and apps acceptance suites, lint and whitespace checks pass on Linux;GOOS=windows/darwinbuild and vet pass. The gate-lookup failure has no acceptance test: the manifest URL has no override, so it is covered by the clicompat and state unit tests.Merge Danger
Door: two-way while the gate is off
With the gate off it reads one small file and settles the version on the first run of each release, sends
SKIPPED_CI/SKIPPED_OPTOUTevents when those apply, and does nothing else. Turning the gate on (acli-compat.jsonedit on main) is the one-way step: installs it causes are not rolled back by turning it off again.Blast Radius: every command (post-run)
aitools install/updatetoday: agent CLIs run in their own process group, with Ctrl-C forwarded to it.~/.databricks/aitools/auto-install.jsonand two lock files next to it.This pull request and its description were written by Isaac.