Skip to content

design: a foundation for schema generation and dependency management - #474

Draft
BigGold1310 wants to merge 1 commit into
crossplane:mainfrom
BigGold1310:design-schema-foundation
Draft

BigGold1310 wants to merge 1 commit into
crossplane:mainfrom
BigGold1310:design-schema-foundation

Conversation

@BigGold1310

Copy link
Copy Markdown
Contributor

Description of your changes

Adds a one-pager proposing a shared foundation for schema generation and
dependency resolution. Document only — no code changes.

Schema generation is wired independently into seven commands today, and each
wiring drops something another one remembers. Four filed issues (#215, #428,
#242, #370) plus one unfiled bug — pins silently moving on unrelated builds —
all trace back to the same missing piece: there is no single implementation
to fix once.

The proposal is that piece: a Resolver ("what does this project depend
on?") and an Engine ("turn that into files"), reachable only through one
constructor, writing one versioned, grouped lock. Nothing generates until a
full resolved set exists, and groups are regenerated whole and swapped in
atomically — which is what closes #215 and #428 structurally rather than
per-call-site.

Two constraints shape it: no behavior change for existing users in the PR
that lands it, and no second foundational rewrite — it has to accommodate
run-level generators (#170, #374), resource-level filtering, and a future
crossplane schema command group without being reshaped. A 7-step roadmap
shows each of those as a layer-3 change.

#435 and #430 reach the same diagnosis by conflicting routes; the
Alternatives section explains why this adopts their ideas as a foundation
rather than picking either as a replacement.

I have:

Need help with this checklist? See the cheat sheet.

Signed-off-by: Cyrill Näf <cyrill.naef@gmail.com>

@adamwg adamwg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for writing this up, @BigGold1310, and sorry for taking so long to review. This plan looks sound to me overall - I would definitely like to make schema generation more consistent across commands, and I like the idea of an updated lock file format to support better generation/re-generation. I've left a few suggestions and questions inline.

}
}
},
"apis": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should call this local (to contrast with dependencies, which are external to the project). Thinking of this open PR that adds support for generating schemas from local CRDs in the project, outside of apis/ - I think that fits more with apis than dependencies, since there's no external source involved.

Comment on lines +349 to +353
staling every group in every project. The tradeoff: the KCL and Python
images are pinned in CLI source ([`kcl.go:54`][kcl-gen],
[`python.go:51`][python-gen]), so bumping either moves the fingerprint,
while a change to the in-process Go or JSON generator does not and surfaces
only on the next source change.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer that this be consistent across languages. The simplest option would be to omit the fingerprint and not re-generate until a source changes or the users explicitly requests it (and we can make that simple with a new command like crossplane schema update). We can document in release notes when re-generation is recommended. This also wouldn't preclude implementing automatic re-generation in the future.

If we really want re-generation on generator changes to be automatic, we could figure out a fingerprinting mechanism for the built-in generators (rust, go, json). One way would be to have a "golden" example source, generate from it, and use a hash of the ouptut as the generator's fingerprint. Ideally we'd do this fingerprinting as part of the build process (or enforce that it's up-to-date via CI) so we're not running each generator on every call to figure out the fingerprint. Definitely interested if anyone else has ideas for how to do this.

Comment on lines +358 to +359
modified `apis/`. A dependency has both; `apis/` has only `hash`, having no
upstream pin.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand this. I believe hash is intended to be a hash of the output (the generated schemas), while pin is a content-based identifier for the input (git commit hash, xpkg digest, etc.). I believe that idea is still relevant for apis - the pin would be a hash of the input files, while hash is a hash of the output files. This is how we determine today whether apis needs to have schemas re-generated.

Comment on lines +437 to +438
generator whose output describes an entire run — TypeScript's npm package,
Rust's crate — implements it, and the engine merges parts instead of letting

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Python and json generators could use this as well:

  • The Python generator produces a pyproject.toml file that is shared by all modules (but its contents is the same regardless of what was generated, so it can safely live in the generator).
  • The JSON generator needs to construct an index.schema.json file based on the generated files. This is done as a post-process step in the manager today (same as the Rust Cargo.toml generation), but it would be nice for it to live in the generator instead.

Comment on lines +366 to +369
separate "which sources fed this file" bookkeeping. Per-resource hashes
don't change regeneration granularity, since a changed source's own `hash`
still invalidates its whole group; they let `Verify` report *which* resource
drifted, and let a later `apiGroups` filter be reasoned about straight from

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this work? I thought the hash recorded in resources was the hash of the input (CRD or OpenAPI doc), but Verify would need a hash of the output to determine which resources have drifted, wouldn't it? And wouldn't that also need to be per-language?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

crossplane dependency update-cache does not add XRD types into schemas/ (inconsistent with crossplane project build)

2 participants