Repository navigation
design: a foundation for schema generation and dependency management - #474
BigGold1310 wants to merge 1 commit into
Conversation
Signed-off-by: Cyrill Näf <cyrill.naef@gmail.com>
adamwg
left a comment
There was a problem hiding this comment.
Thanks for writing this up, @BigGold1310, and sorry for taking so long to review. This plan looks sound to me overall - I would definitely like to make schema generation more consistent across commands, and I like the idea of an updated lock file format to support better generation/re-generation. I've left a few suggestions and questions inline.
| } | ||
| } | ||
| }, | ||
| "apis": { |
There was a problem hiding this comment.
I wonder if we should call this local (to contrast with dependencies, which are external to the project). Thinking of this open PR that adds support for generating schemas from local CRDs in the project, outside of apis/ - I think that fits more with apis than dependencies, since there's no external source involved.
| staling every group in every project. The tradeoff: the KCL and Python | ||
| images are pinned in CLI source ([`kcl.go:54`][kcl-gen], | ||
| [`python.go:51`][python-gen]), so bumping either moves the fingerprint, | ||
| while a change to the in-process Go or JSON generator does not and surfaces | ||
| only on the next source change. |
There was a problem hiding this comment.
I'd prefer that this be consistent across languages. The simplest option would be to omit the fingerprint and not re-generate until a source changes or the users explicitly requests it (and we can make that simple with a new command like crossplane schema update). We can document in release notes when re-generation is recommended. This also wouldn't preclude implementing automatic re-generation in the future.
If we really want re-generation on generator changes to be automatic, we could figure out a fingerprinting mechanism for the built-in generators (rust, go, json). One way would be to have a "golden" example source, generate from it, and use a hash of the ouptut as the generator's fingerprint. Ideally we'd do this fingerprinting as part of the build process (or enforce that it's up-to-date via CI) so we're not running each generator on every call to figure out the fingerprint. Definitely interested if anyone else has ideas for how to do this.
| modified `apis/`. A dependency has both; `apis/` has only `hash`, having no | ||
| upstream pin. |
There was a problem hiding this comment.
Not sure I understand this. I believe hash is intended to be a hash of the output (the generated schemas), while pin is a content-based identifier for the input (git commit hash, xpkg digest, etc.). I believe that idea is still relevant for apis - the pin would be a hash of the input files, while hash is a hash of the output files. This is how we determine today whether apis needs to have schemas re-generated.
| generator whose output describes an entire run — TypeScript's npm package, | ||
| Rust's crate — implements it, and the engine merges parts instead of letting |
There was a problem hiding this comment.
The Python and json generators could use this as well:
- The Python generator produces a
pyproject.tomlfile that is shared by all modules (but its contents is the same regardless of what was generated, so it can safely live in the generator). - The JSON generator needs to construct an
index.schema.jsonfile based on the generated files. This is done as a post-process step in the manager today (same as the RustCargo.tomlgeneration), but it would be nice for it to live in the generator instead.
| separate "which sources fed this file" bookkeeping. Per-resource hashes | ||
| don't change regeneration granularity, since a changed source's own `hash` | ||
| still invalidates its whole group; they let `Verify` report *which* resource | ||
| drifted, and let a later `apiGroups` filter be reasoned about straight from |
There was a problem hiding this comment.
How does this work? I thought the hash recorded in resources was the hash of the input (CRD or OpenAPI doc), but Verify would need a hash of the output to determine which resources have drifted, wouldn't it? And wouldn't that also need to be per-language?
Description of your changes
Adds a one-pager proposing a shared foundation for schema generation and
dependency resolution. Document only — no code changes.
Schema generation is wired independently into seven commands today, and each
wiring drops something another one remembers. Four filed issues (#215, #428,
#242, #370) plus one unfiled bug — pins silently moving on unrelated builds —
all trace back to the same missing piece: there is no single implementation
to fix once.
The proposal is that piece: a
Resolver("what does this project dependon?") and an
Engine("turn that into files"), reachable only through oneconstructor, writing one versioned, grouped lock. Nothing generates until a
full resolved set exists, and groups are regenerated whole and swapped in
atomically — which is what closes #215 and #428 structurally rather than
per-call-site.
Two constraints shape it: no behavior change for existing users in the PR
that lands it, and no second foundational rewrite — it has to accommodate
run-level generators (#170, #374), resource-level filtering, and a future
crossplane schemacommand group without being reshaped. A 7-step roadmapshows each of those as a layer-3 change.
#435 and #430 reach the same diagnosis by conflicting routes; the
Alternatives section explains why this adopts their ideas as a foundation
rather than picking either as a replacement.
I have:
[ ] Run./nix.sh flake checkto ensure this PR is ready for review.[ ] Added or updated unit tests.[ ] Linked a PR or a docs tracking issue to document this change.[ ] Addedbackport release-x.ylabels to auto-backport this PR.Need help with this checklist? See the cheat sheet.