Repository navigation
fix(schema): normalize and cross-check scraped ministry names - #712
Merged
Merged
Conversation
The scraper now keeps only official ministry-page links, converts typographic quotes to ASCII, and fails when that list does not match the cabinet ministers page.
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are identified for the ministry-name scraper changes.
No open issues 🎉
What changed in this PR
Hardens the schema’s ministry-name scraper to exclude stale links and validate names against the cabinet ministers page before updating the schema. The regenerated schema remains separate in #710.
Changes:
- Restricts scraping to ministry-page links and normalizes typographic quotes and whitespace.
- Rejects empty, duplicate, or mismatched ministry lists without falling back to either source.
- Adds unit tests for filtering, normalization, cabinet-title parsing, and list validation.
| File | Description |
|---|---|
schema/script/test/index.test.js |
Adds coverage for scraping and cross-source validation. |
schema/script/index.js |
Filters and normalizes ministry names, then validates both sources before saving. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ministry scraper was writing typographic apostrophes and leftover stale names from the ministries page into the schema.
What changes: The scraper keeps only official ministry-page links under
/ministries-organizations/ministries/<slug>, converts typographic quotes to ASCII apostrophes, and fails the scheduled job when that list is empty, has duplicates, or does not match the cabinet ministers page. It does not fall back to either source. Leftover items that are not official ministry pages, such as the current Tourism, Arts, Culture and Sport/gov/content?id=link, are excluded from the scrape.What could go wrong: If the ministries page and the cabinet page disagree after a shuffle, the scheduled job will fail and will not open a schema pull request until the two pages agree. This pull request does not change stored pubcode records or migrate old ministry names.
How it was verified: 15 unit tests passed in a
node:26container. Running the script locally against live gov.bc.ca pages (2026-11-21, America/Vancouver) passed validation and produced 18 names with a straight apostrophe in Citizens' Services and Technology, and without Tourism, Arts, Culture and Sport. The regenerated schema is applied on #710, not in this pull request.The scheduled ministry-names job opens a new unique branch only when no open pull request already has the title "fix(schema): update ministry names from gov.bc.ca", so it will not overwrite #710 while that pull request stays open.
Thanks for the PR!
Deployments, as required, will be available below:
Please create PRs in draft mode. Mark as ready to enable:
After merge, new images are deployed in: