Skip to content

fix(schema): normalize and cross-check scraped ministry names - #712

Merged
DerekRoberts merged 1 commit into
mainfrom
fix/ministry-list-validation
Nov 21, 2026
Merged

DerekRoberts merged 1 commit into
mainfrom
fix/ministry-list-validation

Conversation

@DerekRoberts

@DerekRoberts DerekRoberts commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

The ministry scraper was writing typographic apostrophes and leftover stale names from the ministries page into the schema.

What changes: The scraper keeps only official ministry-page links under /ministries-organizations/ministries/<slug>, converts typographic quotes to ASCII apostrophes, and fails the scheduled job when that list is empty, has duplicates, or does not match the cabinet ministers page. It does not fall back to either source. Leftover items that are not official ministry pages, such as the current Tourism, Arts, Culture and Sport /gov/content?id= link, are excluded from the scrape.

What could go wrong: If the ministries page and the cabinet page disagree after a shuffle, the scheduled job will fail and will not open a schema pull request until the two pages agree. This pull request does not change stored pubcode records or migrate old ministry names.

How it was verified: 15 unit tests passed in a node:26 container. Running the script locally against live gov.bc.ca pages (2026-11-21, America/Vancouver) passed validation and produced 18 names with a straight apostrophe in Citizens' Services and Technology, and without Tourism, Arts, Culture and Sport. The regenerated schema is applied on #710, not in this pull request.

The scheduled ministry-names job opens a new unique branch only when no open pull request already has the title "fix(schema): update ministry names from gov.bc.ca", so it will not overwrite #710 while that pull request stays open.


Thanks for the PR!

Deployments, as required, will be available below:

Please create PRs in draft mode. Mark as ready to enable:

After merge, new images are deployed in:

The scraper now keeps only official ministry-page links, converts typographic quotes to ASCII, and fails when that list does not match the cabinet ministers page.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the ministry-name scraper changes.

No open issues 🎉

What changed in this PR

Hardens the schema’s ministry-name scraper to exclude stale links and validate names against the cabinet ministers page before updating the schema. The regenerated schema remains separate in #710.

Changes:

  • Restricts scraping to ministry-page links and normalizes typographic quotes and whitespace.
  • Rejects empty, duplicate, or mismatched ministry lists without falling back to either source.
  • Adds unit tests for filtering, normalization, cabinet-title parsing, and list validation.
File Description
schema/​script/​test/​index.test.js Adds coverage for scraping and cross-source validation.
schema/​script/​index.js Filters and normalizes ministry names, then validates both sources before saving.

Review effort: Balanced

@DerekRoberts DerekRoberts self-assigned this Nov 21, 2026
@DerekRoberts DerekRoberts moved this to Active in DevOps (NR) Nov 21, 2026
@DerekRoberts
DerekRoberts merged commit dc8cdcc into main Nov 21, 2026
22 of 24 checks passed
@DerekRoberts
DerekRoberts deleted the fix/ministry-list-validation branch November 21, 2026 18:05
@github-project-automation github-project-automation Bot moved this from Active to Done in DevOps (NR) Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants