Skip to content

Fix DNS driver API inconsistencies - #2066

Closed
aaron-sierra wants to merge 48 commits into
apache:trunkfrom
aaron-sierra:dns-consistent-api
Closed

aaron-sierra wants to merge 48 commits into
apache:trunkfrom
aaron-sierra:dns-consistent-api

Conversation

@aaron-sierra

@aaron-sierra aaron-sierra commented Apr 3, 2025 •

Copy link
Copy Markdown

Fix DNS driver API inconsistencies

Description

Implement #2065

Summary

In general, commit messages provide necessary background regarding the changes made and why.

  • Drop Zerigo DNS
  • Deprecate outdated DNSimple DNS driver
  • Deprecate outdated RcodeZero DNS driver
  • Ignore AuroraDNS (see Remove unavailable cloud vendors: auroracompute; InternetSolutions; dimension data #2057)
  • Add unittest API request checking to remaining drivers
  • Add and leverage common code for domain/host name manipulation
  • Add and leverage common code for <type>[:<name>]-style record IDs
  • Change Record.id format for NS1 and Zonomi providers
  • Change Record.id interpretation for World Wide DNS

Status

  • Ready for review

Checklist (tick everything that applies)

  • Code linting (required, can be done after the PR checks)
  • Documentation
  • Tests
  • ICLA (required for bigger changes)

@aaron-sierra
aaron-sierra force-pushed the dns-consistent-api branch 2 times, most recently from 0fe8ec0 to 46c0e4f Compare April 3, 2025 19:34
@aaron-sierra

Copy link
Copy Markdown
Author

The latest batch of updates includes changes from black and isort.

@aaron-sierra

Copy link
Copy Markdown
Author

The latest changes have been through black, isort, lint, and pylint.

@aaron-sierra
aaron-sierra marked this pull request as ready for review April 3, 2025 20:07

@micafer micafer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the extensive work on making the DNS driver APIs more consistent. The additional HTTP request assertions are particularly valuable.

I found an issue in the new Zone.prefix() implementation that should be addressed before merging.

The expression:

subname.rstrip(".").removesuffix(self.domain).rstrip(".")

removes the zone domain even when it is not preceded by a DNS label separator.

For example, with self.domain == "example.com", calling prefix("notexample.com") returns "not", even though notexample.com is not a hostname within the example.com zone.

Since Zone.hostname() and Zone.fqdn() depend on this method, the issue can silently produce incorrect DNS names.

I suggest checking for an exact domain match or the suffix "." + self.domain before stripping it, and adding regression tests for hostnames that merely share a textual suffix with the zone domain.

Additionally, this PR changes public-facing Record.id, Record.name, and Zone.domain semantics across multiple providers. As we are going to merge this to the new major version these compatibility changes will be explicitly documented and it is not majot issue.

@aaron-sierra

Copy link
Copy Markdown
Author

Hi Miguel,

Thanks for your review. In the example that you described regarding Zone.prefix() for a "sub-domain" that isn't actually related to the Zone instance, what would you suggest as reasonable output? A few options come to mind:

  1. Return None to indicate "no relationship", since the empty string is used to indicate "related, but no prefix"
    • Both None and "" are falsy, which could make it a source of error.
  2. Raise a ValueError exception (to aggressively highlight bugs?)
  3. Something else

Presumably all three of the Zone helpers that operate on sub-domains should behave similarly when offered an unrelated "sub-domain" (i.e. Zone.prefix(), Zone.hostname(), and Zone.fqdn()). Does that sound right to you?

@micafer

micafer commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

I probably prefer to return None when the domain is not related, as in the example notexample.com.
"" should only be returned when using the same domain as a prefix.
What do you think?

@micafer micafer mentioned this pull request Nov 10, 2026
4 of 7 tasks
@micafer
micafer requested a review from tisonkun November 10, 2026 09:51
@micafer

micafer commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

I probably prefer to return None when the domain is not related, as in the example notexample.com. "" should only be returned when using the same domain as a prefix. What do you think?

I thought a bit on this and probably returning the full DNS name can be the best solution: as in the example notexample.com will return notexample.com.

@micafer

micafer commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

I'm still checking this and I found two issues:

Google Cloud DNS — get_record() uses a relative name in the API query.

After parsing the normalized record ID, parts.name contains a relative name such as foo. However, the Cloud DNS API expects a fully qualified name such as foo.example.com. in the name query parameter.

I believe this should use zone.fqdn(parts.name) instead. The corresponding test currently checks the endpoint URL but does not assert the query parameters.

World Wide DNS — update_record() fails when extra=None.

The new condition if "entry" in extra raises TypeError when the optional extra argument is omitted.

Please guard against None and add a regression test covering an update without extra.

@aaron-sierra

Copy link
Copy Markdown
Author

Google Cloud DNS — get_record() uses a relative name in the API query.

After parsing the normalized record ID, parts.name contains a relative name such as foo. However, the Cloud DNS API expects a fully qualified name such as foo.example.com. in the name query parameter.

I believe this should use zone.fqdn(parts.name) instead. The corresponding test currently checks the endpoint URL but does not assert the query parameters.

You're right! I've added API parameter checking to test_get_record(), so this doesn't slip through again:

        self.assertEqual(len(sent.query["type"]), 1)
        self.assertEqual(sent.query["type"].pop(), "A")
        self.assertEqual(len(sent.query["name"]), 1)
        self.assertEqual(sent.query["name"].pop(), "foo.example.com.")

@aaron-sierra

Copy link
Copy Markdown
Author

I probably prefer to return None when the domain is not related, as in the example notexample.com. "" should only be returned when using the same domain as a prefix. What do you think?

I thought a bit on this and probably returning the full DNS name can be the best solution: as in the example notexample.com will return notexample.com.

I think we need to clarify what should happen when a hostname (unrooted) and an FQDN (rooted) are passed into these functions. In the following examples, zone.domain == "example.com".

  1. In the case of an unrelated hostname ("notexample.com") , these functions can't really do much more than treat the input as a prefix and output garbage:
    • zone.prefix("notexample.com") == "notexample.com"
    • zone.hostname("notexample.com") == "notexample.com.example.com"
    • zone.fqdn("notexample.com") == "notexample.com.example.com."
  2. In the case of an unrelated FQDN ("notexample.com.") , these functions know that the input is garbage and can return None (or raise an exception):
    • zone.prefix("notexample.com.") == None
    • zone.hostname("notexample.com.") == None
    • zone.fqdn("notexample.com.") == None

I think that I prefer raising an exception in the unrelated FQDN case, considering how #2065 notes inconsistent usage of None and the empty string by various DNS driver implementations.

@micafer

micafer commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

I think that I prefer raising an exception in the unrelated FQDN case, considering how #2065 notes inconsistent usage of None and the empty string by various DNS driver implementations.

Ok, so let's go in that direction. But we can create an specific exception (something like ZoneNameMismatchError) instead of using the generic ValueError.

@micafer

micafer commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

So please make this changes and merge the trunk branch.

@aaron-sierra

Copy link
Copy Markdown
Author

I think that I prefer raising an exception in the unrelated FQDN case, considering how #2065 notes inconsistent usage of None and the empty string by various DNS driver implementations.

Ok, so let's go in that direction. But we can create an specific exception (something like ZoneNameMismatchError) instead of using the generic ValueError.

How's ZoneNotRelatedError for clarity and specificity?

@aaron-sierra

aaron-sierra commented Nov 13, 2026 •

Copy link
Copy Markdown
Author

World Wide DNS — update_record() fails when extra=None.

The new condition if "entry" in extra raises TypeError when the optional extra argument is omitted.

Please guard against None and add a regression test covering an update without extra.

I'd like to omit this from the current merge request, if you agree. I've spent time trying to implement common/reusable function-call-level API test cases, but it needs review and refinement. I think this warrants a new issue.

  1. At least two other drivers need similar fixes:
    • GandiDNSDriver - create_record() and update_record()
    • GandiLiveDNSDriver - update_record()
  2. Several drivers have explicit quirks that don't play nice with that sort of testing.
  3. The Google DNS driver testing relies on a mocked HTTP authentication endpoint that I have not managed to get working with this class of testing.

Aaron Sierra added 9 commits November 13, 2026 15:16
Aaron Sierra added 22 commits November 13, 2026 15:39
The update_record() and delete_record() functions require a Record
instance in their respective argument lists to identify the DNS record
to manipulate. Record objects instantiated by this driver contain the
assocated "slot" ID within the .id field, so:

1. There is no need for the user to specify the "slot" ID via
   .update_record(..., extra={"entry": ID}). However, since it was
   previously required, an "entry" key will be accepted as long as it is
   consistent with the Record instance.
2. There is no need for .delete_record() to search the Record's zone for
   the "slot" ID of the Record.
The API key is automatically inserted into the request by the
ZonomiConnection class.
The MockHttp function was defined, but the corresponding test was not.
Previously, the Zonomi DNS driver could only return the first record
for a given host, because it used the name of the host as Record.id.

Now, use the new default, <type>[:<name>], format to allow various
records to be differentiated for a given host.
DNS drivers share the need to manipulate host/domain names, so let's add
utility functions to the common Record and Zone classes to allow
redundant code scattered throughout the various drivers to be
consolidated into common and consistent implementations.

  * Record.name - no longer driver-dependent, now the host prefix or ""
  * Record.hostname - the complete host name, including domain part
  * Record.fqdn - the fully qualified domain name, including root dot

  * Zone.domain - no longer driver-dependent, now the unrooted domain
  * Zone.rooted() - format a (un)rooted domain as a rooted domain
  * Zone.unrooted() - format a (un)rooted domain as an unrooted domain
Update testing inputs to conform to spec'ed usage.
Leverage common code from DNSDriver, Record, and Zone classes for host
name and record ID handling.
Drop RackspaceDNSDriver._to_partial_record_name() and
RackspaceDNSDriver._to_full_record_name() in favor of the common
support provided by the Record and Zone classes.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.16318% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.39%. Comparing base (8a0ef62) to head (34d96fa).

Files with missing lines Patch % Lines
libcloud/dns/drivers/worldwidedns.py 36.36% 4 Missing and 3 partials ⚠️
libcloud/dns/drivers/route53.py 75.00% 2 Missing ⚠️
libcloud/dns/drivers/google.py 66.67% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##            trunk    #2066      +/-   ##
==========================================
- Coverage   83.66%   83.39%   -0.27%     
==========================================
  Files         352      350       -2     
  Lines       82445    82990     +545     
  Branches     8890     8867      -23     
==========================================
+ Hits        68974    69207     +233     
- Misses      10591    10934     +343     
+ Partials     2880     2849      -31     
Files with missing lines Coverage Δ
libcloud/dns/base.py 97.97% <100.00%> (+0.70%) ⬆️
libcloud/dns/drivers/dnsimple.py 26.44% <ø> (-67.82%) ⬇️
libcloud/dns/drivers/durabledns.py 88.14% <100.00%> (+0.08%) ⬆️
libcloud/dns/drivers/gandi.py 92.25% <100.00%> (+0.06%) ⬆️
libcloud/dns/drivers/gandi_live.py 89.06% <100.00%> (+0.06%) ⬆️
libcloud/dns/drivers/luadns.py 88.18% <100.00%> (ø)
libcloud/dns/drivers/nsone.py 76.47% <100.00%> (+0.79%) ⬆️
libcloud/dns/drivers/pointdns.py 86.33% <100.00%> (ø)
libcloud/dns/drivers/powerdns.py 67.47% <100.00%> (+0.40%) ⬆️
libcloud/dns/drivers/rackspace.py 77.54% <100.00%> (-0.91%) ⬇️
... and 33 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@micafer

micafer commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

One of the tests are failing:
Please could you update the example: https://github.com/apache/libcloud/blob/trunk/example_dns.py
It uses the deleted Zerigo DNS driver to make the test, could you update it to use any other?

World Wide DNS — update_record() fails when extra=None.

I prefer that, at least in the drivers modified in this PR that this error is solved.
Then we can open another PR to fix the rest.
It requires a minor change:

def update_record(self, record, name, type, data, extra=None):
    record_id = record.id
    if extra and "entry" in extra and str(extra["entry"]) != record_id:
        raise WorldWideDNSError(...)

Comment thread libcloud/dns/base.py

@classmethod
def from_default_id(cls, zone, record_id):
match = cls.DEFAULT_ID_RE.match(record_id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current implementation uses re.match() to parse record IDs. Since re.match() only requires a match at the beginning of the string, malformed IDs can be partially accepted.

For example, "A:" matches the "A" prefix because the optional name group cannot match an empty string. As a result, the trailing colon is silently ignored instead of rejecting the invalid ID.

Would it make sense to use re.fullmatch() instead, so the entire record ID must match the expected format?

A regression test for malformed IDs such as "A:" would also be useful.

@micafer micafer mentioned this pull request Nov 20, 2026
3 of 4 tasks
@micafer
micafer removed the request for review from tisonkun November 20, 2026 08:55
@micafer

micafer commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Moved to: #2233

@micafer micafer closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants