While running a mutation-based check of the test suite at 4b38bec, one structural gap stood out and seemed worth reporting. Nothing here is a bug in tomlkit's code — every mutation below was a deliberate break, and the code as shipped is correct.
What was observed
tests/test_toml_tests.py builds the expected value for every datetime-typed case with tomlkit._utils.parse_rfc3339:
stypes = {
...
"datetime": parse_rfc3339,
"datetime-local": parse_rfc3339,
"date-local": parse_rfc3339,
"time-local": parse_rfc3339,
}
That is the function under test, so for those four types the parser and the expectation drift together. Breaking the negative-offset sign in parse_rfc3339 (if sign == "-": offset = -offset → never negating) produced:
|
red |
| toml-test compliance cases (680) |
0 |
| hand-written tests (378) |
3 (test_parse_rfc3339_datetime ×2, test_datetimes_behave_like_datetimes) |
By contrast, dropping the integer sign in the parser reddened 6 compliance cases and untranslated string escapes 7 — for int and str the corpus is an independent oracle. For datetimes it is not, and the entire guard is three hand-written tests.
Related, smaller: no input anywhere in tests/ (hand-written or corpus) carries a non-zero minute offset, so +05:30 being parsed as +05:00 (dropping minute_offset from the timedelta) survives the whole suite.
Possible fix
Build datetime expectations independently — datetime.fromisoformat accepts RFC 3339 on Python 3.11+ and the project already requires ≥3.9 with a compat shim, or a tiny hand-rolled regex as the toml-test JSON format is fixed — and add one +05:30-style case to test_parse_rfc3339_datetime.
Also noticed
tests/util.py::elementary_test asserts isinstance(v.unwrap(), unwrapped_type) only, so the nine test_*_unwrap tests detect a wrong type but not a wrong value: Bool.unwrap returning not bool(self) passes test_true_unwrap and test_false_unwrap. Passing the expected value to the helper would close that.
Full write-up with the mutation spec and per-test results, reproducible against 4b38bec: https://github.com/ArnauFerma/falsifiable-tests/blob/main/case-studies/tomlkit.md
Happy to send a PR for either if you'd take one.
While running a mutation-based check of the test suite at
4b38bec, one structural gap stood out and seemed worth reporting. Nothing here is a bug in tomlkit's code — every mutation below was a deliberate break, and the code as shipped is correct.What was observed
tests/test_toml_tests.pybuilds the expected value for every datetime-typed case withtomlkit._utils.parse_rfc3339:That is the function under test, so for those four types the parser and the expectation drift together. Breaking the negative-offset sign in
parse_rfc3339(if sign == "-": offset = -offset→ never negating) produced:test_parse_rfc3339_datetime×2,test_datetimes_behave_like_datetimes)By contrast, dropping the integer sign in the parser reddened 6 compliance cases and untranslated string escapes 7 — for
intandstrthe corpus is an independent oracle. For datetimes it is not, and the entire guard is three hand-written tests.Related, smaller: no input anywhere in
tests/(hand-written or corpus) carries a non-zero minute offset, so+05:30being parsed as+05:00(droppingminute_offsetfrom thetimedelta) survives the whole suite.Possible fix
Build datetime expectations independently —
datetime.fromisoformataccepts RFC 3339 on Python 3.11+ and the project already requires ≥3.9 with a compat shim, or a tiny hand-rolled regex as the toml-test JSON format is fixed — and add one+05:30-style case totest_parse_rfc3339_datetime.Also noticed
tests/util.py::elementary_testassertsisinstance(v.unwrap(), unwrapped_type)only, so the ninetest_*_unwraptests detect a wrong type but not a wrong value:Bool.unwrapreturningnot bool(self)passestest_true_unwrapandtest_false_unwrap. Passing the expected value to the helper would close that.Full write-up with the mutation spec and per-test results, reproducible against
4b38bec: https://github.com/ArnauFerma/falsifiable-tests/blob/main/case-studies/tomlkit.mdHappy to send a PR for either if you'd take one.