Skip to content

toml-test datetime cases use tomlkit's own parse_rfc3339 as the expected value, so they cannot detect datetime parsing bugs #603

Description

@ArnauFerma

While running a mutation-based check of the test suite at 4b38bec, one structural gap stood out and seemed worth reporting. Nothing here is a bug in tomlkit's code — every mutation below was a deliberate break, and the code as shipped is correct.

What was observed

tests/test_toml_tests.py builds the expected value for every datetime-typed case with tomlkit._utils.parse_rfc3339:

stypes = {
    ...
    "datetime": parse_rfc3339,
    "datetime-local": parse_rfc3339,
    "date-local": parse_rfc3339,
    "time-local": parse_rfc3339,
}

That is the function under test, so for those four types the parser and the expectation drift together. Breaking the negative-offset sign in parse_rfc3339 (if sign == "-": offset = -offset → never negating) produced:

red
toml-test compliance cases (680) 0
hand-written tests (378) 3 (test_parse_rfc3339_datetime ×2, test_datetimes_behave_like_datetimes)

By contrast, dropping the integer sign in the parser reddened 6 compliance cases and untranslated string escapes 7 — for int and str the corpus is an independent oracle. For datetimes it is not, and the entire guard is three hand-written tests.

Related, smaller: no input anywhere in tests/ (hand-written or corpus) carries a non-zero minute offset, so +05:30 being parsed as +05:00 (dropping minute_offset from the timedelta) survives the whole suite.

Possible fix

Build datetime expectations independently — datetime.fromisoformat accepts RFC 3339 on Python 3.11+ and the project already requires ≥3.9 with a compat shim, or a tiny hand-rolled regex as the toml-test JSON format is fixed — and add one +05:30-style case to test_parse_rfc3339_datetime.

Also noticed

tests/util.py::elementary_test asserts isinstance(v.unwrap(), unwrapped_type) only, so the nine test_*_unwrap tests detect a wrong type but not a wrong value: Bool.unwrap returning not bool(self) passes test_true_unwrap and test_false_unwrap. Passing the expected value to the helper would close that.

Full write-up with the mutation spec and per-test results, reproducible against 4b38bec: https://github.com/ArnauFerma/falsifiable-tests/blob/main/case-studies/tomlkit.md

Happy to send a PR for either if you'd take one.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions