Error in datetime-regex

[Agh42]

Describe the bug

I believe there to be a bug in the "datetime with timezone" regex as described in https://github.com/usnistgov/OSCAL/edit/main/docs/content/reference/datatypes.md

As it stands, the regex will match dates 2017-06-04T00:00:00Z and 1999-04-03T12:13:45Z but it will NOT match i.e. 2400-02-29T00:00:00Z 2022-05-03T02:00:00Z.

This is because the T separating the date from the time part of the regex is only applicable after the fourth of the major matching groups. There has to be an additional group containing the first three major blocks (the ones that are present to differentiate between leap years and normal years).

I think to make the regex work like it's supposed to it needs to be this:

(((2000|2400|2800|(19|20-9))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(.[0-9]+)?(Z|[+-][0-9]{2}:[0-9]{2})

instead of the current

((2000|2400|2800|(19|20-9))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30))T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\.[0-9]+)?(Z|[+-][0-9]{2}:[0-9]{2})

It becomes a bit clearer when you run it through a regex visualizer (using debuggex.com here):

Current regex (notice how T only matches for the last possible year-month-day pattern):

Corrected regex: (T matches after all possible variations of year-month-day):

Who is the bug affecting?

Anyone trying to use the regex pattern defined in the OSCAL (JSON) data types to determine if a date/date-time input is considered a valid date in the OSCAL format. (For instance I tripped over this bug because one of my entered dates was not accepted by an autogenerated REST API stub.)

What is affected by this bug?

See description above, only some valid dates will match the regex currently.

When does this occur?

The regex is present in the documentation, but it is also present in example content and generated OpenAPI specification. It is repeated 36 times in 9 files in the OSCAL repo (mostly XSD schemas)

How do we replicate the issue?

See above.

Expected behavior (i.e. solution)

See suggested change to the regex above.

Other Comments

[Agh42]

I just realized that the regex is already correct in the XSD schemas - just not in the documentation.
This probably lead to the wrong regex being baked into the OpenAPI spec that I used from the tools list.

[aj-stein-nist]

I just realized that the regex is already correct in the XSD schemas - just not in the documentation. This probably lead to the wrong regex being baked into the OpenAPI spec that I used from the tools list.

Thanks for calling that out. We have a triage sync tomorrow and I am sure we can address this sooner rather than later.

[Agh42]

I also noticed that all regexes were removed from the generated JSON schemas on 5/4/22 in 9a03286. I couldn't find a GitHub issue regarding this change, was this done on purpose?

The JSON schemas previously also contained the incorrect regex.

I.e.:

[bradh]

Not everyone uses Gregorian dates, so a string might be more appropriate.

[aj-stein-nist]

I also noticed that all regexes were removed from the generated JSON schemas on 5/4/22 in 9a03286. I couldn't find a GitHub issue regarding this change, was this done on purpose?

I am not 100% certain but I would imagine this was not done on purpose and this might have to do with updating the Metaschema submodule to address fixes usnistgov/metaschema#182.

Will definitely look into this during triage later today!

[Agh42]

Not everyone uses Gregorian dates, so a string might be more appropriate.

I think that there are strong reasons for enforcing the Internet date/time format (RFC 3339). It avoids much confusion and misunderstanding esp. when working with international partners. Conversion to different time zones and formats should not be a problem, as long as the stored value can be interpreted clearly.

(As stated in the RFC: "because no date and time format is readable according to the conventions of all countries, Internet clients SHOULD be prepared to transform dates into a display format suitable for the locality.")

[Agh42]

I'm not sure that this fully closes this issue... I added a comment to the commit.