fix(datadog grok): enable the DOTALL mode by default #1022
Conversation
vladimir-dd
force-pushed
the
vladimir-dd/opa-2298
branch
from
5561067
to
b43bd3a
Compare
vladimir-dd
force-pushed
the
vladimir-dd/opa-2298
branch
from
b43bd3a
to
b5814ad
Compare
vladimir-dd
force-pushed
the
vladimir-dd/opa-2298
branch
from
dd341c9
to
b5814ad
Compare
vladimir-dd
force-pushed
the
vladimir-dd/opa-2298
branch
from
b5814ad
to
d019941
Compare
vladimir-dd
force-pushed
the
vladimir-dd/opa-2298
branch
from
59c01bc
to
72a217f
Compare
changelog.d/1022.breaking.md
Outdated
| @@ -0,0 +1,2 @@ | |||
| The `parse_groks` VRL function has the multiline mode enabled by default. | |||
There was a problem hiding this comment.
I'm not sure whether it makes sense to introduce an optional parameter for users to control this, keep the existing behaviour or make a breaking change for this function? 🤔 Datadog's grok implementation enables the DOTALL mode by default(see the csv filter for example), which seems like a reasonable behaviour. Users can always use (?s) and (?-s) modifiers to control this behaviour.
Also please note that the parse_grok function does not use the datadog/grok lib, so in this case they would behave differently unless we change it too..
Any thoughts @jszwedko @bruceg?
There was a problem hiding this comment.
I don't have as much context on vrl but my 2cents - agreed that making this the default behavior feels reasonable, and I think it would also make sense to eventually update parse_grok to have parity. If I'm looking at the vrl docs I would assume they share this base behavior.
vladimir-dd
changed the title
src/datadog/grok/parse_grok_rules.rs
Outdated
| let mut pattern = String::new(); | ||
| // In Oniguruma the (?m) modifier is used to enable the DOTALL mode(dot includes newlines), | ||
| // as opposed to the (?s) modifier in other regex flavors. | ||
| // \A, \z - parses from the beginning to the end of string, not line(until \n) | ||
| pattern.push_str(r"\A"); | ||
| pattern.push_str(r"(?m)\A"); // (?m) enables the DOTALL mode by default | ||
| pattern.push_str(&context.regex); | ||
| pattern.push_str(r"\z"); | ||
|
|
||
| // our regex engine(onig) uses (?m) mode modifier instead of (?s) to make the dot match all characters | ||
| pattern = pattern.replace("(?s)", "(?m)").replace("(?-s)", "(?-m)"); |
There was a problem hiding this comment.
nit: This could be done all in one go:
let pattern = [
// In Oniguruma the (?m) modifier is used to enable the DOTALL mode(dot includes newlines),
// as opposed to the (?s) modifier in other regex flavors.
// \A, \z - parses from the beginning to the end of string, not line(until \n)
r"(?m)\A", // (?m) enables the DOTALL mode by default
&context.regex.replace("(?s)", "(?m)").replace("(?-s)", "(?-m)"),
r"\z",
].concat();There was a problem hiding this comment.
Good idea, updated.
Co-authored-by: Bruce Guenter <[email protected]>
Enables the DOTALL mode(dot includes \n) for the Datadog grok lib by default.