feat: add multiline string finder in helper script

[b31ngd3v]

[terriko]

I like the idea. Is there any way we could add a test for this functionality?

[BreadGenie]

Looks good!

There are some scenarios where the script could possibly improve

• Catching version string when the lines that matter are not adjacent but is after a couple lines. Example:

  cve-bin-tool/cve_bin_tool/checkers/gnomeshell.py

  Line 18 in 28c8842

  r"var PACKAGE_NAME = 'gnome\-shell';\r?\n/\* The version of this package \*/\r?\nvar PACKAGE_VERSION = '([0-9]+\.[0-9]+(\.[0-9]+)?)';"

• Catching version string when the line with version is before and the other relevant string (with package name) is after.

[b31ngd3v]

• Catching version string when the lines that matter are not adjacent but is after a couple lines. Example:

  cve-bin-tool/cve_bin_tool/checkers/gnomeshell.py

  Line 18 in 28c8842

  r"var PACKAGE_NAME = 'gnome\-shell';\r?\n/\* The version of this package \*/\r?\nvar PACKAGE_VERSION = '([0-9]+\.[0-9]+(\.[0-9]+)?)';"

what should be the maximum line gap between the product string and version string? if the line gap is too high then the signature won't be that good, right? @BreadGenie

[BreadGenie]

@b31ngd3v We could give a maximum line gap but we can also do without it. Example:
r"var PACKAGE_NAME = 'gnome\-shell';(?:(?:\r?\n.*)*)var PACKAGE_VERSION = '([0-9]+\.[0-9]+(\.[0-9]+)?)';" for the above example.

The (?:(?:\r?\n.*)*) part would catch all newline and the characters in between the package name string and version string.

[b31ngd3v]

The (?:(?:\r?\n.*)*) part would catch all newline and the characters in between the package name string and version string.

@BreadGenie don't you think we should use non-greedy mode (?:(?:\r?\n.*?)*) ?

[BreadGenie]

@b31ngd3v yep

[codecov-commenter]

Codecov Report

Merging #1690 (65220dc) into main (0f24b06) will increase coverage by 3.30%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1690      +/-   ##
==========================================
+ Coverage   78.55%   81.86%   +3.30%     
==========================================
  Files         298      298              
  Lines        6216     6258      +42     
  Branches     1011     1023      +12     
==========================================
+ Hits         4883     5123     +240     
+ Misses       1115      931     -184     
+ Partials      218      204      -14     

Flag	Coverage Δ
longtests	81.86% <100.00%> (+3.30%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
cve_bin_tool/helper_script.py	85.32% <100.00%> (+26.49%)	⬆️
test/test_helper_script.py	100.00% <100.00%> (ø)
cve_bin_tool/nvd_api.py	75.20% <0.00%> (-4.80%)	⬇️
cve_bin_tool/extractor.py	55.05% <0.00%> (+1.01%)	⬆️
cve_bin_tool/version_scanner.py	77.51% <0.00%> (+1.93%)	⬆️
cve_bin_tool/cli.py	72.95% <0.00%> (+2.45%)	⬆️
cve_bin_tool/util.py	78.37% <0.00%> (+2.70%)	⬆️
cve_bin_tool/checkers/glibc.py	100.00% <0.00%> (+4.16%)	⬆️
cve_bin_tool/available_fix/debian_cve_tracker.py	83.67% <0.00%> (+6.12%)	⬆️
cve_bin_tool/available_fix/redhat_cve_tracker.py	80.35% <0.00%> (+8.92%)	⬆️
... and 8 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[b31ngd3v]

jackson-databind

gnome-shell

@BreadGenie what do you think?

[b31ngd3v]

these ones don't look good, should we remove the ones with bare version? @BreadGenie

[BreadGenie]

@b31ngd3v it looks nice, but do we need multiline pattern when a single line pattern exist? Or we could mark the single line pattern as a better choice in the output?

[b31ngd3v]

@b31ngd3v it looks nice, but do we need multiline pattern when a single line pattern exist? Or we could mark the single line pattern as a better choice in the output?

good idea! if a single line pattern exists, then we should only print that one.

[b31ngd3v]

these ones don't look good, should we remove the ones with bare version?

@BreadGenie btw what do you think about this?

[BreadGenie]

good idea! if a single line pattern exists, then we should only print that one.

Yep

[BreadGenie]

these ones don't look good, should we remove the ones with bare version? @BreadGenie

We will only need one of those and should be used if we don't have any other choices. Multiline patterns could end up being expensive.

Also I didn't get what you meant by bare version

[b31ngd3v]

these ones don't look good, should we remove the ones with bare version? @BreadGenie

We will only need one of those and should be used if we don't have any other choices. Multiline patterns could end up being expensive.

Also I didn't get what you meant by bare version

by bare version, i meant the second part (after (?:(?:\r?\n.*?)*)) where the version is (in the image), is completely naked it doesn't have anything like version=3.38.4, don't you think we should remove those?

[BreadGenie]

by bare version, i meant the second part (after (?:(?:\r?\n.*?)*)) where the version is (in the image), is completely naked it doesn't have anything like version=3.38.4, don't you think we should remove those?

It could work in multiline mode but it could also generate false positives, so we can remove that.

[b31ngd3v]

cleaned up the output, previously it was printing all combinations of version strings and product strings after comparing the line numbers, now it's only printing the combinations of version strings and the closest product string line to that version string after comparing the line numbers. and also if a one line signature exists then it's going to ignore all multi-line signatures.

gnome-shell [it's printing the same signature two times, because those two are in different lines in the binary]

jackson-databind

[b31ngd3v]

thanks @BreadGenie for helping me out with this PR

[terriko]

This is going to be a very helpful feature, thank you! I have a feeling we might want to do some more precise tests later making sure it found specific expected patterns rather than "just not the max glob" but this should at least get us some basic coverage on the code.