Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial fixed-affected-matching work #1228 #1249

Merged
merged 63 commits into from
Nov 30, 2023
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
1e4079d
Add initial fixed-affected-matching work #1228
johnmhoran Jul 26, 2023
624f047
Explore context and Package class approaches for affected-fixed packa…
johnmhoran Jul 28, 2023
945b811
Add Prefetch and univers-based version comparison #1228
johnmhoran Jul 30, 2023
6497e90
Update affected-fixed package matching #1228
johnmhoran Aug 8, 2023
b6dba78
Improve matching and reporting code and UI #1228
johnmhoran Aug 10, 2023
814cd06
Add univers version, revise sort and related code, update and add new…
johnmhoran Aug 14, 2023
eee1d79
Merge branch 'main' into 1228-fixed-affected-version-matching #1228
johnmhoran Aug 14, 2023
f920ded
Move weakness test #1228
johnmhoran Aug 15, 2023
40c1758
Modify UI, update dictionary and tests #1228
johnmhoran Aug 15, 2023
f370671
Begin replacing strings with objects in package details dictionary #1228
johnmhoran Aug 28, 2023
9f73ea4
Merge branch 'main' into 1228-fixed-affected-version-matching #1228
johnmhoran Aug 28, 2023
9ccaed7
Clean current package details template and related model code #1228
johnmhoran Aug 29, 2023
1a26613
Begin work on major-version issue #1228
johnmhoran Aug 30, 2023
b626762
Complete first round of major-version vetting #1228
johnmhoran Sep 6, 2023
005c094
Remove major-version code, clean comments etc. #1228
johnmhoran Sep 8, 2023
ab80c46
Merge branch 'main' into 1228-fixed-affected-version-matching #1228
johnmhoran Sep 8, 2023
55f9678
Begin test refactoring #1228
johnmhoran Sep 11, 2023
cc45e2a
Merge branch 'main' into 1228-fixed-affected-version-matching #1228
johnmhoran Sep 11, 2023
f904f09
Merge branch 'main' into 1228-fixed-affected-version-matching #1228
johnmhoran Sep 11, 2023
b750d65
Finish package details code and template, refactor/create package-rel…
johnmhoran Sep 12, 2023
fe23fec
Commit the initial refactoring changes from last week #1228
johnmhoran Sep 18, 2023
4cdb629
Merge branch 'main' into 1228-fixed-affected-version-matching #1228
johnmhoran Sep 18, 2023
89aab8f
Refactor package details-related code #1228
johnmhoran Sep 21, 2023
04824e8
Merge branch 'main' into 1228-fixed-affected-version-matching #1228
johnmhoran Sep 21, 2023
e2e4e13
Update Package details UI and Package API #1228
johnmhoran Sep 27, 2023
c76a43d
Merge branch 'main' into 1228-fixed-affected-version-matching #1228
johnmhoran Sep 27, 2023
7a8f83b
Save test experiments including commented-out variations #1228
johnmhoran Sep 27, 2023
2fdc594
Fix 1 of 4 failing API tests #1228
johnmhoran Sep 27, 2023
1bb4f20
Add initial fixed-affected-matching work #1228
johnmhoran Jul 26, 2023
dff950f
Explore context and Package class approaches for affected-fixed packa…
johnmhoran Jul 28, 2023
627d117
Add Prefetch and univers-based version comparison #1228
johnmhoran Jul 30, 2023
65d82a7
Update affected-fixed package matching #1228
johnmhoran Aug 8, 2023
3b34e46
Improve matching and reporting code and UI #1228
johnmhoran Aug 10, 2023
eb69a01
Add univers version, revise sort and related code, update and add new…
johnmhoran Aug 14, 2023
9a9401a
Move weakness test #1228
johnmhoran Aug 15, 2023
5c6838c
Modify UI, update dictionary and tests #1228
johnmhoran Aug 15, 2023
3ef542b
Begin replacing strings with objects in package details dictionary #1228
johnmhoran Aug 28, 2023
1045b36
Clean current package details template and related model code #1228
johnmhoran Aug 29, 2023
57007b9
Begin work on major-version issue #1228
johnmhoran Aug 30, 2023
1ffccbe
Complete first round of major-version vetting #1228
johnmhoran Sep 6, 2023
dc701ca
Remove major-version code, clean comments etc. #1228
johnmhoran Sep 8, 2023
f0bdd3b
Begin test refactoring #1228
johnmhoran Sep 11, 2023
4314820
Finish package details code and template, refactor/create package-rel…
johnmhoran Sep 12, 2023
df7f404
Commit the initial refactoring changes from last week #1228
johnmhoran Sep 18, 2023
e475e5d
Refactor package details-related code #1228
johnmhoran Sep 21, 2023
6a72c58
Update Package details UI and Package API #1228
johnmhoran Sep 27, 2023
f5e267c
Save test experiments including commented-out variations #1228
johnmhoran Sep 27, 2023
ce2c6cf
Fix 1 of 4 failing API tests #1228
johnmhoran Sep 27, 2023
7c412ed
Update API including "lesser" fixed by versions, fix and update faili…
johnmhoran Sep 30, 2023
eecd504
Update APITestCasePackage() class #1228
johnmhoran Oct 3, 2023
9978841
Test lack of "vulnerability" property #1228
johnmhoran Oct 18, 2023
6cd41d3
Update get_affected_vulnerabilities() and test #1228
johnmhoran Oct 26, 2023
e111dbe
Update MinimalPackageSerializer() and missing-vulnerability-key test …
johnmhoran Oct 28, 2023
5eaa9ce
Append inside the if condition #1228
johnmhoran Oct 28, 2023
4ce5d51
Update get_vulnerability() method #1228
johnmhoran Nov 1, 2023
f1a0530
Enable test_models.py and fix failing tests #1228
johnmhoran Nov 22, 2023
9ec2a6a
Update per PR comments #1228
johnmhoran Nov 22, 2023
91c1817
Convert Package method to PackageQuerySet method, clean code and test…
johnmhoran Nov 23, 2023
3dac483
Merge branch '1228-fixed-affected-version-matching' of https://github…
TG1999 Nov 23, 2023
2abcba0
Fix failing tests
TG1999 Nov 24, 2023
0ec7d6c
Add property on functions in models
TG1999 Nov 24, 2023
7a512c2
Merge branch 'main' into 1228-fixed-affected-version-matching #1228
johnmhoran Nov 28, 2023
26c0239
Add and fix tests, address other comments #1228
johnmhoran Nov 29, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 150 additions & 0 deletions vulnerabilities/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
from django.core.validators import MinValueValidator
from django.db import models
from django.db.models import Count
from django.db.models import Prefetch
from django.db.models import Q
from django.db.models.functions import Length
from django.db.models.functions import Trim
Expand All @@ -31,6 +32,8 @@
from packageurl.contrib.django.models import PackageURLQuerySet
from packageurl.contrib.django.models import without_empty_values
from rest_framework.authtoken.models import Token
from univers import versions
from univers.version_range import RANGE_CLASS_BY_SCHEMES

from vulnerabilities.importer import AdvisoryData
from vulnerabilities.importer import AffectedPackage
Expand Down Expand Up @@ -628,6 +631,153 @@ def get_absolute_url(self):
"""
return reverse("package_details", args=[self.purl])

def get_fixed_packages(self, package, fix=True):
"""
With fix=True, return a queryset of all packages that fix a vulnerability and have the
same type, namespace, name, subpath and qualifiers as the `package`.
With fix=False, return a queryset of all packages that have the same type, namespace,
name, subpath and qualifiers as the `package`, whether or not they fix any vulnerability.
"""
filter_dict = {
"name": package.name,
"namespace": package.namespace,
"type": package.type,
"qualifiers": package.qualifiers,
"subpath": package.subpath,
}

if fix:
filter_dict["packagerelatedvulnerability__fix"] = True

return Package.objects.filter(**filter_dict).distinct()

def sort_by_version(self, packages_to_sort):
"""
Use the univers version_class to sort the incoming list of vulnerabilities.models.Package
objects.
"""
univers_version = RANGE_CLASS_BY_SCHEMES[packages_to_sort[0].type].version_class
johnmhoran marked this conversation as resolved.
Show resolved Hide resolved
return sorted(
packages_to_sort,
key=lambda x: univers_version(x.version),
)

@property
def fixed_package_details(self):
"""
For a given package, report the closest subsequent fixed by version for each of that
package's vulnerabilities as well as the closest non-vulnerable version and the most recent
non-vulnerable version.
"""
fixed_packages = self.get_fixed_packages(package=self, fix=True)
qs = self.vulnerabilities.filter(packagerelatedvulnerability__fix=False)
johnmhoran marked this conversation as resolved.
Show resolved Hide resolved
all_sibling_packages = self.get_fixed_packages(package=self, fix=False)

non_vuln_sibs = []
for sib in all_sibling_packages:
if sib.is_vulnerable is False:
non_vuln_sibs.append(sib)

univers_version_class = RANGE_CLASS_BY_SCHEMES[self.type].version_class
later_non_vuln_sibs = []

for non_vuln_sib in non_vuln_sibs:
non_vuln_sib_univers_version = univers_version_class(
PackageURL.from_string(non_vuln_sib.purl).version
)

if univers_version_class(non_vuln_sib.version) > univers_version_class(self.version):
later_non_vuln_sibs.append(non_vuln_sib)

qs = qs.prefetch_related(
Prefetch(
"packages",
queryset=fixed_packages,
to_attr="fixed_packages",
)
)

purl_dict = {}
purl_dict["purl"] = PackageURL.from_string(self.purl)

closest_non_vulnerable_sib = ""
latest_non_vulnerable_sib = ""

if len(later_non_vuln_sibs) > 0:
closest_non_vulnerable_sib = self.sort_by_version(later_non_vuln_sibs)[0]
latest_non_vulnerable_sib = self.sort_by_version(later_non_vuln_sibs)[-1]

purl_dict["closest_non_vulnerable"] = PackageURL.from_string(
closest_non_vulnerable_sib.purl
)
purl_dict["latest_non_vulnerable"] = PackageURL.from_string(
latest_non_vulnerable_sib.purl
)

else:

purl_dict["closest_non_vulnerable"] = None
purl_dict["latest_non_vulnerable"] = None

purl_dict.update({"vulnerabilities": []})

for vuln in qs:
johnmhoran marked this conversation as resolved.
Show resolved Hide resolved
purl_dict["vulnerabilities"].append({"vulnerability": vuln})

closest_fixed_package = ""
closest_fixed_package_vulns = ""
later_fixed_packages = []
sort_fixed_by_packages_by_version = []
vuln_fixed_packages = vuln.fixed_packages

if len(vuln_fixed_packages) > 0:
for fixed_pkg in vuln_fixed_packages:

fixed_pkg_univers_version = univers_version_class(
PackageURL.from_string(fixed_pkg.purl).version
)

if fixed_pkg in fixed_packages and univers_version_class(
fixed_pkg.version
) > univers_version_class(self.version):

later_fixed_packages.append(fixed_pkg)

if later_fixed_packages:
sort_fixed_by_packages_by_version = self.sort_by_version(later_fixed_packages)
closest_fixed_package = sort_fixed_by_packages_by_version[0]
closest_fixed_package_vulns = closest_fixed_package.affected_by

else:
closest_fixed_package = None
closest_fixed_package_vulns = None

for dict_vuln in purl_dict["vulnerabilities"]:
if dict_vuln["vulnerability"] == vuln:

if closest_fixed_package:
dict_vuln["fixed_by_purl"] = PackageURL.from_string(
closest_fixed_package.purl
)

dict_vuln["fixed_by_purl_vulnerabilities"] = [
fixed_pkg_vuln for fixed_pkg_vuln in closest_fixed_package_vulns
]
else:
dict_vuln["fixed_by_purl"] = None
dict_vuln["fixed_by_purl_vulnerabilities"] = []

# Temporary print output during dev/testing.
johnmhoran marked this conversation as resolved.
Show resolved Hide resolved
# from pprint import pprint

# pprint(
# purl_dict,
# sort_dicts=False,
# )
# print("")

return purl_dict


class PackageRelatedVulnerability(models.Model):
"""
Expand Down
117 changes: 101 additions & 16 deletions vulnerabilities/templates/package_details.html
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
<table class="table vcio-table width-100-pct mt-2">
<tbody>
<tr>
<td class="two-col-left" style="width: 175px;">
<td class="two-col-left">
<span
class="has-tooltip-multiline has-tooltip-black has-tooltip-arrow has-tooltip-text-left"
data-tooltip="The package url or purl is a URL string used to identify and locate a software package."
Expand All @@ -33,24 +33,59 @@
</span>
</td>
<td class="two-col-right">
{{ package.purl }}
{{ fixed_package_details.purl.to_string }}
</td>
</tr>
</tbody>
</table>
</div>

{% if affected_by_vulnerabilities|length != 0 %}

<div class="pl-3 pr-3 mb-5 non-vuln">
<table class="table vcio-table width-100-pct mt-2">
<tbody>
<tr>
<td class="two-col-left">
Next non-vulnerable version
</td>
<td class="two-col-right">
{% if fixed_package_details.closest_non_vulnerable.version %}
<a href="/packages/{{ fixed_package_details.closest_non_vulnerable }}?search={{ fixed_package_details.closest_non_vulnerable }}" target="_self">{{ fixed_package_details.closest_non_vulnerable.version }}</a>
{% else %}
<span style="background-color: #ffcccc;">None.</span>
{% endif %}
</td>
</tr>
<tr>
<td class="two-col-left">
Latest non-vulnerable version
</td>
<td class="two-col-right">
{% if fixed_package_details.latest_non_vulnerable.version %}
<a href="/packages/{{ fixed_package_details.latest_non_vulnerable }}?search={{ fixed_package_details.latest_non_vulnerable }}" target="_self">{{ fixed_package_details.latest_non_vulnerable.version }}</a>
{% else %}
<span style="background-color: #ffcccc;">None.</span>
{% endif %}
</td>
</tr>
</tbody>
</table>
</div>

{% endif %}

<div class="content ml-3 mr-3">
<div class="has-text-weight-bold ml-1 mb-0">
Affected by vulnerabilities ({{ affected_by_vulnerabilities|length }})
Vulnerabilities affecting this package ({{ affected_by_vulnerabilities|length }})
</div>

<table class="table is-bordered is-striped is-narrow is-hoverable is-fullwidth">
<thead>
<tr>
<th style="width: 175px;">Vulnerability</th>
<th style="width: 200px;">Vulnerability</th>
<th>Summary</th>
<th style="width: 225px;">Aliases</th>
<th style="width: 300px;">Fixed by</th>
</tr>
</thead>

Expand All @@ -59,11 +94,9 @@
<tr>
<td>
<a href="{{ vulnerability.get_absolute_url }}" target="_self">{{ vulnerability.vulnerability_id }}</a>
</td>
<td>
{{ vulnerability.summary }}
</td>
<td>
<br />
Aliases:
<br />
{% for alias in vulnerability.alias %}
{% if alias.url %}
<a href="{{ alias.url }}" target="_blank">{{ alias }}<i class="fa fa-external-link fa_link_custom"></i></a>
Expand All @@ -74,33 +107,86 @@
{% endif %}
{% endfor %}
</td>
<td style="word-wrap: break-word; word-break: break-word;">
{{ vulnerability.summary }}
</td>
<td style="word-wrap: break-word; word-break: break-all;">
{% if package.purl == fixed_package_details.purl.to_string %}
{% for key, value in fixed_package_details.items %}
{% if key == "vulnerabilities" %}
{% for vuln in value %}
{% if vuln.vulnerability.vulnerability_id == vulnerability.vulnerability_id %}
{% if vuln.fixed_by_purl is None %}
<span style="background-color: #ffcccc;">There are no reported fixed versions.</span>
{% else %}
{% if vuln.fixed_by_purl_vulnerabilities|length == 0 %}
<a href="/packages/{{ vuln.fixed_by_purl }}?search={{ vuln.fixed_by_purl }}" target="_self">{{ vuln.fixed_by_purl.version }}</a>
<br />
<span style="background-color: #b3ffb3;">Affected by 0 other vulnerabilities.</span>
{% else %}
<a href="/packages/{{ vuln.fixed_by_purl }}?search={{ vuln.fixed_by_purl }}" target="_self">{{ vuln.fixed_by_purl.version }}</a>
{% if vuln.fixed_by_purl_vulnerabilities|length != 1 %}
<br />
<span style="background-color: #ffcccc;">Affected by {{ vuln.fixed_by_purl_vulnerabilities|length }} other vulnerabilities.</span>
{% else %}
<br />
<span style="background-color: #ffcccc;">Affected by {{ vuln.fixed_by_purl_vulnerabilities|length }} other vulnerability.</span>
{% endif %}

<div class="dropdown is-hoverable has-text-weight-normal is-right">
<div class="dropdown-trigger">
<i class="fa fa-question-circle ml-0 fa-sm has-background-white has-text-link"></i>
</div>
<div class="dropdown-menu dropdown-vuln-list-width" id="dropdown-menu4" role="menu">
<div class="dropdown-content dropdown-instructions-box-shadow">
<div class="dropdown-item">
<div style="max-height: 200px; overflow-y: auto;">
This version is affected by these other vulnerabilities:
<div style="padding-top: 5px;">
{% for fixed_by_vuln in vuln.fixed_by_purl_vulnerabilities %}
<div>
<a href="/vulnerabilities/{{ fixed_by_vuln.vulnerability_id }}" target="_self">{{ fixed_by_vuln.vulnerability_id }}</a>
</div>
{% endfor %}
</div>
</div>
</div>
</div>
</div>
</div>
{% endif %}
{% endif %}
{% endif %}
{% endfor %}
{% endif %}
{% endfor %}
{% endif %}
</td>
</tr>
{% empty %}
<tr>
<td colspan="3">
This package is not known to be affected by vulnerabilities.
<span style="background-color: #b3ffb3;">This package is not known to be affected by vulnerabilities.</span>
</td>
</tr>
{% endfor %}
</tbody>

</table>
</div>

<div class="content ml-3 mr-3">
<div class="has-text-weight-bold ml-1 mb-0">
Fixing vulnerabilities ({{ fixing_vulnerabilities|length }})
Vulnerabilities fixed by this package ({{ fixing_vulnerabilities|length }})
</div>

<table class="table is-bordered is-striped is-narrow is-hoverable is-fullwidth">
<thead>
<tr>
<th style="width: 175px;">Vulnerability</th>
<th style="width: 200px;">Vulnerability</th>
<th>Summary</th>
<th style="width: 225px;">Aliases</th>
</tr>
</thead>

<tbody>
{% for vulnerability in fixing_vulnerabilities %}
<tr>
Expand Down Expand Up @@ -137,5 +223,4 @@
</div>
</section>
{% endif %}

{% endblock %}
Loading
Loading