Feature/136 trivy integration #139

alexander-dammeier · 2024-11-29T08:02:53Z

No description provided.

As trivy exits with exit code 1 when it encountered an error, we will use exit code 10 if trivy scanned successful and found vulnerabilities. Also don't use RuntimeException, which is blocked by Jenkins.

To exclude CVEs from trivy scan, add a .trivyignore file to the workspace. The workspace is mounted to the docker container and therefore the .trivyignore file is integrated into the scan process, if it exists

also update compiler dependencies

This reverts commit e6736d9. The problem with this commit was, that now the jenkins pipeline only creates empty jars. It only works on local machines when we call our groovy tests directly

…es-build-lib into feature/136_trivy_integration

#136 update to latest temurin 11 jdk temurin is the successor of openjdk

…ovy_build

alexander-dammeier · 2024-11-29T08:14:20Z

src/com/cloudogu/ces/cesbuildlib/Trivy.groovy

+        String additionalFlags = "--db-repository public.ecr.aws/aquasecurity/trivy-db --java-db-repository public.ecr.aws/aquasecurity/trivy-java-db",
+        String trivyReportFile = "trivy/trivyReport.json"
+    ) {
+        Integer exitCode = docker.image("aquasec/trivy:${trivyVersion}")


We should not hardcode the trivy image as Docker Hub has a rate limiting, which makes trouble already. If we make this configurable we could use a mirror, like a artifact repo in our own cloud.

alexander-dammeier · 2024-11-29T08:27:11Z

src/com/cloudogu/ces/cesbuildlib/Trivy.groovy

+                fileExtension = "txt"
+                break
+            default:
+                script.error("This format did not match the supported formats: " + format)


Why no support for custom formats?

Because this PR was made to combine the functionality in dogu-build-lib and ces-build-lib and we didn't have custom format options yet. Custom formats could be its own issue/PR.

alexander-dammeier · 2024-11-29T08:37:44Z

src/com/cloudogu/ces/cesbuildlib/Trivy.groovy

+     * @param formattedTrivyReportFilename The file name your report files should get, without file extension. E.g. "ubuntu24report"
+     * @param trivyReportFile The "trivyReportFile" parameter you used in the "scanImage" function, if it was set
+     */
+    void saveFormattedTrivyReport(String format = TrivyScanFormat.HTML, String formattedTrivyReportFilename = "trivyReport", String trivyReportFile = "trivy/trivyReport.json") {


My intuition was that the order of parameters should be: trivyReportFile, format, formattedTrivyReportFilename.

Also it feels strange to expect the user to leave out the file extension. This could also be a problem when we want to support custom formats as then we have no clue, which extension we should add via the code but it is not so easy to provide a default parameter with a matching filename. Maybe we could use the format parameter for the default parameter.

A third thing is, that there is no way at the moment to set the output path for the formattedReport.

Suggested change

void saveFormattedTrivyReport(String format = TrivyScanFormat.HTML, String formattedTrivyReportFilename = "trivyReport", String trivyReportFile = "trivy/trivyReport.json") {

void formatReport(String inputReportFile = "trivy/trivyReport.json", String format = TrivyScanFormat.HTML, String outputReportFile = "$trivyDirectory/trivyReport.${format}") {

If the user wants to use the default report file, he/she can just call saveFormattedTrivyReport(TrivyScanFormat.HTML). If we use your order, you always have to declare the (default) report file name, which you may not even know as a new user of this function: saveFormattedTrivyReport("trivy/trivyReport.json", TrivyScanFormat.HTML). This would be much less convenient for the user.

alexander-dammeier · 2024-11-29T08:41:08Z

src/com/cloudogu/ces/cesbuildlib/TrivySeverityLevel.groovy

+    static String HIGH = "CRITICAL,HIGH"
+
+    /**
+     * Medium or higher vulnerabilities.
+     */
+    static String MEDIUM = "CRITICAL,HIGH,MEDIUM"


Suggested change

static String HIGH = "CRITICAL,HIGH"

/**

* Medium or higher vulnerabilities.

*/

static String MEDIUM = "CRITICAL,HIGH,MEDIUM"

static String HIGH_AND_ABOVE = "CRITICAL,HIGH"

/**

* Medium or higher vulnerabilities.

*/

static String MEDIUM_AND_ABOVE = "CRITICAL,HIGH,MEDIUM"

meiserloh · 2024-11-29T10:10:35Z

README.md

+Note that the flags "--db-repository public.ecr.aws/aquasecurity/trivy-db --java-db-repository public.ecr.aws/aquasecurity/trivy-java-db"
+are set by default to avoid rate limiting of Trivy database downloads.


should we specify here, that these defaults are implemented in a way, that they are not set if the additionalFlags is used?

meiserloh · 2024-11-29T10:12:26Z

src/com/cloudogu/ces/cesbuildlib/Trivy.groovy

+     * Notes:
+     * - Use a .trivyignore file for allowed CVEs
+     * - This function will generate a JSON formatted report file which can be converted to other formats via saveFormattedTrivyReport()
+     * - Evaluate via exit codes: 0 = no vulnerability; 1 = vulnerabilities found; other = function call failed


It has to be exit code 10 for vulnerabilities or am I not understanding the code correctly?

Suggested change

* - Evaluate via exit codes: 0 = no vulnerability; 1 = vulnerabilities found; other = function call failed

* - Evaluate via exit codes: 0 = no vulnerability; 10 = vulnerabilities found; other = function call failed

This seems to be an old comment which doesn't apply any more as the function now has a boolean return value

meiserloh · 2024-11-29T10:24:42Z

src/com/cloudogu/ces/cesbuildlib/Trivy.groovy

+     * Notes:
+     * - Use a .trivyignore file for allowed CVEs
+     * - This function will generate a JSON formatted report file which can be converted to other formats via saveFormattedTrivyReport()
+     * - Evaluate via exit codes: 0 = no vulnerability; 1 = vulnerabilities found; other = function call failed


see below. Should be exit code 10 instead of 1, shouldn't it?

Suggested change

* - Evaluate via exit codes: 0 = no vulnerability; 1 = vulnerabilities found; other = function call failed

* - Evaluate via exit codes: 0 = no vulnerability; 10 = vulnerabilities found; other = function call failed

This seems to be an old comment which doesn't apply any more as the function now has a boolean return value

meiserloh · 2024-11-29T10:45:59Z

src/com/cloudogu/ces/cesbuildlib/Trivy.groovy

+        }
+        docker.image("aquasec/trivy:${trivyVersion}")
+            .inside("-v ${script.env.WORKSPACE}/.trivy/.cache:/root/.cache/") {
+                script.sh(script: "trivy convert --format ${formatString} --output ${trivyDirectory}${formattedTrivyReportFilename}.${fileExtension} ${trivyReportFile}")


Feels like there is a slash missing

Suggested change

script.sh(script: "trivy convert --format ${formatString} --output ${trivyDirectory}${formattedTrivyReportFilename}.${fileExtension} ${trivyReportFile}")

script.sh(script: "trivy convert --format ${formatString} --output ${trivyDirectory}/${formattedTrivyReportFilename}.${fileExtension} ${trivyReportFile}")

In this method, trivyDirectory already holds the slash (line 109): String trivyDirectory = "trivy/"

meiserloh · 2024-11-29T10:46:26Z

src/com/cloudogu/ces/cesbuildlib/Trivy.groovy

+            .inside("-v ${script.env.WORKSPACE}/.trivy/.cache:/root/.cache/") {
+                script.sh(script: "trivy convert --format ${formatString} --output ${trivyDirectory}${formattedTrivyReportFilename}.${fileExtension} ${trivyReportFile}")
+            }
+        script.archiveArtifacts artifacts: "${trivyDirectory}${formattedTrivyReportFilename}.*", allowEmptyArchive: true


Feels like there is a slash missing

Suggested change

script.archiveArtifacts artifacts: "${trivyDirectory}${formattedTrivyReportFilename}.*", allowEmptyArchive: true

script.archiveArtifacts artifacts: "${trivyDirectory}/${formattedTrivyReportFilename}.*", allowEmptyArchive: true

In this method, trivyDirectory already holds the slash (line 109): String trivyDirectory = "trivy/"

We should not hard code this directory. Please use the field of the trivy class for the working directory. then @meiserloh s comment is valid again.

You are right, we should use Trivy.trivyDirectory here and remove the trivyDirectory declared in this method.

meiserloh · 2024-11-29T11:04:18Z

src/com/cloudogu/ces/cesbuildlib/Trivy.groovy

+     * @param formattedTrivyReportFilename The file name your report files should get, without file extension. E.g. "ubuntu24report"
+     * @param trivyReportFile The "trivyReportFile" parameter you used in the "scanImage" function, if it was set
+     */
+    void saveFormattedTrivyReport(String format = TrivyScanFormat.HTML, String formattedTrivyReportFilename = "trivyReport", String trivyReportFile = "trivy/trivyReport.json") {


We could add the option to filter by severity, too. This way we could scan for all severities and create reports by severity or something.

https://trivy.dev/v0.48/docs/references/configuration/cli/trivy_convert/

I think this is an additional feature and should have its own issue/PR.
There may be a debate if we even want to implement this, because setting the severity level in the scan-methods is reducing the scan runtime, which would always be at maximum, if you filter the report after scanning.

meiserloh · 2024-11-29T13:08:03Z

test/com/cloudogu/ces/cesbuildlib/TrivyTest.groovy

+
+        def gotException = false
+        try {
+            doTestScan(imageName, severityLevel, TrivyScanStrategy.FAIL, 10)doTestScan(imageName, severityLevel, TrivyScanStrategy.FAIL, 10)


this looks like double copy pasta?

meiserloh · 2024-11-29T13:09:07Z

test/com/cloudogu/ces/cesbuildlib/TrivyTest.groovy

+    }
+
+    void testScanImage_unsuccessfulTrivyExecution() {
+        // with hopes that this image will always have CVEs


delete this comment

alexander-dammeier · 2024-11-29T14:07:20Z

src/com/cloudogu/ces/cesbuildlib/Trivy.groovy

+     * @param formattedTrivyReportFilename The file name your report files should get, without file extension. E.g. "ubuntu24report"
+     * @param trivyReportFile The "trivyReportFile" parameter you used in the "scanImage" function, if it was set
+     */
+    void saveFormattedTrivyReport(String format = TrivyScanFormat.HTML, String formattedTrivyReportFilename = "trivyReport", String trivyReportFile = "trivy/trivyReport.json") {


"trivyReport" as default for the file name could lead to an override of the initial trivy report if we convert to json again as the default report filename also is "trivyReport.json"

The report file is already in json format (scanImage(): trivy image --exit-code 10 --exit-on-eol 10 --format ${TrivyScanFormat.JSON}...), so there should be no harm in converting it again I guess

As of now, you are right, but it would be problematic if we add for example the severity option to the saveFormattedTrivyReport function.

That's right, let's give it a better name. Any suggestions? How about formattedTrivyReport?

robertauer and others added 30 commits November 18, 2024 13:47

Add Trivy implementation structure; #136

c2897fb

Add invalid image name test; #136

251cdf1

Fail pipeline on failing unit tests

ec14c12

Implement scanImage method; #136

8cc5d40

Handle null references; #136

cc9b3e6

Move parameters to sensible locations; #136

5a97314

try to fix pipeline error

0341304

Fix parameter call

6d6dbb7

#136 fix groovy compile error with constant

d5b41b1

Fix NullPointerException

77ad25b

Use custom exit code on found vulnerabilities; 136

c30624e

As trivy exits with exit code 1 when it encountered an error, we will use exit code 10 if trivy scanned successful and found vulnerabilities. Also don't use RuntimeException, which is blocked by Jenkins.

Implement report conversion function; #136

a65c6e2

Save all trivy report files; #136

a28ad47

Add HTML format conversion; #136

096a0aa

Fail on unsupported scan format; #136

c933342

Trivy ignore file is working

7777d4d

To exclude CVEs from trivy scan, add a .trivyignore file to the workspace. The workspace is mounted to the docker container and therefore the .trivyignore file is integrated into the scan process, if it exists

Enable report file name setting; #136

358b7e2

Avoid rate limits of default Trivy database source; #136

190fb08

Set build status according to strategy; #136

7bd4562

Document Trivy functionality; #136

4183ba2

Adapt tests to new Trivy constructor; #136

5bf7fdf

Add method to scan Dogu images; #136

a32591f

Remove newlines from ssh output; #136

122c87d

#136 implement trivy integration test

326db5c

Merge branch 'develop' into feature/136_trivy_integration

52f5ddf

#136 add comments to integration test

91bf4ca

#136 use javac compiler as default

e6736d9

also update compiler dependencies

Adapt old trivy function to new implementation; #136

f651b32

Revert "#136 use javac compiler as default"

5e5703e

This reverts commit e6736d9. The problem with this commit was, that now the jenkins pipeline only creates empty jars. It only works on local machines when we call our groovy tests directly

Update changelog; #136

dc31bd4

robertauer and others added 5 commits November 28, 2024 09:20

Merge branch 'feature/136_trivy_integration' of github.com:cloudogu/c…

6cbd37d

…es-build-lib into feature/136_trivy_integration

#136 compile to java8

a4651bd

#136 update to latest temurin 11 jdk temurin is the successor of openjdk

Merge branch 'feature/136_trivy_integration' into feature/136_fix_gro…

57db423

…ovy_build

#136 add tests for trivy scans

37ce2f9

#136 refactor tests

94ef45f

alexander-dammeier commented Nov 29, 2024

View reviewed changes

alexander-dammeier force-pushed the feature/136_trivy_integration branch from 9119c08 to 94ef45f Compare November 29, 2024 09:33

meiserloh reviewed Nov 29, 2024

View reviewed changes

alexander-dammeier commented Nov 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/136 trivy integration #139

Feature/136 trivy integration #139

alexander-dammeier commented Nov 29, 2024

alexander-dammeier Nov 29, 2024

alexander-dammeier Nov 29, 2024

robertauer Dec 2, 2024

alexander-dammeier Nov 29, 2024 •

edited

Loading

robertauer Dec 2, 2024 •

edited

Loading

alexander-dammeier Nov 29, 2024

meiserloh Nov 29, 2024

meiserloh Nov 29, 2024

robertauer Dec 2, 2024

meiserloh Nov 29, 2024

robertauer Dec 2, 2024

meiserloh Nov 29, 2024

robertauer Dec 2, 2024

meiserloh Nov 29, 2024

robertauer Dec 2, 2024

alexander-dammeier Dec 3, 2024

robertauer Dec 3, 2024 •

edited

Loading

meiserloh Nov 29, 2024

robertauer Dec 2, 2024

meiserloh Nov 29, 2024

meiserloh Nov 29, 2024

alexander-dammeier Nov 29, 2024 •

edited

Loading

robertauer Dec 2, 2024 •

edited

Loading

meiserloh Dec 2, 2024 •

edited

Loading

robertauer Dec 2, 2024

	void saveFormattedTrivyReport(String format = TrivyScanFormat.HTML, String formattedTrivyReportFilename = "trivyReport", String trivyReportFile = "trivy/trivyReport.json") {
	void formatReport(String inputReportFile = "trivy/trivyReport.json", String format = TrivyScanFormat.HTML, String outputReportFile = "$trivyDirectory/trivyReport.${format}") {

		Note that the flags "--db-repository public.ecr.aws/aquasecurity/trivy-db --java-db-repository public.ecr.aws/aquasecurity/trivy-java-db"
		are set by default to avoid rate limiting of Trivy database downloads.

	* - Evaluate via exit codes: 0 = no vulnerability; 1 = vulnerabilities found; other = function call failed
	* - Evaluate via exit codes: 0 = no vulnerability; 10 = vulnerabilities found; other = function call failed

	script.sh(script: "trivy convert --format ${formatString} --output ${trivyDirectory}${formattedTrivyReportFilename}.${fileExtension} ${trivyReportFile}")
	script.sh(script: "trivy convert --format ${formatString} --output ${trivyDirectory}/${formattedTrivyReportFilename}.${fileExtension} ${trivyReportFile}")

	script.archiveArtifacts artifacts: "${trivyDirectory}${formattedTrivyReportFilename}.*", allowEmptyArchive: true
	script.archiveArtifacts artifacts: "${trivyDirectory}/${formattedTrivyReportFilename}.*", allowEmptyArchive: true

Feature/136 trivy integration #139

Are you sure you want to change the base?

Feature/136 trivy integration #139

Conversation

alexander-dammeier commented Nov 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexander-dammeier Nov 29, 2024 • edited Loading

Choose a reason for hiding this comment

robertauer Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertauer Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexander-dammeier Nov 29, 2024 • edited Loading

Choose a reason for hiding this comment

robertauer Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

meiserloh Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexander-dammeier Nov 29, 2024 •

edited

Loading

robertauer Dec 2, 2024 •

edited

Loading

robertauer Dec 3, 2024 •

edited

Loading

alexander-dammeier Nov 29, 2024 •

edited

Loading

robertauer Dec 2, 2024 •

edited

Loading

meiserloh Dec 2, 2024 •

edited

Loading