Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containerize all non-docker boefjes #3859

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

Donnype
Copy link
Contributor

@Donnype Donnype commented Nov 19, 2024

Changes

As the title suggests, this PR introduces a new OCI image having the requirements of all non-docker boefjes. Through the oci_image and oci_arguments we now specify the right script within this generic image. This, in short, is going to be the biggest jump in containerizing our boefjes, converting a total of 27 boefjes. In follow-up PRs, we will add a specialized worker that pulls tasks for this list of oois, and the 13 docker-boefjes will be turned into specialized images having the same interface as well.

Issue link

Closes #3698

Demo

QA notes

This is going to be a chunk of work, but to really verify these changes work we need to test whether the following boefjes (by their directory name because I could copy that easily) still run and produce the output we expect:

  • kat_dns
  • kat_snyk
  • kat_binaryedge
  • kat_censys
  • kat_crt_sh
  • kat_cve_2023_34039
  • kat_cve_2023_35078
  • kat_dicom
  • kat_dns_version
  • kat_dns_zone
  • kat_external_db
  • kat_fierce
  • kat_green_hosting
  • kat_leakix
  • kat_rdns
  • kat_rpki
  • kat_security_txt_downloader
  • kat_service_banner
  • kat_shodan
  • kat_webpage_analysis
  • pdio_subfinder
  • kat_adr_finding_types
  • kat_cve_finding_types
  • kat_cwe_finding_types
  • kat_kat_finding_types
  • kat_retirejs_finding_types
  • kat_snyk_finding_types

Another interesting note would be to see if there are any significant performance degradations.


Code Checklist

  • All the commits in this PR are properly PGP-signed and verified.
  • This PR only contains functionality relevant to the issue.
  • I have written unit tests for the changes or fixes I made.
  • I have checked the documentation and made changes where necessary.
  • I have performed a self-review of my code and refactored it to the best of my abilities.
  • Tickets have been created for newly discovered issues.
  • For any non-trivial functionality, I have added integration and/or end-to-end tests.
  • I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
  • I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.


Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

import httpx


def main():
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we iterate on this a bit, we should consider a refactor to deduplicate the code between this adapter and the regular one. The behavior is a bit different though, so for both simplicity and clarity, we chose to split them up for now

@Donnype
Copy link
Contributor Author

Donnype commented Nov 22, 2024

QA notes on broken boefjes:

  • Censys: no API key (frontend / UX issue)
  • Reverse DNS: "No module names boefjes.config
  • Security.txt: No connection adapters were found for 'ssl/http://xxx.x.x.x/.well-known/security.txt'
  • Subfinder: no module named 'docker'
  • Maxmind: no API key (frontend / UX issue)
  • KAT CVE 2023 35078: requests.exceptions.MissingSchema: Invalid URL '/mifs/c/windows/api/v2/device/registration': No scheme supplied. Perhaps you meant https:///mifs/c/windows/api/v2/device/registration?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Generic container for non-docker local boefjes
2 participants