Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add http README.md with field mapping and example log #91

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Add http README.md with field mapping and example log #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
5499168
Add http README.md with field mapping
JW-Corelight Nov 8, 2024
4d06d45
Add http_log.raw example log
JW-Corelight Nov 8, 2024
8883523
Update http_log.raw to valid JSON
JW-Corelight Nov 8, 2024
0f1954d
Update to metadata.loggers[] fields
JW-Corelight Nov 12, 2024
931af99
Update to metadata.logged_time
JW-Corelight Nov 13, 2024
741b438
Update http_request header record notes and unmapped .file. field pro…
JW-Corelight Nov 14, 2024
7654740
Add Zeek Field Description context; add Conditional mapping section
JW-Corelight Nov 21, 2024
d245f7d
Changed observables and http_headers to observables[] and http_headers[]
JW-Corelight Nov 21, 2024
8d08572
Convert Static mapping section to table
JW-Corelight Nov 21, 2024
8a1a0e1
Create http_log.ocsf
JW-Corelight Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions mappings/markdown/Zeek/http_log/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Event Dossier: Zeek http.log
### Summary:
- **Description**: Translates a Zeek http.log to OCSF.
- **Event References**:
- https://schema.ocsf.io/1.3.0/classes/http_activity
- https://docs.zeek.org/en/master/logs/http.html
- https://docs.zeek.org/en/master/scripts/base/protocols/http/main.zeek.html#type-HTTP::Info

### Static mapping
| OCSF field | Value |
| ----------------------------------- | ----------------------------------------------- |
| `metadata.version` | "1.3.0" |
| `category_name` | "Network Activity" |
| `category_uid` | "4" |
| `class_name` | "HTTP Activity" |
| `class_uid` | "4002" |
| `metadata.log_name` | "http.log" |
| `metadata.loggers[].log_provider` | "Zeek" |
| `metadata.loggers[].product.name` | "Zeek" |
| `metadata.product.cpe_name` | "cpe:2.3:a:zeek:zeek" |
| `metadata.product.feature.name` | "http.log" |
| `metadata.product.name` | "Zeek" |
| `metadata.product.url_string` | "https://docs.zeek.org/en/current/logs/http.html"|
| `metadata.product.vendor_name` | "Zeek" |
| `severity` | "Informational" |
| `severity_id` | "1" |

### Mapping:
| OCSF | Raw | Zeek Field Description |
| ------------------------------ | ----------------- | --------------------------------------------------------------------------------------- |
| `time` | `ts` | Timestamp indicating when the event occurred. |
| `start_time` | `ts` | Timestamp indicating when the event occurred. |
| `metadata.logged_time` | `_write_ts` | Timestamp indicating when the log entry was written to disk. |
| `metadata.loggers[].name` | `_system_name` | Name of the system or logging subsystem generating the log entry. |
| `metadata.uid` | `uid` | Unique ID for the connection. |
| `src_endpoint.ip` | `id.orig_h` | The originator’s IP address. |
| `src_endpoint.port` | `id.orig_p` | The originator’s port number. |
| `dst_endpoint.ip` | `id.resp_h` | The responder’s IP address. |
| `dst_endpoint.port` | `id.resp_p` | The responder’s port number. |
| `http_request.http_method` | `method` | Verb used in the HTTP request (GET, POST, HEAD, etc.). |
| `http_request.length` | `request_body_len`| Actual uncompressed content size of the data transferred from the client. |
| `http_request.referrer` | `referrer` | Value of the “referrer” header. |
| `http_request.url.hostname` | `dest_host` | (No description available) |
| `http_request.url.path` | `uri` | URI used in the request. |
| `http_request.user_agent` | `user_agent` | Value of the User-Agent header from the client. |
| `http_request.version` | `version` | Value of the version portion of the reply. |
| `http_request.x_forwarded_for` | `proxied` | All of the headers that may indicate if the request was proxied. |
| `http_response.code` | `status_code` | Status code returned by the server. |
| `http_response.length` | `response_body_len`| Actual uncompressed content size of the data transferred from the server. |
| `http_response.status` | `status_msg` | Status message returned by the server. |
| `message` | `tags` | A set of indicators of various attributes discovered and related to a particular request/response pair. |

### Conditional mapping:
Fields described here are subject to dynamic mappings contingent on a conditional evaluation of source data.
| OCSF | Raw | Evaluation Conditions | Zeek Field Description |
| --------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `http_request.http_cookies[].value` | `cookie` | Where `http_request.http_cookies[].name` = "Request Cookie" | Cookie value used by the client machine. |
| `http_request.http_headers[].value` | `accept_language` | Where `http_request.http_headers[].name` = "Accept Language" | |
| `http_request.http_headers[].value` | `accept_encoding` | Where `http_request.http_headers[].name` = "Accept Encoding" | |
| `http_request.http_headers[].value` | `accept` | Where `http_request.http_headers[].name` = "Accept" | |
| `http_request.http_headers[].value` | `post_body` | Where `http_request.http_headers[].name` = "Body" | The post_body information. |
| `http_request.http_headers[].value` | `origin` | Where `http_request.http_headers[].name` = "Origin" | Value of the Origin header from the client. |
| `http_request.http_headers[].value` | `client_headers` | Where `http_request.http_headers[].name` = "Client Headers" | |
| `http_response.http_cookies[].value`| `resp_cookie` | Where `http_response.http_cookies[].name` = "Response Cookie" | |
| `http_response.http_headers[].value`| `server_headers` | Where `http_response.http_headers[].name` = "Server Headers" | |
| `observables[].value` | `username` | Where `observables[].type_id` = "4" | Username if basic-auth is performed for the request. |

### Unmapped (proposed):
| OCSF | Raw | Zeek Field Description |
| ----------------------------------| -------------------------| --------------------------------------------------------------------------------------- |
| `http_request.(file.names[])` | `orig_filenames` | An ordered vector of filenames from the client. |
| `http_response.(file.names[])` | `resp_filenames` | An ordered vector of filenames from the server. |
| `http_request.(file.mime_types[])`| `orig_mime_types` | An ordered vector of mime types. |
| `http_response.(file.mime_types[])`| `resp_mime_types` | An ordered vector of mime types. |
| `http_request.(file.uids[])` | `orig_fuids` | An ordered vector of file unique IDs. |
| `http_response.(file.uids[])` | `resp_fuids` | An ordered vector of file unique IDs. |
| `unmapped` | `trans_depth` | Represents the pipelined depth into the connection of this request/response transaction.|
| `unmapped` | `if_none_match` | |
| `unmapped` | `if_modified_since` | |
136 changes: 136 additions & 0 deletions mappings/markdown/Zeek/http_log/samples/http_log.ocsf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
{
"time": "2024-10-16T02:43:57.734946Z",
"start_time": "2024-10-16T02:43:57.734946Z",
"category_name": "Network Activity",
"category_uid": 4,
"class_name": "HTTP Activity",
"class_uid": 4002,
"severity": "Informational",
"severity_id": 1,
"metadata": {
"version": "1.3.0",
"log_name": "http.log",
"logged_time": "2024-10-16T02:43:57.736852Z",
"loggers": [
{
"name": "sensor",
"log_provider": "Zeek",
"product": {
"name": "Zeek"
}
}
],
"product": {
"cpe_name": "cpe:2.3:a:zeek:zeek",
"feature": {
"name": "http.log"
},
"name": "Zeek",
"url_string": "https://docs.zeek.org/en/current/logs/http.html",
"vendor_name": "Zeek"
},
"uid": "CbNapWwSGFIOYRBzk"
},
"src_endpoint": {
"ip": "172.70.175.90",
"port": 26566
},
"dst_endpoint": {
"ip": "198.71.247.91",
"port": 80
},
"http_request": {
"http_method": "POST",
"length": 28,
"referrer": "anonymousfox.co",
"url": {
"hostname": "lifeisnetwork.com",
"path": "/wp-includes/css/wp-config.php"
},
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36",
"version": "1.1",
"x_forwarded_for": [
"X-FORWARDED-FOR -> 20.115.4.12"
],
"http_cookies": [
{
"name": "Request Cookie",
"value": "JSESSIONID=80DF1E116C9617F3EEAFBE46CF0A8E05"
}
],
"http_headers": [
{
"name": "Accept Language",
"value": "en-US,en;q=0.9,fr;q=0.8"
},
{
"name": "Accept Encoding",
"value": "gzip"
},
{
"name": "Accept",
"value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
},
{
"name": "Body",
"value": "1=echo%22AnonymousFox+%22%3B"
},
{
"name": "Origin",
"value": "http://172.0.0.101"
},
{
"name": "Client Headers",
"value": "HOST: lifeisnetwork.com; CONNECTION: Keep-Alive; ACCEPT-ENCODING: gzip; CF-IPCOUNTRY: US; X-FORWARDED-FOR: 20.115.4.12; CF-RAY: 6bc5aa001b3f6fbb-IAD; CONTENT-LENGTH: 28; X-FORWARDED-PROTO: https; CF-VISITOR: {\"scheme\":\"https\"}; ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8; USER-AGENT: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36; ACCEPT-LANGUAGE: en-US,en;q=0.9,fr;q=0.8; CACHE-CONTROL: max-age=0; REFERER: anonymousfox.co; UPGRADE-INSECURE-REQUESTS: 1; CONTENT-TYPE: application/x-www-form-urlencoded; CF-CONNECTING-IP: 20.115.4.12; CDN-LOOP: cloudflare"
}
]
},
"http_response": {
"code": 404,
"length": 279,
"status": "Not Found",
"http_cookies": [
{
"name": "Response Cookie",
"value": "SSID=eaf1bddcaafb7e25f4fe29a6dc0744f1; HttpOnly"
}
],
"http_headers": [
{
"name": "Server Headers",
"value": "DATE: Sun, 12 Dec 2021 08:43:15 GMT; SERVER: Apache/2.4.41 (Ubuntu); CONTENT-LENGTH: 279; KEEP-ALIVE: timeout=5, max=100; CONNECTION: Keep-Alive; CONTENT-TYPE: text/html; charset=iso-8859-1"
}
]
},
"message": "CVE_2021_44228::LOG4J_RCE",
"observables": [
{
"value": "tomcat",
"type_id": "4",
"type": "Username"
}
],
"unmapped": {
"orig_filenames": [
"payload.zip"
],
"resp_filenames": [
"ISRG Root X1.der"
],
"orig_mime_types": [
"text/plain"
],
"resp_mime_types": [
"text/html"
],
"orig_fuids": [
"FDDthg48f7r5xYMkAf"
],
"resp_fuids": [
"Fa3Nye3upzqg6Rruoa"
],
"trans_depth": 1,
"if_none_match": "\"80424021c7dbd21:0\"",
"if_modified_since": "Fri, 02 Jun 2017 17:39:05 GMT"
}
}
88 changes: 88 additions & 0 deletions mappings/markdown/Zeek/http_log/samples/http_log.raw
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
{
"_path": "http",
"_system_name": "sensor",
"_write_ts": "2024-10-16T02:43:57.736852Z",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"accept_encoding": "gzip",
"accept_language": "en-US,en;q=0.9,fr;q=0.8",
"client_headers": [
"HOST: lifeisnetwork.com",
"CONNECTION: Keep-Alive",
"ACCEPT-ENCODING: gzip",
"CF-IPCOUNTRY: US",
"X-FORWARDED-FOR: 20.115.4.12",
"CF-RAY: 6bc5aa001b3f6fbb-IAD",
"CONTENT-LENGTH: 28",
"X-FORWARDED-PROTO: https",
"CF-VISITOR: {\"scheme\":\"https\"}",
"ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"USER-AGENT: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36",
"ACCEPT-LANGUAGE: en-US,en;q=0.9,fr;q=0.8",
"CACHE-CONTROL: max-age=0",
"REFERER: anonymousfox.co",
"UPGRADE-INSECURE-REQUESTS: 1",
"CONTENT-TYPE: application/x-www-form-urlencoded",
"CF-CONNECTING-IP: 20.115.4.12",
"CDN-LOOP: cloudflare"
],
"cookie": [
"JSESSIONID=80DF1E116C9617F3EEAFBE46CF0A8E05"
],
"dest_host": "lifeisnetwork.com",
"id.orig_h": "172.70.175.90",
"id.orig_p": 26566,
"id.resp_h": "198.71.247.91",
"id.resp_p": 80,
"if_modified_since": "Fri, 02 Jun 2017 17:39:05 GMT",
"if_none_match": "\"80424021c7dbd21:0\"",
"method": "POST",
"orig_filenames": [
"payload.zip"
],
"orig_fuids": [
"FDDthg48f7r5xYMkAf"
],
"orig_mime_types": [
"text/plain"
],
"origin": "http://172.0.0.101",
"post_body": "1=echo%22AnonymousFox+%22%3B",
"proxied": [
"X-FORWARDED-FOR -> 20.115.4.12"
],
"referrer": "anonymousfox.co",
"request_body_len": 28,
"resp_cookie": [
"SSID=eaf1bddcaafb7e25f4fe29a6dc0744f1; HttpOnly"
],
"resp_filenames": [
"ISRG Root X1.der"
],
"resp_fuids": [
"Fa3Nye3upzqg6Rruoa"
],
"resp_mime_types": [
"text/html"
],
"response_body_len": 279,
"server_headers": [
"DATE: Sun, 12 Dec 2021 08:43:15 GMT",
"SERVER: Apache/2.4.41 (Ubuntu)",
"CONTENT-LENGTH: 279",
"KEEP-ALIVE: timeout=5, max=100",
"CONNECTION: Keep-Alive",
"CONTENT-TYPE: text/html; charset=iso-8859-1"
],
"status_code": 404,
"status_msg": "Not Found",
"tags": [
"CVE_2021_44228::LOG4J_RCE"
],
"trans_depth": 1,
"ts": "2024-10-16T02:43:57.734946Z",
"uid": "CbNapWwSGFIOYRBzk",
"uri": "/wp-includes/css/wp-config.php",
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36",
"username": "tomcat",
"version": "1.1"
}