Unable to store reports http 404

[lp-fgo]

Hello,
I am trying to get dmarc reports from IMAP using the master version (as the current version from splunkbase wasn't working) without any success.
So far, i tried App Version 3.2.4 and App Version 3.2.5, each time trying kv store and json

Splunk HF : 8.2.2.1
TA-dmarc App Version 3.2.5

Here is a quick debug from the log file while trying to get results in json mode :

2021-12-16 16:17:36,746 INFO pid=2559 tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_messages: 10898 messages in folder INBOX match subject "Report domain:"
2021-12-16 16:17:36,747 INFO pid=2559 tid=MainThread file=splunk_rest_client.py:_request_handler:105 | Use HTTP connection pooling
2021-12-16 16:17:36,747 DEBUG pid=2559 tid=MainThread file=binding.py:get:677 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/config/TA_dmarc_checkpointer (body: {})
2021-12-16 16:17:36,749 DEBUG pid=2559 tid=MainThread file=connectionpool.py:_new_conn:959 | Starting new HTTPS connection (1): 127.0.0.1:8089
2021-12-16 16:17:36,755 DEBUG pid=2559 tid=MainThread file=connectionpool.py:_make_request:437 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-dmarc/storage/collections/config/TA_dmarc_checkpointer HTTP/1.1" 200 5287
2021-12-16 16:17:36,756 DEBUG pid=2559 tid=MainThread file=binding.py:new_f:73 | Operation took 0:00:00.008821
2021-12-16 16:17:36,756 DEBUG pid=2559 tid=MainThread file=binding.py:get:677 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/config/ (body: {'count': -1, 'offset': 0, 'search': 'TA_dmarc_checkpointer'})
2021-12-16 16:17:36,759 DEBUG pid=2559 tid=MainThread file=connectionpool.py:_make_request:437 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-dmarc/storage/collections/config/?count=-1&offset=0&search=TA_dmarc_checkpointer HTTP/1.1" 200 4439
2021-12-16 16:17:36,759 DEBUG pid=2559 tid=MainThread file=binding.py:new_f:73 | Operation took 0:00:00.003181
2021-12-16 16:17:36,760 DEBUG pid=2559 tid=MainThread file=binding.py:get:677 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/imaps.XxXxXxX_181196 (body: {})
2021-12-16 16:17:36,763 DEBUG pid=2559 tid=MainThread file=connectionpool.py:_make_request:437 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/imaps.XxXxXxX_181196 HTTP/1.1" 404 140
2021-12-16 16:17:36,763 DEBUG pid=2559 tid=MainThread file=binding.py:get:677 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/imaps.XxXxXxX_181197 (body: {})
2021-12-16 16:17:36,765 DEBUG pid=2559 tid=MainThread file=connectionpool.py:_make_request:437 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/imaps.XxXxXxX_181197 HTTP/1.1" 404 140
2021-12-16 16:17:36,766 DEBUG pid=2559 tid=MainThread file=binding.py:get:677 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/imaps.XxXxXxX_181198 (body: {})
2021-12-16 16:17:36,768 DEBUG pid=2559 tid=MainThread file=connectionpool.py:_make_request:437 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/imaps.XxXxXxX_181198 HTTP/1.1" 404 140

Thanks for your help

[jorritfolmer]

The 404s you're seeing are normal. It's the add-on checking if some new email has already been imported by checking the KVstore. Since you're looking at the DEBUG log these micro actions show up but are not indicative of failure.

Below are INFO-level logs that show a happy path. This is on a clean Splunk install so it will fetch all emails before processing. You can use it to compare to your logs. This will probably help you pinpoint where things stop, slow-down or break so you can look for the real root cause.

2022-10-07 10:36:31,925 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | Start processing imap server [imap.gmail.com](http://imap.gmail.com/) with use_ssl True
2022-10-07 10:36:34,700 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_messages: 648 messages in folder INBOX
2022-10-07 10:36:35,010 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_messages: 629 messages in folder INBOX match subject "Report domain:"
2022-10-07 10:36:35,011 INFO pid=5052 tid=MainThread file=splunk_rest_client.py:_request_handler:99 | Use HTTP connection pooling
2022-10-07 10:36:59,661 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | Start processing 629 new messages of 629 on [imap.gmail.com](http://imap.gmail.com/)
2022-10-07 10:36:59,661 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_message_bodies: getting messages 0 to 100
2022-10-07 10:37:17,951 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_message_bodies: getting messages 100 to 200
2022-10-07 10:37:39,604 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_message_bodies: getting messages 200 to 300
2022-10-07 10:37:58,173 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_message_bodies: getting messages 300 to 400
2022-10-07 10:38:17,020 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_message_bodies: getting messages 400 to 500
2022-10-07 10:38:35,651 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_message_bodies: getting messages 500 to 600
2022-10-07 10:38:51,629 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_message_bodies: getting messages 600 to 629
2022-10-07 10:39:25,538 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | Ended processing 629 new messages with 629 attachments
2022-10-07 10:39:25,539 INFO pid=5052 tid=MainThread file=base_modinput.py:log_info:295 | Ended processing imap server [imap.gmail.com](http://imap.gmail.com/)

After this, the add-on will start process the downloaded attachments. It then starts logging Start processing zip file ... for every file it encounters.