ArrayIndexOutOfBoundsException: Index -65536 out of bounds for length 71428 at org.apache.lucene.index.TermsHashPerField.writeByte

[luizedum]

The following error was presented by IPED 4.1.6 during the processing of evidence in E01 format (parts of the file names was replaced as they contained personal information):

2024-09-11 11:44:34 [INFO] [task.index.IndexTask$DocumentsIterable$1] Worker-15 Splitting text of /ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/conta@icloud.com/Apple iCloud (Backup)/conta@icloud.com.ufdr>>report.xml>>report.xml_0
2024-09-11 11:44:34 [INFO] [engine.core.Worker] Worker-15 interrupted on /ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/conta@icloud.com/Apple iCloud (Backup)/conta@icloud.com.ufdr>>report.xml>>report.xml_0 (1950114112 bytes)
2024-09-11 11:44:35 [ERROR] [app.processing.Main] Processing Error:
java.lang.Exception: Worker-15 Error while processing /ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/conta@icloud.com/Apple iCloud (Backup)/conta@icloud.com.ufdr>>report.xml>>report.xml_0 (1950114112bytes)
at iped.engine.core.Worker.process(Worker.java:186) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Worker.run(Worker.java:265) ~[iped-engine-4.1.6.jar:?]
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -65536 out of bounds for length 71428
at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:219) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:241) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:86) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:127) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.TermsHashPerField.initStreamSlices(TermsHashPerField.java:175) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:198) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1224) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:729) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:241) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1532) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1503) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at iped.engine.task.index.IndexTask.process(IndexTask.java:149) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processMonitorTimeout(AbstractTask.java:277) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:192) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.MinIOTask.sendToNextTask(MinIOTask.java:416) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.PythonTask.sendToNextTask(PythonTask.java:278) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.PythonTask.sendToNextTask(PythonTask.java:278) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.PythonTask.sendToNextTask(PythonTask.java:278) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.EmbeddedDiskProcessTask.sendToNextTask(EmbeddedDiskProcessTask.java:116) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Worker.process(Worker.java:177) ~[iped-engine-4.1.6.jar:?]
... 1 more

[lfcnassif]

Thanks for reporting. Are you able to share the file:
/ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/conta@icloud.com/Apple iCloud (Backup)/conta@icloud.com.ufdr>>report.xml

which seems to have caused the exception, so we can try to reproduce the error and validate a possible fix?

[luizedum]

Do you prefer that I share the UFDR file or the 'report.xml' file inside it?
What is the best way to share it while ensuring data confidentiality?

Is it possible to process an evidence in IPED ignoring a specific file so that the processing of the other files can be completed?

[lfcnassif]

Do you prefer that I share the UFDR file or the 'report.xml' file inside it?

At first, I think just the report.xml file could be enough.

What is the best way to share it while ensuring data confidentiality?

You can send it directly to my email in my profile.

Is it possible to process an evidence in IPED ignoring a specific file so that the processing of the other files can be completed?

Yes, the procedure below can be used to ignore specific files:
https://github.com/sepinf-inc/IPED/wiki/Troubleshooting#processing-frozen

[luizedum]

Thank you, the file was shared via email.