Error commiting. Rollback commit started...

[fewirchen]

Please, I have tried process a big case and get an error after about 14 hours.

The syntax was:
iped.exe
-d "2_DataExtracted\target1"
-d "2_DataExtracted\target2"
-d "2_DataExtracted\target3"
-d "2_DataExtracted\target4"
-d "2_DataExtracted\target5"
-o "3_DataIndexed" -profile default –portable

The biggest folders: 178GB (target2), 232GB (target4) and 635GB (target 5).

The version of IPED is 3.18.14.

This happended several times ( Prepare commit started…
Commit finished in 38s) but after 14 hours, an error.

And the error was:
…
2022-04-02 09:15:24 [ERROR] [indexer.process.Manager$1] Error commiting. Rollback commit started...
2022-04-02 09:15:24 [WARN] [process.task.ParsingTask] ParsingThread-583 SAX error while extracting subitem 02_DataExtracted\target5\target5.Drive_033.zip>>Drive/Incoming/UNADJUSTEDNONRAW_mini_72fb.jpg org.apache.tika.sax.TaggedSAXException: Error writing: Drive/Incoming/UNADJUSTEDNONRAW_mini_72fb.jpg

org.xml.sax.SAXException: Error writing: Drive/Incoming/UNADJUSTEDNONRAW_mini_72fb.jpg

java.io.IOException: Pipe closed
2022-04-02 09:15:24 [WARN] [process.task.ParsingTask] ParsingThread-494 SAX error while extracting subitem 02_DataExtracted\target5\target5.Drive_034.zip>>Drive/Incoming/UNADJUSTEDNONRAW_thumb_b1ed.jpg org.apache.tika.sax.TaggedSAXException: Error writing: Drive/Incoming/UNADJUSTEDNONRAW_thumb_b1ed.jpg

org.xml.sax.SAXException: Error writing: Drive/Incoming/UNADJUSTEDNONRAW_thumb_b1ed.jpg

java.io.IOException: Pipe closed
2022-04-02 09:15:24 [INFO] [indexer.process.Worker] Worker-0 interrupted on 02_DataExtracted\target5\target5.Drive_033.zip (1619242131 bytes)
2022-04-02 09:15:24 [INFO] [indexer.process.Worker] Worker-7 interrupted on 02_DataExtracted\target5\target5.Drive_032.zip (2147981570 bytes)
2022-04-02 09:15:24 [INFO] [indexer.process.Worker] Worker-6 interrupted on 02_DataExtracted\target5\target5.Drive_034.zip (1879925388 bytes)
2022-04-02 09:15:24 [INFO] [indexer.process.Worker] Worker-5 interrupted on 02_DataExtracted\target5\target5.Drive_031_1.zip (2148692719 bytes)
2022-04-02 09:15:24 [INFO] [indexer.process.Worker] Worker-4 interrupted on 02_DataExtracted\target5\target5.Drive_030_1.zip (2147734742 bytes)
2022-04-02 09:15:25 [WARN] [process.task.ParsingTask] ParsingThread-561 SAX error while extracting subitem 02_DataExtracted\target5\target5.Drive_032.zip>>Drive/Incoming/UNADJUSTEDNONRAW_mini_9085.jpg org.apache.tika.sax.TaggedSAXException: Error writing: Drive/Incoming/UNADJUSTEDNONRAW_mini_9085.jpg

org.xml.sax.SAXException: Error writing: Drive/Incoming/UNADJUSTEDNONRAW_mini_9085.jpg

java.io.IOException: Pipe closed
2022-04-02 09:15:24 [INFO] [indexer.process.Worker] Worker-2 interrupted on 02_DataExtracted\target5\target5.Drive_033_1.zip (2149608547 bytes)
2022-04-02 09:15:24 [INFO] [indexer.process.Worker] Worker-3 interrupted on 02_DataExtracted\target5\target5.Drive_032_1.zip (2088624395 bytes)
2022-04-02 09:15:25 [INFO] [indexer.process.Worker] Worker-1 interrupted on 02_DataExtracted\target5\target5.Drive_031.zip (1807934733 bytes)
2022-04-02 09:15:25 [WARN] [process.task.ParsingTask] ParsingThread-580 SAX error while extracting subitem 02_DataExtracted\target5\target5.Drive_032_1.zip>>Drive/Incoming/UNADJUSTEDNONRAW_thumb_7b34.jpg org.apache.tika.sax.TaggedSAXException: Error writing: Drive/Incoming/UNADJUSTEDNONRAW_thumb_7b34.jpg

org.xml.sax.SAXException: Error writing: Drive/Incoming/UNADJUSTEDNONRAW_thumb_7b34.jpg

java.io.IOException: Pipe closed
2022-04-02 09:15:26 [ERROR] [indexer.process.Manager$1] Rollback commit finished.
2022-04-02 09:15:26 [ERROR] [gpinf.indexer.IndexFiles] Processing Error:
org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (cannot commit - no transaction is active)
at org.sqlite.core.DB.newSQLException(DB.java:941) ~[sqlite-jdbc-3.25.2.jar:?]
at org.sqlite.core.DB.newSQLException(DB.java:953) ~[sqlite-jdbc-3.25.2.jar:?]
at org.sqlite.core.DB.throwex(DB.java:918) ~[sqlite-jdbc-3.25.2.jar:?]
at org.sqlite.core.DB.exec(DB.java:178) ~[sqlite-jdbc-3.25.2.jar:?]
at org.sqlite.SQLiteConnection.commit(SQLiteConnection.java:404) ~[sqlite-jdbc-3.25.2.jar:?]
at dpf.sp.gpinf.indexer.process.task.ExportFileTask.commitStorage(ExportFileTask.java:741) ~[iped-engine-3.18.14.jar:?]
at dpf.sp.gpinf.indexer.process.Manager$1.run(Manager.java:544) ~[iped-engine-3.18.14.jar:?]

[lfcnassif]

Could you share the full processing log here or privately? Have you tried the --continue option to resume processing? Just use the same command line and add --continue at the end.

[fewirchen]

Please, "--continue" will bypass the process and will continue from where stopped or another bahavior?

[paulobreim]

Suggestion:
Try to run each one separately and from the second put the --append parameter.
So you will have each one with separate log and time.

[fewirchen]

Sorry, will bypass the error and will continue from where stopped or another bahavior?

[lfcnassif]

It'll try to continue the processing from last commit point: https://github.com/sepinf-inc/IPED/wiki/User-Manual#--continue-and---restart

Thanks for the log. Looking at it, seems you are running out of physical memory:
2022-04-02 09:00:18 [WARN] [process.task.ExportFileTask] Error exporting XXXXXX.HEIC org.sqlite.SQLiteException: [SQLITE_NOMEM] A malloc() failed (out of memory)

If --continue doesn't help, I recommend using other machine with more RAM to process this case. How much does your computer have?

[fewirchen]

I’ve tried the --continue option to resume processing but I got the same error.

The computer has 16GB of memory, I will try in a computer with 32GB.

If I put the structure already processed (folder indexador, FileList.csv etc) in this computer with 32GB, is possible resume?

[lfcnassif]

The computer has 16GB of memory, I will try in a computer with 32GB.

This is not a bad machine, this is looking weird...

If I put the structure already processed (folder indexador, FileList.csv etc) in this computer with 32GB, is possible resume?

I've never tried that, but since your index is in the case folder (result of indexTempOnSSD = false), it should work.

[fewirchen]

A feedback about this issue.

As I said, I put this case to resume in another computer with 32GB of memory, the same way, input in a external HD with 2TB and output in another HD with 2TB, it ran about more 20 hours and “IPED finished”. The total time was about 34 hours.

One thing that called my attention was:
input ===> the biggest folders: 178GB, 232GB and 635GB
output ===> 367GB
I have the idea that input and output must have similar size.

Two doubts?
In a way to confirm, should I run again?
Is viable turn on the OCR?

Thanks.

[lfcnassif]

Thanks for the feedback. Usually output case folder is much smaller than input evidences, but this can change depending on input data and processing options (e.g. if you configure CategoriesToExportConfig.txt)

This output size seems compatible with input size, I think there is no need to process again. But if you want OCR, currently you will need to process everything again from scratch.

I recommend to change indexTemp default value, pointing it to a SSD drive, that should speed up processing a lot.