Stream finished in an invalid state, uncompression failed

[kilsen]

I'm getting the error "Stream finished in an invalid state, uncompression failed" on some ZIP files. Do you know what might be the cause? When it happens, is there a proper way to shut down the stream so that it no longer produces entries?

[mhr3]

Difficult to say without the zip file itself, might be a fixable bug, might be that the zip file was produced in a way that makes it impossible to process by streaming.

From README:

Please note that the zip file format isn't really meant to be processed by streaming, though this library should succeed in most cases, if you do have complete zip file available, you should consider using other libraries which read zip files from the end - as originally intended (for example yauzl or decompress-zip).

[kilsen]

Yep, that’s what I figured, since I’m actually reading it from a stream that’s coming from an Amazon S3 object :-). Best to switch the approach, I guess, and copy the object from the S3 bucket to a local file, then process it. - Kevin

…

On Feb 4, 2018, at 5:05 PM, Michal Hruby ***@***.***> wrote: Difficult to say without the zip file itself, might be a fixable bug, might be that the zip file was produced in a way that makes it impossible to process by streaming. From README: Please note that the zip file format isn't really meant to be processed by streaming, though this library should succeed in most cases, if you do have complete zip file available, you should consider using other libraries which read zip files from the end - as originally intended (for example yauzl or decompress-zip). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOsDQ_Zxp_EhewMa3vohHXbsvNNoc2bks5tRinHgaJpZM4R4rK3>.

[mhr3]

If you have control over the original zip file you can just try to compress it with a different program - for example the Mac archive utility should produce zips that can be always processed with a streaming unzip.

[kilsen]

Thanks — I’ll keep that in mind, but for this project I need to process several thousand ZIP files that already exist. - Kevin

…

On Feb 4, 2018, at 5:10 PM, Michal Hruby ***@***.***> wrote: If you have control over the original zip file you can just try to compress it with a different program - for example the Mac archive utility should produce zips that can be always processed with a streaming unzip. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOsDQWXvymimt7YpDjUb4WDADvXa5SFks5tRirLgaJpZM4R4rK3>.

[mhr3]

Fair enough, if you have an example zip that can be shared, I can have a look if it's something fixable.

[qburst-akhil]

Hi,
I got this same issue, if I downlaod the zip file and uncompress it locally its works without any problem.
But if I try to stream the file over network and uncompress it using pipe it throws this error.

var readStream = sftp.createReadStream(/Import/${fileNameArray[0]}); readStream.pipe(unzip.Parse()) .pipe(stream.Transform({ objectMode: true, transform: (entry, encoding, callback) => { console.log(Inside transform callback....); const outputPath = path.join(dir, entry.path); entry.pipe(fs.createWriteStream(outputPath)) .on('finish', () => { console.log(Done writing file ${entry.path}); }) .on('error', err => { console.log(Error writing stream ${JSON.stringify(err)}); }); } }));

Erro thrown is as shown below:

Error: Stream finished in an invalid state, uncompression failed
at UnzipStream._flush (/home/user/Nodejs/fileDownload/node_modules/unzip-stream/lib/unzip-stream.js:734:19)
at UnzipStream.prefinish (_stream_transform.js:137:10)
at emitNone (events.js:106:13)
at UnzipStream.emit (events.js:208:7)
at prefinish (_stream_writable.js:603:14)
at finishMaybe (_stream_writable.js:611:5)
at endWritable (_stream_writable.js:622:3)
at UnzipStream.Writable.end (_stream_writable.js:573:5)
at ParserStream._flush (/home/user/Nodejs/fileDownload/node_modules/unzip-stream/lib/parser-stream.js:33:22)
at ParserStream.prefinish (_stream_transform.js:137:10)

[mhr3]

@qburst-akhil #14 (comment)

[pomelio]

I got same error.

Before unzip, I print out the file size. The file size is correct.
when the program second time unzip the file. It runs successfully.

(node:25) UnhandledPromiseRejectionWarning: Error: Stream finished in an invalid state, uncompression failed
appstore-api-server_1_450bccf02a4c | at UnzipStream._flush (/node_modules/unzip-stream/lib/unzip-stream.js:734:19)
appstore-api-server_1_450bccf02a4c | at UnzipStream.prefinish (_stream_transform.js:137:10)
appstore-api-server_1_450bccf02a4c | at emitNone (events.js:106:13)
appstore-api-server_1_450bccf02a4c | at UnzipStream.emit (events.js:208:7)
appstore-api-server_1_450bccf02a4c | at prefinish (_stream_writable.js:602:14)
appstore-api-server_1_450bccf02a4c | at finishMaybe (_stream_writable.js:610:5)
appstore-api-server_1_450bccf02a4c | at endWritable (_stream_writable.js:621:3)
appstore-api-server_1_450bccf02a4c | at UnzipStream.Writable.end (_stream_writable.js:572:5)
appstore-api-server_1_450bccf02a4c | at Extract._flush (/node_modules/unzip-stream/lib/extract.js:41:22)
appstore-api-server_1_450bccf02a4c | at Extract.prefinish (_stream_transform.js:137:10)

[drmrbrewer]

Getting the same error occasionally:

Error: Stream finished in an invalid state, uncompression failed
	at UnzipStream._flush (/usr/src/app/node_modules/unzip-stream/lib/unzip-stream.js:734:19)
	at UnzipStream.prefinish (_stream_transform.js:142:10)
	at UnzipStream.emit (events.js:314:20)
	at prefinish (_stream_writable.js:628:14)
	at finishMaybe (_stream_writable.js:636:5)
	at endWritable (_stream_writable.js:656:3)
	at UnzipStream.Writable.end (_stream_writable.js:595:5)
	at ParserStream._flush (/usr/src/app/node_modules/unzip-stream/lib/parser-stream.js:33:22)
	at ParserStream.prefinish (_stream_transform.js:142:10)
	at ParserStream.emit (events.js:314:20)

For some context, I am piping a read stream into unzip-stream from a remote file (fetched using gaxios using responseType: 'stream'). Mostly this works fine. When it doesn't work (with the above error) it is always when the node on which this is running is also fetching and processing lots of other files (in other node apps running on the same node)... so resources are somewhat limited. My guess is that the remote fetch operation times out after some time, so that the stream into unzip-stream ends prematurely. Is there any way to avoid this?

Incidentally, I get an equivalent error under the same circumstances with related node-unzipper... see here... I came to this library in the hope that it performed better in these circumstances but it looks like it fails in a similar way.