Chunker not converting world and taking too long to download

[kleines-L]

Chunker Version

1.4.0-main-86becbf

Input World Version

Bedrock 1.21.40

Output World Version

Bedrock 1.21.40

Platform / Operating System

win32-x64-10.0.22631

Issue Description

Error Displayed: A fatal error occurred during conversion.
Stack Trace:

java.util.concurrent.CompletionException: org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
	at com.hivemc.chunker.scheduling.task.Environment.cancel(Environment.java:85)
	at com.hivemc.chunker.conversion.WorldConverter.cancel(WorldConverter.java:609)
	at com.hivemc.chunker.conversion.WorldConverter.logFatalException(WorldConverter.java:441)
	at com.hivemc.chunker.scheduling.task.executor.TaskExecutor.handleException(TaskExecutor.java:109)
	at com.hivemc.chunker.scheduling.task.WrappedTask.lambda$setupFuture$0(WrappedTask.java:57)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
	at com.hivemc.chunker.scheduling.task.executor.TaskExecutorSupplier.run(TaskExecutorSupplier.java:44)
	at com.hivemc.chunker.scheduling.task.executor.TaskExecutor.threadLoop(TaskExecutor.java:178)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.OutOfMemoryError: Java heap space
	at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:550)
	at org.iq80.leveldb.impl.DbImpl.createWriteBatch(DbImpl.java:1072)
	at com.hivemc.chunker.conversion.encoding.bedrock.v1_18_30.writer.ColumnWriter.writeMetadata(ColumnWriter.java:112)
	at com.hivemc.chunker.scheduling.function.ThrowableConsumer.invoke(ThrowableConsumer.java:20)
	at com.hivemc.chunker.scheduling.function.ThrowableConsumer.invoke(ThrowableConsumer.java:8)
	at com.hivemc.chunker.scheduling.task.WrappedTask.apply(WrappedTask.java:84)
	at com.hivemc.chunker.scheduling.task.WrappedTask$WrappedTaskConstant.apply(WrappedTask.java:129)
	at com.hivemc.chunker.scheduling.task.WrappedTask.get(WrappedTask.java:77)
	at com.hivemc.chunker.scheduling.task.executor.TaskExecutorSupplier.run(TaskExecutorSupplier.java:42)
	... 2 more
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.nio.HeapByteBuffer.<init>(Unknown Source)
	at java.base/java.nio.ByteBuffer.allocate(Unknown Source)
	at org.iq80.leveldb.util.ZLib.uncompress(ZLib.java:49)
	at org.iq80.leveldb.table.Table.readRawBlock(Table.java:202)
	at org.iq80.leveldb.table.Table.readBlock(Table.java:157)
	at org.iq80.leveldb.table.Table.openBlock(Table.java:137)
	at org.iq80.leveldb.table.Table.openBlock(Table.java:121)
	at org.iq80.leveldb.table.Table.lambda$iterator$0(Table.java:116)
	at org.iq80.leveldb.table.Table$$Lambda$406/0x000000080022ddc0.apply(Unknown Source)
	at org.iq80.leveldb.iterator.TwoLevelIterator.initDataBlock(TwoLevelIterator.java:135)
	at org.iq80.leveldb.iterator.TwoLevelIterator.skipEmptyDataBlocksForward(TwoLevelIterator.java:108)
	at org.iq80.leveldb.iterator.TwoLevelIterator.internalNext(TwoLevelIterator.java:80)
	at org.iq80.leveldb.iterator.ASeekingIterator.next(ASeekingIterator.java:97)
	at org.iq80.leveldb.iterator.InternalTableIterator.next(InternalTableIterator.java:66)
	at org.iq80.leveldb.iterator.TwoLevelIterator.internalNext(TwoLevelIterator.java:80)
	at org.iq80.leveldb.iterator.ASeekingIterator.next(ASeekingIterator.java:97)
	at org.iq80.leveldb.iterator.MergingIterator.internalNext(MergingIterator.java:87)
	at org.iq80.leveldb.iterator.ASeekingIterator.next(ASeekingIterator.java:97)
	at org.iq80.leveldb.impl.DbImpl.doCompactionWork(DbImpl.java:1398)
	at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:638)
	at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:562)
	at org.iq80.leveldb.impl.DbImpl$$Lambda$424/0x000000080023ad18.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	... 1 more

So, I tried converting a world and replacing glass with structure voids but every time I try it, the chunker app crashes at 40-50% converting status! If I try it on the website, everything works except for the download! It says it downloads the 130mb world in 50KB-500KB per second and after around 25% of the file being downloaded, the download speed goes to 0 B/s, then stops and cancels the download!

[FormallyMyles]

For the running out of memory it may be related to #125, or you don't have enough memory and might want to try a different PC.

Regarding the website, it might be related to your internet connection as worlds bigger than that usually download fine for me. You might want to try on a different web browser.

You can use the command-line version of the app and specify how much RAM you want to use if you're confident in doing that (the guide is in the README on the main GitHub page).

[kleines-L]

I don't think that the problem is my internet connection but rather something server related to chunker. I tried it on a different browser and again the download speed is only 100kb/s to 500kb/s but I have a internet connection of 25MB/s! If I use the app it takes up to 5GB of RAM and a lot of CPU Power but at 50% it crashes randomly again! I don't really have another PC laying around better than mine. I have 16GB of ram and a intel i5-11600KF that should handle this. Usually it works but with this specific world not and I don't know why. It is mostly a void world with blocks placed from 0, 0 to 5k, 5k and a lot of structure voids. Could that maybe the problem?

Also can you guide me through using the command-line version because I can't really figure it out. Do I have to install something other than the app or the source code? And where do I get Java 17 from because I think that I have it already installed. And do you think it would be necessary to try this because chunker already uses enough RAM so I don't think that that should be the problem.

[FormallyMyles]

For big maps Chunker unfortunately can consume quite a bit of RAM (which seems like it given it's potentially 5000x5000).

Would you mind providing what input to output version / the goal of using Chunker is? (Since your original message said Bedrock 1.21.40 -> Bedrock 1.21.40).

The current algorithm for Chunker means the UI uses 25% of your RAM as a maximum, meaning it will only be using up to 4GB on your system, we'll hopefully improve this in the next release if you just want to wait for that (likely in the next few days).

In terms of running the command-line version, it would be opening command prompt in the directory of where you have the chunker-cli.jar and doing java -Xmx8G -jar chunker-cli.jar -i input_world_folder -o output_world_folder -f BEDROCK_R21_40. If you have Java installed it should in theory work, that command uses 8GB of ram as a maximum and you'd need to put the input world in the same folder / change the input_world_folder parameter. There are no other requirements other than Java 17 or above.

The current 1.4.0 chunker-cli jar is on our releases page - chunker-cli-1.4.0.jar.

Regarding the web version, it seems like the connection just gets interrupted, perhaps try something like Firefox / Edge assuming you're using Chrome. Otherwise, if you want to email the world to chunker-public@hivemc.com we can have the team look further into the connectivity issues to see if they can replicate it.

[kleines-L]

I managed to use the cli version but the same things happen. It gets stuck at 36.93% then stops and gives me the same error. I used blockmapping every time with structure_voids replacing glass. Also these (errors):

Missing block mapping for {name:"minecraft:deny",states:{},version:18163713}
Missing entity mapping for ChunkerCustomEntityType{identifier='star:bossbar'}
Missing entity mapping for ITEM
[174.688s][warning][gc,alloc] leveldb-db-0: Retried waiting for GCLocker too often allocating 261762 words
[212.531s][warning][gc,alloc] Task Processor 3: Retried waiting for GCLocker too often allocating 66960 words
[269.794s][warning][gc,alloc] Task Processor 2: Retried waiting for GCLocker too often allocating 51061 words

Also I tried Edge, Chrome and Opera but all three browsers have the same download speed. I even tried it on my phone but there chunker just "says something went wrong" and wants me to restart.
I've also sent the world to the email.

[FormallyMyles]

I managed to use the cli version but the same things happen. It gets stuck at 36.93% then stops and gives me the same error. I used blockmapping every time with structure_voids replacing glass. Also these (errors):

Missing block mapping for {name:"minecraft:deny",states:{},version:18163713} Missing entity mapping for ChunkerCustomEntityType{identifier='star:bossbar'} Missing entity mapping for ITEM [174.688s][warning][gc,alloc] leveldb-db-0: Retried waiting for GCLocker too often allocating 261762 words [212.531s][warning][gc,alloc] Task Processor 3: Retried waiting for GCLocker too often allocating 66960 words [269.794s][warning][gc,alloc] Task Processor 2: Retried waiting for GCLocker too often allocating 51061 words

Also I tried Edge, Chrome and Opera but all three browsers have the same download speed. I even tried it on my phone but there chunker just "says something went wrong" and wants me to restart. I've also sent the world to the email.

If you're still getting the memory errors you would need to increase the memory allowed, java -Xmx10G -jar chunker-cli.jar... etc where 10G is ten gigabytes of RAM, if it's not set it will use 4GB of RAM by default.

[FormallyMyles]

Just to update you on this, we did try to replicate the issue with the online version with your map and we weren't able to, and it converted / downloaded fine.