Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Micro102]

Well, was playing agrarian skies, finished the seared brick blocks quest, selected the heart. Turns out it was a rotten heart (really stupid that they put it as a reward, in which it's sole purpose was to try to trick people into dieing) which deleted my world. I know I had at least 2 lives left, so there goes hours of progress because of a bad joke and what I assume to be a bug.

[Skyrunner]

Interesting

The file itself is around 400 MB, but it has a json internal representation (2), then Java gets its own version (3), so at least 1.2 GB of memory. 

For the first problem, it's probably a logic error. The second one, I wouldn't mind being able to make "subregions" (4, probably), but I might be able to get around it by lazily initiating the array. Right now I just initialize all 500x500x500 blocks. Which made me realize one issue: the height is hardcoded as 255 >_> Will fix that when I get home. Another method will be to only add Y arrays when they're needed.

[MagmaMcFry]

Oooh, how about this: I could make it so you can stream multiple JSON objects containing either a level definition or only one region each (which gets immediately written out), which means that there'll never be any significant memory buildup at all, since there's always at most one region loaded in memory of either program.

[Skyrunner]

A level definition?

As for the region thing, 1 region is already too large to fit in the memory of two programs at once, it seems! 500 is only a tad smaller than 512.

[scrdest]

Yay, new update for TerraFirmaCraft is out! I can finally experience malnutrition due to poor food storage practices!

[Skyrunner]

MagmaMcFry: Here's the next error!
updated gist

java.lang.ClassCastException: org.json.JSONObject$Null cannot be cast
ng.Integer
        at mcfry.MCJsonDimension.create(MCJsonDimension.java:27)
        at mcfry.MCJsonWorld.create(MCJsonWorld.java:17)
        at mcfry.JsonWorldGenerator.main(JsonWorldGenerator.java:23)

I assume you still have the spheregen.py-generated files.


I found that a 512x256x512 file uses about 2.2 GB of RAM. That doesn't sound very good :c

[MagmaMcFry]

A level definition?

As for the region thing, 1 region is already too large to fit in the memory of two programs at once, it seems! 500 is only a tad smaller than 512.

Only if you actually define every single block. Interesting info: Minecraft subdivides its chunk NBT format into sixteen 16x16x16 "sections", of which only those that aren't all air need to be defined. My program generates these sections on demand, and stores them in native arrays in memory, so it uses up only 4kb per section you touch.

MagmaMcFry: Here's the next error!
updated gist

java.lang.ClassCastException: org.json.JSONObject$Null cannot be cast
ng.Integer
        at mcfry.MCJsonDimension.create(MCJsonDimension.java:27)
        at mcfry.MCJsonWorld.create(MCJsonWorld.java:17)
        at mcfry.JsonWorldGenerator.main(JsonWorldGenerator.java:23)

I assume you still have the spheregen.py-generated files.

This happens because None values in your Python array would convert to Java nulls, which would complain when you converted them to integers.

Anyway, I did stuff to fix the memory issues.

I rewrote the JSON format again, and now you can basically send commands to the worldgen thing by writing JSON objects to STDOUT. There are three types of commands: Block commands which let you define a cuboid of blocks and a rectangle of biomes anywhere in any dimension (by specifying an initial (X, Y, Z) offset), level commands which generate a level.dat, and region export commands which tell Java to export a specific region to a region file and clear it from memory. This means that your Python program only ever needs to hold in memory the contents of a single sphere, which it can completely flush to Java before rendering the next sphere. And don't worry about holding several regions in Java's memory at once: I've estimated a region to take up at most 100MB of space in that memory, and that will only happen if you touch every single chunk section of that region (which you definitely won't by a large margin), but try to export those regions that you're sure you won't touch again.

To avoid overwriting spheres with the blank space caused by the cube of air around later spheres, you can now use the block IDs of None or -1 to denote 'don't overwrite this block please'.

Also note that you may not have any characters or spaces or newlines between adjacent JSON commands in the pipe.

Oh, and for any other programmers in this thread: Obviously you don't need to send commands using Python; any language will do (as long as you have some sort of JSON library available).

[frostshotgg]

Just a thought: Could you just cut off anything above 255Y? If you publish this, which you totally should, all contingencies should be covered.

[Skyrunner]

Right now, Java crashes with an error message instead of failing silently. I think the former Is mkre preferable than the latter, but probably with a better error message.

[MagmaMcFry]

What's the error message? The current program always displays at least one error message because I was too lazy to filter out the JSON end-of-input error message from all the other error messages. It also prints out a line like "Successfully parsed N JSON objects", and if N equals the number of commands you sent, then everything went right (even though that error message is displayed). I may add a "Thanks I'm done" command type later that you can send after all the other commands, just so that doesn't show up again.

Just a thought: Could you just cut off anything above 255Y? If you publish this, which you totally should, all contingencies should be covered.

I fixed the code to accommodate for that, and it'll be fixed in the next build, but I'm not going to publish this until at least I've added functionality to load existing level and region files and modify those.

[MagmaMcFry]

-doublepost-

[Skyrunner]

What's the error message? The current program always displays at least one error message because I was too lazy to filter out the JSON end-of-input error message from all the other error messages. It also prints out a line like "Successfully parsed N JSON objects", and if N equals the number of commands you sent, then everything went right (even though that error message is displayed). I may add a "Thanks I'm done" command type later that you can send after all the other commands, just so that doesn't show up again.

Just a thought: Could you just cut off anything above 255Y? If you publish this, which you totally should, all contingencies should be covered.

I fixed the code to accommodate for that, and it'll be fixed in the next build, but I'm not going to publish this until at least I've added functionality to load existing level and region files and modify those.

It was meant as a reply to frostshotgg

[Skyrunner]

Wow, that Bay12 crash.

Here's an actual error:



Strange thing is, it works perfectly fine on Linux. Maybe it's some Windows-Linux difference, like line endings?

Also, I think a nice convenience function to have would be "datatype":"exportAll", since I might not know the regions i touched.

[MagmaMcFry]

Oh, I see your problem: You had an older MCWorldgen version on Windows. Updating should fix that problem (it's always the same link).

For your convenience, I added new data types "exportAll" to export all remaining chunks, and "thanks" to quit without displaying an error.

[Skyrunner]

Well, that makes sense. How embarrassing >_>


edit: Hmm :/ I feel bad for cluttering the thread with a bunch of errors.