WU Map/Terrain Generator Development

[Sklo:D 2,210]

We'll hopefully have documentation of the file format out soon, though your guess isn't too far off I don't think.

Edited September 30, 2015 by Sklo:D

[Jonneh 82]

More teaser from my map gen.

 

[Ricowan 199]

 

Reposted from Steam...did not realize all the WU discussion was here as I stopped posting then lurking here long before my deeds got reclaimed to forest.

 

I call my terrain project 'Kraz Muze Isles'

<snip>

 

I really like this.  Do you have any of your code published, or maybe a Blog discussing your techniques?  I wrote some island generator code for a friend's game a couple years ago, but never got erosion working to my satisfaction.

[Warlander 2,730]

More teaser from my map gen.

 

Just keep in mind that maps must be surrounded by water and larger or equal to 1024X1024 (otherwise server will crash on startup).

[enoofu 206]

Always wondered if the heightmap = size of the map since each corner has a different height ingame

[ElvenElder 139]

The heightmap is the first step in the process of map generation. It can either be the same size as the map, or a smaller scaled version that's later used to product the final map. For example, you can have a 4096 map made by bicubic interpolation of a 512 heightmap and so on. I assume at at the end of the process you will have to provide data to the server for each tile of the actual map. So for a 4096 x 4096 map you will have to provide the same size of input data which is 4096 x 4096 tile data.

 

Generally making a map is much more than the heightmap. If you look closely at Jonneh's map sample there are coast lines and beaches with sand, which is the work of post processing agents on top of the heightmap.

Edited September 30, 2015 by ElvenElder

[enoofu 206]

Basically I'm asking is the height-map

 On a 1:1 image to map conversion, one pixel from the image file represents a corner of 4 tiles.

 

Example 1 tile represented by 4 pixel, 4 tiles are represented by 8 pixels. 

 

"Markus_Persson

For wurm online, we have maps of 4096*4096 tiles. Each tile is 4*4 meters.
We use world machine to generate the base heightmaps, then process them through a custom tool to plant all trees and so."

[Budda 2,675]

It depends on what map generator you're using enoofu. For the one that made the WO maps, the heightmap is 1:1 pixel:tile ratio, so it takess in a 4096x4096 heightmap and outputs a 4096x4096 map (meaning 16km^2).

 

As ElvenElder said, you can quite easily take in a heightmap of any size and output tiles of a different size with a bit of interpolation, just depends on how you want to program it yourself.

[yarnevk 257]

See back earlier in the thread for the mesh plot most terrain engines use, importing a 16b greyscale heightmap the pixels would represent the mesh intersections - the corners. So yes 4 pixels is one tile, but it is actually 9 pixels defines four tiles.   It could be unusually done the other way that the pixel represents the tile between the corners, but then the engine has to interpolate what each corner height would be.  I would guess since in Wurm digging is done at mesh corners and not tile centers it is done the usual way.

 

Wurm is further complicated with the dirt vs. rock layer - are those both height maps or is one a delta from the other.  I would guess both height maps, since in game you dig surface mine on both maps.

 

Then there would be another map saying what the texture terrain type for each tile is, where the tile is the area between the corners so that map is size-1.  Some engines the textures are disconnected from the terrain mesh because they blend and interpolate multiple textures, but Wurm the texture is tied to the tiles with optionally edges being blended.

 

Rather than just java API why not support 16b greyscale .tiff for height maps and and 8b .png maps for tile types.  That would make it easier for non programmers using existing terrain tools that can usually output a image map or even those that want to use gimp/photoshop to have an image reader that writes to Wurm thru the API.

Edited October 1, 2015 by yarnevk

[yarnevk 257]

Just keep in mind that maps must be surrounded by water and larger or equal to 1024X1024 (otherwise server will crash on startup).

Edited October 1, 2015 by yarnevk

[akaedis 429]

Just keep in mind that maps must be surrounded by water and larger or equal to 1024X1024 (otherwise server will crash on startup).

 

Way to crush peoples my hopes and dreams of having a map with mostly land with only rivers and streams throughout it.

[yarnevk 257]

I really like this.  Do you have any of your code published, or maybe a Blog discussing your techniques?  I wrote some island generator code for a friend's game a couple years ago, but never got erosion working to my satisfaction.

 

Not disclosing my fractal synthesis method, but here is a trick - islands with river channels and lakes are not made by erosion nor fractals at all but by a fundamental matrix equation multiplying terrains.  It only works because all water is at sea level in Wurm.

 

Y=(M.*X.*W)-B 

 

.* is a element wise matrix multiply in matlab, but you can get matrix/vector libraries in other languages or just loop thru each array element.

 

X is a regular terrain

 

M is modified terrain that originally had both sea floor (-) and mountains (+) just simply take the absolute value.  Wherever there was a coast is now a river channel that looks natural even though it was not even done with erosion!  A steep coast turns into a narrow river, shallow coasts turn into wide channels, Don't use islands for M unless you want river channels flowing in circles.  That leads me next to W - what do you think ocean boundary is - but a circular river channel?!

 

W is terrain that is nothing but a dome mountain to turn everything into an island  I do mine using fractal synthesis so this is already contained in M and it looks more natural than a math function (or window function as they are called in DSP).  Nothing wrong with bowl functions as is used in the squiggly looking terrain in this thread.  I did the exact same thing in my original version where terrain was hollowed out asteriods - needed the bowl to keep the air in the valleys.  Invert the bowl to a dome and you have the required ocean ring.  You will never notice from the ground that this is essentially nothing more than a paint program gradient stencil!

 

Use B to raising/lower the terrain sea level, adjust it until you are happy with the size of channels and lakes.

 

I think all of the above operations could be done by grayscale paint programs if you cannot code.  Either by painting yourself or clipping earth maps.

 

So my water features are not actually done with erosion, the erosion just makes these details look more natural.  You cannot do real rivers in wurm because there is only one water layer.  You also cannot do beach erosion, because a heightmap cannot do cliff overhangs (even if you could it stretches textures vertically and looks ugly) so I refuse to do unrealistic Wurm style uneroded cliffs meeting water (sorry cave shipbuilders use the other generators trying to be more Wurmian) .  Most games implement cliffs using static 3D objects, but then you cannot surface mine those but they do look nicer.

 

My erosion was inspired by one called the raindrop algorithm, which conceptually is simple drop a raindrop and let it flow down and decide when to break dams and dig/drop dirts - I presume most commercial generators can do that.  The trick with erosion algorithms is knowing if you are you a civil engineer that needs a fluid sediment simulator, or are you faking it for a game to look real. Even when faking it they can take a very long time to run if you try to erode 2k terrains all at once.   

 

I may leave erosion out though because I am not convinced those extra details will look good with Wurm LOD nor the tiled texture maps nor even be safe to walk on without getting stuck - but maybe WU rewrites those limitations?  Or maybe the lack of detail is just the archaic terrain generator Rolf uses and the engine can handle it?

Edited October 1, 2015 by yarnevk

[ElvenElder 139]

Ricowan, contrary to what most programmers wanna make it look like, all those techniques are publicly available in scientific literature. The following site sumarizes them pretty well. Scroll under the terrain generation section.

 

http://pcgbook.com

[Jonneh 82]

A few people have asked me about my map renderer now, so I'll share it here.

 

https://mega.nz/#!CBdiCTrJ!d2RWx_KA9xwdLQZAvSbgJF0XiZPvTFj2L6Bj2qUkgUs

 

Run it with the arguments size, input file and output file. Like so:

java -jar render.jar 4096 example.map render.png

I have included an older one of my generated map files for an example.

 

The input file needs to be in the format described in my previous post but without the size header. See http://forum.wurmonline.com/index.php?/topic/130605-wu-mapterrain-generator-development/?p=1347645

[Ricowan 199]

@yarnevk, thank you for your explanation, very helpful!  I had tried various interations of the raindrop algorythms and was either dissatisfied by the results, frustrated by the huge amount of time it added to the terrain generation, or both! 

 

@ElvenElder, nice link!  Some of the information on that page is new to me, I appreciate it.

 

I love this thread!

[ElvenElder 139]

Very interesting Jonneh My approach uses the same color darkening/softening technique based on gradient of [x][y] - [x-1][y-1] but I can't seem to get the colors to display identically to the wurm maps.

 

As for the projection to make the map look kindaish 3D the vertical displacement based on height can be a variable of the angle at which you look the map like this

 

public static Terrain project(Terrain input, double angle)

    {

        Terrain projection = new Terrain(input.getWidth(), input.getHeight());


        for(int i =0; i < input.getWidth()-1; i++)

            for(int j =0; j < input.getHeight()-1; j++)

            {

                double displacement = input.getAltitude(i,j) * Math.tan(Math.toRadians(angle));

                int y = j - (int)displacement;

                if(y<0)

                    y=0;

                for(int k = y; k < j; k++)

                    projection.setColor(i,k,input.getColor(i,j));

            }


        return projection;

    }

Edited October 1, 2015 by ElvenElder

[yarnevk 257]

@yarnevk, thank you for your explanation, very helpful!  I had tried various interations of the raindrop algorythms and was either dissatisfied by the results, frustrated by the huge amount of time it added to the terrain generation, or both! 

 

@ElvenElder, nice link!  Some of the information on that page is new to me, I appreciate it.

 

I love this thread!

 

@Ricowan I will post pics of the layer multiplication this weekend.  Yes raindrop is expensive but it is how most commercial generators get those mountains with all the folds in the sides, some of them are so detailed they take weeks to run even with parallel CPU/GPU.  I think the trick is starting with a more realistic terrain generator then running less aggressive erosion, which is why I am using earth stats in mine.  If you tried to detail Wurms smooth mountains I think the results would be a disaster.  Another trick I use is global sized raindrops that move more dirt along with local sized raindrops.

 

Another classical terrain generator is Perlin noise  which is surely how that squiggly pirate islands was made. Even though that may look unrealistic that map would be a blast to play.  Shame though about the ocean rule since I think everyone spawning in the safe lake middle then having to fight to get the the resource mountains is a good idea, more resources than putting them in a middle mountain.    I never liked the even distributions in WO to keep the vocal minority paying their fee rather than ragequit because they had to explore..    I have used those algorithms before for texture blending (something WO cannot do but I hoping WU client is improved)

 

@ElevenElder I would appreciate it if others would respect the programmers right to keep their algorithms confidential and not try to discredit others.  My algorithms are indeed unique as they are a blending of earth sciences with acoustics, a technique that computer graphics experts have never discovered because they do not think cross discipline and few acoustic engineers even know how to do this math and even if they could they do not have the computer engineering skills to write code.  Not only have I spent a month downloading and analyzing the earth terrain databases I have also spent years  working on it and reading all the research papers to ensure what I have does not infringe on others techniques.  I am not about to share things I invent on a public forum with people that play a game they monetize for profit unless it is something I learned elsewhere like layering tricks and raindrop.  If you are happy working on open source terrain generators to enable better servers when people have already declared they intend to monetize their servers to compete with WO, well a hobby is indeed about the enjoyment of loosing money and time even if others gain.  I choose not to participate in that but WU is about respecting others right to play how they want, so please give others the same respect.

Edited October 1, 2015 by yarnevk

[ElvenElder 139]

I don't think anyone's pressuring you to reveal how your terrain generator is working. But as long as there's people here interested in learning and tempted to give this a try, I think those that are already steps ahead can at least point the rest to resources that can give them a jump start. I think your reply is more suited to the poll topic regarding an open source map generator tool. Well I'm letting you know that even if I'm the creator of the topic, I'm also a bit inclined against providing such tools to everyone for free, mainly because a lot of people will try to monetize their products as you correctly mentioned. Been there, seen that in other games.

 

But putting that aside, sharing code and sharing links to scientific resources are two completelly different things. I don't believe the second is that harmful.

Edited October 1, 2015 by ElvenElder

[yarnevk 257]

I was referring your comment in this thread as that comment had no place here The only thing I am keeping to myself is my specific method of fractal synthesis because that is not known research. I started my participation in this thread saying I intended to exchange ideas but it is my choice to restrict that information sharing to that which is known elsewehere.  Maybe I have dreams of Sony buying my code for a game engine like the voxelgen they bought for Everquest Next or like they are promoting No Mans Sky or maybe I just do not want Rolf to profit from my hobby anymore than he already has.  The reasons I have for not sharing code is irrelevant, but I would appreciate it you refrain from comments that imply programmers are hiding behind known research or hiding database destruction in their code.

Edited October 1, 2015 by yarnevk

[enoofu 206]

Is a nice site for real world height-map would still need to go though another program for proper heightmap since it has 0,0,0 being water

http://terrain.party/

[joedobo 1,187]

This site has a lot of gray scale height maps for all of earth. I'm not sure how easy the data is to work with. The fact WU servers must have water on all sides makes using actual world maps more challenge. Seems they could be modified to have water on the edges.

 

Shuttle Radar Topography Mission (SRTM)

https://lta.cr.usgs.gov/SRTM

[yarnevk 257]

Biggest problem using real world maps is people will expect the rivers, seas and lakes to have water but they will not because all water in Wurm is sea level.  Depends on the map survey some reflected water/snow surface, some got underwater/under ice surfaces. either way porting them to Wurm they would be dirt not water.  I can dig thru my old code what downloads I got - the web based retrieval services was low rez maps and the other ftp service with hirez maps was extremely unreliable I was not joking when I said I spent months downloading the world only to discover maps that are missing grid points either due to bad downloads or bad surveys - they required a lot of post process cleanup.  I ended up working on work servers as I ran out of mem and disk at home.  

 

Also you are not going to get hirez 4m flat tile maps, they are fractional degree maps, but nothing wrong with taking a lo-rez map of the United Kingdom and shrinking it down in scale to fit Wurm at the small scale nobody is going to nitpick the sphere to flat warpings.  

 

By compositing maps as I described you avoid the issue of it having to look exactly like an existing place, and you can get the sea level rivers and lakes looking OK even though they are all at sea level. To make it work though you need the underwater surface maps not flat water maps - but the idea also works with mountain/valley maps since you can set your own sea level.

 

edit here is the better world map survey

 

https://www.ngdc.noaa.gov/mgg/global/global.html

 

 

I used someones Matlab code that allowed you to iterate over the world downloading the hi-rez ftp files but it got left at my old job so  so you will have to make do using their lo-rez world map or their map zoom selector that is not as good.  Interestingly when I plotted fractal scale it was amazing how much of the ocean is faked interpolations - you could literally see the trawler paths running out of hawaii where the real data is.  It turns out we know more about Mars survey than our own oceans.

Edited October 2, 2015 by yarnevk

[joedobo 1,187]

The grayscale height maps I saw looked fine. You don't need high res 4m flat tile maps. As a matter of fact, you're better off totally ignoring the 4m per a tile thing. We just need realistic looking height maps at pixel resolution of 512, 1024, 2048, 4096, 8192. Each pixel on this map would be the NW corner of a tile. Then, take the elevation data and randomly generate tile types using different qualification, like elevation. You could end up with a map file that has 4 data points for each pixel: x, y, z, tile type.

 

I'd use Earth's topography maps to get a realistic looking heightmap. This way a person doesn't have to deal with artificial algorithms. The goal is not about trying to copy some area perfectly.

 

IDK...at this point its hard to say as I haven't seen WU map related content/code.

[yarnevk 257]

There is no confirmation yet if Wurm maps are grid or tile based.  Certainly textures are tile based but digging is clearly grid based (corners of the tile) sowhich system does the power of 2 sizes apply - I would guess the texture tiles so that distant texture maps can be power of 2.  So your Z and tiletypes might not be at the same points. (this is the way Unity Terrain works and is commonly done in game engines).   It is unlikely to be a tile is stored with four ordered Z corners because that data is redundant with it's neighbor tile. 

 

The X,Y would be the index of the Z array, or it might just be a row/col major/minor Z list. Inefficient to store the X,Y because it is a regular grid either way.  A delta list would be more efficient as it could use a smaller datatype but I doubt that since digging and raising can go to extremes so it must be a straight forward dirt/rock height array.

 

The NOAA site explains both indexing systems since they offer both interpolations.   https://www.ngdc.noaa.gov/mgg/global/gridregistration.html  Of course if you are ripping a sample which way Wurm does it vs. NOAA is irrelevant since this is not intended to be an accurate flight simulator, nobody will notice your map has a half pixel shift with interpolation errors.  You will need to scale the height if you are scaling down X,Y so you do not end up with everything being cliff slopes.

 

The greyscale maps look fine because our eye cannot easily resolve 16b gradient differences, you will notice the interpolated parts of those maps only once you stand on the terrain and see odd blocks of smoothed out terrain or noise spikes. Map building will involve a lot of iterative in-game validation flying around at GM speed.

Edited October 2, 2015 by yarnevk

[Jonneh 82]

which system does the power of 2 sizes apply

 

Both. The height given for a tile is for the NW corner. If I recall correctly the client gets around not having the height data for the south and east edges of the map by just not rendering the last row & column of tiles, there's an example of this from islands cut off at the south & east edges of the map on the deliverance server, there's and extra row & column visible in the map dumps.

Edited October 2, 2015 by Jonneh