Elevation lines, Gradients, Smoothed Fractal LODs

[FishOfTheNorthStar]

relevant snippet below. I'm using defines in my fork so you'd want to change that bit around some.

// ....

#elif defined(DEBUG_ELEVATION_LINES)
	// Show elevation lines
	vec3 __pixel_pos = (INV_VIEW_MATRIX * vec4(VERTEX,1.0)).xyz;
	float __camera_dist = length(v_camera_pos - __pixel_pos);
	float __region_line = 0.5;		// Region line thickness
	__region_line = .1*sqrt(__camera_dist);
	float _span_gap = 20.0;
	float _spot_y = __pixel_pos.y;
	float _mod_lum = __region_line - max(0.0, (fract(_spot_y / _span_gap) * _span_gap) - (_span_gap-__region_line) );
	_mod_lum = pow(abs(_mod_lum * 2.0 - 1.0), 0.5);
	ALBEDO = vec3(0.2) * _mod_lum;
	ROUGHNESS = 0.85;
	SPECULAR = 0.;
	//NORMAL_MAP = vec3(0.5, 0.5, 1.0);  // Kinda nice with the texture normals still visible, might be a good option to toggle on off.

// ...

edit: kept messing around with it and have a version that also has a grid overlay, and dark/light mode selection. Here's light mode:

and dark mode, the more colorful option so far:

you can blend the modes (it's lerping their relevant values anyways, by a dark_light_mode float). Here's 85% light mode, it just adds a little color to the lines:

The code got a little crazy and I'm still making it presentable. When it's cleaned up I'll edit this with new code in case you want to try it out.

[TokisanGames]

Looks like a great feature.

The shader preprocessor was a bit clunky when I wrote the INSERT system. I was thinking of expanding it to offer ending demarcations and replacement blocks. However using the preprocessor is probably a better way to go. It would allow the use of conditionals and be easier to maintain over time. I would accept a PR to strip out the INSERT system and go entirely with the Godot preprocessor.

I would accept a PR for these elevation lines.

• It should be additive like vertex grid so it works with other debug views.
• Variables should prefaced with two __. None for public vars that show up in the inspector, _ for private vars in the main shader, __ for debug views to guarantee they won't conflict with the main shader or the user's shader.

[FishOfTheNorthStar]

Update:
Added feature detection outlining.
Added optional opacity by distance.

Shown is "light mode" with 100% "mask material", ie it fades to flat white (in light mode, black in dark mode). In light mode the lines darken the base, in dark mode the add emit.

The "features" it's detecting are "vertical" = cliffs, "hazard" = steep incline, "obstructed" = some incline, "flat" = easily travelled, "flatter" = more flat, likely to collect water, and "puddle", a subset of flatter that might eventually modulate against world noise.

also shown in background is a little bit of a "feature based shader" i'm working on that is essentially just the auto shader but my own spin on it. Same basic functionality but lets you assign 6 texture id's per 'feature set', keys off the same 'feature identification' the overlay is using, which i'll describe below a little (it's very short).

Still cleaning up code, done soon(ish). It will be implemented as two gdshaderinc's and a define, at the main shader start. if the define is true it'll include it's requisite function defs from the first include, in the header, and the second include is a very short in-line stub that sits where debug view's generally do their thing now. That way I can implement it in my fork, and you can use it in the main branch if you like, and gradually we can shift core functionality into the new include-based format. Hopefully with a minimum of hiccups or snags.

const vec4 FEATURE_BREAKS = vec4(0.5, 0.65, 0.85, 0.985);
vec4 get_features_by_normal(vec3 __for_normal ) {
	// externally supplied vec4 roughly equal to vec4(0.5, 0.65, 0.85, 0.985);
	vec4 __breaks_at = FEATURE_BREAKS;
	return 	min( vec4(1.), max( vec4(0.), vec4( 
			clamp( dot( __for_normal, vec3(0.,1.,0.) ), 0., 1. ) ) 
			- __breaks_at ) / ( vec4( __breaks_at.yzw, 1.0 ) - __breaks_at ) ); }

So that expects a world normal, and returns a vec4 where the x is 0->1 response for vertical, y is hazard, z is obstructed, and w is flat.

flat-inner (super flat) is derived by subtracting the sum of the xyzw parts from 1.0, anything left is by definition 'more flat than flat'.

Puddles is calculated elsewhere but with a basically identical process except it only responds to one range at a time not four.

In my local tests, I then store the result of that, summed together into a float, (.x+.y+.z+.w + flat_inner). The integer value of that float is which role to use as a material, the int value + 1 is the role material it is blending into, and the fract() portion is how much it is blending into it. Then the actual texture id's are looked up from an int array of six elements, and eventually the provider of that array might vary it dynamically based on local biome or something. For now it's pretty cool even as just one set, but eventually there could be several it modulates through.

This approach seems to work pretty well, it's very performant, and by keeping the role responses in isolated normal ranges (ie 0->1), we could safely interpolate between them, unlike texture id's which get silly if you try to lerp one to another.

I'll get this finished up and post the code here soon.

edit: another image, in dark mode, distance faded, partial masking

[FishOfTheNorthStar]

OT but I thought this screenshot was looking nice, it's using that 'feature based shader' concept above, but I also added a modulation factor so inside any one feature, vertical etc, if there is nothing for it to blend to (ie, its already all the way vertical, or flat, etc), it randomly blends in an alternate material ID for that region. So vertical cliffs can have little bits of vine fading in, etc.

Anyways, full source on this soon, it's a mess right now still ironing stuff out. Promising though. Thanks for this! Fun stuff, it's great you'd put it out there freely.

btw, when I post screenshots, do you have to pay to host them? or do you run out of space eventually? I could delete them after a while.

[TokisanGames]

Looks great.
No, this is all part of the free plan on github. Public repos are no cost. We also have a terrain showcase channel on our discord.

BTW, for texturing stretched verticals, Roope and I had planned on implementing 3D projection but he got stalled. It's from the same author and project that gave us the indexed map, texturing based on 4 adjacent vertices. cc @Xtarsia

#65
https://github.com/cdxntchou/IndexMapTerrain

PS. Also posted some of your screenshots on twitter
https://twitter.com/TokisanGames/status/1784777712860852479

[FishOfTheNorthStar]

Thanks, that project looks really cool, rotating and projecting the splat textures looks fantastic.

Another thing I've found really nice for steep verticals is a pair of gradients, each applied by height, but one is for vertical surfaces, the other horizontal, and maybe in between gets a mix. For cliff-like strata etc, or adding shore-lines, and greener grasses in low-valleys, etc.

I found this part really important: let those gradients influence spec/roughness etc (via alpha, maybe). for metallic/brighter/glittering horizontal streaks in vertical surfaces, looks fantastic. The horizontal surface gradients would generally leave spec alone, but you can put a boost at the sea level and that looks great, like wet sand.

I'll be (re)implementing the strata gradients here soon. One thing I'd love to figure out with it though, and I think that project you linked to can help with it, is encoding an up/down tilt of the normal in the gradient too, at least the vertical one, so streaks can jut-out-from or etch-in-to their vertical surfaces. But rotating the normal has always given me trouble, when I've tried that in the past.

[TokisanGames]

I came across this video about rotating normals correctly and bookmarked it without watching. Also remember UE uses DirectX normals.
https://www.youtube.com/watch?v=iXqCszt3LbE

[FishOfTheNorthStar]

A number of interesting developments. So it started with: "before I call this done, how about just-one-more-thing...". So I put in an honest attempt at a cell-shaded view, with mixed results. I mean it's ok but it's no borderlands. Maybe good for design time though. Requires no textures, just some defined color values

And then I really wanted to see what strata gradients would look like in there. So I set up a real simple float / vec3 array lookup based gradient function in glsl, and pasted in a gradient I use for earth tones. now in "cell-shaded" mode, it looks like this

What's happening there is it actually has two gradients it cycles through based on elevation, one for vertical surfaces, one for horizontal. It cycles through those gradients at different rates, slowly for vertical, fast for horizontal (because it doesn't change height much by definition). Then it samples both (but through those functions, not texture lookups), and mixes by vertical<->horizontal-ness (with some pow() applied for sharper transitions). The elevation it samples from is modified by the tint noise 1 and tint noise 2 variables, along with another 3d noise I calc per vertex in my version here. That creates the jittered, irregular cross section lines.

So in regular view mode, with those gradients applied, it looks more like this (excuse the fog, i'm making an apocalypse style environment)

(edit: replaced image, new one better)

closer up you can see what it's doing a little more, with the gradients:

One other thing about those gradients is the vertical one can adjust spec/rough/metal, through alpha (like described in earlier post), but the horizontal one can adjust AO, and create darkening edges around it's banding shapes. Because the darkening is AO based, not Albedo, it can push shadows down more canceling out their ambient component too. I use this to create a slight shadowing effect around green->dark brown contrast blobs that appear from noise-ing up the horizontal strata gradient. So it's a bit like leaves, with shadow blobs around the leaves.

In this image, you can see a lot more specular response in the cliff sides with that strata spec mod in effect, it really adds a nice bit of depth I think.

( Elevation / Grid / Feature / Cell-Shaded ) Overlay view mode almost ready. I'm calling those the 'cartography' views so far, but 'topo-view' sounds good too. I might carve Feature and Cell-shaded off into their own thing that runs off the same basic system of injecting it's results into the fragment function, because those require feature identification, and that's still coming together here, how it works might change a bit. But showing the results (ie, flat, not-flat, vertical), that's probably pretty stable.

edit: and one more, I liked this configuration here too

More soon.

[FishOfTheNorthStar]

at the time of this post, this is the gradients used and the function to sample them:

#define STRATA_GRADIENTS_SUPPORT

const int _GRAD_DATA_LEN = 37;

const int TEST_GRAD_OFFSET = 0;
const int TEST_GRAD_LEN = 31;

const int TEST_NORM_GRAD_OFFSET = 31;
const int TEST_NORM_GRAD_LEN = 6;


const int MY_TEST_GRAD_WINDOW_OFFSET = 0;
const int MY_TEST_GRAD_WINDOW_COUNT = 8;  // This says there will be 8 portions "windows" defined that mark the indexes necessary to respond to U values between that window's start and stop normalized values (x/total portions -> x+1/total portions)
const int ANOTHER_GRAD_WINDOW_OFFSET = 8;
const int ANOTHER_GRAD_WINDOW_COUNT = 1;  // Because this gradient is so short, it's window is just it's start and stop index, ie window count 1

const vec4[_GRAD_DATA_LEN] _GRAD_COLORS = {
vec4( 0.2854815, 0.2786366, 0.2557304, 0.0165 ),
vec4( 0.52393, 0.554962, 0.431105, 0.55 ),
vec4( 0.52393, 0.554962, 0.431105, 0.85 ),
vec4( 0.72, 0.60192, 0.4248, 0.25 ),
vec4( 1., 0.985, 0.7, 0.35 ),
vec4( 0.6, 0.6848, 0.568, 0.625 ),
vec4( 0.449237, 0.472517, 0.409896, 0.75 ),
vec4( 0.501828, 0.492807, 0.347508, 0.85 ),
vec4( 0.1312416, 0.1340626, 0.133792, 0.01 ),
vec4( 0.30297, 0.321987, 0.273123, 0.9 ),
vec4( 0.646057, 0.569328, 0.491916, 1. ),
vec4( 0.3217443, 0.2482282, 0.2073914, 0.57 ),
vec4( 0.0596259, 0.0416335, 0.0218797, 0.90137 ),
vec4( 0.678981, 0.516589, 0.431332, 0.34 ),
vec4( 0.83843, 0.63584, 0.476065, 1. ),
vec4( 1.0, 0.8567344, 0.92304, 0.15 ),
vec4( 0.82, 0.70192, 0.5248, 0.9 ),
vec4( 0.2564323, 0.19531793, 0.12385168, 0.17 ),
vec4( 0.830987, 0.615895, 0.438144, 0.8 ),
vec4( 0.856008, 0.713562, 0.589941, 1. ),
vec4( 0.196, 0.17872, 0.10528, 0.135 ),
vec4( 0.540448, 0.524066, 0.434087, 0.75 ),
vec4( 0.792157, 0.670588, 0.498039, 1. ),
vec4( 0.54369, 0.43421, 0.282027, 0.73 ),
vec4( 0.443975, 0.2609797, 0.0441082, 1. ),
vec4( 0.800305, 0.65918, 0.485817, 1. ),
vec4( 0.831373, 0.615686, 0.439216, 0.8 ),
vec4( 0.802576, 0.656002, 0.482411, 1. ),
vec4( 0.898039, 0.784314, 0.466667, 1. ),
vec4( 0.805077, 0.690615, 0.435805, 0.71 ),
vec4( 0.668368, 0.552822, 0.39042, 1. ),

vec4(   0.501828, 0.692807, 0.347508, 1.0 ),
vec4(  0.501828, 0.692807, 0.347508, 1.0 ),
vec4( 0.564323, 0.9531793, 0.2385168, 1.0 ),
vec4( 0.312416, 0.340626, 0.33792,0.8 ),
vec4( 0.312416, 0.540626, 0.33792, 0.4 ),
vec4( 0.21, 0., 0.26, 0.0 )
};
const ivec2[9] _GRAD_WINDOWS = {
	ivec2(0,3),
	ivec2(2,8),
	ivec2(7,11),
	ivec2(10,17),
	ivec2(16,22),
	ivec2(21,25),
	ivec2(24,29),
	ivec2(28,30),
	ivec2(31,36)
	};
	
const float[_GRAD_DATA_LEN] _GRAD_OFFSETS = {
	0.0, 
	0.0594262, 
	0.0819672, 
	0.135295, 
	0.137295, 
	0.162651, 
	0.212851, 
	0.237705, 
	0.266393, 
	0.293033, 
	0.327869, 
	0.381148, 
	0.391393, 
	0.407787, 
	0.453815, 
	0.467213, 
	0.485656, 
	0.55123, 
	0.561475, 
	0.586066, 
	0.60041, 
	0.614943, 
	0.645594, 
	0.67418, 
	0.680328, 
	0.756148, 
	0.770492, 
	0.85041, 
	0.866803, 
	0.938525, 
	1. ,
	
	0.0, 
	0.25, 
	0.42, 
	0.63, 
	0.88, 
	1. 
	
	};


vec4 SAMPLE_GRAD(int _grad_offset, int _grad_len, float _u) {
	vec4 resp = vec4(0.);
	ivec2 _window = _GRAD_WINDOWS[_grad_offset + clamp(int(_u*float(_grad_len)),0, _grad_len-1)];
	for (int _i = _window.x; _i <_window.y; _i++) {
		bool range_valid = _u >= _GRAD_OFFSETS[_i] && _u <= _GRAD_OFFSETS[_i+1];
		float _lerpval = (_u - _GRAD_OFFSETS[_i]) / (_GRAD_OFFSETS[_i+1] - _GRAD_OFFSETS[_i]);
		resp += mix(vec4(0.), mix(_GRAD_COLORS[_i],_GRAD_COLORS[_i+1],_lerpval), float(range_valid)); }
	return resp; }

then to take a sample, you pass it the starting index offset of the "window" array, and how many window slices that gradient uses, From that, it looks up a start and finish index of gradient points that might affect the given U coord, iterates through them, and adds their effect to a return value if the current iteration is the valid range.

like this:

float my_testgrad_u = /* figure out a normalized coord however you do that.  should be 0->1 */
vec4 _gradSamp = SAMPLE_GRAD(MY_TEST_GRAD_WINDOW_OFFSET, MY_TEST_GRAD_WINDOW_COUNT, my_testgrad_u);

Future versions may wrap these 'start window index / total window count' int pairs in a ivec2 that can be defined constant and passed more easily.

[FishOfTheNorthStar]

The 'auto-veg' concept described above

In this image I'm letting the veg response kick height up a little bit, but haven't bothered to properly calc normals with that in effect so that's a bit off, but the shadow blobing effect that's built into the gradient it's using is helping a lot

also trying out some puddles, key'd off a really tight window to the surface normal versus true-up, the puddles aren't great yet but the idea seems to work ok. I'd like to get some animated rain drops in there. The other non-puddle surfaces pick up some extra spec/shine for a wetness effect.

[FishOfTheNorthStar]

I was wondering if I've just gone off the rails off topic here when something occured to me, kinda funny: Plot twist: The plants above are still elevation lines! They always were. Working off the exact same principle, but a gradient instead of a line, only half the gradient is actually used, and the Y is modulated by noise.

[TokisanGames]

Yes, totally off the rails, but it's an interesting and self-contained exploration. I changed the title so now it's on the rails. I'm excited to see what comes out of it. These experiments can lead to usable features. When you're ready, please package up an individual feature into a PR like shader includes, or elevation lines or something so others can use them.

[FishOfTheNorthStar]

Hey great. PRs will be coming in soon. It's mostly usable as is but there are some issues to iron out, and I'm trying to structure it so it's not like you have to change everything in order to use any of it. The shader includes is going to be the only part that the others require, ie we use that to carve the main shader into 3 main functional blocks, vertex, fragment, and material blending. The base-level shader include PR will change nothing about how anything looks or works with the main branch, it'll just portion the code into multiple external files and establish interfaces to that portioned off code, ie which external variables it'll need, and which ones it provides.
Then those main blocks will each have their own define's they negotiate to resolve things like 'where is the normal coming from', so they'll sort that out internally but externally they just provide a final, mixed, transformed normal vector etc for other functions that work upon that.

One thing I've found from my tests so far is that it's definitely easier to work on / adjust / tweak Custom Shaders if as much as possible of the shader is defined in external includes. Updates apply immediately and there's no weirdness with Godot refusing to apply new dynamically injected defines because the file is currently open (which seems to be the case with generated shaders, for now).

Also by portioning functionality up like that, we can use Godot's shader syntax highlighter to help identify loose/external references in the shader code, because it'll throw warnings and such about not recognizing this or that, and to make those go away you have to add in another define block that establishes those values if the file was not included from inside the main file, so it declares some const standins etc and that keeps the syntax highlighter happy. I thought we could have two such define blocks at the top of each module's included files, and the first for external variables the contained code will expect access to, and the other is externally defined variables that the code will be exporting globally for other modules to access. Then we actually define those external variables in another isolated include so failing to find access to those variables intentionally throws red nag text in the shader editor, and both those define blocks with all variables covered will be required for no-red-text. But not actually required to function, it'll function fine even with the red text nags etc, if included correctly.

btw I improved the separation between plant and the ground below it, you can really see how the math-based edge stays nice and sharp no matter how close, like vector images.

then about half the distance to that central patch of dirt

then half the distance again, basically as close as I could get it

[TokisanGames]

ie we use that to carve the main shader into 3 main functional blocks, vertex, fragment, and material blending. The base-level shader include PR will change nothing about how anything looks or works with the main branch, it'll just portion the code into multiple external files and establish interfaces to that portioned off code, ie which external variables it'll need, and which ones it provides.

I don't mind if we have our development shaders spread out among multiple files. But I hesitate to give the user 3 shader overrides. I'm not sure if that's what you're suggesting. I'd like to continue to allow them to generate a single shader based on the options selected and they can adjust the one file as they see fit. We can inject things into it if needed (like the debug views).

Or are you suggesting that all of the development shaders are pushed into the public file system. For development work sure that would be easier. But perhaps easier for users to mess up.

I'll wait to see what you produce rather than guess. Ultimately the guiding principle I'll evaluate from is if it has good usability for intermediate users, who would be writing shaders.

---

The generated textures are certainly interesting options for aerial views. Perhaps we need a new category of extras shaders people can use separate from the debug shaders. In addition to your various cartographic and generated natural shaders, we also need a flat shaded low-poly shader.

[FishOfTheNorthStar]

Oh to clear that up, no I was not suggesting 3 shader overrides. Still just the one. But when you actually open the generated shader in the shader editor, it will be a much shorter text file, because big chunks of it's functionality would be abstracted away into those 'three main chunks' idea, and each of those is inside another file and those files don't change much at all (by end users). Implementing them would be one or several include file lines, and they'd basically handle unpacking control data into a struct, any sort of low level color correction, repositioning vertex's in the vertex shader to their elevated position, any view<->local<->world<->clip space transforms necessary, setting all the varyings, etc. That example is for vertex stuff, the frag and mixing would have their own low level things they do.

So basically all of that wouldn't be in the custom shaders, it'd be in the included file line within custom shaders, and users can pretty much leave that stuff alone and forget about it. Then the file maybe has some comment lines in it to the effect of 'above this line, leave it alone, setting up core stuff. put your custom code after this line but before this other line and it'll be applied.

The goal of this is two part. One, makes it easier to write, manage, and occasionally update custom shaders. So it's a win for end users. but also two, it'll make it far easier for us to manage the implementation of new features and ideas, because people adding stuff can keep their changes focused in on one 'main chunk' at a time, and code changes can stay more atomic and manageable. Added bonus, putting all the core stuff in external files seems to make the actual compilation of the shader and sending it to the card happen much faster, i think Godot is doing some 'chunk caching' in the background, it's just much smoother.

It's coming together here and I think it'll make a ton more sense when you can see it. Will post when it's ready. BTW feel free to just nuke this thread entirely if its too long and weird, I just wanted to communicate some ideas and I think we've done that.

edit:
mandatory random screenshot:

This set of noise data I'm using for world-noise generated topography seems to be making some nice 'water ways' for like, rivers and lakes (more on that soon, one thing at a time). That using fastnoiselite for the noise supplier with settings 'noise_type' = FNL_NOISE_PERLIN and 'fractal_type' = FNL_FRACTAL_PINGPONG

[FishOfTheNorthStar]

Fractional Octaves -> Smoother transitions

original/current:

float world_noise(vec2 p) {
    float a = 0.0;
    float b = 1.0;
    vec2  d = vec2(0.0);

    int octaves = int( clamp(
	float(world_noise_max_octaves) - floor(v_vertex_xz_dist/(world_noise_lod_distance)),
    float(world_noise_min_octaves), float(world_noise_max_octaves)) );
	
    for( int i=0; i < octaves; i++ ) {
        vec3 n = noise2D(p);
        d += n.yz;
        a += b * n.x / (1.0 + dot(d,d));
        b *= 0.5;
        p = mat2( vec2(0.8, -0.6), vec2(0.6, 0.8) ) * p * 2.0;
    }
    return a;
}

Becomes:

float world_noise(vec2 p) {
	float a = 0.0;
	float b = 1.0;
	vec2  d = vec2(0.0);

	float f_octave = clamp(
		float(world_noise_max_octaves) - (v_vertex_xz_dist/(world_noise_lod_distance)),
		float(world_noise_min_octaves), float(world_noise_max_octaves));
	int octaves = int(f_octave);
	f_octave = fract(f_octave);
	vec3 n;
	for( int i=0; i < octaves; i++ ) {
		n = noise2D(p);
		d += n.yz;
		a += b * n.x / (1.0 + dot(d,d));
		b *= 0.5;
		p = mat2( vec2(0.8, -0.6), vec2(0.6, 0.8) ) * p * 2.0;
	}
	n = noise2D(p);
	d += n.yz;
	a += (b * n.x / (1.0 + dot(d,d))) * f_octave;
	return a;
}

Didn't want to junk up your PR list with a million edits so here you go, if you want your octaves to blend into each other this seems to do the trick, doesn't appreciably impact performance for me.

[FishOfTheNorthStar]

It seems like checking distance and doing a mesh or pixel normal off of that doesn't offer the performance I'd expect it to, and I think that's because since the distance isn't a constant value, the compiler can't optimize that conditional, so it turns into a parallelism issue and the end result not much speed improvement regardless what you set the distance to.

So I tried this instead, seems to work much better for me. Instead of testing distance, test against MODEL_MATRIX[0].x where 1 is close and greater values is more distant, since (I believe), that index is the scale.x(?) Pretty sure, couldn't find docs on which index is exactly what but I think [0].x,[1].y,[2].z is basically the scale. Anyways testing against the x being less than 8 seems to work pretty well for me, so it draws the first three stages with mesh normals, than anything farther than that with pixel. And because MODEL_MATRIX is a constant, I can definitely tell the distance optimization is having desired result, the compiler is pruning the branch.

edit: confirmed those are the index's for scale. I've always been terrible at matrix math, finally found some reference.