Adds retro-v2-softlight shader

[rbreaves]

Preview pics here.
https://www.reddit.com/r/RetroArch/comments/y2pw5p/retrov2_screen_shader_wip/

Nothing to note really, based on retro-v2 shader and blendoverlay, but I swapped out blendoverlay for softlight. Also partly based on code from gb-pass-5 to figure out how to more cleanly overlay the contents together as blendoverlay with the luminance feature was more complicated than I needed softlight to be. Luminance plays no role at all in this shader.

All I really wanted the handheld shaders to do was give me glare on light areas but not dark scenes so I will likely add more shaders specifically for handheld games later on.

If there are any issues or questions then please let me know.

[hizzlekizzle]

Ah, nice! Thanks for the PR!

This looks good but we'll need to lose that errant .gitignore before we can merge.

[rbreaves]

So Android may need something additional, not sure about iOS, but I am seeing that on the left, right and top there is a blurred region that needs cropping out.

[rbreaves]

This looks good but we'll need to lose that errant .gitignore before we can merge.

Done.

Not sure if you want to merge yet or not though since I would like to resolve the bleeding or overscan issue that I am seeing on Android, and I assume it may appear under iOS as well. Not sure what I will need to introduce to keep it strictly to the source dimensions - I know there some weird math in there from the gb-pass-5 glsl that I probably just need to tweak.

I am just guessing, but I am assuming I need to tweak the glsl file around line 60

void main()
{
    gl_Position =   MVPMatrix * VertexCoord;
    vec2 scale  =   (OutputSize.xy / InputSize.xy) / SCALE;
    vec2 middle =   vec2(0.5, 0.5) * InputSize.xy / TextureSize.xy;
    vec2 diff   =   TexCoord.xy - middle;
    TEX0.xy     =   middle + diff * scale;
}

[rbreaves]

@hizzlekizzle ok both items should be fixed now. Also made the default more obvious while keeping the original as the "-subtle" variation. I felt like after the fix for the border bleeding issue I was seeing the effect became too subtle for some odd reason.

Either way I'd prefer to keep both so that users can use whatever works best for them and their display, although adding an opacity option may work better long term.

[rbreaves]

I think the actual difference I was seeing earlier, in the strength of the gradient effect, after fixing the border glitch issue was actually due to me moving my testing from a PC monitor to a projector hooked up to an Android TV lol. So as long as you have good contrast on your display then subtle is likely the better one to pick imho.

[rbreaves]

Just now realizing the color palette actually did shift a little bit and I prefer what I had in my original screenshot which had an additional thing in it from a gb-dmg color palette.. so I will fix that tomorrow :/. References to the color palette I was using got missed entirely.

textures = COLOR_PALETTE;BACKGROUND;BORDER
COLOR_PALETTE = resources/dmg-palette.png
COLOR_PALETTE_linear = false
BACKGROUND = resources/dmg-background.png
BACKGROUND_linear = false
BORDER = resources/gba-border-square-4x-shine2.png
BORDER_linear = false

I like the slightly more muted color palette and I think it is ultimately providing a bit more contrast and depth for some reason.

[rbreaves]

Maybe my eyes were messing with me on the color stuff earlier I dunno, either way I am going to make one more update later this evening to have proper scaling parameters from 0.25-4x and 2 aspect ratios as well.

[hizzlekizzle]

Did you figure out the border thing? The issue is that GLES2 doesn't support the border clamping method that's used everywhere else, so you have to manually black-out the stuff that's outside of the viewport. I usually do it with a comparison of the texcoord for anything >1.0 and <0.0.

Also, would you mind adding a line calling out the MIT license. You've already fulfilled the attribution requirement by linking to the repo, but explicitly including the license will be helpful for anyone who wants to copy it in the future.

[rbreaves]

Did you figure out the border thing? The issue is that GLES2 doesn't support the border clamping method that's used everywhere else, so you have to manually black-out the stuff that's outside of the viewport. I usually do it with a comparison of the texcoord for anything >1.0 and <0.0.

Still working out the border thing for 8:7, it is a non-issue at 4:3 aspect, but since I am now trying to include both.. Also I think for some reason I sorta like the 8:7 better as it must be doing a slightly better job with scaling and thus even color somehow but I think the color thing is more an illusion from 4:3 not being as pixel perfect, but it is still really close and looks well depending on your viewport size. (I have to sometimes resize my floating window on a mac or PC to have the 4:3 aspect ratio to look proper, but it isn't ever wildly off even when it is slightly off scaling wise)

That is some good info though that I know nothing about as I am really new to this glsl stuff.

Also, would you mind adding a line calling out the MIT license. You've already fulfilled the attribution requirement by linking to the repo, but explicitly including the license will be helpful for anyone who wants to copy it in the future.

Sure thing, I was following the example set by blendoverlay but I don't mind adding that at all. Also I am now thinking I will follow this up with an additional shader pack of sorts that allows users to play with all of the blend based glsl shaders, but I will also need to create additional layer masks of sorts so that they can dial in something that works well for them.

Wishing I could find some documentation on how to provide parameter options that are not simply numeric based. I would love to allow users to select a different gradient mask with descriptive names and not create individual glslp's or use non-descriptive numbers for a single glslp.

[hizzlekizzle]

Parameters have to be numeric, but you can put anything into the name field that you want, so some people make dummy parameters with, for example, all zeros, just so they can put some text (or a bunch of hyphens to make a dividing line) into the parameters menu.

[rbreaves]

So I have found a couple of things that I think were intended to apply the clamp type fix or mask fix that you alluded to earlier, but having difficulty getting it to apply to either my glsl shader or applying it to the retro-v2 shader.

Not sure if it matters where it is applied - but I think this is actually more of a bug with the retro-v2 shader but perhaps it is in part due to how I am using it. I think this was never really explored in the retro-v2 shader because typically you had a proper aspect ratio handheld image overlaid on top.

Not exactly how I have it in my code - but what it looked from the areas I was looking at.

vec3 outColor = COMPAT_TEXTURE(Source, vTexCoord);
    /* TODO/FIXME - hacky clamp fix */
    vec2 bordertest = gl_FragCoord.xy;
    if ( bordertest.x > 0.0001 && bordertest.x < 0.9999 && bordertest.y > 0.0001 && bordertest.y < 0.9999)
        outColor.rgb = outColor.rgb;
    else
        outColor.rgb = vec3(0.0);
   FragColor = vec4(outColor, 1.0);

screen *= Mask(gl_FragCoord.xy*1.0001);

[rbreaves]

I think what I really need is a function like this from bigblur but greatly simplified.. I just need a very plain blacker border to cover up the mess that retro-v2 is leaving behind and I could do that in my shader just fine without duplicating a variation on retro-v2 I believe.

Either that or I need you to expand on your idea @hizzlekizzle as I am bit lost at the moment. With all the messing around I have been doing though I have found that I can actually scale things at 0.05 steps and still have the grid looking good from retro-v2 so I am going to keep that instead of the 0.25 step scale I was using.

vec4 border(vec2 texture_size, vec2 video_size, vec2 output_size,
	float frame_count, vec2 tex, sampler2D decal, vec2 tex_border, sampler2D ORIG)
{
	vec4 effect = COMPAT_TEXTURE(decal, tex_border);
	effect += vec4(vec3(BRIGHTNESS), effect.w);
	
	vec2 coord = (interp_toggle < 0.5) ? tex : interp_coord(tex, texture_size);
	vec4 frame = COMPAT_TEXTURE(ORIG, coord);
	frame = scanlines(frame, tex, texture_size, video_size, output_size);
	vec2 fragcoord = (tex.xy * (TextureSize.xy/InputSize.xy));
	if (fragcoord.x < 1.0 - OS_MASK_RIGHT && fragcoord.x > 0.0 + OS_MASK_LEFT &&
		fragcoord.y < 1.0 - OS_MASK_BOTTOM && fragcoord.y > 0.0 + OS_MASK_TOP)
		return frame;
	
	else return effect;
}

In my mind I need to modify OutSize or OutputSize but I see practically no examples of that and mask stuff really is just overlaying on top of existing textures/content.. which is not what I thought mask would be, or would only be I should say. I am guessing if I understood masking better I could apply a black border.. I just feel like since I am not trying to emulate a CRT border that the math to do it ought to extremely simple, but instead I am getting all of this overly complicated stuff that is meant for bending the border, overlaying an image or applying special effects. 😂

I'm sure if I was doing something really complicated all of those code examples would be great..

[rbreaves]

Ok, it's done, I added a background border with 1 width black pixel on the left and right side - bit of a hack but it causes it to expand to the video emulation happening in the center. And I also care about supporting 4:3, which already works well w/o any sort of background hack. The 8:7 ratio I zoom into slightly (6%) so that the height ought to fit perfectly, and then the background hack takes care of the rest! The bg hack only applies to 8:7, it does not load on the 4:3 ratio at all as there’s no reason to render it.

Let me know if there is anything else, but I am pretty sure this knocks out the rest of the issues. Also the MIT License is now mentioned in the glsl file.

Also even though I sorta prefer the 8:7 ratio I am defaulting to 4:3, it looks just as good really and I expect that is what most people will be expecting.

[rbreaves]

Found a few more areas that I was able to improve - and also when the user selects a lesser overlay gradation I actually try and boost the contrast more by merging the white gradient with a black backdrop. Some retro games like Super Mario 3 have extremely light color palettes compared to Super Mario Bros 1, so that should help significantly.

There are also differences in how the border gets handled btwn desktop and mobile use, at least for my GPUs, so I have included the option to enable thick borders to hide potential glitching on the edges. The normal border is of an average thickness I found to work typically though. If you or anyone wants to rewrite that part of this shader then please do, otherwise I believe I am calling this shader done so I can just enjoy it personally.

Another side effect is that this actually makes the retro-v2 shader more widely usable to any user as well, aside for the non-optional soft-light shader, but if you lower the OverlayMix to 0 that effectively only leaves retro-v2 running, or a user can turn the shaders down from 2 to 1 and have a more usable retro-v2 experience that is not limited to gameboy roms.

[hizzlekizzle]

this sounds good. That's getting to be quite a lot of images, though, and the 'subtle' one is pretty large. can it/they be losslessly compressed at all?

[rbreaves]

this sounds good. That's getting to be quite a lot of images, though, and the 'subtle' one is pretty large. can it/they be losslessly compressed at all?

I might be able to get away with doing a smaller 256x240 version since nes and snes games all have roughly the same resolution and is the target audience for this, if I see any banding though then 640x480 may be the lowest option for that.

And for shineblack variation I probably can save that one as a jpg instead.

Be great if I understood glsl better and could just create these gradients w/ pure code - but sadly I don't have that level of skill atm. I may have to look at this tomorrow though or sometime next week as I am kinda tired out atm lol.. part of me wants to just keep pushing so we can get this merged, but it's fine. I will work on it more when I am fresh.

If I slow down a little I think I probably can glean how to properly mask the borders from that image-adjustments.glsl file and eliminate those 3 border options/images I added. I can probably get this commit back down to just using 2 images, a white transparency (diagonal), and the same thing but with a black background.

Also for some reason when I add image-adjustments to the 3rd glslp file it sorta prevents the soft-light effect from disappearing on black areas for some odd reason. That should not be happening, but I may leave it whether I solve it or not & changing the order does not help because if image-adjustment goes first or in the middle it will mess up retro-v2 for some reason, so it has to go last.

[hizzlekizzle]

no hurry :)

unfortunately, we can only use pngs for lookup textures. it won't work with jpgs

gradients are pretty easy to generate programatically, since the builtin vTexCoord already does the heavy lifting for us by starting at 0.0 at one side of the screen and ending at 1.0 on the other. To go from transparent black in the bottom left to opaque white in the upper-right, try: vec4((vTexCoord.x + (1.-vTexCoord.y))/2.). For a black texture with an alpha gradient, you would do vec4(0.0, 0.0, 0.0, (vTexCoord.x + (1.-vTexCoord.y))/2.).

(note: the 1.-vTexCoord.y is to work around a peculiarity of RetroArch's shader backend; for other programs, it would probably just be vTexCoord.y without the '1.0 -' inversion)

[rbreaves]

On the dev side of things are there particular tools I ought to be using that would help me debug and/or visualize glsl shaders more easily before runtime inside of the actual emulator? @hizzlekizzle @hunterk @Hyllian

I would like to speed up my dev time significantly on this type of work if that is possible. I think this work has tensed up my neck, but in part because my tv sits behind me and I have probably been turning my head more times than I can count since Mali GPUs and Intel GPUs handle the shader slightly differently.

It'd be fantastic if I could also emulate a Mali GPU and its quirks lol.

Also I imagine I will need to tweak that equation to follow the rule I was trying to go by in the image based gradient, it is diagonal, but I also start it at 100% outside of the edge, so in reality it probably starts at 90% white in the corner and then progresses to 0% by the time it is half through the diagonal trajectory. Hopefully the overlay will look more or less the same when mixed, but it doesn't have to be exact, just close enough.

I know trying to mix a black background with a transparent white gradient in code failed to look the same, it created a noticeable hard line transition where the gradient ended - compared to just pre-merging the images and using a solid png going from white to black. So I am hoping that type of situation wouldn't occur when mixing the equation based gradient with a black background (from code).