Request: Control Phong shading material properties

[p-e-w]

My understanding from reading the Curv documentation is that Phong shading is used to render 3D shapes. But currently, the parameters of the Phong reflection model are not exposed to the user.

It would be nice if those parameters could be controlled on a per-point basis, just like it is already possible for colors with the colour function. Proposed interface:

make_shape {
  ...
  material p = {
    specular_reflectivity = 0.2;
    diffuse_reflectivity = 0.8;
    ambient_reflectivity = 0.35;
    shininess = 0.015;
  };
}

The data structure returned by the material function can grow as needed to satisfy evolving modeling requirements. For example, if a future (hypothetical) 3D printer supported printing using multiple physical materials in the same model, material selection could be realized by extending the structure with an appropriately named field whose value is one of several supported materials.

I would suggest that what is currently colour be absorbed into this more generic function as well, if it weren't for the fact that material properties don't make much sense with 2D images so it is probably better to keep colour separate.

[doug-moen]

I like your proposal. CC'ing @sebastien who might have comments.

Although my research notes have a few references to "Phong shading" and "Blinn/Phong shading", I never tried to implement it. Instead, the existing lighting model was copied and pasted from a shadertoy written by Inigo Quilez, and I don't know how it works. There's nothing sacred about this code, I'm happy to replace it with something better.

// in ro: ray origin
// in rd: ray direction
// out: rgb colour
vec3 render( in vec3 ro, in vec3 rd, float time )
{ 
    //vec3 col = vec3(0.7, 0.9, 1.0) +rd.z*0.8;
    vec3 col = background_colour;
    vec4 res = castRay(ro,rd, time);
    float t = res.x;
    vec3 c = res.yzw;
    if( c.x>=0.0 )
    {
        vec3 pos = ro + t*rd;
        vec3 nor = calcNormal( pos, time );
        vec3 ref = reflect( rd, nor );
        
        // material        
        col = c;

        // lighting        
        float occ = calcAO( pos, nor, time );
        vec3  lig = normalize( vec3(-0.4, 0.6, 0.7) );
        float amb = clamp( 0.5+0.5*nor.z, 0.0, 1.0 );
        float dif = clamp( dot( nor, lig ), 0.0, 1.0 );
        float bac = clamp( dot( nor, normalize(vec3(-lig.x,lig.y,0.0))), 0.0, 1.0 )*clamp( 1.0-pos.z,0.0,1.0);
        float dom = smoothstep( -0.1, 0.1, ref.z );
        float fre = pow( clamp(1.0+dot(nor,rd),0.0,1.0), 2.0 );
        float spe = pow(clamp( dot( ref, lig ), 0.0, 1.0 ),16.0);
        
        vec3 lin = vec3(0.0);
        lin += 1.30*dif*vec3(1.00,0.80,0.55);
        lin += 2.00*spe*vec3(1.00,0.90,0.70)*dif;
        lin += 0.40*amb*vec3(0.40,0.60,1.00)*occ;
        lin += 0.50*dom*vec3(0.40,0.60,1.00)*occ;
        lin += 0.50*bac*vec3(0.35,0.35,0.35)*occ;
        lin += 0.25*fre*vec3(1.00,1.00,1.00)*occ;
        vec3 iqcol = col*lin;

        //col = mix( col, vec3(0.8,0.9,1.0), 1.0-exp( -0.0002*t*t*t ) );
        col = mix(col,iqcol, 0.5); // adjust contrast
    }

    return vec3( clamp(col,0.0,1.0) );
}

[sebastien]

@doug-moen, do you think it would be possible to reify that code into Curv itself, ie. have a standard implementation for view(shape,x,y,t) that then has a pure-Curv implementation of the render function defined above?

This would be a really interesting feature for people (like me), who are not only interested in making printable 3D models, but who are generally interested in exploring generative art, both in 2D and 3D. For instance works like what Sean Zellmer or Kyndinfo are doing would be hard to do with Curv at the moment because it requires custom shading.

So my proposition would be: 1) implement a default view(...) function that does the shading as it is done at the moment, and 2) if a sketch implements a view(...) function, use this one for shading.

Also, in general, I think it's great to have high level scene-like constructs available in the language, but I think we should also make sure that these high level constructs are made on a flexible low level foundation so that people can customize at will.

[doug-moen]

@sebastien: Yes, this seems like a good idea. Yes, a render function could be written in Curv.

In issue #73, I talked about View records, a new kind of graphical value that Curv knows how to render onto a display, and a make_view function for constructing a View record, given 3 arguments: a shape, a camera, and a lights array.

So what does the View record actually contain? Because that is the low level interface you are requesting. I think it contains a shape, camera, and render function.

What is the interface to a render function?

I eventually plan, for performance reasons, to have ray-casting, ambient occlusion, and rendering run in 3 separate compute shaders, quite different from the current architecture, which puts everything in a single fragment shader. This architecture is based on mTec.

So the render function cannot call "castRay", "calcNormal" or "calcAO". Instead, the results of these function calls must be passed in as arguments (or as uniform variables). So the render function will have an interface more like this:

vec3 render(vec3 raydir, vec3 normal, vec3 colour, float occ) // returns colour

And there also needs to be a Material argument, because using an mTec-like architecture, the render function cannot call the shape's material function directly.

[sebastien]

This makes sense, the only thing that worries me both for the make_view and the render functions is that they might be too specific. The notions of lighting, materials and occlusion factor are all part of specific shading models -- they are common, that's for sure, but you might need less, or more parameters to implement your custom shading model.

For instance, make_view looks more like a make_scene (lights, materials, cameras, objects are typically part of a scene graph). It's totally fine to have a high-level scene-graph-like API and probably an interesting evolution of Curv towards complex scene rendering.

So what would a pure-Curv rendering pipeline would look like?

• The shape SDF: it can be implemented using make_shape
• A shape's color/texture: it can be implemented using the colour function
• A shape's shading model: right now, it cannot be overridden
• The raymarching algorithm (from a screen (x,y) to a (r,g,b,a) colour, using all of the above), which also cannot be overridden.

Let's imagine that we have a sketch with two spheres: one that uses a toon-like shading and the other one that uses a reflective metallic shading. For this, we would need at least three separate functions:

• the toon shading function for the first sphere
• the reflective metallic shading function for the second sphere
• the ray marching algorithm that projects the rays and bounce them on the spheres. Given that there is reflection on one sphere, it should bounce at least once when crossing that sphere.

What seems clear to me is that we'll need to pass an open-ended environment/context structure that holds any information that might be required to compute the shading (normal, occlusion, lights, etc). This information would definitely vary depending on the type of shading.
For the ray marching function, we would need a way to do custom blending of the the current ray colour with the sampled colour from the shape's shading, and also tell if the ray marching should continue (transparent/refractive) and/or bounce (reflective). The ray marching function would then populate the environment/context structure passed to the shapes shading functions.

If I try to decompose the process of going from a screen (x,y) pixel value to an (r,g,b,a) colour, that would look like this:

1. Get the directional vector normal to the camera plane for the (x,y) screen pixel
2. Initialize the ray shading context (lights, initial colour, time, etc)
3. Ray march based on a set of shapes until one intersects
4. Calculate the shading of the shape at the given (x,y,z) coordinate, by passing the ray and its shading context. The shading might affect the ray context (returning/blending a colour) and spawn new rays (reflections/refractions)
5. The ray marching algorithm decides how to blend the information returned by the shading function into the ray context.
6. Once the ray terminates, which is determined by the ray marching algorithm, the ray context is converted to the (r,g,b,a) pixel value (blending).

Each of this is a rendering pass, and we could imagine multiple rendering passes being combined together.

An infrastructure like this, if it is reified into the language, would make it possible to not only experiment with procedural shape generation, but also experiment with custom rendering strategies and get even more visually interesting results.

[doug-moen]

The proposed make_view function (based on @p-e-w's proposal) would be written in Curv, and would construct a View record with a render function. It would be one of many possible user-defined functions for constructing views. Renaming it as make_scene seems reasonable.

I want to make Curv's rendering faster and more scalable. Right now, if you union together a large number of shapes, the frame rate plummets. There are multiple open source projects that demonstrate techniques I can use to speed up rendering. To achieve my performance goals, I will need to experiment with multiple techniques (and measure their performance impact), and I'll need to combine multiple techniques together in the final design.

To run these experiments, I need the Curv code that I'm running performance tests on to be well modularized. For example, the dist, colour and render functions are independent, and don't have dependencies that force them to all run in the same shader. I expect that all of the performance-enhancing magic will occur in OpenGL code. I don't have a plan right now for implementing a high performance GPU render pipeline entirely in Curv. The cost of sphere-tracing a union of size N is O(N), and that needs to be reduced to something more like O(log N). Lots of OpenGL magic will be deployed to reduce the cost of raycasting, and I don't know if that code can be written in Curv. I don't actually know the final architecture yet.

I was intending to only support raycasting at this stage, which means: you cast a ray from the camera position, in a straight line, until it hits the surface of the shape, then you stop and compute the lighting at the point where you hit. If you want reflections, then that is ray tracing, which is a lot more complicated. Generalizing my proposed high performance renderer to support real-time raytracing is very ambitious, and it's not in my short or medium term plan.

@sebastien, maybe you can accomplish your goals with a ShaderToy like API, where you write a Curv function similar to ShaderToy's mainImage, and you code your own raymarcher/raytracer and lighting model from scratch. The code would then run in a single fragment shader, not using the optimized rendering pipeline that I'm planning for the future. That is doable. We almost support this today. You could use make_shape to define a 2D shape, and put all of your shadertoy code into the colour function. What's missing is 3D camera control.

So I'm going to replace my previous "View record" proposal with a new version that works at a lower level, and includes a rendering function that maps (x,y) viewport coordinates to a colour, inspired by ShaderToy. I don't have the details worked out yet. Then @p-e-w's interface can be implemented in Curv on top of this low level interface. If you want to experiment with custom rendering strategies, and explore ways to modularize the rendering process, that can be prototyped in Curv.

[sebastien]

@doug-moen That sounds fair, provided we can reuse some of the lower-level rendering building blocks provided by Curv. For instance, there's little benefit to re-implementing the raymarching algorithm if it's already available. Looking forward to seeing this new API!

[doug-moen]

I don't have a written design, but how about this. Maybe we should convert most of the application logic in the Viewer window from C++ code to Curv code, and place it in a Curv Viewer value. Use the model/view/controller paradigm. The view component is a Curv function that is compiled into a fragment shader. The controller is a function that takes mouse and keyboard events as arguments, and maps the old application state to the new application state. The model is the application state, which under current circumstances is the camera state.

The high level building blocks that are composed to create a Viewer are put into a Curv library. If you want to explore a different way to decompose Viewer values into modular building blocks, just create another library: this part of the design does not need to be built in.

[doug-moen]

What are the parameters to the view function?

In the current design, the fragment shader that displays 3D shapes depends on the following parameters:

• The configuration, set when the fragment shader is constructed:
  • The following command line arguments: -Obg=colour, -Oaa=supersampling_factor, -Otaa=supersampling_factor, -Ofdur=frame_duration
• The camera, modified by mouse and keyboard events (aka the model):
  • vec3 u_eye3d
  • vec3 u_centre3d
  • vec3 u_up3d
• The viewport size, modified when the viewer window is resized:
  • vec2 iResolution
• Shader parameters that change every frame:
  • (x,y) -- viewport pixel coordinate
  • t -- time

In the general form of this feature, where you define the model, view and controller by writing Curv code, the view function would look like this:

config -> model -> viewport_size -> (x,y,t) -> pixel_colour

The config parameter is a record, constructed from all of the -O options passed on the command line. The model is also a record, where each field is represented in the fragment shader by a uniform variable. It is by default the camera state, but this is determined by the Viewer record. The VIewer contains a model field that specifies the initial model value, and a controller field that transforms the model based on mouse and keyboard events.

[doug-moen]

There was a discussion in October last year where several people requested more flexibility and configurability in how the Viewer UI interprets keyboard and mouse events for viewing a shape and moving around in space. This MVC design would address those requirements, and also provide the ability to create simple interactive animations in Curv.

[sebastien]

@doug-moen Regarding MVC (3 posts up), that would be great! I would however leave the controller part out of Curv and delegate it to the host (curv's command line viewer, or curved web preview) -- the rationale for that is that Curv is a DSL for generative/procedural modeling and rendering, and that supporting the controller part would mean pulling in a ton of supporting constructs (user input, events, etc) that would dilute Curv's focus and be redundant with the embedding host.

This sums up the ideal scenario: good defaults that you can decompose, override and recompose in pure-Curv:

The high level building blocks that are composed to create a Viewer are put into a Curv library. If you want to explore a different way to decompose Viewer values into modular building blocks, just create another library: this part of the design does not need to be built in.

Also completely agree on the following interface

config -> model -> viewport_size -> (x,y,t) -> pixel_colour

This would be great for Shadertoy-like exploration, but I think we also need a default rendering function that makes use of lower-level primitives, like raymarching, lighting model/shading. In pseudo-code, it could look like this:

// Raymarch returns the intersecting shape and the position where an intersection happens
let (shape,position) := raymarch (ray, scene);

// Shading can decide to continue the raymarching to have reflection/refraction or simply
// calculate the colour using the scene's lights or the shape's material
let shaded_colour := shade (shape,position,ray,scene);

// We blend the computed shaded colour with the original pixel colour (so that we can compose passes)
colour := blend(original_colour, shaded_colour);

Here raymarch, shade and blend could all have default implementations, that could be overriden. For instance, there's less interest in having a custom raymarching than a custom shading.

[p-e-w]

I have refrained from commenting on this thread for the past few days because I wanted to be really sure how I feel about these proposals before responding but here it is now.

When I started to seriously explore Curv around 6 months ago, I realized that this type of scope expansion was something that might happen, and I was very much hoping that it would not.

What the various proposals described above have in common is that they are essentially turning Curv from a DSL for describing three-dimensional shapes (I will ignore 2D here) into a frontend for GLSL. The part of the renderer that is not written in Curv would essentially be limited to OpenGL boilerplate, creating a quad, and then executing a fragment shader that is a thin wrapper around whatever render implementation is provided by the shape/view/etc.

Here is why I think this might be problematic:

1. It ties Curv to GLSL

No matter how the API looks in the end, it will be shaped by how GLSL fragment shaders operate, that is, it will require an underlying rendering system that roughly resembles the Shadertoy API.

Raytracing software, the 3D printing ecosystem, virtual reality etc., all of which are valid environments for shapes represented in Curv, do not provide this type of interface. As a result, there will be a bunch of Curv code that doesn't apply to some output environments because it has nothing to do with shapes per se (I realize that this is also true for the proposal in #73, and perhaps this is actually an argument for not having those parameters in Curv itself but in an external configuration file).

2. It is unlikely to be as good as hand-written GLSL code

The current GPU compiler generates code such as this:

vec3 colour(vec4 r0)
{
  const float r5 = 2.0;
  const float r11 = 3.6489549033077056e-35;
  const float r12 = 0.217637640824031;
  const float r13 = 0.0;
  vec3 r14 = vec3(r11,r12,r13);
  float r1 = r0[0];              \
  float r2 = r0[1];              |
  float r3 = r0[2];              |
  float r4 = r0[3];              |
  float r6 = r1/r5;              |-- ???
  float r7 = r2/r5;              |
  float r8 = r3/r5;              |
  vec4 r9 = vec4(r6,r7,r8,r4);   |
  vec4 r10=r9;                   /
  return r14;
}

Note that much of the code just shuffles values around without any effect on the result. Of course, there must be lots of low-hanging fruit here and the output of such functions can probably be much improved with a moderate amount of effort, but I am highly sceptical of it ever approaching human-written GLSL code in terms of quality and performance, as projects like the GNU Compiler Collection strongly indicate that automatic optimization to that degree requires an effort measured in thousands of man-years.

3. It is not necessary

Professional 3D artists use rendering systems that are highly configurable, but not necessarily "pluggable" in the sense proposed here. I believe the same will work fine for Curv.

Of course, I need to have control over the camera position and perspective. Of course, I want to be able to place lights anywhere, and fine-tune material reflectivity, transparency, refraction and other optical properties.

But do I really need to be able to replace the rendering code? Not sure why, as long as I have control over all of the above. I want the best-looking output possible, as fast as possible. Having a renderer that is compiled from Curv instead of hand-written is unlikely to further that goal, and offers me almost nothing of value in exchange. If there is something the renderer cannot do (such as transparency), it should be added to the renderer, behind a configuration flag, for the benefit of all Curv users.

---

In summary, I believe there should be one (adaptive and highly configurable) piece of rendering code, handwritten in GLSL, that is output by Curv to render shapes defined by their distance functions. I do not think that this renderer should be, or needs to be, written in Curv, as it breaks the abstraction Curv currently provides and offers nothing that cannot also be accomplished by the default renderer in a configurable fashion.

What Curv desperately needs is better actual modeling capabilities, such as an easy way to construct polyhedra or essential operations like taking the convex hull of another shape. It also needs the high-level configurability provided by every 3D modeling software with respect to materials, lights etc. But arbitrary control over per-pixel rendering output strikes me as somewhere between niche and unnecessary. Of course, I understand that others might legitimately feel differently about this subject.

[doug-moen]

@p-e-w: I agree with the high level goals that you stated:

• What Curv desperately needs is better actual modeling capabilities, such as an easy way to construct polyhedra or essential operations like taking the convex hull of another shape. It also needs the high-level configurability provided by every 3D modeling software with respect to materials, lights etc.
• Raytracing software, the 3D printing ecosystem, virtual reality etc., [are all] valid environments for shapes represented in Curv.
• There should be one (adaptive and highly configurable) piece of rendering code ... to render shapes defined by their distance functions.

There are lots of Curv programs that can be 3D printed, but not rendered on screen at an interactive frame rate. So, better rendering performance is a big part of giving Curv better modeling capabilities.

To improve Curv's modeling capabilities, in priority order, I am working on:

1. New Shape Compiler: A complete rewrite of the Shape Compiler (est. 3 months). The subset of Curv understood by the Shape Compiler is called SubCurv. I want to extend SubCurv to support tuples and records, to have better array support (including matrix multiplication), and to support overloaded functions. SubCurv needs to contain a larger subset of GLSL. If you transliterate GLSL into SubCurv, then compile the resulting program back into GLSL, the performance should be as good or better. Getting "as good" performance means not destroying the high-level structure of the code that you wrote in SubCurv. Getting better performance can be achieved by an optimizing compiler that performs domain-specific optimizations. I only optimize referentially transparent Curv expressions, which is a much simpler job than reproducing all the logic in an optimizing C++ compiler.

2. Volume Data Structures: A "volume data structure" (VDS) represents the distance field, as well as other properties, of a particular class of shapes. It enables that shape class to be rendered more efficiently than otherwise, and it enables new operations that are specialized to members of the shape class. Some VDSs, such as voxel grids and bounding volume hierarchies, are quite general purpose, while others are specialized, eg for representing polyhedra, splines or fonts.

   Some Curv programs will construct VDSs algorithmically, or load them from various file formats, before the shape is displayed. The New Renderer will have built-in knowledge of some VDSs, and will display them in an optimized way, while other VDSs will just be Curv data structures plus code that interprets them to provide a distance field. This will expand the range of shapes that can be modeled. Some new operations can't be generally implemented on signed distance fields, but can be implemented on shape subtypes that are constructed from specific volume data structures. Eg, convex hull can be implemented for polyhedra.

3. New Renderer: The new renderer is faster. My intention is to follow the lead of other open source projects, such as mTec, which demonstrate high performance interactive rendering of signed distance fields. The new renderer will put together a number of techniques pulled from different projects, and the final design will be based on performance measurements. Most of these techniques require a GPU rendering pipeline consisting of multiple compute shaders, a feature that's only available on newer GPU hardware.

   The new renderer will support a programmable and configurable lighting model, with support for multiple materials. The details and limitations of how the lighting model works will be conditioned by the architecture and limitations of the new rendering pipeline. For example, I expect that lighting computations occur in a separate compute shader, which has no access to the distance function.

   The new renderer API is portable to an offline raytracer, or to a VR environment.

The New Renderer is the last item in my priority list due to technical constraints. Compute shaders are not available in WebGL 2, or in the OpenGL subset supported by MacOS. However, I could use WebGPU, an API that is still being designed, but which is already available in alpha release form as C, C++ and Rust libraries, and in the 3 major web browsers (as an experimental feature you must explicitly enable). WebGPU will require a GPU manufactured in 2012 or later. The old renderer, based on a single shadertoy-style fragment shader, is capable of running on older hardware.

Hypothetically, WebGPU might be far enough along in 6 months to support the subset of capabilities needed by the new renderer. Maybe development could start then, but the old renderer would still be needed to support older hardware, and to support web browsers that don't have the WebGPU flag enabled. Once people start using the new renderer, new models will be created that can't be rendered by the old renderer. I'm bothered by the idea of maintaining two renderers in parallel, so ultimately, the old renderer will be removed.

This means that the rendering API that Sebastien is requesting is an evolutionary dead end: it requires the old renderer. Nevertheless, it might be useful to implement Sebastien's API as temporary scaffolding, because it would make it easier to prototype ideas for the new lighting model that is part of the new renderer. What I intend is that we would build various high level abstractions on top of Sebastien's API, then port those abstractions to the new renderer once the new renderer becomes available.

Sebastien's API also depends on the New Shape Compiler.