GLSL Programming/Unity/Toon Shading

This tutorial covers toon shading (also known as cel shading) as an example of non-photorealistic rendering techniques.
It is one of several tutorials about lighting that go beyond the Phong reflection model. However, it is based on per-pixel lighting with the Phong reflection model as described in . If you haven't read that tutorial yet, you should read it first. 
Non-photorealistic rendering is a very broad term in computer graphics that covers all rendering techniques and visual styles that are obviously and deliberately different from the appearance of photographs of physical objects. Examples include hatching, outlining, distortions of linear perspective, coarse dithering, coarse color quantization, etc.
Toon shading (or cel shading) is any subset of non-photorealistic rendering techniques that is used to achieve a cartoonish or hand-drawn appearance of three-dimensional models. 
Shaders for a Specific Visual Style.
John Lasseter from Pixar once said in an interview: “Art challenges technology, and technology inspires the art.” Many visual styles and drawing techniques that are traditionally used to depict three-dimensional objects are in fact very difficult to implement in shaders. However, there is no fundamental reason not to try it.
When implementing one or more shaders for any specific visual style, one should first determine which features of the style have to be implemented. This is mainly a task of precise analysis of examples of the visual style. Without such examples, it is usually unlikely that the characteristic features of a style can be determined. Even artists who master a certain style are unlikely to be able to describe these features appropriately; for example, because they are no longer aware of certain features or might consider some characteristic features as unnecessary imperfections that are not worth mentioning.
For each of the features it should then be determined whether and how accurately to implement them. Some features are rather easy to implement, others are very difficult to implement by a programmer or to compute by a GPU. Therefore, a discussion between shader programmers and (technical) artists in the spirit of John Lasseter's quote above is often extremely worthwhile to decide which features to include and how accurately to reproduce them.
Stylized Specular Highlights.
In comparison to the Phong reflection model that was implemented in , the specular highlights in the images in this section are plainly white without any addition of other colors. Furthermore, they have a very sharp boundary.
We can implement this kind of stylized specular highlights by computing the specular reflection term of the Phong shading model and setting the fragment color to the specular reflection color times the (unattenuated) color of the light source if the specular reflection term is greater than a certain threshold, e.g. half the maximum intensity. 
But what if there shouldn't been any highlights? Usually, the user would specify a black specular reflection color for this case; however, with our method this results in black highlights. One way to solve this problem is to take the opacity of the specular reflection color into account and “blend” the color of the highlight over other colors by compositing them based on the opacity of the specular color. Alpha blending as a per-fragment operation was described in . However, if all colors are known in a fragment shader, it can also be computed within a fragment shader. 
In the following code snippet, codice_1 is assumed to have already a color assigned, e.g. based on diffuse illumination. The specular color codice_2 times the light source color codice_3 is then blended over codice_4 based on the opacity of the specular color codice_5:
 if (dot(normalDirection, lightDirection) &gt; 0.0 
 // light source on the right side?
 &amp;&amp; attenuation * pow(max(0.0, dot(
 reflect(-lightDirection, normalDirection), 
 viewDirection)), _Shininess) &gt; 0.5) 
 // more than half highlight intensity? 
 fragmentColor = _SpecColor.a 
 * vec3(_LightColor0) * vec3(_SpecColor)
 + (1.0 - _SpecColor.a) * fragmentColor;
Is this sufficient? If you look closely at the eyes of the bull in the image , you will see two pairs of specular highlights, i.e. there is more than one light source that causes specular highlights. In most tutorials, we have taken additional light sources into account by a second render pass with additive blending. However, if the color of specular highlights should not be added to other colors then additive blending should not be used. Instead, alpha blending with a (usually) opaque color for the specular highlights and transparent fragments for other fragments would be a feasible solution. (See for a description of alpha blending.)
Stylized Diffuse Illumination.
The diffuse illumination in the image of the bull consists of just two colors: a light brown for lit fur and a dark brown for unlit fur. The color of other parts of the bull is independent of the lighting. 
One way to implement this, is to use the full diffuse reflection color whenever the diffuse reflection term of the Phong reflection model reaches a certain threshold, e.g. greater than 0, and a second color otherwise. For the fur of the bull, these two colors would be different; for the other parts, they would be the same such that there is no visual difference between lit and unlit areas. An implementation for a threshold codice_6 to switch from the darker color codice_7 to the lighter color codice_8 (multiplied with the light source color codice_3) could look like this:
 vec3 fragmentColor = vec3(_UnlitColor); 
 if (attenuation 
 * max(0.0, dot(normalDirection, lightDirection)) 
 &gt;= _DiffuseThreshold)
 fragmentColor = vec3(_LightColor0) * vec3(_Color); 
Is this all there is to say about the stylized diffuse illumination in the image ? A really close look reveals that there is a light, irregular line between the dark brown and the light brown. In fact, the situation is even more complicated and the dark brown sometimes doesn't cover all areas that would be covered by the technique described above, and sometimes it covers more than that and even goes beyond the black outline. This adds rich detail to the visual style and creates a hand-drawn appearance. On the other hand, it is very difficult to reproduce this convincingly in a shader.
Outlines.
One of the characteristic features of many toon shaders are outlines in a specific color along the silhouettes of the model (usually black, but also other colors, see the cow above for an example). 
There are various techniques to achieve this effect in a shader. Unity 3.3 is shipped with a toon shader in the standard assets that renders these outlines by rendering the back faces of an enlarged model in the color of the outlines (enlarged by moving the vertex positions in the direction of the surface normal vectors) and then rendering the front faces on top of them. Here we use another technique based on : if a fragment is determined to be close enough to a silhouette, it is set to the color of the outline. This works only for smooth surfaces, and it will generate outlines of varying thickness (which is a plus or a minus depending on the visual style). However, at least the overall thickness of the outlines should be controllable by a shader property. 
Are we done yet? If you have a close look at the donkey, you will see that the outlines at its belly and in the ears are considerably thicker than other outlines. This conveys unlit areas; however, the change in thickness is continuous. One way to simulate this effect would be to let the user specify two overall outline thicknesses: one for fully lit areas and one for unlit areas (according to the diffuse reflection term of the Phong reflection model). In between these extremes, the thickness parameter could be interpolated (again according to the diffuse reflection term). This, however, makes the outlines dependent on a specific light source; therefore, the shader below renders outlines and diffuse illumination only for the first light source, which should usually be the most important one. All other light sources only render specular highlights.
The following implementation uses the codice_10 instruction to interpolate between the codice_11 (if the dot product of the diffuse reflection term is less or equal 0) and codice_12 (if the dot product is 1). For a linear interpolation from a value codice_13 to another value codice_14 with a parameter codice_15 between 0 and 1, GLSL offers the built-in function codice_16. This interpolated value is then used as a threshold to determine whether a point is close enough to the silhouette. If it is, the fragment color is set to the color of the outline codice_17:
 if (dot(viewDirection, normalDirection) 
 &lt; mix(_UnlitOutlineThickness, _LitOutlineThickness, 
 max(0.0, dot(normalDirection, lightDirection))))
 fragmentColor = 
 vec3(_LightColor0) * vec3(_OutlineColor); 
 
Complete Shader Code.
It should be clear by now that even the few images above pose some really difficult challenges for a faithful implementation. Thus, the shader below only implements a few characteristics as described above and ignores many others. Note that the different color contributions (diffuse illumination, outlines, highlights) are given different priorities according to which should occlude which. You could also think of these priorities as different layers that are put on top of each other.
Shader "GLSL shader for toon shading" {
 Properties {
 _Color ("Diffuse Color", Color) = (1,1,1,1) 
 _UnlitColor ("Unlit Diffuse Color", Color) = (0.5,0.5,0.5,1) 
 _DiffuseThreshold ("Threshold for Diffuse Colors", Range(0,1)) 
 = 0.1 
 _OutlineColor ("Outline Color", Color) = (0,0,0,1)
 _LitOutlineThickness ("Lit Outline Thickness", Range(0,1)) = 0.1
 _UnlitOutlineThickness ("Unlit Outline Thickness", Range(0,1)) 
 = 0.4
 _SpecColor ("Specular Color", Color) = (1,1,1,1) 
 _Shininess ("Shininess", Float) = 10
 SubShader {
 Pass { 
 Tags { "LightMode" = "ForwardBase" } 
 // pass for ambient light and first light source
 GLSLPROGRAM
 // User-specified properties
 uniform vec4 _Color; 
 uniform vec4 _UnlitColor;
 uniform float _DiffuseThreshold;
 uniform vec4 _OutlineColor;
 uniform float _LitOutlineThickness;
 uniform float _UnlitOutlineThickness;
 uniform vec4 _SpecColor; 
 uniform float _Shininess;
 // The following built-in uniforms (except _LightColor0) 
 // are also defined in "UnityCG.glslinc", 
 // i.e. one could #include "UnityCG.glslinc" 
 uniform vec3 _WorldSpaceCameraPos; 
 // camera position in world space
 uniform mat4 _Object2World; // model matrix
 uniform mat4 _World2Object; // inverse model matrix
 uniform vec4 _WorldSpaceLightPos0; 
 // direction to or position of light source
 uniform vec4 _LightColor0; 
 // color of light source (from "Lighting.cginc")
 varying vec4 position; 
 // position of the vertex (and fragment) in world space 
 varying vec3 varyingNormalDirection; 
 // surface normal vector in world space
 #ifdef VERTEX
 void main()
 mat4 modelMatrix = _Object2World;
 mat4 modelMatrixInverse = _World2Object; // unity_Scale.w 
 // is unnecessary because we normalize vectors
 position = modelMatrix * gl_Vertex;
 varyingNormalDirection = normalize(vec3(
 vec4(gl_Normal, 0.0) * modelMatrixInverse));
 gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
 
 #endif
 #ifdef FRAGMENT
 void main()
 vec3 normalDirection = normalize(varyingNormalDirection);
 vec3 viewDirection = 
 normalize(_WorldSpaceCameraPos - vec3(position));
 vec3 lightDirection;
 float attenuation;
 if (0.0 == _WorldSpaceLightPos0.w) // directional light?
 attenuation = 1.0; // no attenuation
 lightDirection = normalize(vec3(_WorldSpaceLightPos0));
 else // point or spot light
 vec3 vertexToLightSource = 
 vec3(_WorldSpaceLightPos0 - position);
 float distance = length(vertexToLightSource);
 attenuation = 1.0 / distance; // linear attenuation 
 lightDirection = normalize(vertexToLightSource);
 
 // default: unlit 
 vec3 fragmentColor = vec3(_UnlitColor); 
 // low priority: diffuse illumination
 if (attenuation 
 * max(0.0, dot(normalDirection, lightDirection)) 
 &gt;= _DiffuseThreshold)
 fragmentColor = vec3(_LightColor0) * vec3(_Color); 
 
 // higher priority: outline
 if (dot(viewDirection, normalDirection) 
 &lt; mix(_UnlitOutlineThickness, _LitOutlineThickness, 
 max(0.0, dot(normalDirection, lightDirection))))
 fragmentColor = 
 vec3(_LightColor0) * vec3(_OutlineColor); 
 
 // highest priority: highlights
 if (dot(normalDirection, lightDirection) &gt; 0.0 
 // light source on the right side?
 &amp;&amp; attenuation * pow(max(0.0, dot(
 reflect(-lightDirection, normalDirection), 
 viewDirection)), _Shininess) &gt; 0.5) 
 // more than half highlight intensity? 
 fragmentColor = _SpecColor.a 
 * vec3(_LightColor0) * vec3(_SpecColor)
 + (1.0 - _SpecColor.a) * fragmentColor;
 
 gl_FragColor = vec4(fragmentColor, 1.0);
 
 #endif
 ENDGLSL
 
 Pass { 
 Tags { "LightMode" = "ForwardAdd" } 
 // pass for additional light sources
 Blend SrcAlpha OneMinusSrcAlpha 
 // blend specular highlights over framebuffer
 GLSLPROGRAM
 // User-specified properties
 uniform vec4 _Color; 
 uniform vec4 _UnlitColor;
 uniform float _DiffuseThreshold;
 uniform vec4 _OutlineColor;
 uniform float _LitOutlineThickness;
 uniform float _UnlitOutlineThickness;
 uniform vec4 _SpecColor; 
 uniform float _Shininess;
 // The following built-in uniforms (except _LightColor0) 
 // are also defined in "UnityCG.glslinc", 
 // i.e. one could #include "UnityCG.glslinc" 
 uniform vec3 _WorldSpaceCameraPos; 
 // camera position in world space
 uniform mat4 _Object2World; // model matrix
 uniform mat4 _World2Object; // inverse model matrix
 uniform vec4 _WorldSpaceLightPos0; 
 // direction to or position of light source
 uniform vec4 _LightColor0; 
 // color of light source (from "Lighting.cginc")
 varying vec4 position; 
 // position of the vertex (and fragment) in world space 
 varying vec3 varyingNormalDirection; 
 // surface normal vector in world space
 #ifdef VERTEX
 void main()
 mat4 modelMatrix = _Object2World;
 mat4 modelMatrixInverse = _World2Object; // unity_Scale.w 
 // is unnecessary because we normalize vectors
 position = modelMatrix * gl_Vertex;
 varyingNormalDirection = normalize(vec3(
 vec4(gl_Normal, 0.0) * modelMatrixInverse));
 gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
 
 #endif
 #ifdef FRAGMENT
 void main()
 vec3 normalDirection = normalize(varyingNormalDirection);
 vec3 viewDirection = 
 normalize(_WorldSpaceCameraPos - vec3(position));
 vec3 lightDirection;
 float attenuation;
 if (0.0 == _WorldSpaceLightPos0.w) // directional light?
 attenuation = 1.0; // no attenuation
 lightDirection = normalize(vec3(_WorldSpaceLightPos0));
 else // point or spot light
 vec3 vertexToLightSource = 
 vec3(_WorldSpaceLightPos0 - position);
 float distance = length(vertexToLightSource);
 attenuation = 1.0 / distance; // linear attenuation 
 lightDirection = normalize(vertexToLightSource);
 
 vec4 fragmentColor = vec4(0.0, 0.0, 0.0, 0.0);
 if (dot(normalDirection, lightDirection) &gt; 0.0 
 // light source on the right side?
 &amp;&amp; attenuation * pow(max(0.0, dot(
 reflect(-lightDirection, normalDirection), 
 viewDirection)), _Shininess) &gt; 0.5) 
 // more than half highlight intensity? 
 fragmentColor = 
 vec4(_LightColor0.rgb, 1.0) * _SpecColor;
 
 gl_FragColor = fragmentColor;
 
 #endif
 ENDGLSL
 } 
 // The definition of a fallback shader should be commented out 
 // during development:
 // Fallback "Specular"
One problem with this shader are the hard edges between colors, which often result in noticeable aliasing, in particular at the outlines. This could be alleviated by using the codice_18 function to provide a smoother transition.
Summary.
Congratulations, you have reached the end of this tutorial. We have seen:
Further Reading.
If you still want to know more