Eurographics Symposium on Rendering (2022)

A. Ghosh and L.-Y. Wei (Guest Editors)

Generalized Decoupled and Object Space Shading System

D. Baker and M. Jarzynski

Oxide Games, USA

Figure 1: Rendered castle model (left) and the associated virtualized shadel sheet (right) with level of detail system (similar to a mipmap).

Abstract
We present a generalized decoupled and object space shading system. This system includes a new layer material system, a
dynamic hierarchical sparse shade space allocation system, a GPU shade work dispatcher, and a multi-frame shade work
distribution system. Together, these new systems create a generalized solution to decoupled shading, solving both the visibility
problem, where shading samples which are not visible are shaded; the overshading and under shading problem, where parts
of the scene are shaded at higher or lower number of samples than needed; and the shade allocation problem, where shade
samples must be efficiently stored in GPU memory. The generalized decoupled shading system introduced shades and stores
only the samples which are actually needed for the rendering a frame, with minimal overshading and no undershading. It does
so with minimal overhead, and overall performance is competitive with other rendering techniques on a modern GPU.
CCS Concepts
• Computing methodologies → Rasterization;

1. Introduction well as the practical problems posed by compositing samples, frag-

ment wastage and randomized memory access.

Decoupled shading, also known as object space shading, is a The advantages of decoupling shading are well known. Until re-
method of rendering specifically designed to address well known cent advances in GPU architecture, decoupling shading remained
weaknesses present in both forward and deferred renderers. Decou- impractical due to both performance consideration and lack of
pled shading helps address mathematical stability problems caused generalized GPU architectures. In 2016, the Nitrous game engine
by evaluating shade samples at different locations each frame, as demonstrated that limited decoupled and object space rendering

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.
 D. Baker & M. Jarzynski / Decoupled Shading

was practical for D3D12 and Vulkan class desktop GPU. The first tributes and subsequent shading occurs on these captured at-
Nitrous title, Ashes of the Singularity, [Bak16] demonstrated that tribute buffers. This distinct advantage over other rendering tech-
despite wastage in overshading and general overhead of decoupled nologies, such as forward, deferred, or even the Reyes architec-
shading, the increases in efficiency for shading in a decoupled man- ture, is among one of the fundamental reasons to use decoupled
ner overcame these issues. shading.
• Efficiently store in GPU memory shade samples. Nitrous 1.0’s
However, Nitrous 1.0 and Ashes of the Singularity did not repre-
shade packing scheme required significant wastage due to gutter
sent a generalized solution for object space and decoupled shading.
spaces and other problems. Though the memory was never read
Because the game is a top-down strategy game, the general prob-
or written to, it occupied valuable space in the high performance
lem of samples which were not visible being shaded was not overly
memory for the GPU.
severe, the projected texel density was relatively constant for most
• Competitive with other real time rendering architectures such as
objects, and the terrain system was built with a limited variable rate
deferred, forward, and forward+ on a modern GPU. We define
stitch map with shading controlled by the CPU.
competitive to be within 10% performance of a similar scene
Furthermore, artists were able to tune art assets to circumvent implemented via forward or deferred.
object space and Nitrous 1.0 issues, with techniques such as not • Allow arbitrary complex materials with relatively strong robust-
texture mapping the underside of units, or mapping it with limited ness. Materials should render with high quality without massive
texel density. This introduced additional cost and time to art assets effort from shader authors. Capabilities such as complex mate-
relative to deferred and forward rendering architectures. Further- rial layering should be supported. Decoupled shading materials
more, decoupled and object space shading produced other prob- should be a super set of any material authorable in other render-
lems; the need to overshade required substantial amount of shade ing architectures.
buffer space, potentially more storage space even than a deferred • Be compatible with a variety of rendering technologies such as
render, though the total bandwidth used was typically far less. rasterization or ray-tracing.
• Have simple, easy to understand, and real time adjustable perfor-
Though this solution worked well for Ashes of the Singularity,
mance controls.
it was not a generalized solution for decoupled and object space
shading. The same rendering technique would perform subopti- With these constraints in mind, we believe that the generalized
mally where there is both significant occlusion and large variation decoupled shading system solves all these problems and can be
in shading samples across the domain of a single object. used to render most types of scenes created for real time systems
today.
In earlier versions of Nitrous, we attempted to solve the over-
shading and large texel variation by breaking up objects into
smaller chunks. While this approach had merits and mitigated some 2. Related Work
of the aforementioned problems, it still required more memory and The Reyes Rendering Architecture [CCC87] implemented a tech-
performance was less than was desirable. Additionally, the burden nique called object-based shading. Reyes subdivides surfaces into
of building art assets broken into small chunks represented a non- micropolygons, that are approximately the size of a pixel, which are
trivial cost to production. Part of this cost was related to art assets inserted into a jittered grid – a super-sampled z-buffer, to introduce
needing to be built with a methodology which was considerably noise and avoid aliasing. In object-based shading, shading happens
different from conventional art pipelines and tool support. before rasterization, as opposed to forward shading where shading
For Nitrous 2.0 we therefore desired, based on our experience takes place during rasterization in screen space, and deferred shad-
and the experience of our partners, a solution with a number of ing [DWS*88] takes place after rasterization. Since shading occurs
constraints and goals: before occlusion testing in the z-buffer, overdrawing can occur.
Burns et al. [BFM10] built upon the Reyes by making two major
• Ability to consume art assets with minimal variation from exist-
improvements: no longer requiring surfaces to be subdivided into
ing tools and art processes. Most art assets have a normal map,
micropolygon size and shades 2×2 blocks of pixels on demand af-
albedo map, a color map, and various attribute maps, sometimes
ter performing visibility testing, reducing the number of samples
stored in vertices. The only extra burden for creation of assets
that are shaded but not used. Fascione, et al. [FHL*18] created
which is acceptable is the creation of a unique UV chart. We
Makuna, a modern implementation of Reyes. While not developed
note that the creation of the UV chart can be automated in many
for real time rendering, it did make many improvements to Reyes
cases.
including decoupling shading and path sampling.
• Ability to amortize shading (or parts of the shading) across mul-
tiple frames, decoupled temporally from rasterization, allowable Ragan-Kelley et al. [RLC*11] proposed a hardware extension
on a material by material basis. based on decoupled sampling, sampling visibility and shading sep-
• Have near perfect shade sample and shading coverage. Pixels on arately, and applying to depth of field and motion blur. Liktor and
the screen should be neither overshaded nor undershaded, nor Dachsbacher [LD12] used a deferred shading system where shad-
should back facing or occluded shading samples necessarily be ing samples are cached when computed, to speed up the rendering
rendered. of stochastic supersampling, depth of field, and motion blur. Clar-
• Maintain the filtering advantages of decoupled and object space berg et al. [CTH*14] proposed hardware extensions for computing
shading. One advantage of object space shading is that most shading in texture space, reducing the overshading problem and al-
aliasing occurs only once during the capture of the shading at- lowed for bilinear filtering.

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.
 D. Baker & M. Jarzynski / Decoupled Shading

Andersson, et al. [AHTA14] used an approach called texture-

space shading where each triangle is tested for culling and shades
its surface, while a geometry shader computes each visible trian-
gle’s size to determine where in a structure similar to a mipmap to
insert the triangle.
Hillesland and Yang [HY16] combined texture-space shading
and caching concepts and used a compute shader in D3D11 to gen-
erate a mipmap-like structure of object-based shading results. Sav-
ing triangle IDs in a visibility buffer [BH13], so that the vertex
attributes can be later accessed for interpolation.
Nitrous 1.0 and Ashes of the Singularity had a non-generalized
solution for decoupled shading [Bak16]. Baker used object-space
shading where objects can have any number of materials by uti- Figure 2: A pre-alpha screenshot of our upcoming title: Ara: His-
lizing masks. Several large master textures were allocated and an tory Untold
objects estimated area was assigned a proportion of the master tex-
ture, scaled to make all the shade requests fit. Texture-based shad-
ing was performed in a compute shader, where each material is
frame can co-execute on a compute queue in the D3D12 or Vulkan
accumulated in the assigned master texture. Mipmap levels where
API, running in parallel to the raster frame.
computed for the master textures. Finally, objects where rasterized
with the master textures used to shade them. The raster frame process begins by collecting the scene for
the GPU rendering. It then dispatches this scene the shadel mark
Mueller, et al. [MVD*18] created a system to atlas a large scene, prepass step. This step’s purpose is not to rasterize, but to mark
broken into parts, with CPU coarse culling, and streaming the shad- which shadels will be needed for the rasterization of the scene.
ing at a lower FPS than the rasterization rate. We implemented a Once this is complete, this data is sent to the shade frame loop.
similar system, including streaming the shading with a 100ms sim-
ulated delay, and shared shading between viewports for VR. Unfor- The raster frame then may either proceed to rasterize the scene
tunately, our analysis showed memory requirements and shading or wait for the shade frame to complete its work. If the raster frame
visibility was far too coarse to meet our goals. Our conclusion at proceeds to render, it may optionally compare its current shadels to
the time was that such an approach was not currently feasible for the shadels which were marked in the shadel mark prepass. Shadels
production. Thus, this effort was abandoned in favor of our current which are missing are shaded immediately, so that there will be no
approach. cracks, holes, or undersampling of the scene. Once the shadels are
shaded, the scene can then be rasterized.

3. Architecture In the shade frame loop, the process begins by performing a layer
space support expander to allocate shadels which are not directly
The generalized decoupled shading engine consists of several com- visible but which may contribute to the scene in indirect ways.
plex sub-systems which interact to render a provided scene. Be- Once this is done, the shadels themselves are allocated. During
cause of the general complexity and latency of CPU and GPU in- the allocation, data is collected for the actual use of shadels and
teraction, the systems and control flow remain largely on the GPU. a global adjustment factor is computed to adjust the shading rate to
The primary responsibility of the generalized decoupled shading fit into the desired shade space to not exceed the number of shadels
engine is the allocation, generation, processing, and management allocated.
of shade elements, which we call shadels.
The next step is for the work queues to be generated, allocated,
The process works by running two primary loops which may run and filled. Shadels must be dispatched for every object, for each
at different frequencies. For our upcoming title, Ara: History Un- material instance on that object, and for each layer on that object.
told, we always run them at the same temporal frequency. The first Once shadels have been processed, they are sent back to the render
loop is the raster frame, which corresponds roughly to what is typ- frame. Figure 3 illustrates these steps and the raster frame interac-
ically thought of in deferred or forward renderers’ entire rendering tion with the shade frame.
pipeline. The second loop is the shade frame, which performs the
actual shading. In the typical configuration, the shade frame runs at
3.1. Objects, Shade requests and Materials
no more than 30 times per second, while the raster frame runs at a
much higher rate, ideally as fast as the display device can display In Nitrous 2.0, the following things which are rendered through the
which may be upwards of 200 times per second. decoupled shading system are: objects which consist of triangles, a
material instance which goes with the object, dynamic and constant
The raster frame and the shade frame have very different com-
data, which is generated by the application, and a group of arbitrary
putational needs. The raster frame involves mostly the use of ras-
resources, usually collections of textures.
terization hardware such as triangle rasterization and depth buffer,
where the shade frame requires only the use of compute shaders. All objects together represent a scene and anything which may
Because they can be run in parallel, in one configuration the shade contribute to the final rendering of the screen should be in this

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.
 D. Baker & M. Jarzynski / Decoupled Shading

triangle for each pixel. In the decoupled shading system and in ob-
ject space rendering triangles are not used for shading, rather each
shadel has a collection of corresponding input attributes.
In our first implementation, triangulated meshes were converted
to have shadel input attribute textures. This process involved ren-
dering the model from the 2D texture parameter space into a buffer
(thereby capturing the rendered attributes) either repeating this pro-
cess for each shadel level (analogous to a mip level) or performing a
downsampling filter. In our Nitrous 2.0, rather than capturing each
individual attribute, we follow the approach of Hillesland and Yang
[HY16] and only capture the triangle ID. When the shading occurs,
the mesh index buffer and vertex buffers are bound as inputs to the
shadel shader. The triangle ID is used to look up the vertices, which
Figure 3: The Raster and Shade frames. The raster frame col- are used to interpolate the input attributes from the vertex data in
lects the scene and prepares it for the GPU, then dispatches it to nearly the exactly same way as would occur during a forward or
the shadel mark prepass. The shadel mark prepass marks which deferred renderer. This has the advantage of using far less data, and
shadels need to be rasterized and sends this data to the shade frame also kept attributes more consistent with any corresponding values
loop. Once the shadels have been processed in the shade frame loop that might come from rasterized buffers such as shadow maps.
they are sent back to the render frame for rasterization.
To capture the triangle ID, we rasterize the triangles in texture
space, with the output being only the triangle ID. The resolution of
this texture is captured at is controlled by a setting for each indi-
vidual asset. This is an asset cooking step and does not occur while
the game is running. This triangle ID is captured for each mip level
of the triangle ID texture. One problem that arose was that some-
times triangle or section of triangles on the mesh resulted in no
triangle ID being captured, due to triangles falling in between cov-
erage rules for rasterization. This can later result in geometry being
rendered and having no shadels which represent it.
To solve these problems, we make some important modifications
to other approaches to capture triangle IDs. After the attributes are
first generated as previously described, we capture them again, but
this time switch the rasterization mode to conservative rasterization
Figure 4: A pre-alpha screenshot of Ara: History Untold. Every- [AA05][HAO05]. This changes the coverage rule such that all tri-
thing in this screenshot are rendering through the decoupled shad- angles will emit an attribute to any sample they touch. This process
ing system, except for the trees, people, and special effects such as is repeated similarly to the aforementioned process.
fire. Next, the two triangle ID maps are merged. Non-conservative
rasterization is preferred, however if a sample exists in the conser-
vative rasterization where no sample exists in the non-conservative
scene. This scene is created implicitly by tracking all objects which version, the merged version uses the conservative rasterization sam-
have been requested to be rendered. Scene culling and object refine- ple. This process means that there is no chance that a triangle when
ment occur before the decoupled shading system begins its process- applied to rasterization, does not have any captured triangle IDs.
ing. Figure 4 is a scene from Ara: History Untold. See Figure 5.

Objects have two distinct stages of existing. They may be instan- We have found that these improvements greatly increase the ro-
tiated, which means that they exist somewhere in the world and will bustness of triangulated meshes such that in general the Nitrous
have an object instance ID, and they may be also requested for ren- engine can consume most assets created for forward or deferred
dering - which means they will be rendered into the scene if they renderers. In addition, by saving only the triangle ID, the only ad-
are visible. Multiple objects may share material instances, and vir- ditional memory for an asset is the creation of a triangle ID texture.
tually any buffer or GPU resource with the exception that they will
never share shadel space.
3.3. Virtualized shadel allocation
When objects are instantiated and could possibly render (but may
3.2. Attribute Processing
not be actually requested to render for a particular frame), they are
Triangulated meshes which we use typically have some attributes allocated shade space inside the virtualized shade space system. We
stored in the vertices including: normal, positions, texture coordi- call them shadels as they are distinct from both texels and pixels,
nates, specular power, albedo, etc., which are interpolated across a because they could be implemented in a variety of ways depending

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.
 D. Baker & M. Jarzynski / Decoupled Shading

Figure 5: Merging triangle ID maps between non-conservative and

conservative rasterization. Preferring conservative rasterization
where samples exist and falling back to non-conservative where
conservative samples do not exist.
Figure 6: Virtualized shadel allocation. 8×8 chunks of shadels
from a 2D shadel remap buffer are stored in a shadel storage buffer.

on the specific hardware being used. On some platforms, for exam-

ple, it might be more efficient to store shadels in a typeless buffer
rather than a texture. Currently, we use textures to store them but
have experimented with other storage methods. ShadelChunkStartIndex + countbits(∼ (ShadelSubIndex − 1) & OccupancyField) (1)
The shadels are stored inside a large virtualized 2D space,
where virtualized 2D dimensions have been tested as large as For more extreme scenes, we experimented with an additional
512K×512K. Each object is allocated with the maximum possi- level of indirection. In this scenario, we introduced a secondary
ble shadel density which could be needed for any subsection of the remap buffer and perform a similar allocation step. This configura-
object, including detail which might be introduced via a material tion introduces an additional dependent texture read, but increases
shade. practical virtualized shade space dimensions to the 4M×4M range
at a small decrease in performance. Our experience was that this
This massive virtual shadel sheet creates enough space such that additional shade space was unnecessary. However, it is possible to
objects that are instantiated in the scene need not bound their max- use virtual resident textures to map pages in/out as necessary which
imum scale other than a very coarse maximum. Though the virtu- should allow these same resolutions with no additional passes and
alized texture is beyond the scope of what could practically be on minimal additional overhead.
a GPU now or in the near future, the working set of shade samples
The shadel storage buffer also has a level of detail system sim-
which could actually be touched is a tiny percent of this space.
ilar to a mipmap in a texture. However, even though the remap
The virtualized shadel sheet is created as a two level hierarchy, buffer may be stored in a 2D texture, the mip hardware and tex-
but implemented in such a way as to require only one level of de- ture addressing capabilities are not used. This is due to unordered
pendent reads. The shadels themselves are stored in 2D blocks, access view (UAV) mapping challenges and alignment require-
which we implemented as 8×8 chunks of shadels. The virtualized ments which require specific rules for mip block alignment. Specif-
shadel space consists of two memory buffers, implemented for con- ically, because a shadel chunk is set to 8×8 and each entry the
venience as two 2D textures: a 2D remap buffer and a shadel stor- remap buffer therefore corresponds to 64 shadels, single allocations
age buffer. The remap buffer contain three values: a 32-bit shadel smaller than 64 shadels can’t be made in the shade space, even for
block start offset which marks the beginning index location of smaller mip levels.
the shadel storage buffer, the object instance ID which represents
Figure 6 shows an actual breakdown of the 2D remap buffer
which object the shadels belong to, and the occupancy field, a 64-
along with the shadel detail levels. Note that past the first few en-
bit value which represents 1 bit for each of the chunks of shadels
tries, each level of detail level takes the same amount of space. We
if it is occupied or not occupied. Therefore, each entry in the 2D
determined that for our own use, we did not need to allocate shadel
remap buffer represents 8×8 chunks of shadels, if a shadel chunk
chunks smaller than 256 shadels in our shadel space. This observa-
is also 8×8 then each entry in the remap buffer represents 64×64
tion meant that lower mip levels would all align to the same size,
contiguous shadels.
which effectively means that some occupancy bits can never be set
To address a particular shadel, the 2D shadel location is indexed on the edges of the lower mip level and some entries in the 2D
to the remap buffer which corresponds to it, and the sub index of shadel remap buffer are never written to nor read from.
the shadel chunk is also calculated, which will corresponds to 1 of
the 64 bits in the remap buffer. The address of the shadel chunk is
3.4. Shadel Mark Prepass
then calculated by Formula 1. This gives the location of the shadel
chunk, where each shadel can easily be sub-indexed. The shadel The shadel remap buffer allows locations in the virtual shadel
allocation ID number provides an index value which will be used storage buffer to be physically mapped. Before the shadels can
for the dispatching of work by the GPU. be mapped, the renderer must mark the actual shadels which are

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.
 D. Baker & M. Jarzynski / Decoupled Shading

Figure 8: Pipeline of our pixel shader. It takes advantage of the

fixed function rasterization, atomic operators, early z depth, write
Figure 7: The marking process where an occupancy bit is set using combiners, and efficient caches.
‘interlocked or’ corresponding to a 8×8 chunk of shadels.

actual amount of writes to system memory are on the order of only

needed. This process is called the shadel mark prepass. After this 2 to 4 bits per shadel chunk. If the shadel chunk size is 8×8, a 4K
prepass is complete, every shadel chunk which is needed for the screen writes only about 1 MB of actual data.
rasterization will be marked.
As fast as ‘interlocked or’ operations are, the overhead is still
This process starts by clearing all bits from the bitfield element too high. Because all bits were initialized to 0 and only 1 bit at a
of shadel remap buffer. If we chose to use a texture for the shadel time is set during each operation, a simple optimization is to read
remap buffer, this clear is essentially free on most GPUs. Next, we the bit field first, and only perform the interlocked operation if the
perform a proxy render of the scene. For Nitrous 2.0, this means bit needs to be set. This results in significant performance increases
running the corresponding shader stages which render the objects since the bitfields are so small as to usually be in the L1 or L2 cache
on the screen. This can include any type of shader, such as vertex on the GPU.
shader, hull shader, domain shader, a mesh shader, etc. Since GPUs
have complex and efficient pixel shader and rasterization hardware, Another important performance optimization is to use the ear-
we want to repurpose the hardware. lydepthstencil pixel shader attribute to allow the hardware
to early exit work of shadels which cannot be visible. By using this
To repurpose the pixel shader, the goal is to use the pixel shader attribute, only shadels which will actually end up contributing to
similar to a simple object rasterization. Each object has an associ- the scene will be marked.
ated start location and dimension entry given to it when it allocates
space in the virtualized shadel storage buffer. Because each object Figure 8 illustrates how we set up a pixel shader pipeline to take
has a unique UV chart, this chart is used to index shadels similarly advantage of fixed function rasterization, atomic operators, early z
to a texture. depth, write combiners, and efficient caches. These aforementioned
hardware units have already been implemented on modern GPU
Because it is not a texture, an equivalent trilinear or anisotropic architectures and combined with the locality of pixel processing
filter need to be implemented manually. During the mark prepass, effectively repurpose the GPU to make the pixel shader efficient as
the actual shadels will not be loaded but rather the shadels which marking the shadels with only one prepass of the geometry.
would be accessed during a filter need to be marked (8 in the case
of trilinear). This approach works for any type of filtering desired,
with old techniques such as bilinear able to give modest perfor- 3.5. Shade Space Allocator
mance gains on lower end hardware.
Once the shadel chunks are marked, the shadels must be allocated
The marking process, illustrated in Figure 7, consists of the set- in the shadel storage buffer. This process is performed by examin-
ting an occupancy bit corresponding to a chunk of shadels (which ing each bitfield which corresponds up to 64 shadel chunks. The
we usually set to 8×8). Every chunk of shadels allocated in the number of chunks needed for that shadel chunk cluster is the num-
virtualized shadel space has a single bit which corresponds to it. ber of bits set in the bitfield.
Setting a single bit could be problematic with hardware since the
This can be performed efficiently by first subdividing the shade
bit is co-located with other bits. We note that bits only need to be
texel storage buffer into N subdivisions. The chunks can then be
set, and only 1 bit would ever be set for each shadel with most
allocated in parallel performing an atomic exchange increment on
shadels mapping to the same occupancy bit.
one of the subdivisions by using any number of simple hash func-
We use ‘interlocked or’ to set the bits for each shadel, with each tions to map a shadel group to a subdivision. The actual location of
pixel on screen possibly setting 8 bits. Since most pixels in a GPU an individual shadel chunk is computed by examining the set bits
are processed in the same unit as other nearby pixels, we take ad- of the shadel remap texture. Although this requires slightly more
vantage of the write combiners on modern hardware such that the reading of data for addressing, we note that it reduces the opera-

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.
 D. Baker & M. Jarzynski / Decoupled Shading

Table 1: Memory requirements to store shadels per resolution with

4x MSAA.

Resolution Back Buffer Size Total Shadel Memory

1920×1080 16 MB 64 + 50 MB
2560×1600 32 MB 128 + 50 MB
3840×2160 64 MB 256 + 50 MB

tion to only one dependent read and the remap locations small size
means they are typically inside the L1 cache of the GPU.

3.6. Global Shade Space Adjuster

A typical scene produces large numbers of shadel chunks to shade.
For reasonable efficiently mapped objects, we have noted that as a
rule of thumb we need twice the number of shadels as we do pixels
in the screen resolution to be able to guarantee perfect shade cov-
erage as defined by similar texel coverage in a deferred or forward
renderer. This is given a certain amount of overlap due to trans-
parency and alpha blending. We note that this comparison is some-
what unfair to decoupled shading since a given pixel on the screen
has computed both it’s maximum LOD and at least one LOD less
to provide smooth shadel downfiltering. Thus, the number of sam-
ples used for a frame has a lower bound of about 25% more than a
forward or deferred renderer.
Because scenes can vary in complexity, it is possible for a given
scene to exceed the number of shaded samples in the shade allo-
cation buffer. This would cause incorrect rendering as the shade
samples have no memory to be stored. To mitigate this potential
problem, the current number of shadels is continuously uploaded Figure 9: Pre-alpha render of Sappho, one of the leaders in Ara:
to the host CPU. As it approaches certain thresholds the system in- History Untold. The skin shader uses multiple layers and succes-
creases a global mip bias value, which can fractionally adjust itself sive passes.
to keep the number of shading samples to fit within the shade allo-
cation buffer. Because very small amounts of mip biasing decrease
the number of samples dramatically, we found that in the very rare
event the shade space hit our 2x rule, the perceived quality differ- then computing a normal from that layer by processing the heights.
ence was imperceptible as the fractional mip bias never exceeded In another example, a skin shader can use successive passes to per-
the 0.1 mark. form a blur on the shaded layer. Both of these cases are used by
materials in Ara: History Untold. Figure 9 is an example of the
Table 1 refers to the amount of memory needed to store the
layering process.
shadels. Experimentally, a virtualized 2D map of 256K×256K
worked well for most scenes, so long as some effort was taken to To support layers, generalized decoupled shading uses the same
bind maximum shadel density and reallocate it as objects move fur- allocation location to correspond and to address multiple image
ther and closer to the screen layer sample planes, which can store these possible intermediate
image layers. However, because a layer may need support for a ker-
nel operation (e.g. a wider neighborhood to compute a normal), this
3.7. Layer Shading
could create problems where these samples were never evaluated
As part of the material system, Nitrous supports the concept of lay- because the shadel of that location would never be used for the fi-
ered materials. Material layers exist as a series of layer images, with nal composition in this scene. In practice, however, we never found
a layer program executing on each layer image. Each layer image issues, the support needed for a kernel didn’t exceed the naturally
is evaluated before the next layer can process, and layers can read shown visible shadels. If this become a problem, a pass causing
and write from arbitrary previous layers. visibility expansion could be added.
This is useful for any operation which requires neighborhood Layer images can also store values across frames. In this use
information. For example, a height map can be converted into a case, we buffer the remapping buffers for the previous frame and
normal map by a material creating a height map in one layer, and allow access to the previous layer images directly from any mate-

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.
 D. Baker & M. Jarzynski / Decoupled Shading

rial. Because any material can read and write to any layer image as preprocess can be run at lower resolution often without noticeable
it desires, it is up to the individual material for how it might reuse issues. Problems can arise from certain types of charting on some
any intermediate value. high triangle count meshes, but can be mitigated by LOD choices
and carefulness in art preprocess. In our current implementation,
The retrieval of previous frames layer image samples works by
we chose to keep the resolutions the same, because we had a need
making a function call in the material which uses the previous
to have a screen resolution Z prepass for other rendering needs.
frame shadel’s allocation and storage. If a sample exists, it is re-
turned, otherwise, the function returns false so that the material can Our data indicates, that the shadel mark preprocess step is often
decided how to proceed. This feature is exploited extensively in our less than 1ms on modern GPUs even at high resolutions, therefore
terrain material, where we cache the expensive blended attributes we typically run the process at the full resolution to guarantee wa-
that need to be generated. tertightness with lower resolutions reserved for lower performance
systems.

3.8. Work Dispatch Aggregator and Allocator

4. Results
Once shadel storage has been allocated and the system knows
which shadels need to be processed, it still needs to generate the 4.1. Frame Analysis
actual GPU commands to process the shadels. In Nitrous 2.0, all One critical requirement for decoupled rendering is that to be us-
shadels are processed via a compute shader, but shadels can be pro- able, it must be both general purpose, and also be comparable to
cessed easily by any capable computing device. other rendering architectures in terms of performance and memory.
This remains the key obstacle to using decoupled shading.
A naïve method is to evaluate shade samples is to evaluate the
entire shadel remap buffer for each object which has been sub- While the memory requirement is straightforward to determine,
mitted and for each layer. Early testing of this indicated that for the performance comparison is difficult. To be useful, this metric
small workloads, this is sufficient however the larger the virtualized should be applied to a full game scene. Unfortunately, it is not fea-
shadel space, the more prohibitive this becomes since GPUs are ef- sible to implement the scene of our current title with an entirely dif-
ficient at terminating work early, this quickly becomes an O(n2 ) ferent rendering architecture (some of our materials cannot be used
operation. in a forward or deferred renderer), therefore we use some other
analysis to indicate if we met our goal with being competitive with
Instead, the entire shadel remap buffer is examined, and a work
other rendering architectures.
list is generated for each object which is to be rendered. This work
list consists of the virtual remap location of each shadel chunk and Because our decoupled rendering architecture is fully compati-
the associated shadel detail level. ble with forward rendering, there is some reasonable analysis which
we can perform to get a good estimate of a direct performance com-
The work queue buffer is shared among all objects, and since parison. By taking a frame trace of a typical scene, we can isolate
allocated into segments depending on how many work items are the particular parts of the frame which are specific to decoupled
needed for each object, each work item being effectively a shadel shading and the parts which would be identical to or are similar to
chunk. Some objects are not visible at all, due to occlusion, frustum a forward renderer. The ratio of these two numbers is a reasonable
or other conditions which might mean that their samples are not estimate of the cost differential between the two rendering types,
touched. These objects will have a work count of 0 and take no with some caveats to be discussed.
space in the work queue buffer.
To perform this analysis, we created a typical scene in our game.
When the work queue buffer is populated, a work dispatch pa- We ran this scene on an Nvidia GeForce 1080 Ti at 1080p, with
rameter buffer is also populated with the start location of the work 4x MSAA, and it had an average frame time of around 24ms. This
queue to read and the number of items which will require shading. scene has 8 million triangles, 6500 draw calls, 6GB of GPU Mem-
Each entry in the work dispatch parameter buffer is referenced ory, 8MB CPU to GPU data upload per frame, and around a 5%
via a dispatch indirect call, which corresponds to all the state re- variance of frame time.
quired for the shading and processing of shadels for a particular One important note about the scene, is that our foliage does not
object with one indirect dispatch per material layer per object. run through decoupled shading. This is because the material for
our foliage is a simple, mostly lambertian material, meaning that
3.9. Frame Render there is limited visual benefit for decoupled shading. Our foliage is
almost entirely fillrate/front end bound, and uses very little shader
The frame rasterization occurs as a similar process to the shadel compute resources.
mark prepass, however this time instead of marking the shadel
In the timing for the scene, the total time directly required for
chunks which will be used, the shadels are read from the shadel
decoupled shading is about 2ms. This represents around 10% of the
storage buffer using the already populated shadel remap buffer.
total frame time. The shading represents around 40% of the frame.
The frame render and shade mark preprocess should ideally run At first glance, this would appear to be directly related to decoupled
at the same resolution and exactly replicate one another to guaran- shading, however, in a forward renderer this shading would occur
tee watertightness. Practically speaking if layer edge expansion is during the rasterization pass, so this represents a transfer of work
enabled and because shadels are marked in chunks, the shadel mark rather than overhead due to decoupled shading.

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.
 D. Baker & M. Jarzynski / Decoupled Shading

Table 2: A frame analysis of a typical frame in our Decoupled Shading Engine (DSE) from Ara: History Untold, which featured 8 million
triangles and 6,500 draw calls.

Operation Description Timing (ms) DSE Only

DSE Mark buffer clear DSE Setup 0.25 Y
Terrain Overlay Terrain UI 0.5 N
Shadow maps Shadow Depth Raster 1 N
Rasterization Z Prepass 0.75 N
Rasterization DSE Shade Marking 1 Y
DSE Work Dispatch Control logic for DSE 0.75 Y
Shading Material Shading 10 N
Rasterization Final Scene Composite 1.25 N
Rasterization Foilage 4 N
Volumue Rendering Volumetric Effects 1 N
UI UI Effects 1 N
DOF Post Process 0.75 N
Color Curve Post Process 0.25 N

The major difficulty with the shading section is that the num- poral anti-aliasing. The second observation is that a detailed anal-
ber of samples evaluated from a forward renderer and a decoupled ysis revealed that there is likely a large performance gain simul-
is not the same. A decoupled renderer will typically process more taneously rendering foliage with shading. This is because foliage
samples, though this is not always the case since small triangles is opaque and using alpha to coverage, it may render before any-
will over shade in a forward renderer due to the fact pixels must thing which needs shading. Since the foliage uses little compute
evaluate on a quad. Whereas in a decoupled renderer such as ours, resources and mostly uses rasterization resources, on an architec-
the shading granularity is higher and the renderer takes more sam- ture which allows asynchronous compute it should be possible to
ples on anisotropic edges. co-execute direct rendered opaque objects while shading is occur-
ring. This could yield a performance increase of 2-4ms.
However, we can use a couple of observations to estimate they
are likely approximately the same. We have also observed higher Overall, our analysis indicates that performance should be with-
GPU occupancy shading in decoupled shading than with forward ing 10% of other rendering architectures, and if other mentioned
rendering due to better cache use and dispatching, with occupancy factors are considered it may be able to outperform other rendering
often being near perfect if GPU register usage isn’t too high. There- architectures in some cases.
fore, while decoupled shading is likely processing more samples,
it can evaluate samples somewhat faster. Additionally, we observe
that 30-50% of a frame time spent shading is typical for many 4.2. Quality Feature Improvements
games, based a variety of frame captures we have both down on our We have already noted that decoupled shading is competitive with
own prior titles and the types of captures we have seen from other other rendering architectures, but performance is only one factor to
games and engines. Therefore, 40% of our time spent in shading is consider when building a rendering system.
roughly in-line with our expectations.
Regardless of performance advantages, decoupled shading has
We also ran timings on a similar scene on an AMD Radeon RX intrinsic quality improvements. One of the biggest problems with
6800 GPU. According to UserBench, this GPU is nearly identical shading techniques is shading aliasing and shading instability to in-
performance to the Nvidia GeForce GTX 1080 Ti. However, we terpolation of non-linear data. A common example of this is shim-
noted that while most of the timings were nearly identical to the mering results when a high frequency normal map is used with
1080 Ti as expected, the shading time was less than half at around a low roughness factor. This results in high frequency highlights
4ms, with the entire frame time around 18ms. In this GPU, shading which shimmer frame to frame as the micro variations in the sam-
would represent only around 25% at most of the frame time. We ples can cause widely different shading results.
are still investigating the reason for the performance difference.
It could be due to a newer, more efficient architecture for com- There are many techniques designed to help mitigate this prob-
pute shading, or perhaps some missed optimizations in the Nvidia lem such as LEAN Mapping [OB10], but there exists no general-
Driver. On this GPU architecture, it seems unlikely that any for- ized solution to this problem for forward or deferred, nor can there
ward rendering of our scene could be significantly faster than our be since real world materials are often too complex to fully solve
current approach. these issues. Part of our design requirements was shading robust-
ness, where any shader will have some degree of anti-aliasing re-
There are a few more considerations worth discussing that would
gardless of the attention spent to anti-alias it.
further benefit decoupled shading. One is that due to intrinsic tem-
poral and shader anti-aliasing, our renderer does not require tem- While decoupled shading still aliases, the samples used per

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.
 D. Baker & M. Jarzynski / Decoupled Shading

frame are invariant. This is distinct even from architectures such as multi-pass layers, such as when normals are generated from com-
Reyes where triangles are generated relative to the current frame. posited heights and human skin.
The advantage of this is that while decoupled shading may be in-
The primary hardware features which make this feasible for effi-
correct, it is incorrect in the same way each frame thereby removing
cient rendering are decent speed interlocked bitwise atomics, good
one class of rendering artifacts.
L1 caches, and good write-combiners. On newer GPU architectures
Another feature decoupled shading brings is the aforementioned such as RDNA 2 the rendering is particularly efficient, with shading
concept of material layers. Simply put, shadels can have precise accounting for only 25% of a typical frame time.
access to their complete neighborhood, as well as precise access to
The estimated cost over a forward renderer of similar complexity
previous temporal samples. Additionally, similar to forward render-
is around 10%, with the hope that further optimizations will make
ing, decoupled shading has no real limit to the number of different
the performance nearly the same or better. The primary overhead
material and material instances which can be used.
for decoupled shading is twofold. First is we require an additional
Finally, it is a trivial matter in decoupled shading to adjust the prepass on the geometry of the scene, and the second is that there
sampling rate for specific objects, areas of objects, materials, etc. is some amount of fixed overhead aggregating the results of this
Shading sampling and super sampling is controlled by a single prepass to dynamically dispatch GPU work.
number in the shader. Foveated rendering can be implemented with
The Nitrous 2.0 rendering system provides the first, comprehen-
a handful of shader code. Performance can also be easily adjusted
sive, and complete decoupled rendering solution. Decoupled shad-
by varying total shading samples across the scene independent of
ing is practical and efficient on modern GPUs. Decoupled shading
resolution.
can be used with many rendering techniques including, ray tracing,
point rendering, triangle rasterization, tile rendering, MSAA, alpha
5. Further Work blending, and many other GPU features. Additionally, decoupled
shading has intrinsic quality and flexibility benefits that cannot be
Decoupled Shading can integrate with ray tracing hardware in a
matched by forward or deferred rendering architectures [Bak16].
variety of ways. To ray trace inside a material, the scene is also up-
Finally, we believe with additional GPU modifications, decoupled
dated and maintained in one or more bounding volume hierarchies
shading could exceed forward and deferred rendering in almost all
(BVH), as is typical for real time ray tracing. At this point, any
work loads
shadel can request a ray trace in the same manner as a pixel shader
could, allowing full integration with ray tracing.
7. Acknowledgements
Ray tracing can be integrated more deeply into a decoupled shad-
ing system. If various surface properties are collected into different Ara: History Untold is published by Xbox Game Studios.
layers, traced rays can look up their values into the populated shadel
remap and storage buffer, marking the shadels in the remap buffer
References
so that they become available in future frames.
[AA05] A KENINE -M ÖLLER, T OMAS and A ILA, T IMO. “Conservative
Additionally, decoupled shading allows for additional interesting and Tiled Rasterization Using a Modified Triangle Set-Up”. J. Graph-
modes of operation. Rather than trace rays directly in the material, ics Tools 10 (Jan. 2005), 1–8. DOI: 10 . 1080 / 2151237X . 2005 .
the ray origin and direction can be stored into one or more layers. 10129198 4.
This layer is dispatched to ray tracing hardware which populates [AHTA14] A NDERSSON, M., H ASSELGREN, J., T OTH, R., and
another layer with the results of the ray trace shader. A KENINE -M ÖLLER, T. “Adaptive Texture Space Shading for Stochastic
Rendering”. Comput. Graph. Forum 33.2 (May 2014), 341–350. ISSN:
By dispatching large clusters of rays at once, the decoupled shad- 0167-7055. DOI: 10.1111/cgf.12303 3.
ing engine can sort and group the rays for a much faster trace [Bak16] BAKER, DAN. “Object Space Lighting”. Mar. 2016. URL:
through the scene, avoiding costly shading during the hit shaders. https://www.gdcvault.com/play/1023511/Advanced-
Graphics-Techniques-Tutorial-Day 2, 3, 10.
[BFM10] B URNS, C HRISTOPHER, FATAHALIAN, K AYVON, and M ARK,
6. Conclusion W ILLIAM. “A Lazy Object-Space Shading Architecture With Decoupled
Sampling”. Jan. 2010, 19–28 2.
The described process produces scenes which render quickly and
efficiently on modern hardware, despite the fact that the hardware [BH13] B URNS, C HRISTOPHER A. and H UNT, WARREN A. “The Visibil-
ity Buffer: A Cache-Friendly Approach to Deferred Shading”. Journal of
tested was not designed for this type of rendering. This rendering Computer Graphics Techniques (JCGT) 2.2 (Aug. 2013), 55–69. ISSN:
architecture can replace many forward rendering architectures for 2331-7418. URL: http://jcgt.org/published/0002/02/
a production title at scale, with approximately the same perfor- 04/ 3.
mance characteristics, and is competitive in overall performance [CCC87] C OOK, ROBERT L., C ARPENTER, L OREN, and C ATMULL, E D -
with other rendering techniques. Our current title has over 5,000 WIN . “The Reyes Image Rendering Architecture”. SIGGRAPH Comput.
assets, ranging from characters, to terrain, to buildings compatible Graph. 21.4 (Aug. 1987), 95–102. ISSN: 0097-8930. DOI: 10.1145/
with this decoupled shading architecture. 37402.37414 2.
[CTH*14] C LARBERG, P ETRIK, T OTH, ROBERT, H ASSELGREN, J ON, et
In addition, this rendering architecture can co-exist with forward al. “AMFS: Adaptive Multi-Frequency Shading for Future Graphics Pro-
rendering, and is generally a superset. Many of our materials now cessors”. ACM Trans. Graph. 33.4 (July 2014). ISSN: 0730-0301. DOI:
rely on some of these additional features, especially materials with 10.1145/2601097.2601214 2.

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.
 D. Baker & M. Jarzynski / Decoupled Shading

[DWS*88] D EERING, M ICHAEL, W INNER, S TEPHANIE, S CHEDIWY,

B IC, et al. “The Triangle Processor and Normal Vector Shader: A VLSI
System for High Performance Graphics”. SIGGRAPH Comput. Graph.
22.4 (June 1988), 21–30. ISSN: 0097-8930. DOI: 10.1145/378456.
378468 2.
[FHL*18] FASCIONE, L UCA, H ANIKA, J OHANNES, L EONE, M ARK, et
al. “Manuka: A Batch-Shading Architecture for Spectral Path Tracing in
Movie Production”. ACM Trans. Graph. 37.3 (Aug. 2018). ISSN: 0730-
0301. DOI: 10.1145/3182161. URL: https://doi.org/10.
1145/3182161 2.
[HAO05] H ASSELGREN, J ON, A KENINE -M ÖLLER, T OMAS, and O HLS -
SON , L. “Conservative Rasterization”. GPU Gems 2. Ed. by P HARR ,
M ATT. 2005, 677–690. URL: https : / / developer . nvidia .
com/gpugems/GPUGems2/gpugems2_chapter42.html 4.
[HY16] H ILLESLAND, K. E. and YANG, J. C. “Texel Shading”. Pro-
ceedings of the 37th Annual Conference of the European Association
for Computer Graphics: Short Papers. EG ’16. Lisbon, Portugal: Euro-
graphics Association, 2016, 73–76 3, 4.
[LD12] L IKTOR, G ÁBOR and DACHSBACHER, C ARSTEN. “Decoupled
Deferred Shading for Hardware Rasterization”. Proceedings of the ACM
SIGGRAPH Symposium on Interactive 3D Graphics and Games. I3D
’12. Costa Mesa, California: Association for Computing Machinery,
2012, 143–150. ISBN: 9781450311946. DOI: 10 . 1145 / 2159616 .
2159640 2.
[MVD*18] M UELLER, J OERG H., VOGLREITER, P HILIP, D OKTER,
M ARK, et al. “Shading Atlas Streaming”. ACM Trans. Graph. 37.6 (Dec.
2018). ISSN: 0730-0301. DOI: 10.1145/3272127.3275087. URL:
https://doi.org/10.1145/3272127.3275087 3.
[OB10] O LANO, M ARC and BAKER, DAN. “LEAN Mapping”. Proceed-
ings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graph-
ics and Games. I3D ’10. Washington, D.C.: Association for Computing
Machinery, 2010, 181–188. ISBN: 9781605589398. DOI: 10 . 1145 /
1730804.1730834 9.
[RLC*11] R AGAN -K ELLEY, J ONATHAN, L EHTINEN, JAAKKO, C HEN,
J IAWEN, et al. “Decoupled Sampling for Graphics Pipelines”. ACM
Trans. Graph. 30.3 (May 2011). ISSN: 0730-0301. DOI: 10 . 1145 /
1966394.1966396 2.

© 2022 The Author(s)

Eurographics Proceedings © 2022 The Eurographics Association.