Tech on zhangdoa

Rendering analysis - Cyberpunk 2077

zhangandhang@gmail.com (Hang Zhang) — Sat, 12 Dec 2020 10:30:00 +0000

(Spoilers-free post!)

Quite a lot of people I know inside or outside this industry are playing the long-waited “gargantuan maelstrom” now. After a couple few hours immersed around the stunning view (and bugs of course) in the Night City, I was wondering how one frame is rendered in Cyberpunk 2077 (every graphic programmer’s instincts!). So I opened RenderDoc and PIX, luckily REDengine didn’t behave unfriendly like some other triple-that (yes Watch Dogs 2 I mean you), I got a frame capture without any problems. (Disclaimer: I’m pretty sure later others like Alain Galvan, Anton Schreiner, Adrian Courrèges or even guys from CDPR would bring us a more detailed and accurate demonstration about how things works, I’m just chilling out around and welcome for any discussions and corrections! )

Overview

I’m running the game in Ultra configs in 1080p without ray-tracing and DLSS enabled, for the sake of my years “old” GTX1070
Two captured final results, one is around the main character’s apartment at day (let’s name it main street) and another one is around some river bank at night (Nobody dislikes eye candy!):
Some pictures’ color is manually clamped to easier visualize
It’s a DirectX 12-only game on Windows (if they started the development around 2013 or slightly earlier, I’m curious that how much effort it costs to upgrade the engine)
There are 2 Descriptor Heaps allocated just, one for half million CBV-SRV-UAVs and another for 2k~ Samplers

There is one Graphic queue, one Compute queue, and one Copy queue, the GPU work balance is quite even
ID3D12GraphicsCommandList::CopyBufferRegion is frequently called between passes to copy RT results
Quite a lot D3D12_RESOURCE_FLAGS::ALLOW_UNORDERED_ACCESS removal recommendation was thrown out by PIX, not sure if it’s really just my capture occasionally didn’t use some resources or, they do need some optimizations (I had experiences that unordered access was quite hurting sometimes)
Frame Timeline: Intensive at some early geometry stages, then deferred works kick in

Pre-geometry passes

Billboard and GUI

The entire frame starts by preparing the resources for the in-game billboards and screens
First few copy operations executed on 3 textures: The destination textures’ format is DXGI_FORMAT_R8_UNORM, the first is 480x272 and another two are at half size, the source texture should be a mega texture which is the destination of lots later copy operations. (Question: what’s the purpose for these copies? In the night scene it doesn’t have these operations, is it related to streaming?)
One of the three very first copied textures:
Next pass they are used as the shader input:
Billboards are drawn to one texture where all mips tightly packaged together
Billboards:
Then the elements for the cyberish-GUI are drawn and also copied to the mega texture
Part of the GUI elements:

Sky visibility/top-view shadow/mini map

The RT of the next pass is a single D16_UNORM 2k texture, around 400~ instanced draw calls executed in the captured scene (not sure how this RT later used)

Shadowmap from top:

Unknown compute pass 01

There are 2 dispatches over a 64x64x64 R16G16B16A16_FLOAT 3D texture in this pass (looks like voxelized normal/position?). The thread group count at the first time is 2x2x64, the second time it’s 16x16x16 (the RT is used later multiple times in terrain pass, ocean wave pass, depth-pre pass, and base material pass, definitely something crucial for geometry information)
Some slices of the 3D Normal:

video: title: "04_3DNormal_Slice": /attachments/04_3DNormal_Slice.mp4

Unknown compute pass 02

3 dispatches made up by 63x1x1, 3x1x1 then again 63x1x1 thread groups (in the night scene capture it’s 38x1x1, 3x1x1 and 38x1x1), couple-few ByteAddressBuffer and StructuredBuffer are bound as UAV (all of them are used later in the base material pass)

Terrain

1 R8G8B8A8_UNORM 2 slices 2D texture array and 1 R16_UINT 2D texture are drawn in 1k resolution (looks like they are chunks of terrain)
2 draw calls and RTs:

video: title: "05_Terrain": /attachments/05_Terrain.mp4

Unknown color pass 01

2 draw calls issued (only at the main street capture) but nothing is drawn due to the clip (and the meshes are barely comprehensible, looks like they are 2 levels of a LOD-ed mesh), RT is 1 256x256 R16_UINT 2D texture

Ocean wave

The top-view ocean map mask and a generated wave noise texture are bound as input, RT format also changes to R16_UNORM, no tessellation shader stage, some regular grid plate meshes are used to directly draw different chunks
Ocean wave noise in R16G16B16A16_FLOAT and mask in BC7_SRGB:

video: title: "06_OceanMask": /attachments/06_OceanMask.mp4:

Geometry passes

Depth-pre pass

The RT is in D32_S8_TYPELESS format, surely it would help to optimize later pixel-heavy works
Depth-pre pass result:

Base material passes

All vertex buffers uploaded to GPU memory are packed as SOA:

// VB0
nshort4 POSITION0;
// VB1
half2 TEXCOORD0;
// VB2
xint NORMAL0;
xint TANGENT0;
// VB3
unbyte4 COLOR0;
half2 TEXCOORD1;
// VB7
float4 INSTANCE_TRANSFORM0;
float4 INSTANCE_TRANSFORM1;
float4 INSTANCE_TRANSFORM2;

In pixel shader sometimes an R16G16B16A16_UNORM 64x64 noise texture is bound as input, only RG channels contain data, similar to what we’d use in SSAO for random normal rotation or offset
All material textures are accessed through the descriptor table
RT0 in R10G10B10A2_UNORM is the albedo, alpha channel is used as object mask, animated meshes are marked as 0x03
RT1 in R10G10B10A2_UNORM is world-space normal packaged by offset $0.5x + 0.5$
RT2 in R8G8B8A8_UNORM is metallic-roughness and other attributes, animated meshes are not rendered here, the alpha channel should be the transparent or emissive property for emissive material objects
RT3 in R16G16_FLOAT is Screen-space motion vector:
Depth-stencil is D32_FLOAT_S8X24_UINT format, stencil bit 0x15 is used to mark character body meshes, 0x35 for face, 0x95 for hair and 0xA0 is for trees, brushes, and other foliage
Drawing order:
1. Static Mesh
2. Ground and emissive objects
3. Animated objects and destructible (?)
4. Foliage
5. Decals
6. Mask for fur
7. Fur

video: title: "How the base material passes are drawn on on of the main street": /attachments/08_BaseMaterialPass_MainStreet.mp4 video: title: "How the base material passes are drawn near a river at night": /attachments/08_BaseMaterialPass_RiverBank.mp4 video: title: "The MRT": /attachments/08_BaseMaterialPass_RiverBank_MRT.mp4

DS convert passes

The D32S8_TYPELESS Depth-stencil buffer is converted to R32_TYPELESS in full-screen size and then downsampled to half size as R16_FLOAT and R8_UINT in 3 passes, for easier pixel sampling later.

Motion-stencil pass

All moving objects and foliage and furs are marked with some different bits in this pass regarding with last frame screen-space positions, and then further extended for few pixels (Exactly the same solution as Temporal Antialiasing in Uncharted 4: Better anti-ghost!)
Motion-stencil pass:

Mask and LUT passes

Reflection mask

Parts of some objects are drawn onto an R8_UNORM RT and used later in SSR and TAA passes

Compute pass to noise normal

The RT is used for sun shadow, AO(?), direct sunlight, SSR and indirect skylight passes

Noise sample:

Some passes to mask the sky out

All of them are in quite low resolution and mipmaped, the mipmap calculation executes multiple times

video: title: "13_SkyMaskMipmap": /attachments/13_SkyMaskMipmap.mp4

And finally, there is a graphic pass to draw a quite accurate sky mask on R32_FLOAT:

(HB)AO (?)

All in 960x540, looks like some sort of AO-required normal data:

video: title: "14_AO": /attachments/14_AO.mp4

Color-grading LUT

Color-grading LUT:

video: title: "15_ColorGradingLUT": /attachments/15_ColorGradingLUT.mp4

Ocean wave noise

Around 22~ 32x32x1 compute works are dispatched to generate noise for water, which is used in previous ocean wave pass

Light and shadow passes

CSM passes for direct light

A 2k R16_TYPELESS texture is used first to render a few simple meshes (some kinds of mesh proxy?), but the RT is never used
Then 4 classic CSM cascades are rendered and stored as 4 slices in a 4k R16_TYPELESS 2D texture array, no geometry shader RT index is involved, it just executes draw calls 4 times on each cascade

video: title: "16_CSM": /attachments/16_CSM.mp4

Omni shadow maps for point/area lights

Every omni shadow map is first rendered as R32_TYPELESS in 1k, then converted to 10 slices R16G16UNORM (should be VSM?)

Cloud distribution

Sky cubemaps

7 mipmap levels in R11G11B10_FLOAT format:

video: title: "18_SkyCubemap": /attachments/18_SkyCubemap.mp4

And a stereographic projected sky radiance is also generated:

Coat mask (?)

Clustered light index (?)

Some compute-only passes generate a few 3D textures that contain only indices:

video: title: "20_ClusteredIndices": /attachments/20_ClusteredIndices.mp4

Direct light shadow

In full screen resolution, it’s coming from sun at the day and moon at the night

Local lights mask (?)

In a 120x68 60x34 resolution

Unconfirmed compute dispatches to update some StructuredBuffers

Environment Radiance capture

First, cubemap version is generated video: title: "23_EnvironmentCaptureCubemap": /attachments/23_EnvironmentCaptureCubemap.mp4
Then all of them are converted to 2D dual-paraboloid textures in 512x256, each one contains 6 mips, and there are 32 captures stored as slices in one 2D texture array, 6~ texture arrays in the main street current scene video: title: "23_EnvironmentDualParaboloid": /attachments/23_EnvironmentDualParaboloid.mp4

64x64x64 texture that is R32G32_UINT format, 2 channels contain index-like data
Slice 19 for example

Landscape maps

A few layers drawn here, from normal to a kind of heatmap in 1k resolution

Local volumetric fog

2 mips from 240x136 to 120x68, each one has 128 slices and mip 0 is from main camera view, it should be some kind of ray-marching:

video: title: "26_LocalVolumetricFog": /attachments/26_LocalVolumetricFog.mp4

Head mask

The most hilarious one, should be used later to render detailed facial expressions

Direct and emissive lights

Tiled-lighting, each tile is 16x16 by size and all executed as indirect compute command:

video: title: "28_DirectAndEmissiveLight": /attachments/28_DirectAndEmissiveLight.mp4

Sky

Only masked sky region is drawn

SSR

A low resolution depth mask is drawn first:
Then the full-screen size SSR is drawn in R11B11G10_FLOAT with another R8_UNORM RT for reflectivity. The previously noised world-space normal and motion stencil RT is used as inputs, also the last frame’s downscaled TAA output is used for sampling:

video: title: "31_SSR": /attachments/31_SSR.mp4

Indirect light

Could see some artifacts (Light leaking because of VCT?):

All lights composition pass

2 RTs, specular reflections are stored on RT1:

Transparent LUT

Compute shader is involved a few times to generate some LUTs for later transparent and holographic objects:

Skin and eyes

Skin irradiance is rendered in screen space:
Then eyes are rendered:

video: title: "36_Eyes": /attachments/36_Eyes.mp4

AO(?)/GI shadow

Add additional shadows to pixels that should not be lighted too much: video: title: "37_AO": /attachments/37_AO.mp4

Volumetric fog

Just a simple image composition pass:

video: title: "38_VolumetricFog": /attachments/38_VolumetricFog.mp4

Water reflection

Few passes generate the water reflection, including all related masks and noises:

video: title: "39_Water": /attachments/39_Water.mp4

Post-processing passes

TAA

Next it’s a full-screen no-suprise TAA pass:

video: title: "40_TAA": /attachments/40_TAA.mp4

Blur

First, the TAA-ed result is downscaled to 1/2, 1/4, and 1/8 size in 1 compute pass and stored in 3 mips, then each of the mips is blurred horizontally and vertically

video: title: "41_Blur": /attachments/41_Blur.mp4

Unknown transparent

Cloud

The distribution noise of the cloud is pre-generated:

video: title: "42_Cloud": /attachments/42_Cloud.mp4

Holo and transparent objects

All futurism holographic objects are drawn next (a significant number of small triangle count meshes), including water surface and some local light scattering:

video: title: "43_Transparent": /attachments/43_Transparent.mp4

Post-TAA

2 downscaled images are generated again in 1/2 and 1/4 sizes with all transparent objects on them, and finally, it’s converted back to full-screen size and gets sharpened

video: title: "44_PostTAA": /attachments/44_PostTAA.mp4

HDR LUT(?)

A 256x256 R16_FLOAT, a 256x64 R16G16B16A16_FLOAT and a 256x1 R16G16B16A16_FLOAT images are generated by 3 compute dispatches, and finally are used as part of the inputs for the next compute pass to generate a structured buffer, which is used widely when rendering sky cubemaps and reflection probes

Bloom

The full-screen images are half-sized 6 times and get bloomed

video: title: "46_Bloom": /attachments/46_Bloom.mp4

Camera lens effects

Rendered on a half-screen size RT

Color grading

The color-graded image is then converted to LDR

Gamma correction

The Gamma correction is executed in a weird resolution 456x256

GUI elements

All GUI elements are drawn on a full HD RT and the corresponding mipmaps are generated, and then a few compute passes add bloom to them

Film Grain

The noise LUT is generated on-the-fly

Swap chain image

Finally, the result of the all above is presented to the screen

Undrawnable conclusions

Well, what else I could say, 2 decades have passed already in the 21st century, Cyberpunk 2077 basically is the summing of the modern game rendering techniques which progressed during the years. Even without the enabling of ray-tracing, the overall appearance of the graphics are magnificent, it just bursts my euphoria when I’m pacing around Night City. Despite all the controversies over the release and gameplay glitches which mainly caused by not robust enough physics system and overestimated streaming implementation, the game’s rendering is quite well-crafted, but still, I expected to see more state-of-the-art architectural practices such as a heavier GPU-driven rendering pipeline (which’s been more and more popular since 2016-2017). How’s your opinion about it? And what could change if we come back 5 years later to have a review? Maybe we’d laugh at the “outdated” techniques in Cyberpunk 2077, but I’m sure we’ll still keep admitting that real-time rendering is always full of exciting directions to explore!

How expensive it is to create your own AAA-level game audio solution?

zhangandhang@gmail.com (Hang Zhang) — Wed, 06 May 2020 01:15:00 +0000

Don’t be greedy, unless it’s really cheap

I’ve been in the game industry for almost 5 years since I broke up with traditional music post-production, and the official direction which I spend every 8*5 (of course, sometimes more) hours a week and get paid is not surprisingly, game audio. It doesn’t matter how you’re familiar with this domain, more or less you’d hear about some names like FMOD or Wwise recent years, even you’re just coming from a typical gamer’s point of view. Nowadays anybody who struggles to pick up a sound acceleration card would be called a perfectionist, a rational mind would just spend some hundreds or thousands whatever-your-local-currency-it-is to get some decent speakers or headphones, and that’s ALL for gaming.

The game audio tech is quite saturated, the hardware capability for game audio is quite saturated. It’s not so tempting for the entire industry to figure out something really rejuvenate, the VBAP algorithm is good enough, the traditional reverb is good enough, the shoe box spatialization model is good enough, the geometry-based sound propagation model is good enough, nobody would spend some expensive runtime resources to resolve a 1% more accurate sound scenario. But I’d make a clear clarification, I’m not pessimistic about this, it’s again an economic problem, the nature of the game is the art and entertainment on real-time simulation software, and all we needed is just something good enough to please ourselves. The techs in current game audio are really suitable for almost all the current games, whether you’re playing GTA V in 2013 or 2020, your CPU would hardly complain about the audio tasks from it, but you’d keep trying to augment the graphics with mods and purchasing new graphics cards because the holy grail of real-time rendering is still there, it’s not good enough yet!

So almost all the progresses you’d see among the vendors and studios recent years about game audio are typically related to working pipelines, toolchains and design, sometimes a voxel-based or wave simulation solution would pop up, but maybe they are not eye-catching enough or not robust enough or whatever reasons, the audio community didn’t react actively about them. So I accumulated a question from all those what I saw and experienced so far, what is the core unsolved problem in the game audio?

In some years of SIGGRAPH there is a course called Open Problems in Real-Time Rendering which covers the unsolved problems in real-time rendering topic. I’d think often that, how many open problems in the game audio we still have now? The sound acoustic propagation would be one, anyway we didn’t find an acceptable enough general one-for-all answer yet. Also, maybe the sound replay mechanism or so-called 3D sound would be another one? A one-dimensional PCM wave sample file transferred into a stereo channel headphones, think about it how many compromises we’d given compare with how sound wave travel into our ears in real life! Despite those physical-related problems, is it really comfortable and easy for audio artists to design and implement something for a game nowadays? Every audio software and middlewares would bring you their philosophy of working and force you to adopt into their models, and their features growing constantly every release. Too much trivial knowledge we’d have to learn!

What’s the matter?

There is a famous quote “If Your Only Tool Is a Hammer Then Every Problem Looks Like a Nail”, I’d plagiarism and modify it a little to “If you have a problem and a Swiss Army knife then you’ll never find a way to solve it” for some of my experiences with some software. I’m not criticizing how many unused abilities they provided for us to solve problems, but about how messy those abilities are delivered to the user. I always need just 10%-20% functionalities of them for my problems, another friend of mine needs another 10%, and I have some friends occasionally use them too, so finally the software dev released a 150% feature complex version. Yes, I’m not only targeting Photoshop or Word or Unreal Engine or Wwise, but I’m also talking about all these frustrate experiences when every time I’d spend equal time to search documentations and solve my problems. But thanks all gods we have Unix philosophy, at least some genius invention half-century ago which finally save me from totally being a slave of click-and-drag. I’d list one version of them again because I like these principles so much, it’s even not limited to software developments:

Make it easy to write, test, and run programs.

Interactive use instead of batch processing.

Economy and elegance of design due to size constraints (“salvation through suffering”).

Self-supporting system: all Unix software is maintained under Unix.

Gosh, look at what we are doing every day now! I’d always remember the feeling when I installed Chocolatey on Windows 10 at my home computer and started to just choco install cmake, if you never used yum or apt-get (or brew on macOS) you’d never understand how free it feels like when you have a package manager for all your software on the computer. The problem here is “why I always have to download software installer by myself?” and the solution is “find/build a package manager”.

So now let’s take a look at our poor game audio working pipelines. Audio designer exports some .wav assets from some colorful but messy DAWs, and import them into the editor of the middlewares, and then export them again as the middleware’s asset format, and then import them again into the editor of the game, and then again maybe need to package them as a tight pack for game’s release. Even you’d automate this procedure, still, you’re wasting time and complexing the situation. But wait, what we actually need to do in order to produce audio for a game?:

graph TD
	subgraph Game
 game((.exe/.bin))
	style game fill:#8cb8ff,stroke:#000,stroke-width:4px;
	end

	subgraph Asset
 wav[.wav]-->game
	end

 subgraph Code
 logic[.cpp]-->game
	end

This is an over-simplified version, but what else? The designer produces assets and programmer produces codes, naturally, it has been working like this from the 90s in the game industry. But the borderline between the assets and codes keeps changing during the years, the designer wants something programmable or configurable but in a higher level and in a visual-friendly way, and the programmer doesn’t want so many low-level details, and then the programmable-asset framework became a trend:

graph TD
	subgraph Game
 game((.exe/.bin))
	style game fill:#8cb8ff,stroke:#000,stroke-width:4px;
	end

	subgraph Metadata
 metadata{scripts and descriptions}-->game
	end

	subgraph Asset
 wav[.wav]-->game
	wav-->metadata
	end

 subgraph Code
	logic-->metadata
 logic[.cpp]-->game
	end

Thanks to this trend, game audio middleware solutions like FMOD and Wwise become more and more popular. But they are not simple enough if you remember what I’ve complained about above, you have to compromise some of your freedom in working flow to gain the benefit of another kind of freedom, that you finally could get rid of the low-level details and leave them to the middleware to deal with. But is it really so scary to get our hands dirty with so-called “low-level details” nowadays?

During the research (search actually at the most of the time) about this problem, I found a nice presentation The Next-Gen Dynamic Sound System of Killzone Shadow Fall in GDC 2014 by speakers from Guerrilla Games which answered my question and convinced me about another type of possibilities, that it’s not a behemoth level of task to fully get control back of the audio in your game. Killzone: Shadow Fall is a successful 1st-party FPS game released on PS4 7 years ago, if a 200+ employee (worldwide top-level) studio could afford to develop their own game audio solutions for a 3A game in practice, it simply means that it’s not always an impossible task for other kinds of devs to do the same, even you’ve considered the time and energy cost, it’s maybe not cheaper than a license fee of middleware, but it’s also not more expensive than keeping thinking that the “low-level details” is untouchable. So I decided to recreate their solution by myself. 2 hands + 1 brain and IDE, let’s take a review about how far I have been going until now.

The architecture

The ideal situation about this solution is, there would be a minimal working flow barriers between audio assets, codes for audio, and the game itself. So if we try to follow the architecture what they demonstrated, then we need at least such few modules:

I would try to add as more dependencies on the open-source projects as I could to implement each module, because I want to keep the code iteration speed fast to see the actual results (a little bit sarcastic actually), I’ll leave those businesses which are not directly related to my problems to them. And the final winner for every module are:

Graph Editor: imgui and imgui-node-editor. I’ve had some nice experiences in the past with Dear ImGui, it is quite stable and easy-to-use, the API is super intuitive and it’s lightweight enough if compared with some other GUI libs like Qt. And imgui-node-editor is a project built over imgui (thanks thedmd for hard work on it!) which provide facilities to build node editor easier than from scratch.
Graph Compiler: I decided to generate C++ source code from the node graph by myself, and then build it with CMake and the usually used build toolchains like MSVC.
I/O: I’ve implemented a small WAV loader and parser module before, so I’d keep extending it as the audio asset I/O. For all the other metadata like node graph and node descriptions, it’s obviously we need something human-readable and easy to serialize. So far nlohmann’s json is the most handsome JSON lib for C++ I’ve used, and why don’t we choose it?
Audio Player: The same approach to choose a graphics API, because I don’t want to stick with a platform or OS so tightly, and I prefer something like bgfx in rendering to wrap all the platform differences but still leave me the actual platform-irrelevant low-level details, after a brief gambling research, I found miniaudio (I like single header C lib, because you could just blame it when something not working:)), it’s exactly what I imagined about, an opaque wrapper layer over the OS and audio backends.
Plug-in Hook: I’d like to build some WYSIWYG that the user could hear the result immediately as soon as they change the node graph. It’s the first time I got touch with dynamic library hot-reload topics, my lack of deep knowledge about linking didn’t bring so many barriers, since I found cr, it would automatically manage all the underlying vtable relocation and scope related things for you, and the API is really concise like the document said, only 3 lines of codes.

I didn’t try to make the decision before writing the actual code, all the architecture related topics were progressed during the development, and there were several times of significant refactoring, so if you would like to choose different approaches it’s totally possible.

The implementation

It’s always easy to draw a beautiful architecture picture than successfully build the project, there are still lots of trivial problematic nuances in the details when I implemented this project, but luckily it looks the code quality is convergence nowadays, so I think maybe I didn’t overestimate the challenge at the beginning.

API style

Functional programming is all about calculation, feeding inputs to function and getting outputs, staying far from the global scope and reducing all state changes, and finally, you could go to the pub rather than keep working on bugs on Friday night. But sadly it’s not always easy to do so, it’s not very performance-critic about an audio lib here, so I would like to maintain some state machines, and all the API would return the execution state result like a typical Windows HRESULT:

namespace Waveless
{
	enum class WsResult
	{
		NotImplemented,
		Success,
		Fail,
		NotCompatible,
		IDNotFound,
		FileNotFound
	};
}

And any modifications over the input of function should be applied on pointers:

using namespace Waveless;
WsResult foo(float* bar, float foobar)
{
	if(bar == nullptr)
	{
		return WsResult::Fail;
	}

	if(*bar > 512.0f)
	{
		*bar += foobar;
	}
	else
	{
		*bar -= foobar;
	}
	return WsResult::Success;
}

The lowest low-level details

All software runs on hardware, we need logger to print log, we need a timer to check the time, we need memory manager to play with heap memory, we need some not so dumb and naive math functions… Actually I could choose some open-source facilities libs again, but since I’ve implemented some of them from another project then I’d directly reuse them here. After all, they are not something quite complex at the beginning if you are happy with STL. Let’s take a look at an example:

namespace Waveless
{
	enum class TimeUnit { Microsecond, Millisecond, Second, Minute, Hour, Day, Month, Year };

	struct Timestamp
	{
		uint32_t Year;
		uint32_t Month;
		uint32_t Day;
		uint32_t Hour;
		uint32_t Minute;
		uint32_t Second;
		uint32_t Millisecond;
		uint32_t Microsecond;
	};

	class Timer
	{
	public:
		static WsResult Initialize();
		static WsResult Terminate();

		static const uint64_t GetCurrentTimeFromEpoch(TimeUnit time_unit);
		static const Timestamp GetCurrentTime(uint32_t time_zone_adjustment);
	};
}

The Object

I’m against the irresponsible OOP mindset, so I won’t choose to simply encapsulate behaviors and data together. But still, it’s necessary to have some very common properties among all the “object"s, and then:

namespace Waveless
{
	enum class ObjectState
	{
		Created,
		Activated,
		Terminated
	};

	struct Object
	{
		uint64_t UUID;
		ObjectState objectState = ObjectState::Created;
	};
}

Depends on how maniac you are about ECS, you could inherit from Object structure or just insert it as a field to other types (or rename it to ObjectMetadataComponent or TheMotherOfAllComponent?). I’m okay with harmless inheritance.

WAV instance

There are too many reversions of WAV standard during the years, if you want to parse them by yourself I’d like to say it’s not a so wisdom choice. Anyway, I’ve done such meaningless thing and my WAV loader class supports almost all types of WAV now, from the standard 44 bytes header WAV to BWF and RF64 (Recommendations for sound designer and musicians: don’t use strange DAW). And the WAV instance simply contains the header and the pointer to the sample array on heap. About the parsing details you could take a look at the actual source file.

	struct WavHeader
	{
		RIFFChunk RIFFChunk;
		JunkChunk JunkChunk;
		fmtChunk fmtChunk;
		factChunk factChunk;
		bextChunk bextChunk;
		dataChunk dataChunk;
		int ChunkValidities[6] = { 0 };
	};

	struct WavObject
	{
		WavHeader header;
		char* samples;
		int count;
	};

Audio Player

My previous experience with DSP audio programming is all quite “academic”. When you’re just expecting numpy to give you a correct small array everything looks like from fairy tales, while miniaudio is a C library, and now I’m going to deal with real audio samples from music and sound effects. The audio rendering is quite similar to the graphics rendering, you have a rendering frame for a certain amount of time between each two final rendering result presentations, and you then could process the audio buffer during this period by your desired DSP algorithms. But for graphics rendering all of these happens on GPU and audio rendering typically executed on CPU (we won’t touch the hardware audio chips), so it’s actually easier to manage the synchronization since we only have to take care of one device now.

The audio buffer is just a bunch of raw memory chunks, the underlying data type is from 16-bits integer to 64-bits double float, depends on how precise you’d like to maintain when you’re performing DSP. The size of the buffer shouldn’t be too large because we have a more strict time restriction now, we must finish all the processes on the current buffer before the last frame’s buffer finishes playing on the playback devices, otherwise, we’ll hear discontinuities. A common song lasts around 4-5 minutes, if we pour all the samples of it into the audio buffer then of course, we need to wait quite a long time to hear it if the DSP algorithms are expensive. A better choice is every frame we just consume a little bit data from the entire samples, and discard it after playing:

The functionality of buffer processing in miniaudio is provided through a callback mechanism, we need to initialize the audio device and bind our callback function, and then just start the device and it will call our callback every rendering frame. Finally, we’ll perform our DSP algorithms and write to the output audio buffer in our callback function. I wrapped miniaudio into an audio engine class:

// ignore other details
	void data_callback(ma_device* pDevice, void* pOutput, const void* pInput, ma_uint32 frameCount);

	ma_device_config deviceConfig;
	ma_device device;
	ma_decoder_config deviceDecoderConfig;
	ma_event terminateEvent;

	WsResult AudioEngine::Initialize()
	{
		deviceDecoderConfig = ma_decoder_config_init(ma_format_f32, 2, MA_SAMPLE_RATE_44100);

		deviceConfig = ma_device_config_init(ma_device_type_playback);
		deviceConfig.playback.format = deviceDecoderConfig.format;
		deviceConfig.playback.channels = deviceDecoderConfig.channels;
		deviceConfig.sampleRate = deviceDecoderConfig.sampleRate;
		deviceConfig.dataCallback = data_callback;
		deviceConfig.pUserData = nullptr;

		if (ma_device_init(NULL, &deviceConfig, &device) != MA_SUCCESS)
		{
			Logger::Log(LogLevel::Error, "Failed to open playback device.");
			Terminate();
			return WsResult::Fail;
		}

		ma_event_init(device.pContext, &terminateEvent);

		if (ma_device_start(&device) != MA_SUCCESS)
		{
			Logger::Log(LogLevel::Error, "Failed to start playback device.");
			Terminate();
			return WsResult::Fail;
		}

		return WsResult::Success;
	}

As you could see here in the callback function there is a parameter called frameCount, the frame here means sample frame, and one sample frame contains the number of channels of samples if the audio file is stored interleaved:

Since I’m writing this project for game audio, typically a sound effect would have multiple instances playing at the same time, so it’s necessary to distinguish between the asset data and the instance data. We would not mangle with the asset data but would have as many instances as we want, so now we could add 2 more new types:

	struct PlayableObject : public Object
	{
		ma_decoder_config decoderConfig;
	};

	struct EventPrototype : public PlayableObject
	{
		WavObject* wavObject;
	};

	struct EventInstance : public PlayableObject
	{
		ma_decoder decoder;
		ma_event stopEvent;
	};

The commonly used name for sound effect instances in the game audio community is voice, I didn’t choose it because for me it sounds a little bit vague. I chose to describe the asset and instance data as “event” instead since I’d like to indicate more about “what happened” than “a sound is playing” (also I’m still a user of FMOD and Wwise, but my version of “event” is slightly different from theirs).

And the overall audio engine would look like this:

And finally let’s take a look at the core of the audio engine - data callback functions:

	void data_callback(ma_device* pDevice, void* pOutput, const void* pInput, ma_uint32 frameCount)
	{
		for (auto i : g_eventInstances)
		{
			if (i.second->objectState == ObjectState::Activated)
			{
				auto l_frame = read_and_mix_pcm_frames(i.second, reinterpret_cast<float*>(pOutput), frameCount);

				if (l_frame < frameCount)
				{
					i.second->objectState = ObjectState::Terminated;
					ma_event_signal(&i.second->stopEvent);
				}
			}
		}

		(void)pInput;
	}

It will iterate over all currently activated event instances and mix them onto the output buffer. If the returned frame count of the event instance is smaller than the current global frame count, then it simply means that we’ve reached the end of the instance’s total sample, and then we signal a stop event for further more cleanup operations.

I copy and modify the submixing function from the example of miniaudio, to fit in my current design:

	ma_uint32 read_and_mix_pcm_frames(EventInstance* eventInstance, float* pOutput, ma_uint32 frameCount)
	{
		float temp[sizeOfTempBuffer];

		ma_uint32 tempCapInFrames = ma_countof(temp) / eventInstance->decoderConfig.channels;
		ma_uint32 totalFramesRead = 0;

		while (totalFramesRead < frameCount)
		{
			ma_uint32 iSample;
			ma_uint32 framesReadThisIteration;
			ma_uint32 totalFramesRemaining = frameCount - totalFramesRead;
			ma_uint32 framesToReadThisIteration = tempCapInFrames;

			if (framesToReadThisIteration > totalFramesRemaining)
			{
				framesToReadThisIteration = totalFramesRemaining;
			}

			// Decode and apply filters
			framesReadThisIteration = ma_decoder_read_pcm_frames(&eventInstance->decoder, temp, framesToReadThisIteration);

			if (framesReadThisIteration == 0)
			{
				break;
			}

			if (eventInstance->cutOffFreqLPF != 0.0f)
			{
				low_pass_filter(deviceDecoderConfig.channels, eventInstance->cutOffFreqLPF, deviceDecoderConfig.sampleRate, eventInstance->sampleStateLPF, temp, framesToReadThisIteration);
			}
			if (eventInstance->cutOffFreqHPF != 0.0f)
			{
				high_pass_filter(deviceDecoderConfig.channels, eventInstance->cutOffFreqHPF, deviceDecoderConfig.sampleRate, eventInstance->sampleStateHPF, temp, framesToReadThisIteration);
			}

			/* Mix the frames together. */
			for (iSample = 0; iSample < framesReadThisIteration * deviceDecoderConfig.channels; ++iSample)
			{
				pOutput[totalFramesRead * deviceDecoderConfig.channels + iSample] += temp[iSample];
			}

			totalFramesRead += framesReadThisIteration;

			if (framesReadThisIteration < framesToReadThisIteration)
			{
				break; /* Reached EOF. */
			}
		}

		return totalFramesRead;
	}

It looks a little bit scary, but the actual algorithm is simple, we use a temporary stack buffer to decode the sample of the event instance, and add it onto the output buffer. Since we don’t know how many samples frames the ma_decoder_read_pcm_frames would return, so we keep decoding until we reach the maximum frame count in global. And I add my version of LPF and HPF here to evaluate the possibility of mixing chain, actually it works! But I’d like to revisit this part later for a more flexible mixing architecture, it’s not enough to just have some filters for game audio.

Graph Editor

If you take a look at the excellent example of a blueprint editor from imgui-node-editor, you’d agree it’s quite elegant as Unreal Engine’s version. And that example is my start point, but there are too many refactoring needed for a truly extensible framework like what Killzone: Shadow Fall’s team did. So I designed a more redundant but decoupled architecture for the graph editor:

Basically it’s a variation of classic MVVM or MVP design patterns, the node, pin and link are all the core objects. When loading a canvas file, each model and pin instance would be created from the manager class (“factory”) by the descriptor instance (“prototype”), and then the corresponding widget instance would be created later by the widget rendering modules. I won’t show too many details about how to implement them since they are not the major problem here, you could take a look at the source files if you want to know something more. But let’s still catch a glimpse about how the actual object classes would be declared:

namespace Waveless
{
	enum class PinType
	{
		Flow,
		Bool,
		Int,
		Float,
		String,
		Vector,
		Object,
		Function,
		Delegate,
	};

	enum class PinKind
	{
		Output,
		Input
	};

	enum class NodeType
	{
		ConstVar,
		FlowFunc,
		AuxFunc,
		Comment
	};

	using PinValue = uint64_t;

	struct PinDescriptor : public Object
	{
		const char* Name;
		PinType Type;
		PinKind Kind;
		PinValue DefaultValue;
		int ParamIndex;
	};

	struct ParamMetadata
	{
		const char* Name;
		const char* Type;
		PinKind Kind;
	};

	struct FunctionMetadata
	{
		const char* Name;
		const char* Defi;
		int ParamsCount = 0;
		int ParamsIndexOffset = 0;
	};

	struct NodeDescriptor : public Object
	{
		const char* RelativePath;
		const char* Name;
		NodeType Type = NodeType::ConstVar;
		int InputPinCount = 0;
		int OutputPinCount = 0;
		int InputPinIndexOffset = 0;
		int OutputPinIndexOffset = 0;
		FunctionMetadata* FuncMetadata;
		int Size[2] = { 0 };
		int Color[4] = { 0 };
	};
}

The descriptor’s metadata file for the wave file loader node:

{
 "NodeType": 2,
 "Parameters": [
 {
 "Name": "FilePath",
 "PinType": "String",
 "PinKind": 1,
 "ParamIndex": 0
 },
 {
 "Name": "WaveObject",
 "PinType": "Object",
 "PinKind": 0,
 "ParamIndex": 1
 },
 {
 "Name": "EventPrototypeID",
 "PinType": "Int",
 "PinKind": 0,
 "DefaultValue": 0,
 "ParamIndex": 2
 }
 ],
 "Color": {
 "R": 64,
 "G": 64,
 "B": 255,
 "A": 255
 }
}

And the function definition:

void Execute(std::string& in_FilePath, WavObject& out_WaveObject, uint64_t& out_eventPrototypeID)
{
	out_WaveObject = WaveParser::LoadFile(in_FilePath.c_str());
	out_eventPrototypeID = AudioEngine::AddEventPrototype(out_WaveObject);
}

I limited the possible data type of pin connection since it’s just a graph editor, the user shouldn’t be bothered by how a 32-bits unsigned integer works differently from a 16-bits signed version. The reference from node descriptor to pin descriptors are a typical indirection design, we could use the global offset and count in the descriptor instance array to access them.

The model classes:

namespace Waveless
{
	struct NodeModel;

	struct PinModel : public Object
	{
		PinDescriptor* Desc;

		NodeModel* Owner = 0;
		const char* InstanceName = 0;
		PinValue Value;
	};

	enum class NodeConnectionState { Isolate, Connected };

	struct NodeModel : public Object
	{
		NodeDescriptor* Desc;

		NodeConnectionState ConnectionState = NodeConnectionState::Isolate;
		int InputPinCount = 0;
		int OutputPinCount = 0;
		int InputPinIndexOffset = 0;
		int OutputPinIndexOffset = 0;
		float InitialPosition[2];
	};

	enum class LinkType { Flow, Param };

	struct LinkModel : public Object
	{
		LinkType LinkType = LinkType::Flow;
		PinModel* StartPin = 0;
		PinModel* EndPin = 0;
	};
}

A uint64_t would be convenient to store any constant primitive value of the pin directly in the pin model, or a pointer to the actual value’s address. As you could see there are some duplicated fields, I would prefer to eliminate them but it’s still acceptable for editor-related assets.

Graph Compiler

For the sake of simplicity, I assume the user wouldn’t make any loops between nodes in the canvas, so the graph finally becomes a DAG structure. In order to generate the correct C++ source file, we need to sort the nodes by the execution flow, it would be simple if we mark links with different connection types. My sort algorithm is quite naive, I just brutally iterates over all the node models and store them based on the execution flow index, it didn’t hurt the performance so much until now, but I assume there would be a lot of more elegant and faster approaches:

struct NodeOrderInfo
{
	NodeModel* Model;
	uint64_t Index;
};

	//......
	// part of void GetNodeOrderInfos()
	auto l_startNode = m_StartNode;
	auto l_endNode = m_EndNode;

	auto l_currentNode = l_startNode;

	// Iterite over the execution flow
	while (l_currentNode)
	{
		// Parameter dependencies
		AddAuxFunctionNodes(l_currentNode, *l_links);

		NodeOrderInfo l_nodeOrderInfo;
		l_nodeOrderInfo.Model = l_currentNode;
		l_nodeOrderInfo.Index = m_CurrentIndex;
		m_NodeOrderInfos.emplace_back(l_nodeOrderInfo);

		// Next node
		auto l_linkIt = std::find_if(l_links->begin(), l_links->end(), [l_currentNode](LinkModel* val) {
			return (val->StartPin->Owner == l_currentNode) && (val->LinkType == LinkType::Flow);
		});

		if (l_linkIt != l_links->end())
		{
			auto l_link = *l_linkIt;
			l_currentNode = l_link->EndPin->Owner;
			m_CurrentIndex++;
		}
		else
		{
			l_currentNode = 0;
		}
	}
	//......

After getting all the node order information then it’s time to transform them into the actual C++ source file. It’s again a quite ugly and handcrafted stringstream, I’ll show some examples here:

void WriteExecutionFlows(std::vector<char>& TU)
{
	WriteStartNode(TU);

	for (auto node : m_NodeOrderInfos)
	{
		if (node.Model != m_StartNode)
		{
			if (node.Model->Desc->Type == NodeType::ConstVar)
			{
				WriteConstant(node.Model, TU);
			}
			else
			{
				WriteLocalVar(node.Model, TU);

				WriteFunctionInvocation(node.Model, TU);
			}
		}
	}
}

WsResult GenerateSourceFile(const char* inputFileName, const char* outputFileName)
{
	GetNodeOrderInfos();
	GetNodeDescriptors();

	std::vector<char> l_TU;

	WriteIncludes(l_TU);

	WriteFunctionDefinitions(l_TU);

	WriteScriptSignature(inputFileName, l_TU);

	std::string l_scriptBodyBegin = "\n{\n";
	std::copy(l_scriptBodyBegin.begin(), l_scriptBodyBegin.end(), std::back_inserter(l_TU));

	WriteExecutionFlows(l_TU);

	std::string l_scriptBodyEnd = "}";
	std::copy(l_scriptBodyEnd.begin(), l_scriptBodyEnd.end(), std::back_inserter(l_TU));

	auto l_outputPath = "..//..//Asset//Canvas//" + std::string(inputFileName) + ".cpp";

	if (IOService::saveFile(l_outputPath.c_str(), l_TU, IOService::IOMode::Text) != WsResult::Success)
	{
		return WsResult::Fail;
	}
	return WsResult::Success;
}

And finally, invoke the shell command to build CMake project and build the plugin:

WsResult BuildPlugin()
{
	auto l_cd = IOService::getWorkingDirectory();

	std::string l_command = "mkdir \"" + l_cd + "../../Build-Canvas\"";
	std::system(l_command.c_str());

#ifdef _DEBUG
	l_command = "cmake -DCMAKE_BUILD_TYPE=Debug -G \"Visual Studio 15 Win64\" -S ../../Asset/Canvas -B ../../Build-Canvas";
#else
	l_command = "cmake -DCMAKE_BUILD_TYPE=Release -G \"Visual Studio 15 Win64\" -S ../../Asset/Canvas -B ../../Build-Canvas";
#endif // _DEBUG

	std::system(l_command.c_str());

	l_command = "\"\"%VS2017INSTALLDIR%/MSBuild/15.0/Bin/msbuild.exe\" ../../Build-Canvas/Waveless-Canvas.sln\"";
	std::system(l_command.c_str());

#ifdef _DEBUG
	l_command = "xcopy /y ..\\..\\Build-Canvas\\Bin\\Debug\\* ..\\..\\Build\\Bin\\Debug\\";
#else
	l_command = "xcopy /y ..\\..\\Build-Canvas\\Bin\\Release\\* ..\\..\\Build\\Bin\\Release\\";
#endif // _DEBUG

	std::system(l_command.c_str());

	return WsResult::Success;
}

One of the test canvas:

And the generated header and source files:

#define WS_CANVAS_EXPORTS
#include "../../Source/Core/WsCanvasAPIExport.h"
#include "../../Source/Core/Math.h"
#include "../../Source/IO/WaveParser.h"
#include "../../Source/Runtime/AudioEngine.h"
using namespace Waveless;

#pragma pack(push, 1)
struct MinimalTest_InputData
{
	bool in_Start;
	bool in_Stop;
	Vector in_ListenerWorldPosition;
	Vector in_EmitterWorldPosition;
};
#pragma pack(pop)
WS_CANVAS_API void EventScript_MinimalTest(MinimalTest_InputData inputData);

#include "MinimalTest.h"
void Execute_WaveLoader(std::string& in_FilePath, WavObject& out_WaveObject, uint64_t& out_eventPrototypeID)
{
	out_WaveObject = WaveParser::LoadFile(in_FilePath.c_str());
	out_eventPrototypeID = AudioEngine::AddEventPrototype(out_WaveObject);
}

void Execute_MinimalTest_Output(uint64_t in_eventInstanceID)
{
}

void Execute_MinimalTest_Input(bool in_Start, bool in_Stop, Vector in_ListenerWorldPosition, Vector in_EmitterWorldPosition)
{
}

void Execute_WavePlayer(bool in_Start, bool in_Stop, uint64_t in_eventPrototypeID, float in_Gain, float in_LPFCutOff, float in_HPFCutOff, uint64_t& out_eventInstanceID)
{
	if (in_Start)
	{
		out_eventInstanceID = AudioEngine::Trigger(in_eventPrototypeID);
	}
	if (in_Gain != 0.0f)
	{
		AudioEngine::ApplyGain(out_eventInstanceID, in_Gain);
	}
	if (in_LPFCutOff != 20000.0f)
	{
		AudioEngine::ApplyLPF(out_eventInstanceID, in_LPFCutOff);
	}
	if (in_HPFCutOff != 0.0f)
	{
		AudioEngine::ApplyHPF(out_eventInstanceID, in_HPFCutOff);
	}
}

void Execute_Attenuator(Vector& in_ListenerWorldPosition, Vector& in_EmitterWorldPosition, float& out_gain)
{
	auto l_distanceFromListener = (in_ListenerWorldPosition - in_EmitterWorldPosition).Length();
	if (l_distanceFromListener)
	{
		out_gain = Math::Linear2dBAmp(1.0f / (l_distanceFromListener * l_distanceFromListener));
	}
	else
	{
		out_gain = 0.0f;
	}
}

WS_CANVAS_API void EventScript_MinimalTest(MinimalTest_InputData inputData)
{
	Execute_MinimalTest_Input(inputData.in_Start, inputData.in_Stop, inputData.in_ListenerWorldPosition, inputData.in_EmitterWorldPosition);
	std::string NoName_3764011140541903891 = "..//..//Asset//testC.wav";
	WavObject WaveObject_3285133974272340789;
	uint64_t EventPrototypeID_2726935047208135274;
	Execute_WaveLoader(NoName_3764011140541903891, WaveObject_3285133974272340789, EventPrototypeID_2726935047208135274);
	float Gain_3479971340518051646;
	Execute_Attenuator(inputData.in_ListenerWorldPosition, inputData.in_EmitterWorldPosition, Gain_3479971340518051646);
	float NoName_3109558878669456172 = 5000.000000;
	float NoName_2953546187121584618 = 500.000000;
	uint64_t EventInstanceID_3403570235478685868;
	Execute_WavePlayer(inputData.in_Start, inputData.in_Stop, EventPrototypeID_2726935047208135274, Gain_3479971340518051646, NoName_3109558878669456172, NoName_2953546187121584618, EventInstanceID_3403570235478685868);
	Execute_MinimalTest_Output(EventInstanceID_3403570235478685868);
}

I didn’t believe that the entire implementation of the compiler would be on this level of easiness, just use some basic language features in C++ and I really could get some compilable code. But indeed it is nothing more than some simple string parsers and sorting algorithms, what I’d say, a dirty codebase is not so bad!

Plug-in Hook

It won’t be quite complex to use cr, I wrote a plugin manager class as the host module to manage multiple plugin instances, while the plugin hook entrance would be compiled into the dynamic library when the code was generated. The entrance source file varies between different graphs, so I use a template file to generate them:

#include "../../Source/Core/Logger.h"
#include "../../GitSubmodules/cr/cr.h"
#include "PluginName.h"

static unsigned int CR_STATE version = 1;

CR_EXPORT int cr_main(struct cr_plugin *ctx, enum cr_op operation)
{
	if (operation == CR_UNLOAD)
	{
		Logger::Log(LogLevel::Verbose, "Plugin unloaded version ", ctx->version);
		return 0;
	}
	if (operation == CR_LOAD)
	{
		Logger::Log(LogLevel::Verbose, "Plugin loaded version ", ctx->version);
		return 0;
	}

	if (ctx->version < version)
	{
		Logger::Log(LogLevel::Verbose, "A rollback happened due to failure: ", ctx->failure);
	}

	version = ctx->version;

	auto l_inputData = reinterpret_cast<PluginName_InputData*>(ctx->userdata);

	if (l_inputData != nullptr)
	{
		EventScript_PluginName(*l_inputData);
	}

	return 0;
}

This plugin’s main function would be invoked every time when the host calling cr_plugin_update(), and if the input data is valid then we’d trigger the actual function:

video: title: "Editor hot-reload": /attachments/node-graph-editor-hot-reload.mp4

As you could see above on the video, when I changed the file name from testA.wav to testC.wav and built the graph, the actual dynamic lib automatically reloaded and the global state was preserved correctly (the version of the plugin in this case), and the execution flow change to the updated version.

Robust is the king

The overall development time of this project is around 50-100 coding hours, which I achieved by 5 months, and I assume the maximum coding hour would be around 200-300 hours if consider a further extension and refactoring for several improvements. Yes, I didn’t measure the sLOC but time, because it’s my opinion that the software architecture rules the codebase size and complexity in a rational and controllable project, the only error introduced during the development cycle is always human error.

But you may criticize like “are you sure this demo-level of stuff could work in the real game development?” The answer is, I don’t know, because all that I’ve done is on the direction of R&D, there is no game actually used what I made. But let’s analysis a practical situation:

The audio designer needs a new node to perform a new DSP effect or some logics;
The audio programmer writes the node description and the node function source files;
The audio programmer submits just the files directly to the version control system;
The audio designer reopens the editor, adds the new node, builds the dynamic library, and in the game it’s automatically hot-reloaded.

Which step would be unpredictable? For step 1, the innovative idea would always be harmless. Then the implementation in step 2, if the node function didn’t rely on any global or external states, it’s easy to make them pure (generally speaking we should prefer to use the pure function in the node graph to make everything procedural!). A VCS submit error should not be counted here in step 3. And the last step, apparently we don’t want to see something like Build: 0 succeeded, 1 failed appearing in the console after hitting the compile button.

So, is the node compiler robust? I’d say never. A C++ compiler should only compile a text file which written within the grammar set of C++ standards, and a node graph compiler should only compile a node graph which written by the grammar of node graph standards. But we don’t have any self-constraint node graph standards here! Even the C++ standards are evolving during the years. I think we should be unified with our software development principle in such situations, let’s only focus on the problem (new design) and give the solution (new node), and only if the solution brings new problems (cannot compile) and then deal with it (extend the compiler).

Another practical problem comes from asset management. A real game shouldn’t load small individual files frequently from disk, it’s a meaningless waste. The improvement is clear and has been given by other audio middlewares: Pack audio files into soundbank file. So as soon as the game loaded, you may load an entire soundbank file and use handles to index the actual samples. It’s very easy to implement such design, all you need is the map between the label, the UUID, the global offset, and the sample size. But we would prefer to load individual files for fast iteration in a development environment, middlewares like FMOD Studio and Wwise would not have such flexibility because their runtime modules only support their packed soundbank files. Each time you want to replace a single sample in the bank, you’d have to reload the entire bank if it’s loaded by the game.

The final complaint from the audio engineer should be about the mixing. Here what I built is a solution for game audio logics, while mixing is all about how to control the audio signal chain’s states. Some local scope DSP nodes like a side-chain compressor or EQ could be added, but also an overall mixing editor could be injected into the audio engine module if it’s necessary. I prefer a group or scenario-based game audio mixing approach, while others prefer a more traditional bus-based solution, all of them would require a further extension to the audio engine itself.

If rationally consider, all these concerns are addressable. The more risky problems you’d occur is your team member’s adoption speed to the new solutions. But I think it’s way more concise and easier to learn such a node graph solution if you have some MaxDSP, PureData or Unreal Blueprint experiences, there are no new concepts that differ from what you’ve known from the traditional audio production, and the working flow alternation is minimal. Now you still open your favorite DAW for sample design at the beginning, then the game and the node graph editor are opening too, what you’d keep doing from that point is experimenting with your design idea, keeping push the “compile” button and auditing the result in the game. The working flow barrier disappears naturally!

So what’s the conclusion? My idea is, if your game dev team has at least one guy who gets charge in the game audio related programming, then it’s still totally possible to build your own audio solutions like this. Even you’re targeting the console platform, the only difference is that you now would implement the low-level audio engine by the console’s audio backend API instead, and link against their libs for the timer, logger, memory management and so on. You don’t have to spend time on some highly sophisticated DSP algorithms, the improvement for the final game audio quality more comes from the working pipeline improvement, the same 300 hours you spend on the codebase, you could bring your audio designer a flexible and user-friendly toolchain that they could produce millions of amazing design, rather than just a 10% more realistic acoustic sound propagation algorithm. After all, the game is more about art than tech, don’t you think so?

Reflection in C++... could we nail it now?

zhangandhang@gmail.com (Hang Zhang) — Wed, 25 Dec 2019 17:00:00 +0000

Maybe you’ve seen multiple times the symbol of Ouroboros in different movies and games that about a fancy snake who biting its own tail, despite whether it’s painful (just hope those serpents would have a better control when biting themselves!), these kind of self-dependent problems emerging often when we started to consider one of the most desired features in C++ — how could we inspect the information of the language itself in runtime?

Anybody who has been writing not only std::cout << "hello world!" for a certain amount would know that C++ is a static, nominative, partially inferred programming language, a language designed to be written in a human-readable format and then compiled directly into a bunch of CPU-readable instructions. The only information we could get in touch within a runtime environment is those values stored on heap and stack since after an executable file was loaded into main memory and started to execute, the available scope of the program itself is bound by its own virtual mapped memory space. This would be fine for quite a lot usage cases if we’d only expect C++ as a tool to drive the computer to work for our certain computational tasks, then we’d write the human-readable source code with our well-designed logic business model, and later let the compiler translate them into whatever a CPU wants. But when you started to build a framework, a middleware, a tool for tool production, or anything complicated enough to scare people, there would often appear a requirement that you’d want the program to know itself in a human-readable way and feedback the user these pieces of information in runtime. For example, the simplest case is you want to give the user a magic log printing function, that could automatically print the variable’s name:

int IAmAnInt = 42;

void MagicLogger(int rhs)
{
	std::cout << "The variable ";
	//
	// Imagine a magic here
	//
	std::cout << " is ";
	//
	// Imagine another magic here
	//
}

The result we’d expect is:

The variable IAmAnInt is an Int

Super unfortunately, I randomly grab a version of ISO C++ Standard draft and it is 1448 pages long, which described all the necessary details of the language that could help us keep building our modernism today. We could access a memory address, we could calculate a vector multiplication on CPU by SIMD, all directly with the original syntax. But still, we can’t get the one-line magic logger through the language itself, C++ is still way too low-level! Even with some of the RTTI (Run-time type information) features like typeid() or std::type_info or <type_traits>, what we’d got is only a piece of elementary type information with quite a lot limitations. If we want something fast, then we give up the indirectivity, if we want something noobie-friendly, we give up the speed, what a frustrating reality we are in!

All we need is just a little patience

The ability of a program to inspect itself is called introspection, and if it could furthermore modify itself then it’s called intercession, the combination of these two abilities is named as reflection. As we’ve known, C++ is designed to be statically assembled but with the ability to perform dynamic behavior. All the execution procedures are pre-defined by the programmer, and we’d use the conditional branch and the polymorphism to achieve dynamic in runtime. The actual implementation of the virtual function pointer lookup is called vtable, it is fully hidden by the compiler. Furthermore with the limited RTTI feature like dynamic_cast<T> we could say we’ve almost gotten the intercession in C++.

But when we want some flexible introspection, the language itself didn’t provide us enough facilities. Like the previous example of the magic logger, we can’t get any human-readable information about the source code, since all the executable files or libraries are binary, they are CPU instructions some 0 and 1, unless we would manually bury some strings inside of the program we can’t preserve these pieces of information. So now we’d like to modify a little bit previous example:

struct IAmAnInt
{
	const char* VarName = "IAmAnInt";
	const char* TypeName = "Int";
	int VarValue = 42;
}

void MagicLogger(IAmAnInt rhs)
{
	std::cout << "The variable ";
	std::cout << rhs.VarName;
	std::cout << " is ";
	std::cout << rhs.TypeName;
}

Congratulations! You just invented Java or C#. All these higher-level interpreted languages would like to model all the fundamental types as the object, everything is a certain object and all objects came from, well, another mother-of-all object. Then for what we still play around with C++? If we had to write every variable instance with a type declaration then it’s really a waste of life. So you may come with an idea to use the template:

template <typename T>
struct Var
{
	Var(T&& rhs, const char* varName, const char* typeName)
	{
		VarValue = rhs;
		VarName = varName;
		TypeName = typeName;
	};
	const char* VarName;
	const char* TypeName;
	T VarValue;
};

template <typename T>
void MagicLogger(const Var<T>& rhs)
{
	std::cout << "The variable ";
	std::cout << rhs.VarName;
	std::cout << " is ";
	std::cout << rhs.TypeName;
}

And the user code:

int main()
{
	auto v = Var(42, "v", “Int”);
	MagicLogger(v);
	return 0;
}

It’s more like a Java or C# now. Even before you doubt the waste of the time to write the type name each time, another question comes immediately, what if it’s a user-defined type?

struct MyType
{
	Var<int> AInt;
	Var<float> AFloat;
}

And now you’d have to provide some boilerplate constructor to initialize the name like:

MyType(int v1, float v2)
{
 AInt = Var(v1, "AInt", “Int”);
 AInt = Var(v2, "AFloat", "Float");
}

But you want flexibility, you want extensibility, then later you may go back to the declaration of the Var or MyType and try to modify it to be variadic, and then you may get a:

template <typename... Ts> struct Var {};

template <typename T, typename... Ts>
struct Var<T, Ts...> : Var<Ts...>
{
 Var(T t, Ts... ts) : Var<Ts...>(ts...), tail(t) {}

 T tail;
};

This, basically is a hand-crafted tuple template, and then you’d dig your head into the painful path of the template specialization and workaround for the name string and so on and on… And even you’d like to give up and use std::tuple, another problem about how to generate back the parameter pack would stuck you up for a while. And furthermore, every member of the custom type needs a variable name and type name, just imagine how your custom type would have lots of intrinsic data embedded inside. And this is just the case of the data member, what if the user wants to inspect an array? What about inheritance? We don’t even consider the function yet, oh gosh.

Godmode activated

When the serpent bites its tail, it could never fully swallow itself, and when we want to inspect C++ only by C++ itself, we could never free ourselves from the feature dependency deadlock. But if we stand a step back, what we want is just the information about the source code itself, it is not even fully related to C++, in another language case what we want is still the same thing, and all of them would be written exactly once into the source file by our hands. So rather than we expect the language would have features for inspection, why don’t we generate the information ahead of the actual build time?

Let’s take a look at how a text source file would be processed into a binary executable file:

graph TD;
	subgraph Text
 SourceFile1--Pre-Processor-->TranslationUnitFile1;
 TranslationUnitFile1--Compiler-->AssemblyFile1;
	style SourceFile1 fill:#8cb8ff,stroke:#000,stroke-width:4px;
	end

	subgraph Text
 SourceFile2--Pre-Processor-->TranslationUnitFile2;
 TranslationUnitFile2--Compiler-->AssemblyFile2;
	style SourceFile2 fill:#8cb8ff,stroke:#000,stroke-width:4px;
	end

	subgraph Binary
	ExecutableFile
	style ExecutableFile fill:#8cb8ff,stroke:#000,stroke-width:4px;
	end

	subgraph Binary
	AssemblyFile1--Assembler-->RelocatableObjectFile1;
	RelocatableObjectFile1--Linker-->ExecutableFile;
	style SourceFile1 fill:#8cb8ff,stroke:#000,stroke-width:4px;
	end

	subgraph Binary
	AssemblyFile2--Assembler-->RelocatableObjectFile2;
	RelocatableObjectFile2--Linker-->ExecutableFile;
	style SourceFile2 fill:#8cb8ff,stroke:#000,stroke-width:4px;
	end

…“Preprocessing phase.The preprocessor (cpp) modifies the original C program according to directives that begin with the ‘#’ character…” — Computer Systems - A Programmer’s Perspective, Third edition

The most fully extended version of the source file is the Translation Unit (abbr. TU, or Compilation Unit), it contains all the code we’ve written inside the original source file but also with the replacement of the inclusion directives (#include) and macros (e.g. #ifdef) to the actual content, and you could imagine it should be a longer source file if we included any mega headers like STL or Boost. It seems a good stage for us to get some additional pieces of information.

And now we need to find that manual “Emergency Stop” button of the build toolchain, that we could get the actual TU files in hand. If you are a GCC user then simply gcc -E a source file and you’ll get the preprocessed source code. When switching to Clang it’s exactly the same clang -E, or clang --preprocess. And if you are sticking with MSVC then CL /P /C. Basically the preprocessor is just a text parser with a functionally to replace the directive keyword. And if we could get a TU file, then the only step left is to extract human-readable information from it. It doesn’t matter how we implement the extraction feature, we could write a Python script, we could build yet another C++ program, all choices are possible and flexible, because we won’t have any limitations if we change our goal from “Reflect C++ by C++” to “Extract some text from a file”!

So let’s start from a simplest case, a static primitive type variable declared in global scope:

int IAmAnInt;

And we would like to utilize the introspection info in the same C++ project later, then we could declare a type to describe the data of the data, or we call it metadata:

struct Metadata
{
	const char* VarName;
	const char* TypeName;
};

What we want is something like this:

Metadata IAmAnInt_Metadata = { "IAmAnInt", "Int" };

But this line of the definition should be generated automatically, not handcrafted, and we could query it in runtime, this means whether we need to associate the actual variable to the metadata ahead of runtime or we’d provide a runtime mechanism to map a memory address of IAmAnInt to the IAmAnInt_Metadata. As we’ve talked about before it’s impossible to achieve the last design without any intrusive modification to the original variable. So we’d have to still write some glue code and I would use a macro to solve it:

#define GetMetadata(name) name##_Metadata

And the magic logger would be implemented just like:

void MagicLoggerImpl(const Metadata& rhs)
{
	std::cout << "The variable ";
	std::cout << rhs.VarName;
	std::cout << " is ";
	std::cout << rhs.TypeName;
}

#define MagicLogger(name) MagicLoggerImpl(GetMetadata(name))

And finally in user code things should be as simple as:

	MagicLogger(IAmAnInt);

And you’d get in the console:

The variable IAmAnInt is Int

Now the only thing left is how to generate the metadata! We’ve got all the idea about how to fly to Mars, now what we don’t have is just 1 trillion euros to craft the rocket!

Доброе утро, мистер Матриошка

Quite a lot reflection solutions (this nice post by Jeff Preshing, also this nice post by Konstantin Knizhnik, yet another nice post by Veselin Karaganev, the reflection lib rttr and other uncountable projects) I’ve seen chose an compromised approach, they use macro to generate the metadata by requiring user to write additional marker codes:

struct TestStructA
{
	char* testChar;
	bool testBool;
	int testInt;
};

REFLECT_BEGIN()
REFLECT_STRUCT(TestStructA)
REFLECT_FIELD(char_ptr, testChar)
REFLECT_FIELD(bool, testBool)
REFLECT_FIELD(float, testInt)
REFLECT_END()

This approach is totally fine with a minimum additional overwork, the actual reflected code would be generated directly by the preprocessor, but the overall cons are also significant — it’s macro, nothing would be straightforward within a macro-rich environment. And from another point of view, we’ve provided the unique information once by the structure declaration, then for what to write it again in a different form just to get some info we provided by ourselves? This kind of duplication is quite a torture for people lazy like me. After all my life is finite, I don’t have to repeat a joke twice if a dude can’t catch up with it!

So my criteria are crystal clear: no additional duplicated code for reflection in the original source file. And my only hope now is crafting my own C++ syntax analyzer to interpret the code and extract the metadata. The fundamental idea is simple, we need to implement a string pattern match with certain policies to cover the necessary C++ keywords. But wait, isn’t this the job that a compiler would do every day? They analyze the TU and translate them into an assembly, if they could analyze TU then that means they’ve had a mechanism to decompose a literal C++ file into a logically related syntax tree (it has a name Abstract Syntax Tree, abbr. AST)! Now all the works left for me are just, find a way to stop the compiler’s work in the middle, and ask them to generate the metadata I wanted instead of the assembly code.

Let’s evaluate the most commonly used 3 C++ frontends today:

MSVC: Nevermind, since it’s a proprietary IDE/compiler, we can’t hijack in.
GCC: GCC is open source, so the access to its C++ frontend is available. But it doesn’t have a native architecture for user extension, any additional custom feature support would require us to intrusively modify the frontend. I’d like to recommend you to have a look at an awesome series of blog post by Roger Ferrer Ibáñez if you would want to deal with it.
Clang: Clang is basically a light-weight GCC frontend which initially designed by Apple but open-sourced (somehow sounds like a Sith joined the rebellions), which now developed and maintained by the LLVM Developer Group. It was naturally modeled with a different workflow from GCC that all modules of Clang are individual libs, so it’s always flexible to be used as a general frontend despite the actual usage cases, whether you’d like to just compile your C-family source to binary or you want to customize out some functionalities like code refactoring or formatting, it would be totally possible with a non-intrusive approach to implement them atop Clang.

It’s obvious that Clang is the most suitable frontend that could help us with our goal, there are even multiple choices inside of Clang. It has LibTooling and LibASTMatchers that we could just write the matcher pattern and then query the source files to get the wanted AST node, the introduction and tutorial demonstrated well enough how to use it. Since I’m more eager to build the wheel by myself and lazy to learn the details of Clang’s AST, I’d like to use another more lower-level lib called LibClang, which is described as a “stable high-level C interface to clang” from the page about how to choose the right interface for your application. My goal is an individual tool that could generate reflected C++ code with full awareness of the original source code, so I think it’s better to get my hand dirtier than have some not so necessary dependencies.

Simple == Brutal

The actual implementation is surprisingly stupid, we just need to #include "clang-c/Index.h", and use the functions it provided to parse the TU as our wish by a Visitor pattern. We need to define the visitor callback which would recursively visit all the cursor children like a typing machine:

CXChildVisitResult visitor(CXCursor cursor, CXCursor parent, CXClientData clientData)
{
	return CXChildVisitResult::CXChildVisit_Recurse;
};

As now we just return the visit result without any additional process, but remember this is the core gameplay mechanism. And then we could load the TU. Before that, we need to create an index object “that consists of a set of translation units that would typically be linked together into an executable or library”, and also indicate that we’d like to parse C++ files rather than C or something else. And I assume you’ve had a file name string:

char* args[] = { "--language=c++" };
auto index = clang_createIndex(0, 0);
auto translationUnit = clang_parseTranslationUnit(index, fileName.c_str(), args, 1, nullptr, 0, 0);

And then we trigger the actual visit from the first cursor position of the TU:

auto cursor = clang_getTranslationUnitCursor(translationUnit);
clang_visitChildren(cursor, visitor, nullptr);

And finally dispose all resources after the job done:

clang_disposeTranslationUnit(translationUnit);
clang_disposeIndex(index);

That’s all the code you were promised to write when someone trying to sell you a “Learn C++ in 30 days”. The actual implementation of the visitor callback is definitely underestimated here, just imagine you’d parse a random C++ file that written by one of your teammates, that almost every kind of language features appeared inside. But in practice the most scenes that require reflection support are those custom type declarations, we don’t need to reflect every detail of a procedure or a complex folded variadic template often, so we could start from some simple test case and try to extend it later. Let’s take a look at a previous example:

struct TestStructA
{
	char* testChar;
	bool testBool;
	int testInt;
};

A very simple POD type, and if we just print out the CXCursor’s type information by changing the visitor callback to:

CXChildVisitResult visitor(CXCursor cursor, CXCursor parent, CXClientData clientData)
{
	auto cursorKindName = clang_getCString(clang_getCursorKindSpelling(cursor.kind));
	auto cursorEntityName = clang_getCString(clang_getCursorSpelling(cursor));
	std::cout << " CursorKindName: " << cursorKindName << " CursorEntityName: " << cursorEntityName << "\n";
	return CXChildVisitResult::CXChildVisit_Recurse;
};

And what we’d get in the console is:

CursorKindName: StructDecl CursorEntityName: TestStructA
CursorKindName: FieldDecl CursorEntityName: testChar
CursorKindName: FieldDecl CursorEntityName: testBool
CursorKindName: FieldDecl CursorEntityName: testInt

Seems we’re on the correct track! The actual information of a cursor is more detailed and could be queried by different interfaces, and I’d introduce some of them later by examining some of the fundamental keywords in C++ and how they are identified by the LibClang:

Record

Although the record in C++ could be considered only as structure, we still could treat union, class and even enum and namespace as the record, since they all satisfy the definition of a “record” that provides the direct data access ability. The way to evaluate whether a cursor is a record declaration is by invoking:

unsigned clang_isDeclaration(enum CXCursorKind);

The CXCursorKind type is a member of CXCursor, and some of the fundamental declaration keywords in C++ would be:

C++ keyword	In LibClang
struct	CXCursorKind::CXCursor_StructDecl
union	CXCursorKind::CXCursor_UnionDecl
class	CXCursorKind::CXCursor_ClassDecl
enum/enum class	CXCursorKind::CXCursor_EnumDecl
namespace	CXCursorKind::CXCursor_Namespace

But the result would surprise you a little bit because the access specifier is also interpreted as a declaration. So I’d filter out the CXCursorKind::CXCursor_CXXAccessSpecifier case since they would be handled in a different way later.

Access Specifier

I’d evaluate the access specifier of some cursors separately by querying:

CX_CXXAccessSpecifier clang_getCXXAccessSpecifier(CXCursor);

And the result:

C++ keyword	In LibClang
	CX_CXXAccessSpecifier::CX_CXXInvalidAccessSpecifier
public	CX_CXXAccessSpecifier::CX_CXXPublic
protected	CX_CXXAccessSpecifier::CX_CXXProtected
private	CX_CXXAccessSpecifier::CX_CXXPrivate

As the comment of the query interface said:

If the cursor refers to a C++ declaration, its access control level within its parent scope is returned. Otherwise, if the cursor refers to a base specifier or access specifier, the specifier itself is returned.

And as my observation, for a POD type declaration, if there were any type references in the field of it then the actual access specifier is public. Also if it occurred in an inheritance case, then the cursor kind would be a CXCursor_CXXBaseSpecifier rather than CXCursor_CXXAccessSpecifier, although they are literally the same in the source file.

Type

This is somehow an ambiguous word in this context, here “type” means the information that indicates an entity in the source file what its category is, for example like int in the int testIntdeclaration. We could get this type info by:

 CXType clang_getCursorType(CXCursor C);

And CXType is a structure that could be investigated furthermore by lots of functions, like:

CXType clang_getCanonicalType(CXType T);
unsigned clang_isConstQualifiedType(CXType T);
CXCursor clang_getTypeDeclaration(CXType T);

Or we could simply check the CXTypeKind member of it:

C++ keyword	In LibClang
bool	CXTypeKind::CXType_Bool
int	CXTypeKind::CXType_Int
unsigned long long	CXTypeKind::CXType_ULongLong
float	CXType_Float

The types that are not primitive types also have the corresponding CXTypeKind:

C++ keyword	In LibClang
lvalue reference “&”	CXTypeKind::CXType_LValueReference
rvalue reference “&&”	CXTypeKind::CXType_RValueReference
pointer “*”	CXTypeKind::CXType_Pointer
array “[]”	CXTypeKind::CXType_ConstantArray
custom type	CXTypeKind::CXType_Record

Field

For enumeration type, the field of it would be the enum constants as CXCursorKind::CXCursor_EnumConstantDecl; For a structure, class or union, the field would be the member data as CXCursorKind::CXCursor_FieldDecl. The actual relationship of a record declaration and its member declarations is a simple one-root subtree in the entire AST, so we could choose to store it and use later when we need to evaluate the semantic relationship between the metadatas.

Method

A method in C++ could be a function or any callable object, that basically accepts an (optional) input and return an (optional) result. The cursor kind of normal function is CXCursorKind::CXCursor_CXXMethod. As we all know that a named function (in contrast of a lambda function) must be unique, thus the signature and the resident scope should be different from any of others, so if there were any overloaded or overridden functions then we could and the machine itself could distinguish in between. If you try to print out the display name clang_getCursorDisplayName(cursor) and the entity name (clang_getCursorSpelling(cursor)) of a cursor, you could see the difference when it meets a method, that display name would include the function signature while the entity name is just the function’s literal name. For example, the printed result of the function:

bool testFuncA(int testParm1, const char* testParm2);

Would be:

CursorDisplayName: testFuncA(int, const char *) CursorEntityName: testFuncA

And when writing the export metadata to the file, I use a simple string hash to generate unique but readable names for each function, but you could choose to add a GUID field to the metadata then they could have the same name for display.

Inheritance

We could use CXCursorKind::CXCursor_CXXBaseSpecifier as the pivot to find the base classes because the next few cursors must be CXCursorKind::CXCursor_TypeRef which refers to the base classes, and it would end in a field or a method declaration. For example in a class inheritance hierarchy:

class TestClassA {};
class TestClassB : public TestClassA {};
class TestClassC {};
class TestClassD : public TestClassB, TestClassC {};

The CXCursorKind of them should be:

ClassDecl // "class TestClassA"
ClassDecl // "class TestClassB"
C++ base class specifier // "public"
TypeRef // "TestClassA"
ClassDecl // "class TestClassC"
ClassDecl // "class TestClassD"
C++ base class specifier // "public"
TypeRef // "TestClassB"
C++ base class specifier // "public"
TypeRef // "TestClassC"

And we could assign the inheritance relationship between the metadata later after parsing the entire TU (if we consider this relationship as an abstract data structure, actually it would form a Directional Acyclic Graph).

Inclusion directive

In a real codebase certainly, we would use #include to establish source file relations, but if we have to parse all the codes inside the included files each time then it’s a waste of time and space. LibClang provides us a few useful functions to distinguish the cursor’s origin:

int clang_Location_isInSystemHeader(CXSourceLocation location);
int clang_Location_isFromMainFile(CXSourceLocation location);

We could obtain the source location by:

auto range = clang_getCursorExtent(cursor);
CXSourceLocation location = clang_getRangeStart(range);

Additionally, there is another visitor callback at a higher level to parse TU, that could provide us the inclusion directives’ information:

void inclusionVisitor(CXFile included_file, CXSourceLocation* inclusion_stack, unsigned include_len, CXClientData client_data)

We could dispatch the visit by:

clang_getInclusions(CXTranslationUnit tu, CXInclusionVisitor visitor, CXClientData client_data);

Because we’d like to only parse each file once so it’s important to know whether the included file has been parsed or not, and furthermore with these pieces of information we could later build the exact same inclusion relationships among the generated metadata files.

No free lunch, at least not today

Some of the frameworks (e.g. Unreal Engine, Qt) would give the user the freedom to choose whether to reflect a certain sector or not in the source files, and they require users to markup them by macros. And then the reflected code would be generated by their customized tools. Since we won’t mess us up to such level, we could utilize the custom annotations in C++ which most of the people would ignore — the attributes to get the similar result. Since most of the compilers should simply ignore the unknown attributes if targeting at C++17, we could use them to markup the wanted region in a source file to reflect. One of the design would look like:

#define STRUCT() struct __attribute__((annotate("refl_record")))
#define FIELD() __attribute__((annotate("refl_field")))
#define METHOD() __attribute__((annotate("refl_method")))

STRUCT() TestStructA
{
 FIELD()
	char* testChar;

	FIELD()
	bool testBool;

	FIELD()
	int testInt;

	METHOD()
	float testFunc();
};

But it introduced a tight coupling over the choices of the compiler, that __attribute__ is GNU style while MSVC requires __declspec, and if we use __attribute__ for reflection then we can’t compile the source file successfully with MSVC. A possible workaround would be using another one macro to distinguish between them like:

#ifdef _MSC_VER
#define STRUCT() struct [[refl_struct]]
#define FIELD() [[refl_field]]
#define METHOD() [[refl_method]]
#else
#define STRUCT() struct __attribute__((annotate("refl_struct")))
#define FIELD() __attribute__((annotate("refl_field")))
#define METHOD() __attribute__((annotate("refl_method")))
#endif

Since in my project the overall architecture is more data-driven thus I would choose to reflect on a granularity level between source files, rather than markup the specific sectors inside a type declaration.

And finally, we could implement the actual reflection modules, here is my approach:

Store the LibClang generated metadata in an array, in such kind of format:

struct ClangMetadata
{
	CXString displayName;
	CXString entityName;
	CXCursorKind cursorKind;
	CX_CXXAccessSpecifier accessSpecifier;
	CXTypeKind typeKind;
	CXString typeName;
	bool isPtr = false;
	bool isPOD = false;
	CXTypeKind returnTypeKind;
	CXString returnTypeName;
	size_t arraySize = 0;
	ClangMetadata* inheritanceBase = nullptr;
	ClangMetadata* semanticParent = nullptr;
	size_t totalChildrenCount = 0;
	size_t validChildrenCount = 0;
};

Assign the inheritance relationship (Let’s just consider the single inheritance case for the sake of simplicity):

auto l_clangMetadataCount = m_clangMetadata.size();
for (size_t i = 0; i < l_clangMetadataCount; i++)
{
	auto& l_clangMetadata = m_clangMetadata[i];

	if (l_clangMetadata.cursorKind == CXCursorKind::CXCursor_CXXBaseSpecifier)
	{
		m_clangMetadata[i - 1].inheritanceBase = &m_clangMetadata[i + 1];
	}
}

Generate custom metadata definition by simply writing string into a file (or if you want you could generate the serialization functions too), the runtime metadata has such a structure:

struct Metadata
{
	const char* name;
	DeclType declType;
	AccessType accessType;
	TypeKind typeKind;
	const char* typeName;
	Metadata* typeRef;
	bool isPtr;
	Metadata* base;
};

Include your generated metadata file to the runtime source code.

Here are two header files I used as the test case:

//TestA.h
#pragma once

enum class TestEnumA
{
	case1, case2, case3
};

union TestUnionA
{
	int I;
	float F;
	char* C;
};

struct TestStructA
{
	TestEnumA testEnumA = TestEnumA::case2;
	char* testChar = 0;
	bool testBool = false;
	int testInt = 42;
};

class TestClassA
{
public:
	virtual bool testFuncA(uint64_t testParm1, const char* testParm2) = 0;
	bool testFuncA(float testParm1, const TestUnionA& testParm2);
	short testShort = 4;
	long testLong = 4;
	long long testLongLong = 4;
	float testFloat = 4.20f;
protected:
	double testDouble = 4.2;
private:
	TestStructA testStructA = {};
};

template <typename T>
struct TestTemplateStructA
{
	T testVar;

	template <typename T, typename U>
	bool testFuncA(T testParm1, const U& testParm2);
};

using TestTemplateStructAInt = TestTemplateStructA<int>;

And:

#pragma once

#include "TestA.h"

template <typename ...T>
struct TestTemplateStructB
{
};

template <class T, class... Ts>
struct TestTemplateStructB<T, Ts...> : TestTemplateStructB<Ts...>
{
	TestTemplateStructB(T t, Ts... ts) : TestTemplateStructB<Ts...>(ts...), tail(t) {}

	T tail;
};

using TestTemplateStructBICF = TestTemplateStructB<int, char*, float>;

struct TestStructB
{
	TestEnumA testEnumA = TestEnumA::case2;
	wchar_t* testWchar_t = L"42";
	TestClassA* testClassA = 0;
	int8_t testInt8t = 42;
};

namespace TestNamespace
{
	struct TestStructC : public TestStructB
	{
		int16_t testInt16t = 42;
		int32_t testInt32t = 42;
		int64_t testInt64t = 42;
	};
}

class TestClassB : public TestClassA
{
	bool testFuncA(uint64_t testParm1, const char* testParm2) override { return false; };
};

And the generated metadata files:

//TestA.refl.h
#pragma once

Metadata refl_TestEnumA = { "TestEnumA", DeclType::Enum, AccessType::Invalid, TypeKind::Enum, "TestEnumA", nullptr, false, nullptr };
Metadata refl_TestEnumA_member[3] =
{
	{ "case1", DeclType::EnumConstant, AccessType::Invalid, TypeKind::Enum, "TestEnumA", nullptr, false, nullptr },
	{ "case2", DeclType::EnumConstant, AccessType::Invalid, TypeKind::Enum, "TestEnumA", nullptr, false, nullptr },
	{ "case3", DeclType::EnumConstant, AccessType::Invalid, TypeKind::Enum, "TestEnumA", nullptr, false, nullptr },
};
Metadata refl_TestStructA = { "TestStructA", DeclType::Struct, AccessType::Invalid, TypeKind::Custom, "TestStructA", nullptr, false, nullptr };
Metadata refl_TestStructA_member[4] =
{
	{ "testEnumA", DeclType::Var, AccessType::Public, TypeKind::Enum, "TestEnumA", &refl_TestEnumA, false, nullptr },
	{ "testChar", DeclType::Var, AccessType::Public, TypeKind::SChar, "char", nullptr, true, nullptr },
	{ "testBool", DeclType::Var, AccessType::Public, TypeKind::Bool, "bool", nullptr, false, nullptr },
	{ "testInt", DeclType::Var, AccessType::Public, TypeKind::SInt, "int", nullptr, false, nullptr },
};
Metadata refl_TestClassA = { "TestClassA", DeclType::Class, AccessType::Invalid, TypeKind::Custom, "TestClassA", nullptr, false, nullptr };
Metadata refl_TestClassA_member[8] =
{
	{ "TestClassA_testFuncA_18297869138227429096", DeclType::Function, AccessType::Public, TypeKind::Invalid, "bool (int, const char *)", nullptr, false, nullptr },
	{ "TestClassA_testFuncA_11769634238821699591", DeclType::Function, AccessType::Public, TypeKind::Invalid, "bool (float, const TestUnionA &)", nullptr, false, nullptr },
	{ "testShort", DeclType::Var, AccessType::Public, TypeKind::SShort, "short", nullptr, false, nullptr },
	{ "testLong", DeclType::Var, AccessType::Public, TypeKind::SLong, "long", nullptr, false, nullptr },
};
Metadata refl_TestTemplateStructA = { "TestTemplateStructA", DeclType::ClassTemplate, AccessType::Invalid, TypeKind::Invalid, "", nullptr, false, nullptr };
Metadata refl_TestTemplateStructA_member[2] =
{
	{ "testVar", DeclType::Var, AccessType::Public, TypeKind::Invalid, "T", nullptr, false, nullptr },
	{ "TestTemplateStructA<T>_testFuncA_8483520174557475787", DeclType::FunctionTemplate, AccessType::Public, TypeKind::Invalid, "bool (T, const U &)", nullptr, false, nullptr },
};

And:

#pragma once

#include "TestA.refl.h"

Metadata refl_TestTemplateStructB = { "TestTemplateStructB", DeclType::ClassTemplate, AccessType::Invalid, TypeKind::Invalid, "", nullptr, false, nullptr };
Metadata refl_TestStructB = { "TestStructB", DeclType::Struct, AccessType::Public, TypeKind::Custom, "TestStructB", nullptr, false, nullptr };
Metadata refl_TestStructB_member[4] =
{
	{ "testEnumA", DeclType::Var, AccessType::Public, TypeKind::Enum, "TestEnumA", &refl_TestEnumA, false, nullptr },
	{ "testWchar_t", DeclType::Var, AccessType::Public, TypeKind::WChar, "wchar_t", nullptr, true, nullptr },
	{ "testClassA", DeclType::Var, AccessType::Public, TypeKind::Custom, "TestClassA", &refl_TestClassA, true, nullptr },
	{ "testInt8t", DeclType::Var, AccessType::Public, TypeKind::SInt, "int", nullptr, false, nullptr },
};
Metadata refl_TestStructC = { "TestStructC", DeclType::Struct, AccessType::Public, TypeKind::Custom, "TestNamespace::TestStructC", nullptr, false, &refl_TestStructB };
Metadata refl_TestStructC_member[3] =
{
	{ "testInt16t", DeclType::Var, AccessType::Public, TypeKind::SInt, "int", nullptr, false, nullptr },
	{ "testInt32t", DeclType::Var, AccessType::Public, TypeKind::SInt, "int", nullptr, false, nullptr },
	{ "testInt64t", DeclType::Var, AccessType::Public, TypeKind::SInt, "int", nullptr, false, nullptr },
};
Metadata refl_TestClassB = { "TestClassB", DeclType::Class, AccessType::Invalid, TypeKind::Custom, "TestClassB", nullptr, false, &refl_TestClassA };
Metadata refl_TestClassB_member[1] =
{
	{ "TestClassB_testFuncA_17046919256446738461", DeclType::Function, AccessType::Private, TypeKind::Invalid, "bool (int, const char *)", nullptr, false, nullptr },
};

As you could see here, my data structure design for runtime metadata isn’t so optimized and intuitive, there are tons of unfinished features, for example, add syntax sugars like some getter and logger functions or organize all the metadata into a unified list for memory coherency. But as soon as you started to swim in the water of LibClang, you’ve already been exposed to all the possibilities that a frontend could have in front of you, and you could design your own reflection pipeline based on your specific usage. But there are still too many (not so) edge cases I didn’t talk about, for example how to cover the specialized template since they would be expended in compile-time, and how to solve the multi-references between metadata with the respect of complex inheritance and so on. I would keep enriching the reflection module in my project, you could take a look at the actual code about how it is finally implemented. And since the language support for the reflection that has been postponed to the next standard C++23, until then I thought I might still focus on crafting my reflection mirror for C++. But maybe it’s a good breaking-up moment from C++ and finds new ones like Dlang or Rust? After all, it’s not an easy way to breath if you have to craft oxygen on your own!

Physically Based Rendering - Lighting

zhangandhang@gmail.com (Hang Zhang) — Sat, 07 Dec 2019 18:25:00 +0000

As soon as we modeled the surface’s physical properties that covered a certain range of material in real life, we would need to emit light onto them, in order to finally get the outcome radiance from the surface. If you take a look back at the rendering equation, the outcome radiance is just an integral of the income radiance over the semi-sphere around the normal. This gives us a fundamental assumption that with a brutal algorithm such as path tracing we could get a numerical solution for the problem. We could just model the geometry shape of the light source, then emit single light through all the possible directions and evaluate whether they hit a surface, until the overall energy diminished to a certain threshold. But in real-time rendering, the computational source currently we had in hand won’t be sufficient for such cost of the algorithm. We need to find the analytical replacement of the integral, or at least some cheaper numerical integrals.

Remapping of reality

Before we dive into the detail of the analytical representation of different light shapes, I’d introduce another type of unit system to measure energy and related quantities. If you remembered what I introduced in a previous post about radiometry, they are identically related. Thus the photometry it is, basically it’s a weighted version of radiometry, but with the respect to the nature of human eyes’ sensitivity. Because when we deal with the light in real life, typically the light sources we would get are bulbs, candles, flashlights, and our old friends, sun, we perceive the electromagnetic wave from them by the cell on our retina. And because of the non-linearly of human nature (of course!), our eyes have different sensitivities among different light wavelengths, so finally people invented the photometry quantities to better measure the light we actually “see”.

Here are the common photometry quantities we would occur in real-time rendering:

Quantity		Unit		Notes
Name	Symbol	Name	Symbol
Luminous energy	${Q_v}$	lumen second	${lm·s}$	Energy of electromagnetic radiation with respect of human eyes’ sensitivity
Luminous flux	${\Phi_v}$	lumen	${lm}$	also called Luminous power
Luminous intensity	${I_v}$	candela	${\frac{lm}{sr}}$ or ${cd}$	steradian is similar to angle in 2D space
Illuminance Flux density	${M_v}$ or ${E_v}$	lux	${\frac{lm}{m^2}}$ or ${lx}$	Il-luminance, means the luminance received by a surface
Luminance	${L_v}$	nit	${\frac{lm}{m^2·sr}}$ or ${\frac{cd}{m^2}}$ or ${nt}$

As you could see, it’s basically another version of quantity measurement for the electromagnetic radiation, but with more emphasis on the visible light range. Because the two types of retina cell - rods and cones have quite different responses for different intensity and wavelengths of lights, the actual mapping between radiometry to photometry requires to be done in two different versions. The one related to cones cell typically has a luminance range from ${10 nt}$ to ${10^8 nt}$ and we call it photopic vision in biology, and another one related to rods cell is called scotopic vision with the luminance range from ${10^{-3} nt}$ to ${10^{-6} nt}$. The cones cell could receive the chromatic of the world and the rods cell receives the silhouette of the object. But you may wonder, how much is one $nt$? The SI defines the fundamental unit of the light measurement by ${cd}$, and then the other units are its deduced units. And with the standard definition, one candela means “A $540.0154×10^{12}Hz$ monochromatic light emits $1/683$ Watt per steradian”, then we could say how much a nit it is by adding “per square meter” to it. The reason to choose such a reference wavelength is that one of the three types of cone cells is most sensitive to it, and if you took a look at the visible light spectrum you’d see it’s a greenish color. The magic number 1/683 came from a historical compatibility requirement to transit the old unit system to the modern one.

“…Radiometry deals purely with physical quantities, without taking account of human perception. A related field, photometry, is like radiometry, except that it weights everything by the sensitivity of the human eye. The results of radiometric computations are converted to photometric units by multiplying by the CIE photometric curve, bell-shaped curve centered around 555 nm that represents the eye’s response to various wavelengths of light…” - pg. 271, “Real-Time Rendering”, 4th Edition.

So with a certain wavelength $\lambda$, the mapping from radiometry to photometry is expressed like ${I_v(\lambda)}=683.002\overline{y}(\lambda)I_e(\lambda)$, or we could integrate over the entire visible light wavelength range to get ${I_v}=683.002\int_{380}^{780}\overline{y}(\lambda)I_e(\lambda)d\lambda$. The $\overline{y}(\lambda)$ here is the luminosity function, it’s a bell-shaped distribution function and has different values in photopic and scotopic. The photopic luminosity function would reach 1 in 555nm or 540 THz (the definition frequency of candela) and the scotopic luminosity function is 1 in around 510nm, they demonstrate the eyes’ perception ratio of different wavelengths. And we could furthermore deduce two quantities called luminous efficacy and luminous efficiency from the mapping function. Luminous efficacy $\eta$ is defined as ${\eta}=683.002\frac{\int_{380}^{780}\overline{y}(\lambda)I_e(\lambda)d\lambda}{\int_{380}^{780}I_e(\lambda)d\lambda}$ and the unit is ${\frac{lm}{W}}$, it represents the light source’s ability to emit visible light; And if we normalized the luminous efficacy with ${\frac{1W}{683lm}}$ then we would get luminous efficiency, which is simply defined as $\frac{\int_{380}^{780}\overline{y}(\lambda)I_e(\lambda)d\lambda}{\int_{380}^{780}I_e(\lambda)d\lambda}$. With the help of them we could easily evaluate whether a light source is wasting or saving energy, or to directly convert from the radiometric quantities to photometric version. And we could get that an ideal 540 THz black body would have a 683 ${\frac{lm}{W}}$ luminous efficacy or 100% luminous efficiency.

Since what we’ve been talking so far is all about the real-time rendering, we would use the discrete RGB space instead of a continuous spectrum, and then we could simplify the continuous luminosity function to a discrete version. And we could then give the user a few parameters like the radiometric quantities and the luminous efficiency to adjust the light source. But when we build a physically-based rendering pipeline we’d often go to find references in real life, and almost all the light sources like those bulbs sold in supermarket would print their photometric quantities on the package, it would be more convenient to use photometric quantities in the light system for the final users.

Integration matters

The more crucial part of a light system is how to model the geometry appearance of the light source. When dealing with a non-physically based rendering pipelines, there are often 3 types of analytical light we’d like to use, directional light, point light, and spot light. I’d not discuss any of them here because all of them are not modeling around the actual light source geometry but rather around the appearance of the lighting result, another reason is that there are tons of information online about how to implement them and I’d better not repeat again. I’d indicate them as punctual light sources later since the actual shape of them is infinitesimal. The physically correct light sources all have volume, or if we ignore one of the 3 dimensions with the respect of the nature of projection they still have the area. I’d indicate them as area light source later on. The typical analytical area light source has shapes like sphere, disk, tube and rectangular, or with a trivial shape which requires some advanced approximations.

If we took a look once more at the reflectance equation, let’s write a slightly different version $L_o=\int\limits_{\Omega+} f(v,l) V(l) L_i \langle n·l \rangle dl$ instead, the $V(l)$ here is a Heaviside function to indicate whether a light source is visible along a certain direction, and it describes the shape and the shadow of the light. As long as the shape of the light source is non-trivial, it’s impossible to find a general analytical solution for such hemisphere integral. The punctual light sources all assume the shape is infinitesimal so the integral would be simplified by an integral of cosine over hemisphere and finally we’d get $L_o=\pi f(v,l) \langle n·l \rangle$, that’s the reason why we could write Lo = rho * NDotL when using the simple Lambert diffuse BTDF $f(v,l) = \frac{\rho}{\pi}$, the $\pi$ is just perfectly canceled. For area light we cannot have such one-line-to-rule-them-all enjoyment, there are some practical solutions so far, that all of them would solve the problem to a certain level which is acceptable enough. I’d introduce them one by one below.

Form Factor

The first approach is that we could still try to solve the integral directly. We can’t numerically solve it in runtime, but if we convert it to the integral over the light source’s area instead, then it would bring some certainties because we would have known the shape or the actual area of the light source when shading the object. The rewritten version of the reflectance equation is $L_o=\int\limits_{A} f(v,l) L_i \langle n·l \rangle \frac{\langle n_a·-l \rangle}{r^2} dA$, here $n_a$ is the normal of $dA$, $r$ is the distance from the lighting point to $dA$. Because the effective area along the $dl$ is proportional to the normal orientation of $dA$ and the distance from the shading point to the $dA$, so here we introduced a $\frac{\langle n_a·-l \rangle}{r^2}$ factor.

And then we could move the BSDF out of the integral by assuming it’s not interleaved with light direction so much, for example in a Lambert diffuse BTDF case. Then what we’d solve is $L_o = f(v,l) \int\limits_{\Omega+} L_i \langle n·l \rangle dl$, or $L_o = f(v,l) E(n)$ that we could treat all the income luminance as the illuminance. The analytic method to integrate illuminance has been already developed in problem domain of energy transformation, such like radio transfer or heat transfer, since fundamentally they are all the same type of problems about how to calculate the energy transferred from one object to another, or specifically speaking in our case, from a surface to another one.

“…The view factor from a general surface A1 to another general surface A2 is given by: $\displaystyle F_{1\rightarrow 2}={\frac {1}{A_{1}}}\int {A{1}}\int {A{2}}{\frac {\cos \theta {1}\cos \theta {2}}{\pi s^{2}}},{d}A{2},{d}A{1}$…” [Wiki1]

The actual analytic formulae are more complicated but well-documented in [Mar14] and [HSM10], I’d only give each link of the commonly used geometry shapes below:

Sphere: “Patch to a sphere - Titled” in pg. 10 in [Mar14], and B-43 in [HSM10]
Disk: “Parallel configurations - Patch to disc” in pg. 22 in [Mar14], and B-13 in [HSM10]
Rectangular: “Parallel configurations - Patch to rectangular plate” in pg. 22 and “Perpendicular -configurations - Patch to rectangular plate” in pg. 26 in [Mar14], and B-5 in [HSM10]

The solution in [HSM10] covers more general cases than [Mar14]. The overall runtime cost if we’d implement them naively is not so acceptable for real products, but they bring us the perfect analytic forms, we could use them wisely in products with some optimizations and simplifications, or just leave them as a ground truth faster than the classic path tracing.

A code example for sphere light that modified from [LR14]:

	for (int i = 0; i < NR_SPHERE_LIGHTS; ++i)
	{
		float lightRadius = sphereLightCBuffer.data[i].luminousFlux.w;
		if (lightRadius > 0)
		{
			vec3 unormalizedL = sphereLightCBuffer.data[i].position.xyz - posWS;
			vec3 L = normalize(unormalizedL);
			vec3 H = normalize(V + L);

			float LdotH = max(dot(L, H), 0.0);
			float NdotH = max(dot(N, H), 0.0);
			float NdotL = max(dot(N, L), 0.0);

			float sqrDist = dot(unormalizedL, unormalizedL);

			float Beta = acos(NdotL);
			float H2 = sqrt(sqrDist);
			float h = H2 / lightRadius;
			float x = sqrt(max(h * h - 1, eps));
			float y = -x * (1 / tan(Beta));
			float illuminance = 0;

			if (h * cos(Beta) > 1)
			{
				illuminance = cos(Beta) / (h * h);
			}
			else
			{
				illuminance = (1 / max(PI * h * h, eps))
					* (cos(Beta) * acos(y) - x * sin(Beta) * sqrt(max(1 - y * y, eps)))
					+ (1 / PI) * atan((sin(Beta) * sqrt(max(1 - y * y, eps)) / x));
			}
		}
	}

As you could see here, the form factor approach is suitable for our low-frequency signals - the diffuse part of the BSDF, since it is modeled basically around the phenomenon of energy bounce between diffuse surfaces. But still we need to find a solution for the specular part, and if you want to use some more advanced BTDF than the Lambert model which involves more about the complex subsurface scattering phenomenon, then the form factor solution is not accurate enough.

Representative Point

Another approach is that we won’t solve any of the integral at all, instead, we would try to substitute the problem with our old method, use punctual light to approximate area light. If we still focused on the lighting result, area light is similar to a bunch of point lights. And then we could find a point on the surface of the light source that contributes the most to the lighting result, and use it as the location of a point light to continue our shading process. This method is often called Representative Point or Most Representative Point (abbr. MRP), which has been successfully used in lots of PBR pipelines like [Kar13] and [LR14].

“…These approaches resemble the idea of importance sampling in Monte Carlo integral, where we numerically compute the value of a definite integral by averaging samples over the integral domain. In order to do so more efficiently, we can try to prioritize samples that have a large contribution to the overall average….” - pg. 385, “Real-Time Rendering”, 4th Edition.

When using the MRP method, we could both apply it to diffuse and specular part of BSDF, while it depends on the actual requirement of the products. One of the general algorithms has been developed by [Dro14], it uses the halfway vector method to find MRP for diffuse and cone projection to find MRP for specular. For the diffuse part, the halfway vector could be expressed as $\vec{h} = \frac{\vec{pp_0} + \vec{pp_1}}{||\vec{pp_0} + \vec{pp_1}||}$, $p$ is the shading point, $p_0$ is the intersection point of a ray from $p$ along the reflection vector $\vec{r}$ of the view direction, and $p_1$ is the intersection point of a ray from $p$ along negative direction of the light plane normal $\vec{n’}$. While the demonstration here may sound not complex, in a real scenario with an actual light shape, we often need to adjust the halfway vector method’s MRP result onto the surface of the light source, since it won’t always fall inside the light area. In another word, we need to find the closest point of the diffuse MRP in the light area if it’s outside.

For the specular part, the calculation became more complicated with the respect of the probability distribution functions (abbr. PDF) in the BRDF, we need to generate a cone along the reflection direction based on the PDF and use the geometry center of the intersection area between the cone and the light surface as the MRP. Furthermore, we could pre-compute lookup tables for a specific BSDF and light texture combination we chose, by eliminating some variables we could limit the LUT to 3D tables and free us from runtime calculation. And after we use the MRP to calculate the illuminance we should weight it by the intersection area, to preserve the energy conservation. All the details were well demonstrated in [Dro14], I’d recommend reading if it’s available for you.

Another way to calculate specular MRP is that we could find the point on the light surface with the smallest distance to the reflection ray of view direction, which is demonstrated in [Kar13]. For example, in a sphere light case we could calculate $p_{cr} = (l · r)r − l$ then $p_{cs} = l + p_{cr} · min(1,\frac{radius}{||p_{cr}||})$, while $p_{cr}$ is the point on the reflection ray closest to the sphere center, and $p_{cs}$ is the point on the sphere surface closest to $p_{cr}$.

Linearly Transformed Cosines

The LTC approach tries to solve the integral from another point of view, or more precisely speaking, another space. The reflectance equation involves several variables, and we’d evaluate each direction of the incoming light with a BSDF function, and the complexity is $O(n^2)$ due to the variance of the $dl$ and the BSDF. But if we could control one of them to be some sort of constant, then we would optimize the complexity to $O(n)$. Thanks to the popular usage of parametric BSDF nowadays, we could pre-compute lots of them into a multi-dimensional lookup table and just sample from it in runtime to save our precious frame time. But unfortunately, when we implement such a solution and try to fit it into the rendering pipeline, we’d still need to integrate the illuminance over a semispherical region, with an uncertain shape and orientations of the light surface. But if we consider the solving procedure of the reflectance equation in a linear algebra point of view, then the integral of the incoming light in a “world” spherical space could be solved as the integral of the transformed incoming light in a “local” spherical space. If we could choose a suitable transformation to ensure the linearity then the above hypothesis would become a provable truth. And if this transformation could eliminate the shape factor or the orientation factor of the light then we would be one step closer to our expectation of $O(n)$. Luckily the BSDF is naturally a spherical distribution function, the only problem left is how to find the transformation for a BSDF and find a close form of the integral in that BSDF space, after all, if we can’t solve the reflectance equation easier then all of these works would be nonsense.

The most commonly used distribution function of BSDF today is the GGX approximation for the Cook-Torrance microfacet model, the isotropic version of it only depends on the view direction and the surface roughness $\alpha$, even we add the anisotropicity later on, the number of the variables is still only 3. But even 3 variables the integral is still too expensive to calculate in runtime, we have to continue eliminating variables and transforming the light source until we finally find a space cheaper enough to evaluate the integral in real-time. That’s how the LTC came from, we would use a clamped cosine spherical distribution function as the base space since the integral over it is both cheap and analytic, then find a transformation matrix to approximately transform it to a specific BSDF, then use the inverse of that matrix to transform the vertices of the light source into the base space, and finally calculate the integral by utilising the analytic form of line integral.

The theoretical details are well demonstrated in [Hei16] and the reference C++ source code has been provided on Github, but without too much explanation about what it is doing. Actually, if you have ever implemented any BSDF LUT, the approach behind is similar, we just generate the value pair of all possible parameter combination ahead of runtime, and store them in an easy-to-fetch data structure. What we want to get here is a table of transformation matrices that, we could transform our specific BSDF distribution function to the clamped cosine one bidirectionally. Or mathematically speaking, we need to find the $M$ in $D_{BSDF}(\omega) = M * D_o(\omega)$, where $D_o$ would be the original clamped cosine distribution function we chose, with a form of $D_o(\omega_o = (x, y, z)) = \frac{1}{\pi}max(0, z)$ that $\omega_o$ is the normalized direction vector. As you could see we would employ a 3D vector as usual, so the matrix should be a 3x3 matrix.

The next step would be how to calculate the $M$ for our specific BSDF, for example, what the [Hei16] did for GGX distribution. They use Nelder–Mead method to numerically search the $M$, with a cost/error function implemented by simply comparing the actual BSDF value and LTC fitted value. And because in the paper they only fitted isotropic version of GGX, finally the $M$ only has 5 effective elements on each diagonal.

\begin{bmatrix} a & 0 & b\\ 0 & c & 0\\ d & 0 & e\\ \end{bmatrix}

And with normalization by one of the elements there would be only 4 elements left, which could just be stored into a 4 channels texture and linear sampled in runtime. In the paper, they use $e$ as the normalizer, but later in [Hill16], they admit that it would bring some serious numerical error, and somehow I found that they use $c$ later again in the repo’s history, that should be a measured decision maybe. Also in [Hill16], they mentioned multiple problems when implementing LTC solutions, from how to deal with polygon clipping effectively to how to ensure the robust of inverse trigonometric functions, highly recommend to read.

After you get the LUT for $M$, then could just use it in runtime for the shading. You just need to write some shader codes to transform the light vertices to LTC space and do a spherical line integral by $\frac{1}{2\pi}\sum_{i=1}^{n}acos(v_i · v_j)\frac{v_i \times v_j}{||v_i \times v_j||} · n$. But this numerical integral is not practical for light shapes such as single line, capsule, sphere and disk, since you can’t find a small but smooth enough vertices number for ellipse shapes. Moreover, it’s the natural solution for convex polygons like rectangular light. Luckily later they provided the corresponding integral form for other shapes in [Hei17], where the final appearance and performance are quite plausible for real products in contrast with other solutions. I’d recommend again to find details in their publications.

To be continued.

Bibliography：

[Wiki1] https://en.wikipedia.org/wiki/View_factor

[Mar14] http://webserver.dmt.upm.es/~isidoro/tc3/Radiation%20View%20factors.pdf

[HSM10] http://www.thermalradiation.net/tablecon.html

[Kar13] B. Karis. “Real Shading in Unreal Engine 4”. In: Physically Based Shading in Theory and Practice, ACM SIGGRAPH 2013 Courses. SIGGRAPH ’13. Anaheim, California: ACM, 2013, 22:1–22:8. isbn: 978-1-4503-2339-0. doi: 10.1145/2504435.2504457. url: http://selfshadow.com/publications/s2013-shading-course/.

[LR14] S. Lagarde and C. de Rousiers. “Moving Frostbite to PBR”. In: Physically Based Shading in Theory and Practice, ACM SIGGRAPH 2014 Courses. SIGGRAPH ’14. Vancouver, Canada: ACM, 2014, 23:1–23:8. isbn: 978-1-4503-2962-0. doi: 10.1145/2614028.2615431. url: http://www.frostbite.com/2014/11/moving-frostbite-to-pbr/.

[Dro14] M. Drobot. “Physically Based Area Lights”. In: GPU Pro 5 Advanced Rendering Techniques. ISBN: 978-1-4822-0864-1. url: https://www.crcpress.com/GPU-Pro-5-Advanced-Rendering-Techniques/Engel/p/book/9781482208634

[Hei16] E.Heitz, J.Dupuy, S.Hill and D.Neubelt. “Real-time polygonal-light shading with linearly transformed cosines”. In: ACM Transactions on Graphics (TOG) Volume 35 Issue 4, July 2016 Article No. 41. url: https://eheitzresearch.wordpress.com/415-2/

[Hill16] S.Hill and E.Heitz. “Real-Time Area Lighting: a Journey from Research to Production”. In Advances in Real-Time Rendering in Games, ACM SIGGRAPH 2016 Courses. SIGGRAPH ’16. Anaheim, California: ACM, 2016, url: https://blog.selfshadow.com/publications/s2016-advances/

[Hei17] E.Heitz and S.Hill. “Real-Time Line- and Disk-Light Shading”. In Physically Based Shading in Theory and Practice, ACM SIGGRAPH 2017 Courses. SIGGRAPH ’17. Los Angeles, California: ACM, 2017, url: https://blog.selfshadow.com/publications/s2017-shading-course/heitz/s2017_pbs_ltc_lines_disks.pdf

Walking through the heap properties in DirectX 12

zhangandhang@gmail.com (Hang Zhang) — Wed, 18 Sep 2019 14:00:00 +0000

Indifferent to the difference?

Back to the old times, you never worried about how the physical memory would be allocated when you’re dealing with OpenGL and DirectX 11, GPU and the video memory were hidden behind the driver so well that you might even not realize they were there. Nowadays we get Vulkan and DirectX 12 (of course Metal, but…nevermind), that the “zero driver-overhead” slogan (not “Make XXX Great Again” sadly) become the reality on desktop platforms. And “ohh I don’t know that we need to manually handle the synchronization” or “ohh why I can’t directly bind that resources” and so on and on. Of course, the new generation (already not new actually) graphic API is not for casual usages, your hands get dirtier and your head gets more drizzling, while there are still a bunch of debugging layer warning and error messages keeping pop up. Long story in short, if you want something pretty and simple, turn around and rush to modern OpenGL (4.3+) or DirectX 11 and happy coding; if you want something pretty and fast, then stay with me a while and let’s see what’s going on with the new D3D12 memory model.

The fundamental CPU-GPU communication architecture is quite similar around different machines, you have a CPU chip, you have a GPU chip, you have some memory chips, gotcha! The typical PC with a dedicated graphics card would have 2 memory chips, one we often referred as the main memory and another one as the dedicated graphics card memory, or more commonly used (not so strict) name convention are RAM and VRAM for them. Other architectures like those game consoles, the main memory, and the video card memory would be the same physical one, we name such kind of memory accessing model as UMA - Uniform Memory Access. Also, the functional microchips of CPU and GPU would be put together or closer in some certain designs (for example PS4) to get optimized communication performance. You should remember that you just paid once for your DDR4 16GB fancy “memory” when you’re crafting your state-of-the-art PC right? They are the “main” RAM for the general-purpose, like loading your OS after power-up or put the elements of your std::vector<T> inside. But if you also purchased an AMD or NVIDIA dedicated graphics card, you might notice the printed instructions on the package box that there are other couple-few Gibibytes some sort of memory on it. That’s the VRAM memory where the raw texture and mesh data would stay when you are playing CS:GO and swearing random Ruglish in front of your screen.

So, if you want to render the nice SWAT soldier mesh in CS::GO, you need to load it from your disk into the VRAM and then ask GPU to schedule some parallel rasterization work to draw it. But unfortunately, you can’t access the VRAM directly in your C++ code due to the physical design of the hardware. You could reference a main memory virtual address by semantics like Foo* bar = nullptr in C++, because it would be finally compiled into some machine instructions like movq, $(0x108), $0 (it should be binary instruction data actually, for the sake of human-readability here I use assembly language instead) that your CPU could execute. But generally speaking, you can’t expect the same programming experience on GPU, since it is designed for highly parallel computational tasks thus you can’t refer to some fine-grin global memory addresses directly (there are always some exceptions, but let’s stay foolish at present). The start offset of a bunch of raw VRAM data should be available for you in order to create a context for GPU to prepare and execute works. If you were familiar with OpenGL or D3D11 then you had already used interfaces such as glBindTextures or ID3D11DeviceContext::PSSetShaderResources. These 2 APIs expose the VRAM memory not explicitly to developers, instead, you would get some indirect objects in runtimes like an integer handle in OpenGL or a COM object pointer in D3D11.

A step closer

GPU is a heavily history-influenced peripheral product, as time goes by its ability becomes more and more general and flexible. As you might know the appearing of Unified Scalar Shader Architecture and Highly Data Parallel Stream Processing made GPU become compatible for almost every kinds of parallel computation works, the only thing lying between the developer and the hardware is the API. The old generation of graphics APIs like OpenGL or DirectX 11 were designed with emphasis, that they’d better lead developers to a direction that they’d spend more time with the specific computer graphics related tasks they want to hand on with, rather than too much low-level hardware related details. But the experience told us, more abstraction, more overhead. So when the clock ticking around 2015 that the latest generation of graphics API was released to the mass developer like me, a brand new or I’d rather to say “retro” design philosophy appearing among them, no more pre-defined “texture” or “mesh” or “constant buffer” object models, instead we get some new but lower-level objects such as “resources” or “buffers” or “command”.

Honestly speaking, it’s a little bit painful to transit the programming mindset from OpenGL/D3D11 era to Vulkan/D3D12. It’s quite like a 3-years-old kid who used to ride his cute tiny bike with auxiliary wheels now need to drive a 6-shifts manual gear 4WD car. Previously you call a glGen* or ID3D11Device::Create* interfaces you would get the resource handles in no means more than few milliseconds. Now you even can’t “invoke” functions to let GPU do these works! But wait, could we actually ask GPU to allocate a VRAM range for us and put some nice AK-47 textures inside before? Just the graphic cards vendor’s implementation handled the underlying dirty business for us, all the synchronization of CPU-GPU communication, all the VRAM allocation and management, all the resources binding details, we had even not taken a glimpse about them before! But it’s not as bad as I exaggerated, you just have to take care the additional steps which you don’t obligate to do previously, and if you succeeded you’d not only get more code in your repo but also a tremendous performance boost in your applications.

Let’s forget about the API problems for a couple few minutes and take a look back at the hardware architecture to better understand the physical structure that triggered the API “revolution”. The actual memory management relies on the hardware memory bus (it’s part of the I/O bridge) and the MMU - Memory Management Unit, they work together to transfer data between the processor and different external adapters to RAM, and mapping physical memory address to a virtual one. So when you want to load some data from your HDD to RAM, the data would travel through the I/O bridge to CPU and then after some parsing processes it would be stored into RAM. If you had a performance-focused attitude when writing codes, you may wonder is there any optimizations for usage cases like simply loading an executable binary file to RAM, which doesn’t require any additional processing to the data itself. And yes, we had DMA - Direct Memory Access! With DMA the data doesn’t need to travel through CPU anymore and instead, it would be loaded directly from the HDD to RAM.

As we could imagine, CPU and GPU could have individual RAMs and MMUs and Memory Buses, thus they could execute and load-store data into their RAMs individually. That’s perfect, two kingdoms live peacefully with each other. But the problems emerge as soon as they start to communicate, the data needs to be transferred from the CPU side to GPU side or vice versa, and we need to build a “highway” for it. One of the “highway” hardware communication protocol that widely used today is PCI-E, I’d omit the detail instructions and focus on what we’d care about here. It’s basically another bus-like design and provides the functionality that we could transfer data in between different adapters, such as a dedicated graphics card and main memory. With its help, we could almost freely (sadly highway still need payment, it’s not a freeway yet) write something utilizing CPU and GPU together now.

The bridges are a little bit too many, isn’t it? If you remembered that I’ve briefly introduced a memory architecture called UMA before, it basically just looks like we merging RAM and VRAM together. Since its design requires the chip and memory manufacturers to produce such products, and until now I’ve never seen one in the customer hardware market, we can’t craft it by ourselves. But still, if you had an Xbox One or PS4 you’ve enjoyed the benefit of UMA.

Heap creation

So now it’s time to open your favorite IDE and #include some headers. In D3D12, all the resources would resident inside some explicitly specified memory pools, and the responsibility to manage the memory pool belongs to the developer now. This is the how the interface

HRESULT ID3D12Device::CreateHeap(
 const D3D12_HEAP_DESC *pDesc,
 REFIID riid,
 void **ppvHeap
);

comes. If you’re familiar with D3D11 or other Windows APIs in COM model you could easily understand the function signature style. It is made by the combination of a reference to a description structure instance, a COM object class’s GUID and a pointer to store the created object instance’s address. The return value of the function is the execution result.

Now let’s take a look at the description structure:

typedef struct D3D12_HEAP_DESC {
 UINT64 SizeInBytes;
 D3D12_HEAP_PROPERTIES Properties;
 UINT64 Alignment;
 D3D12_HEAP_FLAGS Flags;
} D3D12_HEAP_DESC;

It apparently follows the consistent code style of D3D12 API, and here we get another property structure to fulfill in:

typedef struct D3D12_HEAP_PROPERTIES {
 D3D12_HEAP_TYPE Type;
 D3D12_CPU_PAGE_PROPERTY CPUPageProperty;
 D3D12_MEMORY_POOL MemoryPoolPreference;
 UINT CreationNodeMask;
 UINT VisibleNodeMask;
} D3D12_HEAP_PROPERTIES;

This structure would inform the device which kind of the physical memory should the heap refer to. Since the documentation of D3D12 is comprehensible enough, I’d rather not talk about too many things which have been listed there. When D3D12_HEAP_TYPE Type is not D3D12_HEAP_TYPE_CUSTOM, then the D3D12_CPU_PAGE_PROPERTY CPUPageProperty should be always D3D12_CPU_PAGE_PROPERTY_UNKNOWN, because the CPU accessibility of the heap has already been indicated by the D3D12_HEAP_TYPE so you shouldn’t repeat the information; Similar reason, D3D12_MEMORY_POOL MemoryPoolPreference should always be D3D12_MEMORY_POOL_UNKNOWN when D3D12_HEAP_TYPE Type is not D3D12_HEAP_TYPE_CUSTOM.

In UMA architecture, there is only one physical memory pool which is both shared by CPU and GPU, the most common case is that you got an Xbox One and start to write some D3D12 games on it. In such case only D3D12_MEMORY_POOL_L0 is available and thus we don’t need to take care of it at all.

The most of the desktop PC with a dedicated graphics card are NUMA memory architecture (although recent years there are something like AMD’s hUMA appeared and gone), in such case D3D12_MEMORY_POOL_L0 is the RAM and D3D12_MEMORY_POOL_L1 is the VRAM.

So now if we set the heap type to D3D12_HEAP_TYPE_CUSTOM, then we could have a more flexible control over the heap configuration. I’ll list a chart below that how different combination of D3D12_CPU_PAGE_PROPERTY and D3D12_MEMORY_POOL would finally look like on NUMA architectures.

	NOT_AVAILABLE	WRITE_COMBINE	WRITE_BACK
L0	Similar as `D3D12_HEAP_TYPE_DEFAULT`, a GPU access-only RAM (but a little bit non-sense configuration for common usage cases)	Similar as `D3D12_HEAP_TYPE_UPLOAD`, it is uncached for CPU read operation so the reading result won’t always stay coherent but write operation is faster because now the memory ordering is trivial and irrelevant, perfect for GPU to read	Similar as `D3D12_HEAP_TYPE_READBACK`, all the GPU write operation would be cached and CPU read operation would get a coherent and consistent result
L1	Similar as `D3D12_HEAP_TYPE_DEFAULT`, a GPU access-only VRAM	Invalid, CPU can’t access VRAM directly	Invalid, CPU can’t access VRAM directly

It looks like that we don’t need a custom heap property structure on NUMA architectures (or single engine/single adapter case), all possible heap types have been already provided by the pre-defined types, there is not too much space for us to maneuver in order to get some advanced optimization. But if your application wants any better customization for all the possible hardware that it would run on, then using custom heap properties is still worth enough to investigate.

And finally, we had a misc flag mask to indicate the detailed usage of the heap:

typedef enum D3D12_HEAP_FLAGS {
 D3D12_HEAP_FLAG_NONE,
 D3D12_HEAP_FLAG_SHARED,
 D3D12_HEAP_FLAG_DENY_BUFFERS,
 D3D12_HEAP_FLAG_ALLOW_DISPLAY,
 D3D12_HEAP_FLAG_SHARED_CROSS_ADAPTER,
 D3D12_HEAP_FLAG_DENY_RT_DS_TEXTURES,
 D3D12_HEAP_FLAG_DENY_NON_RT_DS_TEXTURES,
 D3D12_HEAP_FLAG_HARDWARE_PROTECTED,
 D3D12_HEAP_FLAG_ALLOW_WRITE_WATCH,
 D3D12_HEAP_FLAG_ALLOW_SHADER_ATOMICS,
 D3D12_HEAP_FLAG_ALLOW_ALL_BUFFERS_AND_TEXTURES,
 D3D12_HEAP_FLAG_ALLOW_ONLY_BUFFERS,
 D3D12_HEAP_FLAG_ALLOW_ONLY_NON_RT_DS_TEXTURES,
 D3D12_HEAP_FLAG_ALLOW_ONLY_RT_DS_TEXTURES
} ;

Depends on the specific D3D12_RESOURCE_HEAP_TIER that different hardware support, some certain D3D12_HEAP_FLAGS are not allowed to use alone or combine together. The furthermore detail is well documented on the official website so I’ll not discuss them here. Because some of the enums are just the alias to the others, the actual possible heap flags are less than how many it is defined, and I’ll list a chart below to demonstrate different usage cases and the corresponding flags.

	Tier1	Tier2
All resource types	Not supported	`D3D12_HEAP_FLAG_ALLOW_ALL_BUFFERS_AND_TEXTURES` or `D3D12_HEAP_FLAG_NONE`
Buffer only	`D3D12_HEAP_FLAG_DENY_RT_DS_TEXTURES	D3D12_HEAP_FLAG_DENY_NON_RT_DS_TEXTURES`
Non-RT/DS texture only	`D3D12_HEAP_FLAG_DENY_BUFFERS	D3D12_HEAP_FLAG_DENY_RT_DS_TEXTURES`
RT/DS texture only	`D3D12_HEAP_FLAG_DENY_BUFFERS	D3D12_HEAP_FLAG_DENY_NON_RT_DS_TEXTURES`
Swap-chain surface only	`D3D12_HEAP_FLAG_ALLOW_DISPLAY`	Same as Tier1
Shared heap (multi-process)	`D3D12_HEAP_FLAG_SHARED`	Same as Tier1
Shared heap (multi-adapter)	`D3D12_HEAP_FLAG_SHARED_CROSS_ADAPTER`	Same as Tier1
Memory write tracking	`D3D12_HEAP_FLAG_ALLOW_WRITE_WATCH`	Same as Tier1
Atomic primitive	`D3D12_HEAP_FLAG_ALLOW_SHADER_ATOMICS`	Same as Tier1

As you can see above, the only meaningful difference between Tier1 and Tier2 here is that Tier2 support a D3D12_HEAP_FLAG_ALLOW_ALL_BUFFERS_AND_TEXTURES flag thus we could put all the common resources into one heap. It again depends on what specific task you would like to finish, sometimes you want an all-in-one heap, sometimes it’s better to separate them into different heaps by the usage cases.

Resource creation

After you created a heap successfully, you could start to create resources inside it now. There are 3 different ways to create a resource:

Create resource which has only virtual address inside the already created heap, it requires us to map to the physical address manually later. ID3D12Device::CreateReservedResource is the interface for such a task;
Create resource which has both virtual address and mapped physical address inside the already created heap, the most commonly-used resources are this type. ID3D12Device::CreatePlacedResource is the interface for such a task;
Create placed-resource and an implicate heap at the same time. ID3D12Device::CreateCommittedResource is the interface for such a task.

If you don’t want to manually manage the heap memory at all, then you could choose to use committed-resource with some sacrifices to the performance, but naturally it’s not a good idea to stick with committed-resource heavily in the product code (unless you’re lazy like me who don’t want to write more code in show-case projects). The more mature choice is using placed-resources since we’ve already could create heaps, the only thing left that you have to do now is designing a heap memory management module with some efficient strategies. You could just use as many design patterns and architectures from the experience when you’re implementing the main RAM heap memory management system (still malloc() inside 16ms? No way!). A ring buffer or a double buffer for Upload heap or some linked-list for Default heap or whatever, there are no limitations for the imagination, just analysis your application requirement and figure out a suitable solution (but don’t write a messy GC system for it:). There shouldn’t be too many choices since in the most D3D12 applications like a game, the most of the resources are CPU write-once and others are dynamic buffers which won’t occupy too much space but update frequently.

The more advanced situation which rely on a tremendous memory size, such like mega-texture (maybe you need a 64x64 $km^2$ terrain albedo texture?) or sparse-tree volume textures (maybe you need a voxel-cone-traced irradiance volume?), which would index over the physical VRAM address easily or the actual texture size is beyond the maximum hardware support. In such cases a dynamic virtual memory address mapping technique is necessary. Developers intended to implement a software cache solution for this problem in the past because the APIs didn’t provide any reliable functionalities at that time (before D3D11.2 and OpenGL 4.4 which started to support tiled/sparse textures). The reserved-resources in D3D12 are the fresh new one-for-all solution today, it inherited the design of the tiled-resources architecture in D3D11 but also provided more flexibilities. But still, it depends on the hardware support when you wonder how to fit your elegant and complex SVOGI volume texture into the VRAM, it’s better to query D3D12_TILED_RESOURCES_TIER and see if the target hardware support tiled-resource or not at first.

So many descriptors in Vulkan

zhangandhang@gmail.com (Hang Zhang) — Sun, 21 Apr 2019 10:53:00 +0000

Brainwash is always somewhere

It’s really a mess when I started to port my engine to Vulkan, there are too many new data types that mapped to those came-from-nowhere concepts, which I don’t need to take care about previously. But luckily those concepts are well designed and once after you understand what they are, every pain you occurred would just disappear.

The new generation graphics APIs all transport the responsibility of CPU-GPU communication to the user more or less explicitly, now if you want to ask GPU to do something for you, there won’t be any already defined API, which you just need to feed in some data from your CPU and memory then all the others would be handled by “somebody”.

Let’s say, GPU knows absolutely nothing what you want to do as always, and that “(ex)-somebody”, previously it’s your graphics card vendor’s implementation of those graphics API, they did the “trivial” underlying pipeline works for you. They all have gone now, you have to take care of all what it did for you before. Sounds like a fairly bad break-up!

But what would you benefit from a break-up? (almost) Freedom. Now GPU is more like a general computing server, which exposure itself through the new generation of lower level APIs. As the client, we need to submit the computing work with a detailed enough work description, and then keep feed in data and commands following with the description which we signed with GPU before. But in practice, the most work I did like lots of people who found this article is rendering, and for this purpose, these new APIs were designed still with lots of rendering related specific concepts (because GPU is still “Graphic” Process Unit today:)).

But without considering the details, the whole bunch of things is easy to understand. What we need to do is just fit ourselves into the new CPU-GPU communication model. Create a work description, submit work, repeat, that’s all. Now it’s time to write the code, and you may want to say “whaaaaat” when you type “vk” inside your IDE if it has some kind of code autocomplete features. Yep, too many data types!

Ce este?

“…A descriptor is an opaque data structure representing a shader resource such as a buffer, buffer view, image view, sampler, or combined image sampler. Descriptors are organised into descriptor sets, which are bound during command recording for use in subsequent draw commands…” -13. Resource Descriptors, Vulkan® 1.1.106 - A Specification (with all published extensions)

If you asked me what is the most beneficial thing I got from the journey of writing a game engine, I would answer, “Don’t Panic”. One headache thing when I play with Vulkan is the descriptors, I can’t catch the meaning of it at the very beginning, because with an OpenGL mindset there is no corresponding concept.

But if you think in a fresh point of view, it’s really not a nonsense existence because we have to tell GPU the work description, like where are the resources, how shader will access them and in which kind of view since data are just some bytes inside the GPU memory. So, it’s really better to build a new mindset closer to the GPU pipeline.

“…Descriptors are grouped together into descriptor set objects. A descriptor set object is an opaque object that contains storage for a set of descriptors, where the types and number of descriptors is defined by a descriptor set layout. The layout object may be used to define the association of each descriptor binding with memory or other hardware resources. The layout is used both for determining the resources that need to be associated with the descriptor set, and determining the interface between shader stages and shader resources… -13.2. Descriptor Sets, Vulkan® 1.1.106 - A Specification (with all published extensions)

Actually, there isn’t a “descriptor” data type that we could interact with directly on the CPU side, as the specification said, it’s opaque. The workflow of creating the descriptors is designed as to create a combination of a set(VkDescriptorSet) handle, a buffer or image bound info(VkDescriptorBufferInfo/VkDescriptorImageInfo) and a write or copy operation(VkWriteDescriptorSet/VkCopyDescriptorSet). You acquire one set instance from a pool(vkAllocateDescriptorSets), and all of the set and pool have their own characteristics or usage hints which you would specific before creating them. These characteristics are typically configured with the layout(VkDescriptorSetLayout) and the create info(VkDescriptorSetAllocateInfo and VkDescriptorPoolCreateInfo). After you create the set you have to provide the info about what buffer or image it would bind to, and finally, update this information by a write or copy operation(vkUpdateDescriptorSets). The cons of a plain code example are it’s not so intuitive about the relation between data structure and functions, so I made a little flow graph to demonstrate.

graph TD
	subgraph "create descriptor pool"
 VkDescriptorPoolSize-.->VkDescriptorPoolCreateInfo
 VkDescriptorPoolCreateInfo==vkCreateDescriptorPool==>pool((VkDescriptorPool))
	end
	subgraph "create descriptor set layout"
 VkDescriptorSetLayoutBinding-.->VkDescriptorSetLayoutCreateInfo
	VkDescriptorSetLayoutCreateInfo==vkCreateDescriptorSetLayout==>layout(VkDescriptorSetLayout)
 end
 subgraph "create descriptor set"
	layout-.->VkDescriptorSetAllocateInfo
	pool-.->VkDescriptorSetAllocateInfo
	VkDescriptorSetAllocateInfo==vkAllocateDescriptorSets==>set(VkDescriptorSet)
 end
	subgraph "UBO"
	ubo(VkBuffer)-.->VkDescriptorBufferInfo
	end
	subgraph "Sampler"
	sampler(VkSampler)-.->VkDescriptorImageInfo
	end
	subgraph "update descriptor set"
	set-.->writeCopySets(VkWriteDescriptorSet/VkCopyDescriptorSet)
	VkDescriptorBufferInfo-.->writeCopySets
	VkDescriptorImageInfo-.->writeCopySets
	writeCopySets--vkUpdateDescriptorSets-->device(VkDevice)
 end
	subgraph "create pipeline layout"
	layout-.->VkPipelineLayoutCreateInfo
	VkPipelineLayoutCreateInfo==vkCreatePipelineLayout==>pipelineLayout(VkPipelineLayout)
	end
	subgraph "create pipeline"
	pipelineLayout-.->VkGraphicsPipelineCreateInfo
	VkGraphicsPipelineCreateInfo==vkCreateGraphicsPipelines==>pipeline(VkPipeline)
	end

All the thick line indicate the real object instance is created by a function invocation, while the dotted line means the dependency between the info. I omitted the other dependencies of creating a VkPipeline since they are not related to the topic I’m talking about.

Actually now you may start to feel Vulkan has a really clean architecture model, indeed it is. We now have a far more flexible possibility that we could have as many descriptors in many different descriptor sets in many descriptor pools with many different combinations of configurations. One thing that breaks a little bit of the name convention is the VkWriteDescriptorSet and VkCopyDescriptorSet, they should and only could be created by user directly and submit later rather than acquire from the vKDevice (I was thinking why there isn’t a VkWriteDescriptorSetCreateInfo kind stuff but it would be too redundant to create a VkWriteDescriptorSet, because after all, it’s about an operation around the descriptor, or maybe better call it VkDescriptorSetWriteOp?).

Some usage cases

Let’s have a look at some code examples, which are all coming from some real scenarios I occurred before.

Single UBO data accessed per shader stage

I have a UBO for main camera related data like the projection matrix, only update once per frame, the GLSL code like this:

layout(std140, row_major, set = 0, binding = 0) uniform cameraUBO
{
	mat4 uni_p_camera_original;
};

Then it’s better to answer some questions before creating the descriptor-related data:

How many descriptors will we have inside the pool? Only one.
Which kind of resource type it will be used for? For uniform buffer.
How many different type and number of descriptors will be allocated from this pool? Only one type and only descriptor will it hold.
How many sets it could hold at all? Only one.

VkDescriptorPoolSize l_poolSize = {};
l_poolSize.type = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER;
l_poolSize.descriptorCount = 1;

VkDescriptorPoolCreateInfo l_poolInfo = {};
l_poolInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO;
l_poolInfo.poolSizeCount = 1;
l_poolInfo.pPoolSizes = l_poolSize;
l_poolInfo.maxSets = 1;

Create VkDescriptorPool.

VkDescriptorPool l_pool;
vkCreateDescriptorPool(m_device, &l_poolInfo, nullptr, &l_pool);

Where it will be bound to with the shader? Binding point 0.
How many descriptors will be bound? Only one.
Which shader stage could access it? Vertex shader.

VkDescriptorSetLayoutBinding l_setLayoutBinding = {};
l_setLayoutBinding.binding = 0;
l_setLayoutBinding.descriptorCount = 1;
l_setLayoutBinding.descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER;
l_setLayoutBinding.pImmutableSamplers = nullptr;
l_setLayoutBinding.stageFlags = VK_SHADER_STAGE_VERTEX_BIT;

VkDescriptorSetLayoutCreateInfo l_layoutCreateInfo = {};
l_layoutCreateInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO;
l_layoutCreateInfo.bindingCount = 1;
l_layoutCreateInfo.pBindings = l_setLayoutBinding;

Create VkDescriptorSetLayout.

VkDescriptorSetLayout l_setLayout;
vkCreateDescriptorSetLayout(m_device, &l_layoutCreateInfo, nullptr, &l_setLayout);

Create VkDescriptorSet.

VkDescriptorSetAllocateInfo l_allocInfo = {};
l_allocInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO;
l_allocInfo.descriptorPool = l_pool;
l_allocInfo.descriptorSetCount = 1;
l_allocInfo.pSetLayouts = &l_setLayout;

VkDescriptorSet l_set;
vkAllocateDescriptorSets(.m_device, &l_allocInfo, &l_set)

Which resource it will be bound to? A UBO.
Where to bind? Binding point 0.
Which VkDescriptorSet that the write operation targeted at? The one we just created.

VkDescriptorBufferInfo l_bufferInfo = {};
l_bufferInfo.buffer = m_cameraUBO; // created from somewhere else
l_bufferInfo.offset = 0;
l_bufferInfo.range = sizeof(CameraGPUData);

VkWriteDescriptorSet l_writeDescriptorSet = {};
l_writeDescriptorSet.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET;
l_writeDescriptorSet.dstBinding = 0;
l_writeDescriptorSet.dstSet = l_set;
l_writeDescriptorSet.dstArrayElement = 0;
l_writeDescriptorSet.descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER;
l_writeDescriptorSet.descriptorCount = 1;
l_writeDescriptorSet.pBufferInfo = &l_bufferInfo;

vkUpdateDescriptorSets(
		m_device, // VkDevice handle
		1, // only 1 VkWriteDescriptorSet
		&l_writeDescriptorSet,
		0,
		nullptr);

Then when I need to submit command, the only thing left is a call to vkCmdBindDescriptorSets:

	vkCmdBindDescriptorSets(
	&m_commandBuffer,
	VK_PIPELINE_BIND_POINT_GRAPHICS,
	m_pipelineLayout,
	0, // the first set is set 0
	1, // and one set only
	&l_descriptorSet, 0, nullptr);

Array UBO data accessed per shader stage

My punctual light data is inside an array which contains all the information and will be updated to GPU once per frame, but I’ll iterate through the array for a deferred style light pass inside the fragment shader. The GLSL code like this:

#define MAX_POINT_LIGHT 64
// w component of luminance is attenuationRadius
struct pointLight {
	vec4 position;
	vec4 luminance;
	//float attenuationRadius;
};

layout(set = 0, binding = 2) uniform pointLightUBO
{
	pointLight uni_pointLights[MAX_POINT_LIGHT];
};

The only things change in C++ code is the binding point and the buffer range, since it’s an array.

#define MAX_POINT_LIGHT 64
VkDescriptorSetLayoutBinding l_setLayoutBinding = {};
l_setLayoutBinding.binding = 2;

VkDescriptorBufferInfo l_bufferInfo = {};
l_bufferInfo.buffer = m_pointLightUBO; // created from somewhere else
l_bufferInfo.offset = 0;
l_bufferInfo.range = sizeof(PointLightGPUData) * MAX_POINT_LIGHT; // the total size of the UBO array

VkWriteDescriptorSet l_writeDescriptorSet = {};
l_writeDescriptorSet.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET;
l_writeDescriptorSet.dstBinding = 2;

The command submit is exactly the same as the previous example:

	vkCmdBindDescriptorSets(
	&m_commandBuffer,
	VK_PIPELINE_BIND_POINT_GRAPHICS,
	m_pipelineLayout,
	0, // the first set is set 0
	1, // and one set only
	&l_descriptorSet,
	0,
	nullptr);

Array UBO data accessed per draw call object

The local-to-world space transformation matrix needs to be updated per object, and for this what we could use is the Dynamic Uniform Buffer (forget your glUpdateUniform* things!), I’ll update a UBO array per frame which contain all the drawable meshes information, and will use an offset to access the corresponding part per draw call later.

layout(std140, row_major, set = 0, binding = 1) uniform meshUBO
{
	mat4 uni_m;
};

Now in C++ code, I need to specify a different descriptor type, also a different binding point because I bind other resources at binding point 0, but it’s trivial.

VkDescriptorPoolSize l_poolSize = {};
l_poolSize.type = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER_DYNAMIC;

VkDescriptorSetLayoutBinding l_layoutBinding = {};
l_layoutBinding.binding = 1;
l_layoutBinding.descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER_DYNAMIC;

VkDescriptorBufferInfo l_bufferInfo = {};
l_bufferInfo.buffer = m_meshUBO;
l_bufferInfo.offset = 0;
l_bufferInfo.range = sizeof(MeshGPUData);

Since it would be accessed in a dynamic favor, the buffer range is per block rather than the whole UBO data array.

And when bind the DescriptorSet, we need to specify the dynamic offset, I use such an implementation:

unsigned int l_blockSize = sizeof(MeshGPUData);

for (int i = 0; i < total_meshes_this_frame; i++)
{
	auto l_offset = l_blockSize * i;

	vkCmdBindDescriptorSets(
	&m_commandBuffer,
	VK_PIPELINE_BIND_POINT_GRAPHICS,
	m_pipelineLayout,
	0, // the first set is set 0
	1, // and one set only
	&l_descriptorSet,
	1, // Now we have one dymanic offset
	&l_offset // the offset value
	);

	// draw call
	//...
	//
}

Multiple array UBO data accessed per draw call object

This is the combination of the previous situations, the mesh UBO and material UBO is related to each draw call, but the camera UBO is one frame one update, but still, we could achieve this.

The Vertex shader looks like this:

layout(std140, row_major, set = 0, binding = 0) uniform cameraUBO
{
	mat4 uni_p_camera_original;
};

layout(std140, row_major, set = 0, binding = 1) uniform meshUBO
{
	mat4 uni_m;
};

While the fragment shader looks like this:

layout(std140, set = 0, binding = 2) uniform materialUBO
{
	vec4 uni_albedo;
	vec4 uni_MRAT;
};

Now the C++ code:

How many descriptors will we have inside the pool? Now 3.
Which kind of resource type it will be used for? For normal uniform buffer and dynamic uniform buffer.
How many different type and number of descriptors will be allocated from this pool? 2 types 3 descriptors.
How many sets it could hold at all? Now, still 1.

VkDescriptorPoolSize l_cameraUBOPoolSize = {};
VkDescriptorPoolSize l_meshUBOPoolSize = {};
VkDescriptorPoolSize l_materialUBOPoolSize = {};

l_cameraUBOPoolSize.type = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER;
l_cameraUBOPoolSize.descriptorCount = 1;

l_meshUBOPoolSize.type = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER_DYNAMIC;
l_meshUBOPoolSize.descriptorCount = 1;

l_materialUBOPoolSize.type = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER_DYNAMIC;
l_materialUBOPoolSize.descriptorCount = 1;

VkDescriptorPoolSize l_UBOPoolSizes[] = { l_cameraUBOPoolSize , l_meshUBOPoolSize, l_materialUBOPoolSize };

VkDescriptorPoolCreateInfo l_poolInfo = {};
l_poolInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO;
l_poolInfo.poolSizeCount = 3;
l_poolInfo.pPoolSizes = l_UBOPoolSizes;
l_poolInfo.maxSets = 1;

Other parts are all the same, just change the binding points and the buffer info. And when bind the DescriptorSet, the dynamic offset is an array now:

unsigned int l_meshDataBlockSize = sizeof(MeshGPUData);
unsigned int l_materialDataBlockSize = sizeof(MeshGPUData);

for (int i = 0; i < total_meshes_this_frame; i++)
{
	auto l_meshOffset = l_meshDataBlockSize * i;
	auto l_materialOffset = l_materialDataBlockSize * i;
	unsigned int l_offsets[] = { l_meshOffset, l_materialOffset };

	vkCmdBindDescriptorSets(
	&m_commandBuffer,
	VK_PIPELINE_BIND_POINT_GRAPHICS,
	m_pipelineLayout,
	0, // the first set is set 0
	1, // and one set only
	2, // Now we have two dymanic offsets
	&l_offsets // the offset value array
	);

	// draw call
	//...
	//
}

That’s it! The sampler descriptor is similar to uniform buffer, all the rules work as well as what I applied above. Freedom means more responsibility and caution, but we now have more possibility to optimize the whole rendering pipeline to a new level of efficiency, I would keep investigating how to utilize the power of Vulkan in a more concise way, after all, it’s quite a more complex API than OpenGL!

Welding (THE latest) C++ to Metal, and macOS

zhangandhang@gmail.com (Hang Zhang) — Wed, 17 Apr 2019 19:07:00 +0000

For what?

I’ve previously deployed my game engine to macOS successfully with GLFW and macOS’s “vanilla” version OpenGL 4.1 library, everything was pretty smooth and sweet (and of course naive) at that time, and every few weeks I would port the latest updates from Windows side to macOS in an accumulated way. But as I started to utilize some C++17 new features, yep I mean <filesystem>, which Apple still didn’t give us a certain answer (I’ve heard that there is something new with Apple Clang in the Xcode 10.2 update, that somehow <experimental/filesystem> is magically appearing, but later I’ll talk about it separately), and targeting OpenGL version to the almost latest 4.5 for quite few new features, I have to deprecate macOS for some moments because, why not?

So the frustrated feeling always tangles around when I started to think about anything related to macOS until one day I realized maybe I should just forget about Apple Clang, it’s not the only fruit on the tree. Then I tested the LLVM/Clang package provided by Homebrew (since I use GCC on Linux, it’s better to have another compiler/linker choice on macOS for a wider test scenario), and of course, it works.

But the problem related to OpenGL is still quite unsolvable, and when I heard about macOS 10.14 would deprecate OpenGL officially, there was really a moment that I started to hesitate maybe it’s time to say bye-bye to macOS. I wanted to use Metal, but I really didn’t figure out how to integrate a Swift written module into C++. Then, I threw the idea about the macOS porting to a dusty corner as same as how Apple did with OpenGL decades ago.

And recently I removed the engine’s dependency on GLFW, because I need to use Qt as the editor framework, then choose an architectural design with the native OS window event handling is better and a little bit more flexible. After 2 or 3 days refactoring, I successfully decoupled the entire window event system on Windows with GLFW and plugged Qt into it smoothly (ignore the annoying QtNativeEvent). And then I have to repeat the similar procedure for Linux (currently means Ubuntu), I need to write a native window event system for user input and display there too.

Then how?

And for the sake of these reasons I mentioned above macOS became a far more impossible direction. Previously GLFW wrapped the platform window for me, now what should I do there without it? But GLFW provides a C++ API while standard macOS application is often written in Objective-C (or Swift recently), thus I assume it should have some bridging method to achieve this. So I googled a little bit and saw a weird thing called, Objective-C++.

If you googled that name as same as what I did, then you should have had a brief understanding of how to “connect” Objective C to C++. We just need to configure something on the build pipeline then we could build an Objective C source file mixed with C++ code together. In practice our task is just simply to rename your Objective-C source file’s extension from .m to .mm, and then the XCode build toolchain would interpret it as an Objective-C++ file, and it would try to compile it with a mixed favor (Or you could change to recognize any file explicitly as an Objective-C++ file in your Xcode IDE).

Then I started to build a native Cocoa window accidentally and, why not get some Metal? But I want to integrate the Metal rendering module into my current rendering backend architecture, rather than alter anything in case to fit them. So basically speaking what I need is a bidirectional communication solution between the C++ engine code and the Objective-C code. Use Objective-C++ could only solve the problem on Objective-C side, it’s always impossible to put any Objective-C-language-set-exclusive code in C++ source file directly.

Luckily this kind of bridging hooking attaching mapping snapping problem should be a common pain because I’ve found some nice example and article easily about how to deal with it. The basic idea is using inheritance (OOP is not always so bad taste!) to encapsulate the Objective-C object into a C++ object, and provide some virtual functions as the interface. Then it just becomes a solvable problem, I just need to wrap the window handle into an Objective-C++ class which inherits from a C++ class, and pass the handle or pointer of it to C++ side in runtime.

This is how an Objective-C++ header file looks like, it contains an Objective-C++ class declaration, while it looks just as same as a 100% pure C++ class who inherits from an abstract bridge class of the engine (the private member pointers are Objective-C object pointers, I would explain why it is here later):

#define NO_ANY_OBJECTIVE_C_SXXT_HERE
class MacWindowSystemBridgeImpl : public MacWindowSystemBridge
{
public:
 explicit MacWindowSystemBridgeImpl(MacWindowDelegate* macWindowDelegate, MetalDelegate* metalDelegate);
 ~MacWindowSystemBridgeImpl();

 bool setup(unsigned int sizeX, unsigned int sizeY) override;
 bool initialize() override;
 bool update() override;
 bool terminate() override;

 ObjectStatus getStatus() override;
private:
 ObjectStatus m_objectStatus = ObjectStatus::SHUTDOWN;
 MacWindowDelegate* m_macWindowDelegate = nullptr;
 MetalDelegate* m_metalDelegate = nullptr;
};

And the .mm source file is mixed with the C++ function declaration and Objective-C implementation, “bridging” happens here:

// Personally I feel O-C is interesting to write painful to read
bool MacWindowSystemBridgeImpl::setup(unsigned int sizeX, unsigned int sizeY) {
 NSRect frame = NSMakeRect(0, 0, sizeX, sizeY);

 [m_macWindowDelegate initWithContentRect:frame
 styleMask:NSWindowStyleMaskTitled|NSWindowStyleMaskClosable
 backing:NSBackingStoreBuffered
 defer:NO];

 return true;
}
// And so on...

And it’s quite a shame to talk about, I finally knew Metal support Objective-C at that time as late as I started to search about the window solution. But it’s quite an additional inspire for me because all the barrier on the macOS porting seems would disappear sooner. Now the Objective-C side objects are created in such a way:

// The main entry
int main(int argc, const char * argv[]) {
 [NSApplication sharedApplication];
 AppDelegate* appDelegate = [[AppDelegate alloc] init];
 [NSApp setDelegate:appDelegate];
 [NSApp run];
 return 0;
}

//......
//......
//......
// the AppDelegate initialization, allocate the Objective-C objects

- (id)init
{
 m_macWindowDelegate = [MacWindowDelegate alloc];
 m_metalDelegate = [MetalDelegate alloc];


	// Allocate Objective-C++ bridge implementation class instances, the simple new operator allocation is quite enough, just remember to delete them in dtor

 m_macWindowSystemBridge = new MacWindowSystemBridgeImpl(m_macWindowDelegate, m_metalDelegate);
 m_metalRenderingSystemBridge = new MTRenderingSystemBridgeImpl(m_macWindowDelegate, m_metalDelegate);

 // Start the engine c++ code
	// Pass the bridge instance pointers to the engine
 const char* l_args = "-renderer 4 -mode 0";
 if (!InnoApplication::setup(m_macWindowSystemBridge, m_metalRenderingSystemBridge, (char*)l_args))
 {
 return 0;
 }
 if (!InnoApplication::initialize())
 {
 return 0;
 }


	// the classic while-loop for update
 [NSTimer scheduledTimerWithTimeInterval:0.000001 target:self selector:@selector(drawLoop:) userInfo:nil repeats:YES];

 return self;
}

And now I’ve implemented a bridger (or you could call it as “Adapter Pattern” or “Adopter Pattern” or “Avenger Pattern” or something else, I’m really bad at enumerating those design pattern name) for the communications we imagined before. Native window and Metal Objective-C objects are created by the Objective-C entrance application, and then use them to construct Objective-C++ class instances and pass the pointer as pure C++ classes to the engine’s runtime, everything works so perfectly!

Devils in the details???

And of course, there are some disadvantages beneath. The first thing which needs extra care about is the memory and the life cycle of the Objective-C object. Since the Objective-C has a so-called ARC some sort of reference counter style memory management stuff, in order to safely maintain the life cycle of them, I would never pass any Objective-C object pointer directly to C++ code in case someone would delete it by accident. Instead, it is better to use a std::shared_ptr (or std::make_shared<T> directly) to wrap the raw Objective-C object we created to pass it to C++ object. Or a better and safer solution is, that we’d never inform C++ side anything about Objective-C object, like my example above. But if we have to dance next to the cliff necessarily, then we could use the language keyword like __bridge_retained and __bridge_transfer to transfer the ownership of the raw Objective-C object back and forth:

id<MTLBuffer> oldOCObjectHandle;
// ...Assign a value to oldOCObjectHandle
void* newCppObject = (__bridge_retained void*)(oldOCObjectHandle);
id<MTLBuffer> newOCObjectHandle = (__bridge_transfer id<MTLBuffer>)(cppObject);

And furthermore, as you may have had discovered in my example, actually I add another indirection in between a C++ declaration and an Objective-C implementation. Because if you just pass your Objective-C++ object by a safer std::shared_ptr to C++, you would still occur some weird problems because the object is not always “there” in memory! Maybe its reference count was reset to 0 at some moments, or maybe some garbage collector or defragiler works around and then it was moved to another location, I didn’t dig into the reason so deeply, as the solution what I’ve figured out, that I would choose to use an Objective-C++ object as the bridge like what we’ve talked about in the original design, but don’t implement any Objective object-related features in it directly, instead it just owns a pure Objective object’s reference (I use pointer actually), and dispatch all the invocation to that object.

Another not so bad “side-effect” when I removed the dependency on GLFW is, I could enjoy all the latest C++ features without worry about anything. Because now the platform entrance module is compiled by Apple Clang, and the engine module is compiled by LLVM/Clang, just link the entrance module with the engine module I would have a fully functional application, kill two birds with one stone!

So now the final solution looks like this:

st=>start: C++ Module
e=>end: Objective-C Module
op1=>operation: Objective-C++ Module
st->op1->e

The possible burden would relate to the indirection cost and the code redundancy, in this kind of design if you want any interactions in between, you may have to write lots of wrapper functions, and it quite reminds me of a similar thing which everybody used at least for a while when they were learning 3D programming, da, GLAD/GLEW/GLBLABLA these OpenGL function loader:).

Normal and normal mapping

zhangandhang@gmail.com (Hang Zhang) — Sun, 18 Nov 2018 01:34:00 +0000

A (not) tedium work

Recently I started to port my project to DirectX 11, and it has a lot of interesting differences with OpenGL such like the coordinates and matrix convention, stronger type safety requirement, better shader resources management (I didn’t try OpenGL’s SSBO yet but the constant buffer is really easier to wrap into an elegant layer) and etc. One thing I stuck a little is the normal mapping there since I used the on-the-fly tangent generation in GLSL, it quite confuses me at first when I rewrote it in HLSL.

Always Review

Basically the normal vector is interpreted as one unit direction vector who is perpendicular with a surface (or mathematically speaking in general, normal vector $\vec{n}$ is one gradient vector of the gradient $\nabla f = \left\langle {{f_x},{f_y},{f_z}} \right\rangle$ of a scalar field $f\left( {x,y,z} \right) = k$ in a vector space at a certain point $\left( {{x_0},{y_0},{z_0}} \right)$, who is orthogonal/normal always with the field), and with this surface normal data in hand we could achieve some old-style flat shading which only gives us some discrete results of the surface color. Later Gouraud (maybe not him) invented the vertex normal, as the average of the surface normals where the vertex is in, it could give a smoother shading result with the nature of interpolation in the pixel processing stage.

Typically we would get the precomputed normal vector of the models from those DCC tools such as Blender 3DS Max or Maya (or generated with some cross-product on CPU side by your own), and they are stored in the local space of the model. Then if we want to render anything with its help in our world space (strictly speaking finally in screen space), we need to transform these normal vectors.

The first idea is just to use the model’s local to world transformation matrix to transform the normal vector since it represents a direction, its w component is 0 in the homogeneous 4D space then the translation part (the last column of transformation matrix) won’t have any effect on it. And the rotation part is what we want to apply, but if we scaled an un-unified extension to the model, then the normal vector would be sheared too, the direction changes! So we need to figure out how to cancel the unwanted scaling only on normal vectors, multiple solutions here:

Multiply with the inverse of the scale matrix $N_{ws} = S^{-1} * M * N_{ls}$;
Without multiply the transformation matrix at the first, instead, just multiply a rotation matrix with local space normals $N_{ws} = R * N_{ls}$;
A classic trick called “invert transpose normal matrix”, which means use the transpose of the inverse of the model’s local to world transformation matrix to multiply with the local space normals $N_{ws} = M^{-1T} * N_{ls}$. Since it looks less intuitive, you may ask why this works, well, with the inverse operation we could cancel the scaling, and because the scaling part is in the diagonal of the matrix (the coordinate system’s axes actually), so transpose operation didn’t affect anything. And since rotation matrix is an orthogonal matrix its inverse is its transpose, then the transpose operation just inverse again the rotation part, as a conclusion means we first invert it, then transpose it, same as invert it and invert again, which means nothing happened to the rotation part! Let’s write it in formula: $M^{-1T} = S^{-1T} * R^{-1T} * T^{-1T}$, since $S^{-1T} = S^{-1}, R^{-1} = R^{T}$, then $M^{-1T} = S^{-1} * R * T^{-1T}$. In practice we often shrink the 4x4 matrix to 3x3, just use the upper left part, which removes the useless translation components as $M_{3 \times 3}^{-1T} = S^{-1} * R$, and with this matrix, we could achieve our goal.

Since most of the rendering pipelines(as far as I know) are designed to deal with scene transform on the CPU side, and always passes the already multiplied transformation matrix to GPU, then the method 3 is used commonly. But for HLSL it doesn’t provide a native inverse function like GLSL, that means whether you have to build a special hard-coded version for this purpose or inverse it previously on the CPU side. And with respect to this little cons the method 2 looks like another good choice because you don’t need any inverse operation on the GPU, the cost changes to an additional cbuffer data passed to GPU. Currently, I choose to use method 2, unless it hits me with some painful issues I think I won’t change to method 3 in DirectX.

Messy micro

And then the normal mapping became a little bit annoyed. The technique to store the geometry detail as the offsets in surface/vertex’s tangent space was invented by Blinn in the ’70s, which we called normal mapping commonly now, it is one kind of bump mapping techniques which could give a huge improvement about the surfaces detail without adding more vertices to hurt the performance. Since the common practice is storing normal texture in tangent space, unless we could get the tangent space-to-model’s local/world space transformation, we can’t apply it to our vertex normals. This means we need to construct a space that treats the vertex normals as the positive Z axis, then transform the normal texture data to world space, or transform the other light/position data to tangent space with its inverse matrix. There are still some different approaches:

Use precomputed tangent vector combining with normal vector to calculate the bitangent vector, and use these 3 as the axes to construct a tangent/TBN space;
Compute tangent vector on-the-fly by using texture UV coordinates and vertices data. Since the edge direction of the triangle could be calculated by the vertex position, and at the same time it also could be constructed by the TBN space’s T and N axis and the UV coordinates, then we could combine them together to get the T and N, in formula it is $\vec{E} = \vec{p_1} - \vec{p_2}$, $\vec{E} = \Delta \vec{U}T + \Delta \vec{V}B$, then this is a solvable linear algebra problem.

In practical, if we choose method 1, then we have to precompute the tangent vector and store it offline somewhere, then passed to GPU in real-time. And if we choose method 2, we need to calculate it on CPU side then send to GPU, basically, it has no implementation difference with method 1. I chose method 2 and implement it on shader directly, rather than the original idea to compute it on CPU, it would spend less bandwidth and make the vertex data structure tighter. The idea comes from a nice blog post, it utilizes the built-in shader partial derivative functions to calculate the screen space gradient, then does some cross-product to get the final results. But since OpenGL users commonly use RHS and DirectX users commonly use LHS, also because of the different windows/texture coordinate system of them, it confused me a lot at the beginning.

The texture coordinates of OpenGL starts from bottom-left corner, and when submitting 2D texture data array (1D in memory) to OpenGL, it would fill the texture buffer from the bottom-left corner to top; In DirectX the texture coordinates starts from top-left corner, and DirectX texture buffer would be filled from top-left corner to bottom. That means if you sample a texture data with same coordinates, OpenGL and DirectX would return the same results. And this means actually for every loaded textures data, I don’t need to change anything related with UV coordinates. The misinterpretation of this led me to a wrong UV convert parser at the first.

Then the partial derivative functions, this is easier to map from OpenGL to DirectX, the ddx_fine() and ddy_fine() are exactly same as dFdx() and dFdy(). But then the T and B axes construction is a little bit tricky, I choose to use LHS in DirectX, then have to flip the T and B when the other implementation details are as same as OpenGL. Also, the final TBN 3x3 matrix in OpenGL and DirectX would be as the transpose to each other but meanwhile staying unified with each other’s math convention, so I just need to follow the matrix-vector multiplication rules on each side as usual and always.

Here are some code pieces about:

	// get edge vectors of the pixel triangle
 vec3 dp1 = dFdx(thefrag_WorldSpacePos.xyz);
 vec3 dp2 = dFdy(thefrag_WorldSpacePos.xyz);
 vec2 duv1 = dFdx(thefrag_TexCoord);
 vec2 duv2 = dFdy(thefrag_TexCoord);

	// solve the linear system
 vec3 N = normalize(thefrag_Normal);
 	vec3 dp2perp = cross(dp2, N);
 	vec3 dp1perp = cross(N, dp1);
 	vec3 T = normalize(dp2perp * duv1.x + dp1perp * duv2.x);
 	vec3 B = normalize(dp2perp * duv1.y + dp1perp * duv2.y);

 mat3 TBN = mat3(T, B, N);

	vec3 WorldSpaceNormal = normalize(TBN * (texture(uni_normalTexture, thefrag_TexCoord).rgb * 2.0 - 1.0));


	// get edge vectors of the pixel triangle
	float3 dp1 = ddx_fine(input.thefrag_WorldSpacePos);
	float3 dp2 = ddy_fine(input.thefrag_WorldSpacePos);
	float2 duv1 = ddx_fine(input.thefrag_TexCoord);
	float2 duv2 = ddy_fine(input.thefrag_TexCoord);

	// solve the linear system
	float3 N = normalize(input.thefrag_Normal);

 float3 dp2perp = cross(dp2, N);
 float3 dp1perp = cross(N, dp1);
 float3 T = -normalize(dp2perp * duv1.x + dp1perp * duv2.x);
 float3 B = -normalize(dp2perp * duv1.y + dp1perp * duv2.y);

 float3x3 TBN = float3x3(T, B, N);

 float3 normalInWorldSpace = normalize(mul(t2d_normal.Sample(SampleTypeWrap, input.thefrag_TexCoord).rgb * 2.0f - 1.0f, TBN));

Physically Based Rendering - Material

zhangandhang@gmail.com (Hang Zhang) — Sun, 12 Aug 2018 17:36:00 +0000

I learned the shading magic like how the most of others did, from a classic Blinn-Phong model to some more complex models like Cook-Torrance model, but the understanding would always be confined around the common practices shaped by our slow speed computer, and our eager desires for the more realistic results. With these compensations, tricks, and hacks we may achieve lots of cool and amazing stuff at first, but if we don’t have a clear bird’s-eye view (for example people like me who is always suffering among the formulae), then we would lose ourselves inside just shader code and strange artifacts often and can’t find the solutions without painful testing and doubting.

Then I was thinking, why not implement and learn something from the basic of the physics again? Let’s forget about a little what I’ve already written and start to construct a shading pipeline from scratch, following the physical interpretations, then seeking for solutions to run between each 16 ms. Let’s go back to the start.

So, what is light?

“…In physical optics, light is modeled as an electromagnetic transverse wave, a wave that oscillates the electric and magnetic fields perpendicularly to the direction of its propagation…” - pg. 293, “Real-Time Rendering”, 4th Edition.

(The book already has become the bible of me, it’s a nice working reference, a balanced textbook and a well-formed dictionary, I’ll quote a lot from it later in this ctrl-c + ctrl-v style post (Why not?))

Light, a kind of electromagnetic wave whose property shaped by its wavelengths majorly, is one of the fundamental beings in our universe, and all of our beautiful rendering works would focus on how to calculate the physical transmission of it always. But what about to think in this way, just because it could be received by our eyes (a psychophysical phenomenon) then it had a special name as “light”, and if we could treat it not so special in our theoretical discussion would something become easier and more generally to handle with?

Let’s say, the basic electromagnetic mechanism works well too with light, all and no exceptions (of course it is), then we could just model a general solution in the computer, and feed it with the light wavelengths $\lambda$ we emitted from some origins, and other relating properties of some objects which stops the light propagation to calculate the result (may also contain the influences from the space they are in later, now let’s just work inside an ideal static vacuum). The classic physics model of the electromagnetic mechanism is described by Maxwell’s equations successfully, and with it we could figure out how the electric field propagating through space, thus means to get the correct phenomenon of the electromagnetic wave in our simulation box.

But unfortunately, we don’t have such huge computational resources to run a field propagation simulation in real-time inside our personal computer, representing a non-analytical electric field also requires a 3D data structure at least, we must allow some compensations. Luckily, physicists have already simplified the scope of the problems for us, instead of using the original electromagnetic field related methods, we could try to use the radiation methods to focus on the energy property only, and this would bring us a simplification of only considering about 2 things, the emitter and the receiver of the electromagnetic wave.

A new measurement

“…Light waves are emitted when the electric charges in an object oscillate. Part of the energy that caused the oscillations—heat, electrical energy, chemical energy—is converted to light energy, which is radiated away from the object. …” - pg. 296, “Real-Time Rendering”, 4th Edition.

Then we would arrive at Radiometry, which deals with the measurement of electromagnetic radiation. For the common usage, there are some radiometric quantities which represent the measurement of the electromagnetic radiation energy with respects to other basic physical quantities such like time/distance/area/angle, I’ll list some of them below which are more important to our rendering business, with the reference from the book and Wikipedia.

Quantity		Unit		Notes
Name	Symbol	Name	Symbol
Radiant energy	${Q_e}$	joule	${J}$	Energy of electromagnetic radiation
Radiant flux	${\Phi_e}$	joule per second or watt	${\frac{J}{s}}$ or ${W}$	also called Radiant power
Radiant intensity	${I_e}$	watt per steradian	${\frac{W}{sr}}$	steradian is similar with angle in 2D space
Irradiance Flux density	${M_e}$ or ${E_e}$	watt per square metre	${\frac{W}{m^2}}$	Ir-radiance, means the radiance received by a surface
Radiance	${L_e}$	watt per steradian per square metre	${\frac{W}{m^2·sr}}$

We won’t directly calculate Radiant energy, because usually we would handle with some objects which has shape and some events which happens during a period, it’s more convenient to cancel these quantities at first, that means we better choose to use Radiant intensity, Irradiance Flux density and Radiance.

To evaluate how a light source emitted light or energy, we need to define what is a light source at first. In this blog post, I would choose to discuss the ideal punctual light source only, which has an infinity small shape, the same as a point in space. Also, I would idealize it with an omnidirectional radiation characteristic, which means the radiation wouldn’t variant around different steradians.

We could deduce the definition of Radiant Intensity first, imagine a unit sphere surrounding the point light source, a unit steradian would emit some energy per second, then $I_e = \frac{d\Phi}{d\omega}$.

Similar, we could get Irradiance Flux density (or call it Radiant exitance if we want to emphasize it’s more about emitting rather than receiving, but it assumes we use some area light sources), $M_e = \frac{d\Phi}{dA}$.

And then Radiance is $L_e = \frac{d\Phi}{d\omega} * \frac{d\Phi}{dA \cos\theta} = \frac{d^2\Phi}{ d\omega dA \cos\theta}$, we now need to consider about the angle between the surface normal and the unit steradian because the actual effective area is not same as the original unit area, it is projected ($A\cos\theta$ is called projected area), so here we add a cos to it. This kind of cos-weighted distribution is called Lambert’s cosine law.

Through time and space, once and forever?

“…The oscillating electrical field pushes and pulls at the electrical charges in the matter, causing them to oscillate in turn. The oscillating charges emit new light waves, which redirect some of the energy of the incoming light wave in new directions. This reaction, called scattering, is the basis of a wide variety of optical phenomena. …” - pg. 297, “Real-Time Rendering”, 4th Edition.

If we emit any light/energy in vacuum, it would never change the direction and the energy unless meets with some obstructions (for example some molecules or small dust floating in space or some large giant gas planets like Saturn, or our body/eyes), then it would be absorbed or scattered due to the characteristics of the obstructions. Actually, this is all what we need to care about, the obstructions IS exactly the receiver! Then if we could model how the receiver it is we would solve the problem. But again, the limitation of our computational resources doesn’t allow us to simulate every molecule, instead, we have to follow some macro scope rules to abstract them. We call these single or group of trivial shape objects which influence the wave propagation as particles, and the volume the particles fulfilled with as media.

We live on Earth where the air and water are the most common noticeable medium, who gives us an inspiration for aesthetic creations for thousands of years. For to measure how they influence the light propagation, we need to define a kind of ratio between the original light and the affected light.

“…The ratio of the phase velocities of the original and new waves defines an optical property of the medium called the index of refraction (IOR) or refractive index, denoted by the letter $n$. Some media are absorptive. They convert part of the light energy to heat, causing the wave amplitude to decrease exponentially with distance. The rate of decrease is defined by the attenuation index, denoted by the Greek letter $\kappa$ (kappa). Both n and $\kappa$ typically vary by wavelength. Together, these two numbers fully define how the medium affects light of a given wavelength, and they are often combined into a single complex number $n + i \kappa$, called the complex index of refraction. …” - pg. 298, “Real-Time Rendering”, 4th Edition.

As the book introduced, we use complex IOR to represent the characteristics of a media, but in practice of local illumination, we’d often only care about the real number part $n$, since the attenuation happens around the conductor medium more and we typically imply we would treat the air as our original media (or in the volume rendering business, but I won’t cover about that topic here). Then we could simply get IOR by $n = \frac{c}{v}$, where $c$ is the light speed in vacuum and $v$ is the light speed in the media. For a further detailed discussion about complex IOR, I recommend this post by Sébastien Lagarde, which he talked around the situations that covering all the other possible medium interfaces.

And then we would have a physical law called Snell’s law, which relates the incident angle with refracted angle by IOR of two mediums, written as $\sin(\theta_t) = \frac{n_1}{n_2}\sin(\theta_i)$, where $\theta_i$ is the angle between the interface normal and the incident light direction, $\theta_t$ as the angle between the inverse of the interface normal and the refracted/transmitted light direction. We denote the index of refraction on the “outside” (the side where the incoming, or incident, wave originates) as and the index of refraction on the “inside” (where the wave will be transmitted after passing through the surface) as .

Now if we know the angles (here we abstract the light to a single monochromatic beam, which has “angle” and a single wavelength, but actually we’ve talked before how it is in real situation) and the IOR of the medium, could we calculate anything useful to display on the screen? Well, with Snell’s law we could figure out the direction change of the light, but we didn’t know how much energy would change, also if the light is not monochromatic, we don’t know which range of wavelengths would be reflected or refracted. And the most important problem is, we are receiving light through our eyes, but not always directly from the light source, what we want to receive are those light “reflected” from different surfaces, rather than those directly emitted from the sun or the bulbs!

So actually, we need to figure out such a scenario: Light is emitted from a source, meets with some surfaces and changes, then somehow travels into our eyes. When the light “hit” the surface we then could try to apply Snell’s law. But Snell’s law is just about how light changes at the medium interface in a very ideal situation, we don’t know what would happen next, since the conservation of energy always work in this universe (until the moment I wrote this line it is still valid), the reflected light must be weaker than the incident light, but how much? Also, where the refracted light part goes?

That requires us to give further information about how light continuously traveling. As I mentioned before, media is made up by particles, then the size of particles and the distance of particles should have an influence on the light transmission. But since it’s impossible to calculate everything happened inside the media, we choose to model the region around the surface of the media only which has much more contributions to the light into our eyes.

“…In rendering, we typically use geometrical optics, which ignores wave effects such as interference and diffraction. This is equivalent to assuming that all surface irregularities are either smaller than a light wavelength or much larger. …” - pg. 303, “Real-Time Rendering”, 4th Edition.

“…surface irregularities much larger than a wavelength change the local orientation of the surface. When these irregularities are too small to be individually rendered—in other words, smaller than a pixel—we refer to them as microgeometry. …” - pg. 304, “Real-Time Rendering”, 4th Edition.

“…For rendering, rather than modeling the microgeometry explicitly, we treat it statistically and view the surface as having a random distribution of microstructure normals. As a result, we model the surface as reflecting (and refracting) light in a continuous spread of directions. The width of this spread, and thus the blurriness of reflected and refracted detail, depends on the statistical variance of the microgeometry normal vectors—in other words, the surface microscale roughness. …” - pg. 304, “Real-Time Rendering”, 4th Edition.

The book introduced the microgeometry theory, which is balanced between the computational burden and the credibility of the result when it was adopted into real-time rendering community. Since our screen has limited discrete pixels, then it’s meaningless and wasting to calculate anything too accurately, instead we choose to use some statistical models for a cheaper routine. You may have heard some statistical methods used in rendering like the most famous Monte Carlo method before, here we would follow the similar path to figure out how to get the final light.

The Rendering Equation

Now we could introduce an almost accurate representation of the interaction between light and surfaces. For a single light in wavelength $\lambda$ at time $t$ who hits the surface in a unit area $A$ from a unit steradian $\omega_i$, and is “changed” by the surface then finally “re-emitted” to our eyes by an unit steradian $\omega_o$, we could write such a formula: $L_o(A,\omega_o,\lambda,t) = f_r L_i(A,\omega_i,\lambda,t)$, which describe the incident radiance is weighted by a factor $f_r$ who indicates the surface optic characteristic then contributes to the surface irradiance. Then we could do an integration for every possible incident direction of different light around the hemisphere centered at the surface normal, combine with the original emitted light from the surface itself, to get a more general formula. One summarization formula for rendering in this context is The Rendering Equation, $L_o(p,\omega_o,\lambda,t) = L_e(p,\omega_o,\lambda,t) + \int\limits_{\Omega} f_r(p,\omega_i,\omega_o,\lambda,t) L_i(p,\omega_i,\lambda,t) (n \cdot \omega_i) d\omega_i$, basically it is the same transcript of what I talked before, but with a simplification that we ignore the surface area, just minimize it to a point $p$.

Further more, we would like to treat the light-surface interaction as an time-domain individual event, then we could omit $t$. And because we would finally send a RGB color space data to the screen, so a continuous spectral irradiance is fairly unnecessary, then we would like to also cancel the spectrum by replacing it with 3 individual similar formulae, which have same form but care only about each color channels, thus we could just write one instead as The Reflectance Equation: $L_o(p,\omega_o) = \int\limits_{\Omega} f_r(p,\omega_i,\omega_o) L_i(p,\omega_i) (n \cdot \omega_i) d\omega_i$.

“…Local reflectance is quantified by the bidirectional reflectance distribution function (BRDF), denoted as $f(l, v)$. …” - pg. 310, “Real-Time Rendering”, 4th Edition.

Now if we know how the incident light they are, then we just need to give a $f_r$ weight who described the entire optic characteristic of the media and would get the final results. This $f_r(p,\omega_i,\omega_o)$, like the book introduced, is called BRDF, combining with the microgeometry method I listed above, we would have chance to finally write some codes to calculate. But at first, let’s take a look back at the surface model we used here, we say “surface” not “interface”, the “surface” indicates we are inspecting in a region around the “interface”, so it should have more properties than what Snell’s law could describe. Let’s take a look:

As the media won’t always absorb and scatter all the refracted light inside (for example most of the non-conductors won’t, since there are too less free electrons inside to do the business), some of them would leave the media and enter the previous media again, which we called this kind of phenomenon as Subsurface Scattering. A typical practice is to separate the BRDF to 2 parts, a reflection part as specular and a local subsurface scattering part as diffuse. I would like to use the notation as $f_s$ and $f_d$ later.

Also, sometimes we need to calculate more general subsurface scattering, then we would use a bidirectional scattering distribution function (abbr. BSSRDF) instead, and for the light who travel through the entire media and leave in another surface it becomes bidirectional transmittance distribution function (abbr. BTDF). Together they are called as BxDF.

To make a BxDF physically correct, we need to achieve 3 goals：

$f_r(\omega_i,\omega_o) \ge 0$, a BxDF never results negative weight;
$f_r(\omega_i,\omega_o) = f_r(\omega_o,\omega_i)$, it’s called Helmholtz reciprocity, simply speaking means if we change the incident and the observe direction it should have the same result;
$\forall \omega_i, \int\limits_{\Omega}f_r(\omega_i,\omega_o)(n \cdot \omega_i) d\omega_i \le 1$, means we need to follow the energy conservation law, the weight should never exceed than 1, the relative outgoing light energy should never exceed than the relative incoming light energy.

In practice there is a kind of convenient way to evaluate whether a BxDF is energy conservation or not called White furnace test, I’ll talk about it later.

BRDF

BRDF would be thought as $f_r(\omega_i,\omega_o) = \frac{dL_o(\omega_o)}{L_i(\omega_i)\cos\theta_i d\omega_i}$, which gives us a possibility to measure it in real, for example MERL is one kind of BRDF database. Also, we could write BRDF as $f_r(\omega_i,\omega_o) = f_d(\omega_i,\omega_o) + f_s(\omega_i,\omega_o)$, to indicate that we would like to seperate BRDF to the specular and diffuse parts and solve them independently.

Let’s use $n$ as the macro surface normal vector and $m$ as the micro surface normal vector, and $h\ = \frac{\omega_o + \omega_i}{||\omega_o + \omega_i||}$ as the normalized halfway vector of the view direction and light direction.

Also, for the sake of convenience to discuss BRDF more practically, I’d like to introduce the Directional albedo which measures the amount of light coming from a given direction that is reflected at all, into any outgoing direction in the hemisphere around the surface normal, in formula as $R_s = \int\limits_{\Omega}f(l, v)(n·s)ds$, if the BRDF is Helmholtz reciprocal then could substitute $s$ with $l$ or $v$ freely.

Diffuse part

Simple Lambert model

$f_{Lambert} = \frac{\rho} {\pi}$, $\rho$ as the “color” of the surface, strictly speaking it is the subsurface scattering part of the surface irradiance under a particular lighting circumstance; $\pi$ comes from the fact that we treat the surface as the Lambertian surface, thus it won’t change due to the view and light direction, then the BRDF integral over the hemisphere yield it.

Let’s visualize its directional albedo: (BRDF Explorer/Octave WIP)

This simple Lambert diffuse model would ignore the surface micro scope variation, combines with the Lambert’s cosine law we could already get $L_o(\omega_o) = \int\limits_{\Omega}\frac{\rho} {\pi}L_i(\omega_i)\cos\theta_i d\omega_i$, since the integral of direction $\omega_i$ over the hemisphere is $\pi$, then it would cancel the $\pi$ in the BRDF, so finally we would just get $L_o(\omega_o) = \rho \cos\theta_i$, exactly the same as what we learned first in the real-time rendering class 101!

Cook-Torrance model

The lack of micro scope detail of the simple Lambert model limits us to get further realistic results, luckily as the researchers and scientists work on it for decades, we’ve already had some advanced replacements of the simple Lambert model, one of them which is used most commonly today in real-time rendering community is the microgeometry theory, we’d refer it as microfacet theory here.

R.L. Cook and K. E. Torrance [CT82] wrote a paper in 80’s which is the root of the most popular adopted microfacet model today, with the other nice references [Hei14][LR14] we could conclude the general Cook-Torrance model as $f_{Cook-Torrance}=\frac{1}{|n·\omega_o||n·\omega_i|}\int\limits_{\Omega}f_m(\omega_o,\omega_i,m)G(\omega_o,\omega_i,m) D(m,\alpha)\langle \omega_o·m\rangle \langle \omega_i·m\rangle dm$. It emphasizes that the macro BRDF is correlated with the micro BRDF, and it could be calculated through the integral over the microfacet $m$, with the additional weight functions $G$ and $D$ to help keeping the micro-macro mapping relationship stay correct.

The $D$ function is called Distribution function or Normal Distribution Function(abbr. NDF), gives the spatial/statical distribution of the micro normal $m$ over the macro normal $n$, the $\alpha$ here is a user-controlled variable which describes how “rough” the surface it is, so we call it roughness or smoothness in practice (typically we’d like to build a non-linear mapping between the user-controlled roughness and the real $\alpha$). We would use statical functions in practice since it’s the only possible way to calculate in real-time, about how to mapping from spatial function to statical function I recommend to read this paper [Hei14] for detail understanding.

The $G$ function is called Geometry function or Masking-shadowing function, but strictly speaking, we would better call it $V$ as Visibility function, since the Geometry function is usually used to compose the Visibility function actually, but in literal it is used exchangeably often. It gives a weight about how the microfacets influence themselves by masking each other along the view direction and shadowing each other along the incident light direction, it should be deduced accordingly with the $D$ function we chose.

I’ll list some common used $D$ and $G$ functions below:

$D$ function

Gaussian $D$ function [CT82]

$D_{Gaussian}=ce^{(-\alpha/m)^2}$

Beckmann $D$ function [CT82]

$D_{Beckmann}=\frac{1}{m^2\cos^4\alpha}e^{-[ ,(\tan\alpha)/m ] ,^2}$

For these two $D$ functions, $c$ is an optional scaling factor, $\alpha$ as the angle between $n$ and $h$, $m$ as the RMS of the slope of the microfacet. Since they are computationally expensive, it’s rare to see them in real products, but we’d like to treat them as the offline reference sometimes.

Berry $D$ function [Bur12]

$D_{Berry}=\frac{c}{((n·h)^2(\alpha^2-1)+1)}$

GGX/Trowbridge-Reitz $D$ function [Bur12]

$D_{TR}=\frac{c}{((n·h)^2(\alpha^2-1)+1)^2}$

Generalized GGX/Trowbridge-Reitz $D$ function [Bur12]

$D_{GTR}=\frac{c}{((n·h)^2(\alpha^2-1)+1)^\gamma}$

For these three $D$ functions, $c$ is an optional scaling factor, $\alpha$ is the roughness, $\gamma$ is an optional exponential factor. If we choose $\gamma=10$ it is fairly close to the Beckmann $D$ function. They are most commonly used $D$ functions as far as I know, gives a “long-tailed” visual appearance.

$G$ function

Cook-Torrance $G$ function [CT82]

$G_{cook-torrance} = \min{1,\frac{2(n·h)(n·\omega_o)}{(\omega_o·h)},\frac{2(n·h)(n·\omega_i)}{(\omega_o·h)}}$

This one comes from the original paper itself but I haven’t seen it in any product level shader yet since it could be deduced to other more optimal versions.

Smith $G$ function [Smith67] [Hei14]

$G_{Smith}=\frac{\chi^+(\omega_o·\omega_m)}{1 + \Lambda(\omega_o)}$

The original paper is not publicly available, so I list the deduced version from a later reference. Here the nominator $\chi^+(u)$ is a heavy-side function, when $u>0$ then $\chi^+(u)=1$, otherwise $\chi^+(u)=0$, it ensures the sidedness effect, while the $\Lambda()$ function is an integral over the slopes of the microsurface, which gives the masking probability. So, unless we provide a possible $\Lambda()$ function, this formula can’t be translated to a shader.

Schlick $G$ function [Sch94]

$G_{Schlick}=\frac{n·\omega_o}{(n·\omega_o)(1-k)+k}·\frac{n·\omega_i}{(n·\omega_i)(1-k)+k}$

$k$ is the user-controlled roughness, in practice we could remapping roughness to $\alpha=\frac{Roughness 1}{2}$ [Bur12], $k=\alpha^2/2$ [Kar13] to get a better non-linearity result. Schlick $G$ function is the approximation of Smith $G$ function in $[ ,0,1] ,$, which is kind friendly for our application scene because its parameter requirement is acceptable.

Height correlated Smith $G$ function [Hei14] [LR14]

$G_{CorrelatedSchlick}=\frac{\chi^+(\omega_o·h)\chi^+(\omega_i·h)}{1+\Lambda(\omega_o)+\Lambda(\omega_i)}$

$\Lambda(m)=\frac{-1+\sqrt{1+\alpha^2\tan^2(\theta_m)}}{2}=\frac{-1+\sqrt{1+\frac{\alpha^2(1-\cos^2(\theta_m))}{\cos^2(\theta_m)}}}{2}$, because the microfacet would mask and shadow at the same time, then if we correlate the masking and shadowing parts with the respect of its height would bring some energy loss back. The detailed deduction is math heavily, I’d recommend reading the corresponding papers for further understanding.

Multi-scattering Smith $G$ function [HHED16]

All of the $G$ functions I listed above only take care of a single-scattering phenomenon, while in reality, the rougher surface would have more possibility to bounce light around different microfacets, it’s important to count these part of the energy. The paper [HHED16] gives a stochastic method based ground truth, and later [IW17] gives an implementable compensation way to achieve it, an alternative version of the general formula is given by the book as:

“… $f_{ms}(l, v) = \frac{\overline F \overline {Rs_{F_1}}}{\pi(1-\overline {Rs_{F_1}})(1-\overline F(1-\overline {Rs_{F_1}}))}(1-Rs_{F_1}(l))(1-Rs_{F_1}(v))$ …” - pg. 346, “Real-Time Rendering”, 4th Edition.

Let’s start to decompose this formula from the $F$ Fresnel function. As we discussed before, the Snell’s law could give us the direction of the transmitted light, and the ideal mirror reflection could give us the reflected light direction, but we don’t know how much energy is reflected and how much is transmitted, this is quite an annoy problem. But luckily it has been solved (with some restricted conditions) already in 19th century by French scientist Augustin-Jean Fresnel as Fresnel equations. Long talk in short, since we only care about the ideal sandbox in our real-time compensations, we would look for a nice enough Fresnel function to simulate this phenomenon. The Schlick Fresnel approximation is widely adopted nowadays, it’s has a form as $F_{Schlick}=F_0 + (F_{90} - F_0)(1-(\cos\theta))^5)$, $F_0$ is the specular reflectance from normal incidence, it represents the IOR when the view direction is perpendicular with the surface as $\omega_o \parallel n$; similar $F_{90}$ is the IOR when $\omega_o \perp n$, some of the implementations would assume $F_{90}$ is always 1, and it could cover most of common conductor/dielectric materials type. Finally $\cos\theta=n·\omega_o$ or $\cos\theta=n·\omega_i$, it’s another kind of cosine-weighted contribution appears here.

The $\overline F$ here means the average of $F$ over all the different cosine of angles, if we just use the Schlick Fresnel approximation mentioned above, it could simply be calculated as $\overline F = \frac{20}{21}F_0 + \frac{1}{21}$.

Then we’d move on to another average, $Rs_{F_1}$, it is the directional albedo of , which is the specular BRDF term with set to 1, which could be interpreted as the irradiance of a pure white surface when illuminated by a unit directional light source, and Stephen Hill wrote a nice blog series about it. Basically, if we could implement the single-scattering BRDF, then could just use some discrete numerical integral method (like Importance sampling) to calculate it and save it to a look-up table, which exactly similar like what the split-summing technique of IBL applicated in [Kar13].

Use Simple Lambert model as the microfacet BRDF in Cook-Torrance model [LR14]

$f_m(\omega_o,\omega_i,m) = f_{Lambert} = \frac{\rho} {\pi}$ and then $f_{cook-torrance}=\frac{\rho}{\pi}\frac{1}{|n·\omega_o||n·\omega_i|}\int\limits_{\Omega}G(v,l,m) D(m,\alpha)\max(0,\omega_o·m)\max(0,\omega_i·m)dm$

Still, no analysis solution, but gives a theoretical fundamental about the problem we need to solve, and unified all the problem inside one microsurface theory.

Oren-Nayar model [ON94]

$f_{oren-nayar}=\frac{\rho} {\pi}(A+(B·\max(0, \cos (\omega_i - \omega_o))·\sin(\max(\omega_i, \omega_o))·tan(\min(\omega_i,\omega_o))))$ $A=1-\frac{\alpha}{2\alpha+0.66}$, $B=0.45\frac{\alpha}{\alpha+0.09}$ and $\alpha$ is the roughness, it’s an approximation of the general Cook-Torrance model, when $\alpha = 0$ we’ll get the Simple Lambert model. Could treat Oren-Nayar model as a kind of generalization of Simple Lambert model.

Disney model [Bur12]

Another advanced Lambert-based diffuse model which considers about Fresnel effect, $f_{disney}=\frac{\rho} {\pi}(1+(F_{d90}-1)(1-(n · \omega_i))^5)(1+(F_{d90}-1)(1-(n · \omega_o))^5)$, or written as $f_{Disney}=\frac{\rho} {\pi}F_{Schlick}(1, F_{d90}, n, \omega_o)·F_{Schlick}(1, F_{d90}, n, \omega_i)$, here $F_{d90}=0.5+2(h·\omega_i)^2\alpha$, $\alpha$ is roughness.

Normalized Disney model [LR14]

For the sake of energy conservation, we could remapping the original Disney model to $[ ,0,1] ,$, then $f_{normalizedDisney}=c·\frac{\rho} {\pi}F_{Schlick}(1, F_{d90}, n, \omega_o)·F_{Schlick}(1, F_{d90}, n, \omega_i)$, $c=\frac{1}{1.51}+\frac{0.51}{1.51}\alpha$ it’s a scaling factor, I deduced it here for to better compare with the original version, and now $F_{d90}=0.5+(2(h·\omega_i)^2-0.5)\alpha$, $\alpha=Roughness^\gamma$, in practice the original paper chooses $\gamma=2$, and it find when $\gamma=4$ it’s almost near the result in [Sch14] where $\alpha=(0.3+0.7Roughness)^6$.

Specular part

Phong model

$f_{Phong} = (r·\omega_o)^\alpha$, $r$ is the reflection direction of $\omega_i$, it’s the most famous and commonly used specular model in last few decades, and even programmed inside the graphics hardware, it needs the exponential factor $\alpha$ as the user-controlled parameter.

Normalized Phong model

$f_{normalizedPhong} = c·(r·\omega_o)^\alpha$, $c=\frac{\alpha+1}{2\pi}$, actually Phong model gives a $D$ function in a microsurface view of point, here $\alpha$ thus could be thought as the roughness.

Blinn-Phong model

$f_{Blinn-Phong} = (n·h)^\alpha$, an optimization of Phong model, in practice if we choose mapping $\alpha_{blinn-phong}=4\alpha_{phong}$ then Blinn-Phong model would looks like Phong model [Wiki1].

Normalized Blinn-Phong model

$f_{normalizedBlinn-Phong} = c·(n·h)^\alpha$, $c=\frac{\alpha+2}{4\pi(2-2^{\frac{-\alpha}{2}})}$.

Cook-Torrance model [CT82] [Hei14]

$f_{cook-torrance} = \frac{F(\omega_o, h , f_0, f_{90})D(h, \alpha)G(\omega_o, \omega_i, h)}{4|n·\omega_o||n·\omega_i|}$, the new kids (popular from ~2012) in town! Everything we’ve talked before, the denominator is deduced from the Jacobian Matrix when we change the space from the microfacet space to macro and makes it’s quite elegant, we use the microfacet theory but calculate in macro!

Some sample codes

a. Simple Lambert model + Blinn-Phong model

vec3 CalcDirectionalLight(dirLight light, vec3 normal, vec3 diffuse, vec3 specular, vec3 viewPos, vec3 fragPos)
{ 
 vec3 N = normalize(normal);
 vec3 L = normalize(-light.direction);
 vec3 V = normalize(viewPos - fragPos);
 vec3 H = normalize(V + L);

 float NdotH = max(dot(N , H), 0.0);
 float NdotL = max(dot(N , L), 0.0);
 
 // ambient color
 vec3 ambientColor = diffuse * light.color * 0.04;

 // diffuse color
 vec3 diffuseColor = diffuse * NdotL * light.color;
 
 // specular color
 float alpha = 32;
 vec3 specularColor = specular * pow(NdotH, alpha) * light.color;
 
 return (ambientColor + diffuseColor + specularColor);
}

b. Oren-Nayar model + Normalized Blinn-Phong model

// Oren-Nayar diffuse BRDF
// ----------------------------------------------------------------------------
float orenNayarDiffuse(float LdotV, float NdotL, float NdotV, float roughness) 
{
 float s = LdotV - NdotL * NdotV;
 float t = mix(1.0, max(NdotL, NdotV), step(0.0, s));

 float sigma2 = roughness * roughness;
 float A = 1.0 - (0.5 * sigma2 / (sigma2 + 0.33));
 float B = 0.45 * sigma2 / (sigma2 + 0.09);

 return max(0.0, NdotL) * (A + B * s / t);
}

vec3 CalcDirectionalLight(dirLight light, vec3 normal, vec3 diffuse, vec3 specular, float roughness, vec3 viewPos, vec3 fragPos)
{ 
 vec3 N = normalize(normal);
 vec3 L = normalize(-light.direction);
 vec3 V = normalize(viewPos - fragPos);
 vec3 H = normalize(V + L);
 float LdotV = max(dot(L , V), 0.0);
 float NdotH = max(dot(N , H), 0.0);// ambient color
 vec3 ambientColor = diffuse * light.color * 0.04;

 // diffuse color
 float Fd = orenNayarDiffuse(LdotV, NdotL, NdotV, roughness);
 vec3 diffuseColor = diffuse * Fd * light.color;
 
 // specular color
 float alpha = 32;
 float normalizedScaleFactor = (alpha + 2) / (4 * PI * (2 - pow(2, (-alpha / 2))));
 vec3 specularColor = specular * (1 - Fd) * pow(NdotH, alpha) * normalizedScaleFactor * light.color;
 
 return (ambientColor + diffuseColor + specularColor);
}

c. Normalized Disney model + Cook-Torrance (specular) model, use $D_{TR}$+$G_{CorrelatedSchlick}$+$F_{Schlick}$ combination

(reference Frostbite Engine [LR14])

// Frostbite Engine model
// ----------------------------------------------------------------------------
// Specular/Diffuse BRDF Fresnel Component
// ----------------------------------------------------------------------------
vec3 Frostbite_fresnelSchlick(vec3 f0, float f90, float u)
{
 return f0 + (f90 - f0) * pow(1.0 - u, 5.0);
}
// Diffuse BRDF
// ----------------------------------------------------------------------------
float Frostbite_DisneyDiffuse(float NdotV, float NdotL, float LdotH, float earRoughness)
{ 
 float energyBias = mix(0, 0.5, linearRoughness);
 float energyFactor = mix(1.0, 1.0/1.51, linearRoughness);
 float fd90 = energyBias + 2.0 * LdotH * LdotH * linearRoughness;
 vec3 f0 = vec3 (1.0, 1.0, 1.0);
 float lightScatter = Frostbite_fresnelSchlick(f0, fd90, NdotL).r;
 float viewScatter = Frostbite_fresnelSchlick(f0, fd90, NdotV).r;
 return lightScatter * viewScatter * energyFactor;
}
// Specular BRDF Geometry Component
// ----------------------------------------------------------------------------
float Frostbite_V_SmithGGXCorrelated(float NdotL , float NdotV , float alphaG)
{
 float alphaG2 = alphaG * alphaG;
 float Lambda_GGXV = NdotL * sqrt(NdotV * NdotV * (1.0 - alphaG2) + alphaG2);
 float Lambda_GGXL = NdotV * sqrt(NdotL * NdotL * (1.0 - alphaG2) + alphaG2);
 return 0.5 / max((Lambda_GGXV + Lambda_GGXL), 0.00001);
}
// Specular BRDF Distribution Component
// ----------------------------------------------------------------------------
float Frostbite_D_GGX(float NdotH , float roughness)
{
 // remapping to Quadratic curve
 float m = roughness * roughness;
 float m2 = m * m;
 float f = (NdotH * m2 - NdotH) * NdotH + 1;
 return m2 / (f * f);
}
// ----------------------------------------------------------------------------
vec3 Frostbite_CalcDirectionalLightRadiance(dirLight light, vec3 albedo, float metallic, float roughness, vec3 normal, vec3 viewPos, vec3 fragPos, vec3 F0)
{
 vec3 N = normalize(normal);
 vec3 L = normalize(-light.direction);
 vec3 V = normalize(viewPos - fragPos);
 vec3 H = normalize(V + L);

 float NdotV = max(dot(N , V), 0.0);
 float LdotH = max(dot(L , H), 0.0);
 float NdotH = max(dot(N , H), 0.0);
 float NdotL = max(dot(N , L), 0.0);

 // Specular BRDF
 float f90 = 1.0;
 vec3 F = Frostbite_fresnelSchlick(F0, f90, LdotH);
 float G = Frostbite_V_SmithGGXCorrelated(NdotV, NdotL, roughness);
 float D = Frostbite_D_GGX (NdotH, roughness);
 vec3 Fr = F * G * D;

 // Diffuse BRDF
 float Fd = Frostbite_DisneyDiffuse(NdotV, NdotL, LdotH ,roughness * roughness); 
 
 return (Fd * albedo + Fr) * light.color * NdotL / PI;
}

d.Simple Lambert model + Cook-Torrance (specular) model, use $D_{TR}$+$G_{Schlick}$+$F_{Schlick}$ combination

(reference from Unreal Engine 4[Kar13])

// Unreal Engine model
// ----------------------------------------------------------------------------
// Specular BRDF Distribution Component
// ----------------------------------------------------------------------------
float Unreal_DistributionGGX(float NdotH, float roughness)
{
 float a = roughness*roughness;
 // remapping to Quadratic curve
 float a2 = a * a;
 float NdotH2 = NdotH*NdotH;

 float nom = a2;
 float denom = (NdotH2 * (a2 - 1.0) + 1.0);
 denom = denom * denom;

 return nom / denom;
}
// Specular BRDF Geometry Component
// ----------------------------------------------------------------------------
float Unreal_GeometrySchlickGGX(float NdotV, float roughness)
{
 float r = (roughness + 1.0);
 float k = (r*r) / 8.0;

 float nom = NdotV;
 float denom = NdotV * (1.0 - k) + k;

 return nom / denom;
}
// ----------------------------------------------------------------------------
float Unreal_GeometrySmith(float NdotV, float NdotL, float roughness)
{
 float ggx2 = Unreal_GeometrySchlickGGX(NdotV, roughness);
 float ggx1 = Unreal_GeometrySchlickGGX(NdotL, roughness);

 return ggx1 * ggx2;
}
// Specular BRDF Fresnel Component
// ----------------------------------------------------------------------------
vec3 Unreal_fresnelSchlick(float cosTheta, vec3 F0)
{
 return F0 + (1.0 - F0) * pow(1.0 - cosTheta, 5.0);
}
// ----------------------------------------------------------------------------
vec3 Unreal_CalcDirectionalLightRadiance(dirLight light, vec3 albedo, float metallic, float roughness, vec3 normal, vec3 viewPos, vec3 fragPos, vec3 F0)
{
 vec3 N = normalize(normal);
 vec3 L = normalize(-light.direction);
 vec3 V = normalize(viewPos - fragPos);
 vec3 H = normalize(V + L);

 float NdotV = max(dot(N , V), 0.0);
 float NdotH = max(dot(N, H), 0.0);
 float HdotV = max(dot(H , V), 0.0);
 float NdotL = max(dot(N , L), 0.0);
 
 // Specular BRDF
 vec3 F = Unreal_fresnelSchlick(HdotV, F0);
 float G = Unreal_GeometrySmith(N, V, L, roughness); 
 float D = Unreal_DistributionGGX(N, H, roughness); 
 
 vec3 nominator = D * G * F; 
 float denominator = 4 * NdotV * NdotL;
 vec3 specular = nominator / max(denominator, 0.00001);
 
 // for energy conservation 
 vec3 kS = F;
 vec3 kD = vec3(1.0) - kS; 
 kD *= 1.0 - metallic; 
 
 return ((kD * albedo + specular) * light.color * NdotL) / PI;
}

To be continued.

Bibliography：

[Bur12] B. Burley. “Physically Based Shading at Disney”. In: Physically Based Shading in Film and Game Production, ACM SIGGRAPH 2012 Courses. SIGGRAPH ’12. Los Angeles, California: ACM, 2012, 10:1–7. isbn: 978-1-4503-1678-1. doi: 10.1145/2343483.2343493. url: http://selfshadow.com/publications/s2012-shading-course/.

[CT82] R. L. Cook and K. E. Torrance. “A Reﬂectance Model for Computer Graphics”. In: ACM Trans. Graph. 1.1 (Jan. 1982), pp. 7–24. issn: 0730-0301. doi: 10.1145/357290.357293. url: http://graphics.pixar.com/library/ReflectanceModel/.

[Hei14] E. Heitz. “Understanding the Masking-Shadowing Function in Microfacet-Based BRDFs”. In: Journal of Computer Graphics Techniques (JCGT) 3.2 (June 2014), pp. 32–91. issn: 2331-7418. url: http://jcgt.org/published/0003/02/03/.

[HHED16] E. Heitz, J. Hanika, E. d’Eon, C. Dachsbacher, “Multiple-Scattering Microfacet BSDFs with the Smith Model”. In: ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH 2016, Volume 35 Issue 4, July 2016, ISSN: 0730-0301 E, ISSN: 1557-7368 doi>10.1145/2897824.2925943, url: https://eheitzresearch.wordpress.com/240-2/

[HTSG91] Xiao D. He, Kenneth E, Torrance, Frangois X. Sillion and Donald P. Greenberg. “A Comprehensive Physical Model for Light Reflection”. In: ACM SIGGRAPH Computer Graphics Homepage, Volume 25 Issue 4, July 1991, Pages 175-186, ACM New York, NY, USA, doi>10.1145/127719.122738, url: https://www.graphics.cornell.edu/pubs/1991/HTSG91.pdf

[Sch94] Schlick, Christophe, “An Inexpensive BRDF Model for Physically-based Rendering”, Computer Graphics Forum, vol.13, no.3, Sept.1994, pp.149–162. http://dept-info.labri.u-bordeaux.fr/ ~Schlick/DOC/eur2.html

[Sch14] N. Schulz. “Moving to the Next Generation - The Rendering Technology of Ryse”. In: Game Developers Conference. 2014.

[Wiki1] https://en.wikipedia.org/wiki/Blinn%E2%80%93Phong_shading_model

Row-Major or Column-Major? Some notes about matrix convention

zhangandhang@gmail.com (Hang Zhang) — Sat, 24 Mar 2018 20:14:00 +0000

The mathematical conventions

Generally speaking, when we want to “render” something to the screen, it means we need to calculate the “color” of each pixel on the refreshing LCD in front of us, and then ask some hardware to fulfill those pixels with the color we specified. But typically what we have in our hard disk are just some mesh and texture data files, which only contain some basic information about the object we want to render, like how many vertexes they have, where the vertex’s position is in their own coordinate space, what the “color” of that vertex should look like and etc.. So, how to manipulate them until we get the final color that we could output to the screen’s pixel? Of course, we need some math, I assume you had already learned some basic linear algebra in theory and thus could understand what a matrix it is (or you could reference Wikipedia here for some sorts of overview).

First of all, let’s have a review about some theoretical things, the 4x4 matrix in Row-major mathematical convention is represented as (let’s count from 0 for convenient, also read the row first, then read the column):

\begin{bmatrix} a_{00} & a_{01} & a_{02} & a_{03} \\ a_{10} & a_{11} & a_{12} & a_{13} \\ a_{20} & a_{21} & a_{22} & a_{23} \\ a_{30} & a_{31} & a_{32} & a_{33} \\ \end{bmatrix}

In this convention, we assume the right side ($y$ in $a_{xy}$)of the index (or call it “subscript”) of the element is counted always at first, and as you can see the row-major here means the elements are counted in the row first. This convention is used widely among the world but still not all the people choose to adopt it, I’ll give an opposite example later.

And since we are focusing on the game development, typically we just use a matrix to do some fairly limited linear transformation with another vector (or multiplicate different matrices one by one to get a transformation matrix). And in most cases the vector we cared about is just a 3-dimensional vector. But for cache-friendly purpose (have a look at Wikipedia, the alternate approach is using the additional padding data to make the alignment is the power of 2) and to utilize the power of the ~~Homo Sapiens coordinates~~(Homogeneous coordinates), we’d intense to use 4-dimensional vector instead, and it would bring us the convenience to represent a point or a direction by just modifying one of the components (set the additional w component to 1.0 as a point, 0.0 as a direction).

Then we could define the vector4 in Column-major mathematical convention is represented as:

\begin{bmatrix} x \\ y \\ z \\ w \\ \end{bmatrix}

As you can see, after all, it’s just a matrix with only one column, the column-major here means the elements are listed as a 4x1 matrix instead of the 1x4 matrix.

Typically when we want to do the matrix-vector multiplication with these two mathematical representations, we name it as the right/post-multiplication, indicate that the vector is on the right side or in the post position of the matrix in the equation (restricted by the definition of matrix multiplication, we need to match the dimension between two multiplicators). And as what I listed in the demonstration below, if we calculated the final result’s elements one by one by following the order of elements in vector4, we need to access each row of the matrix first then each element in the row.

4x4 matrix * vector4 (4x1 matrix) :

\begin{bmatrix} x^{'} = a_{00} * x + a_{01} * y + a_{02} * z + a_{03} * w \\ y^{'} = a_{10} * x + a_{11} * y + a_{12} * z + a_{13} * w \\ z^{'} = a_{20} * x + a_{21} * y + a_{22} * z + a_{23} * w \\ w^{'} = a_{30} * x + a_{31} * y + a_{32} * z + a_{33} * w \\ \end{bmatrix}

In memory

If you implement the theory stuffs that we talk about with some C/C++, it would be quite cache-friendly with Row-major memory layout. (And for Fortan/Matlab, it’s cache-friendly with Column-major memory layout.)

Why? So here again we introduced another term came from no where, Row-major memory layout, what is it? As different software frameworks and people have different definitions for it, I will choose to follow one definition based on some nice blog as my definition. For example, I use C++ at the most of times to work out some 01010101 projects, and if I use 2D array like m[A][B] in C++ to represent and store a $A*B$ dimension matrix, the final memory would always laid as the m[0][0] to m[0][B - 1] continuously and then m[1][0] to m[1][B - 1] until m[A - 1][0] to the last element m[A - 1][B - 1].

The 2D array in memory looks like (in C/C++, let’s use a quite small virtual address table, welcome to 70’s???):

0x0000	0x0001	0x0002	0x0003	0x0004	0x0005	0x0006	0x0007	0x0008	0x0009	0x000A	0x000B	0x000C	0x000D	0x000E
m[0][0]	m[0][1]	m[0][2]	m[0][3]	m[1][0]	m[1][1]	m[1][2]	m[1][3]	m[2][0]	m[2][1]	m[2][2]	m[2][3]	m[3][0]	m[3][1]	m[3][2]

For this kind of compiler interpretation order of the operator [], some people would say, C/C++ is Row-major memory layout language. I would rather deprecate this kind of understanding since it would have some conflicts with my own definition because, you could achieve Column-major memory layout in C/C++ too if you just change your mind in deed! Let’s explain just a little bit later.

And if you want to assign the matrix data into this array, you could choose two ways to assign it: set m[x - 1][y - 1] to value of $a_{xy}$ or set m[x - 1][y - 1] to value of $a_{yx}$. Naturally, if you want to match the index of the theoretical matrix and the implemented 2D array, you would choose the first way and that’s how the term came from in my opinion. If you set m[x - 1][y - 1] to the value of $a_{xy}$, that means if anybody iterated the elements of the array continuously they would access the data of the first whole row of the matrix then the second row until the last of rows, in another word, row-major.

But be careful! After we defined Row-major memory layout, it doesn’t means we have to choose the only one way to assign the matrix data, we still could store them in any kind of orders we like. For example as I’ve already talked above, we could just store a 4x4 matrix in Row-major mathematical convention in C++ 2D array with Row-major memory layout like:

0x0000	0x0001	0x0002	0x0003	0x0004	0x0005	0x0006	0x0007	0x0008	0x0009	0x000A	0x000B	0x000C	0x000D	0x000E
m[0][0]	m[0][1]	m[0][2]	m[0][3]	m[1][0]	m[1][1]	m[1][2]	m[1][3]	m[2][0]	m[2][1]	m[2][2]	m[2][3]	m[3][0]	m[3][1]	m[3][2]
$a_{00}$	$a_{01}$	$a_{02}$	$a_{03}$	$a_{10}$	$a_{11}$	$a_{12}$	$a_{13}$	$a_{20}$	$a_{21}$	$a_{22}$	$a_{23}$	$a_{30}$	$a_{31}$	$a_{32}$

or alternatively, we could store the column of the matrix first, that means we choose to use a kind of Column-major memory layout in C++:

0x0000	0x0001	0x0002	0x0003	0x0004	0x0005	0x0006	0x0007	0x0008	0x0009	0x000A	0x000B	0x000C	0x000D	0x000E
m[0][0]	m[0][1]	m[0][2]	m[0][3]	m[1][0]	m[1][1]	m[1][2]	m[1][3]	m[2][0]	m[2][1]	m[2][2]	m[2][3]	m[3][0]	m[3][1]	m[3][2]
$a_{00}$	$a_{10}$	$a_{20}$	$a_{30}$	$a_{01}$	$a_{11}$	$a_{21}$	$a_{31}$	$a_{02}$	$a_{12}$	$a_{22}$	$a_{32}$	$a_{03}$	$a_{13}$	$a_{23}$

As far as we just discussed the convention itself until now, there is nothing ahead that would obstacle us to pick up a preferred way, and actually, you could even store the matrix in memory in reverse order or some crazy encrypted random order. After all, it’s just a mapping relationship between the matrix data and the memory location. We really don’t need to worry about which convention should we choose until the moment we start to write the code and build them into a binary executable file in our project.

Alternative approaches

Similar, we could define a vector4 in Row-major mathematical convention now as:

\begin{bmatrix} x & y & z & w \\ \end{bmatrix}

it’s just a 1x4 matrix with only one row. And then we could get a term as left/pre-multiplication.

vector4 (1x4 matrix) * matrix4x4:

\begin{bmatrix} x^{'} = x * a_{00} + y * a_{10} + z * a_{20} + w * a_{30} \\ y^{'} = x * a_{01} + y * a_{11} + z * a_{21} + w * a_{31} \\ z^{'} = x * a_{02} + y * a_{12} + z * a_{22} + w * a_{32} \\ w^{'} = x * a_{03}+ y * a_{13} + z * a_{23} + w * a_{33} \\ \end{bmatrix}

Again, there is no doubt we need to access the elements in each column of matrix continuously, so the more efficient memory layout is:

0x0000	0x0001	0x0002	0x0003	0x0004	0x0005	0x0006	0x0007	0x0008	0x0009	0x000A	0x000B	0x000C	0x000D	0x000E
m[0][0]	m[0][1]	m[0][2]	m[0][3]	m[1][0]	m[1][1]	m[1][2]	m[1][3]	m[2][0]	m[2][1]	m[2][2]	m[2][3]	m[3][0]	m[3][1]	m[3][2]
$a_{00}$	$a_{10}$	$a_{20}$	$a_{30}$	$a_{01}$	$a_{11}$	$a_{21}$	$a_{31}$	$a_{02}$	$a_{12}$	$a_{22}$	$a_{32}$	$a_{03}$	$a_{13}$	$a_{23}$

And after we investigate some programming languages we would get the conclusion: for C/C++, it’s cache-friendly with Column-major memory layout. (And for Fortan/Matlab, it’s cache-friendly with Row-major memory layout, but who cares about cache friendly in Matlab??? Come on.)

So finally the vector4 mathematical convention and the memory layout have 2 x 2 = 4 different combinations: vector4 in Column-major mathematical convention + Column-major memory layout vector4 in Column-major mathematical convention + Row-major memory layout vector4 in Row-major mathematical convention + Row-major memory layout vector4 in Row-major mathematical convention + Column-major memory layout

And practically speaking, w.r.t. the mapping relationship between the index of the array and the index of the element in the matrix, whether you choose Row-major memory layout or Column-major memory layout, it’s better to choose a corresponding cache-friendly vector4 mathematical convention at the same time. And that means we need to avoid using the combinations I marked with different colors above because who want some stalls in their precious CPU cycles?

Application examples

Let’s have a look at some examples for better comprehension:

If we want to extend a vector4 in Column-major mathematical convention to a translation matrix, we could write it in paper as:

\begin{bmatrix} T_x \\ T_y \\ T_z \\ T_w \\ \end{bmatrix}

Then the matrix is:

\begin{bmatrix} 1 & 0 & 0 & T_x \\ 0 & 1 & 0 & T_y \\ 0 & 0 & 1 & T_z \\ 0 & 0 & 0 & T_w \\ \end{bmatrix}

And let’s define a vector4 as the original point before the translation process,

\begin{bmatrix} x \\ y \\ z \\ w \\ \end{bmatrix}

Then after the translation the result vector4 is:

\begin{bmatrix} x^{'} = 1 * x + 0 * y + 0 * z + T_x * w = x + T_x * w \\ y^{'} = 0 * x + 1 * y + 0 * z + T_y * w = y + T_y * w \\ z^{'} = 0 * x + 0 * y + 1 * z + T_z * w = z + T_z * w \\ w^{'} = 0 * x + 0 * y + 0 * z + T_w * w = T_w * w \\ \end{bmatrix}

Because the ${w}$ component represents the length of the projection line in homogeneous/projective coordinate, let’s assume we’ve already done the Perspective Division and then ${w} = 1$, the previous calculation could be simplified as:

Translation vector4:

\begin{bmatrix} T_x \\ T_y \\ T_z \\ 1 \\ \end{bmatrix}

Translation matrix:

\begin{bmatrix} 1 & 0 & 0 & T_x \\ 0 & 1 & 0 & T_y \\ 0 & 0 & 1 & T_z \\ 0 & 0 & 0 & 1 \\ \end{bmatrix}

The original point:

\begin{bmatrix} x \\ y \\ z \\ 1 \\ \end{bmatrix}

The result point:

\begin{bmatrix} x^{'} = 1 * x + 0 * y + 0 * z + T_x * 1 = x + T_x \\ y^{'} = 0 * x + 1 * y + 0 * z + T_y * 1 = y + T_y \\ z^{'} = 0 * x + 0 * y + 1 * z + T_z * 1 = z + T_z \\ w^{'} = 0 * x + 0 * y + 0 * z + 1 * 1 = 1 \\ \end{bmatrix}

And again, of course here we need to access each row first, then in C++ it’s more convenient to use Row-major memory layout because we will read the 2D array’s index as same as the math convention’s index and won’t have too much cache-missing problem, but also you could use Column-major memory layout, just need to flip everything in your mind and lose some CPU cycles.

And the same, if we choose to use vector4 in Row-major mathematical convention instead, finally we would find it’s more rational to use Column-major memory layout, but still we could use Row-major memory layout.

But what if we want to use both vector4 in Row-major mathematical convention and vector4 in Column-major mathematical convention and want to have the same cache-friendly performance in C++? How the code should be written? Actually, it’s not so complex, if you understood what all I talked above thoroughly, you may figure it out by yourself. As we defined two mathematical conventions for vector4 before, since it is just a specialized matrix, we could think matrix mathematical convention could be row-major and column-major too, that’s why I emphasized the 4x4 matrix in Row-major mathematical convention at the very beginning of this article, we represent the 4x4 matrix in Column-major mathematical convention, and flip everything in theory we discussed before, then put them in the cache-friendly memory layout, that’s all!

And below is how the mathematical representation looks like, remember we still read the row first, then read the column:

\begin{bmatrix} a_{00} & a_{10} & a_{20} & a_{30} \\ a_{01} & a_{11} & a_{21} & a_{31} \\ a_{02} & a_{12} & a_{22} & a_{32} \\ a_{03} & a_{13} & a_{23} & a_{33} \\ \end{bmatrix}

And now the vector4 in Column-major mathematical convention is:

\begin{bmatrix} x = a_{00} & y = a_{10} & z = a_{20} & w= a_{30} \\ \end{bmatrix}

And the translation matrix is:

\begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ T_x & T_y & T_z & 1 \\ \end{bmatrix}

Now the ${T_x}$、${T_y}$、${T_z}$ (and ${T_w}$) is $a_{03}$、$a_{13}$、$a_{23}$（And $a_{33}$）

And finally we get the best choice as:

0x0000	0x0001	0x0002	0x0003	0x0004	0x0005	0x0006	0x0007	0x0008	0x0009	0x000A	0x000B	0x000C	0x000D	0x000E
m[0][0]	m[0][1]	m[0][2]	m[0][3]	m[1][0]	m[1][1]	m[1][2]	m[1][3]	m[2][0]	m[2][1]	m[2][2]	m[2][3]	m[3][0]	m[3][1]	m[3][2]
$a_{00}$	$a_{01}$	$a_{02}$	$a_{03}$	$a_{10}$	$a_{11}$	$a_{12}$	$a_{13}$	$a_{20}$	$a_{21}$	$a_{22}$	$a_{23}$	$a_{30}$	$a_{31}$	$a_{32}$

Actually, as you can see here, nothing changed in memory, what changed exactly is our mathematical convention and our mind. And then I feel it’s “meaningless” to distinguish between “row-major” or “column-major” inside C++, and this should be the reason why some people call C++ as a “row-major” language. But still, we could have a more clear point of view after all of these red tapes just before we start to write something.

But in practice, unfortunately, due to the historical reason of OpenGL (see OpenGL FAQ 9.005), it uses 4x4 matrix in Column-major mathematical convention and a memory layout corresponding to Column-major memory layout in C/C++ (as the document said

The translation components occupy the 13th, 14th, and 15th elements of the 16-element matrix

, means m[3][0] to m[3][3] are ${T_x}$、${T_y}$、${T_z}$ and ${T_w}$). For the sake of different mathematical conventions, glUniformMatrix4fv function has a parameter at the third position to indicate if we want to ask the vendor’s implementation library to flip the input matrix or not. If we choose to use the same mathematical convention and memory layout as OpenGL initially defined then we just set it to GL_FALSE, means no flipping. And if we choose to use 4x4 matrix in Row-major mathematical convention but didn’t change the memory layout in C++, we need to set it to GL_TRUE to flip the input matrix, as same as the situation when we change the memory layout but don’t change the mathematical convention. If you are some messy guy like me who intended not to use DirectXMath and want to achieve a unified math library among different graphics APIs, you’d better have a caution about it.

Actually it is really a confusing topic, because how we read how we count how we use and understand the terms finally influence our implementation, in my game engine I decided to support both two different conventions and until now it works fine, but in large scale products I assume that would become a problem if people didn’t make the clarification well and clearly at the beginning.

Code samples

Some code samples here:

//Column-major memory layout
#if defined (USE_COLUMN_MAJOR_MEMORY_LAYOUT)
	template<class T>
	auto TVec4::toTranslationMatrix(const TVec4<T>& rhs) -> TMat4<T>
	{
		// @TODO: replace with SIMD impl
		TMat4<T> l_m;

		l_m.m[0][0] = one<T>;
		l_m.m[1][1] = one<T>;
		l_m.m[2][2] = one<T>;
		l_m.m[3][0] = rhs.x;
		l_m.m[3][1] = rhs.y;
		l_m.m[3][2] = rhs.z;
		l_m.m[3][3] = one<T>;

		return l_m;
	}
	//Row-major memory layout
#elif defined (USE_ROW_MAJOR_MEMORY_LAYOUT)
	template<class T>
	auto toTranslationMatrix(const TVec4<T>& rhs) -> TMat4<T>
	{
		// @TODO: replace with SIMD impl
		TMat4<T> l_m;

		l_m.m[0][0] = one<T>;
		l_m.m[0][3] = rhs.x;
		l_m.m[1][1] = one<T>;
		l_m.m[1][3] = rhs.y;
		l_m.m[2][2] = one<T>;
		l_m.m[2][3] = rhs.z;
		l_m.m[3][3] = one<T>;

		return l_m;
	}
#endif

//Column-major memory layout
#if defined (USE_COLUMN_MAJOR_MEMORY_LAYOUT)
	template<class T>
	auto toRotationMatrix(const TVec4<T>& rhs) -> TMat4<T>
	{
		// @TODO: replace with SIMD impl
		TMat4<T> l_m;

		l_m.m[0][0] = (one<T> -two<T> * rhs.y * rhs.y - two<T> * rhs.z * rhs.z);
		l_m.m[0][1] = (two<T> * rhs.x * y + two<T> * rhs.z * rhs.w);
		l_m.m[0][2] = (two<T> * rhs.x * rhs.z - two<T> * rhs.y * rhs.w);
		l_m.m[0][3] = (T());

		l_m.m[1][0] = (two<T> * rhs.x * rhs.y - two<T> * rhs.z * rhs.w);
		l_m.m[1][1] = (one<T> -two<T> * rhs.x * rhs.x - two<T> * rhs.z * rhs.z);
		l_m.m[1][2] = (two<T> * rhs.y * rhs.z + two<T> * rhs.x * rhs.w);
		l_m.m[1][3] = (T());

		l_m.m[2][0] = (two<T> * rhs.x * rhs.z + two<T> * rhs.y * rhs.w);
		l_m.m[2][1] = (two<T> * rhs.y * rhs.z - two<T> * rhs.x * rhs.w);
		l_m.m[2][2] = (one<T> -two<T> * rhs.x * rhs.x - two<T> * rhs.y * rhs.y);
		l_m.m[2][3] = (T());

		l_m.m[3][0] = (T());
		l_m.m[3][1] = (T());
		l_m.m[3][2] = (T());
		l_m.m[3][3] = (one<T>);

		return l_m;
	}
	//Row-major memory layout
#elif defined ( USE_ROW_MAJOR_MEMORY_LAYOUT)
	template<class T>
	auto toRotationMatrix(const TVec4<T>& rhs) -> TMat4<T>
	{
		// @TODO: replace with SIMD impl
		TMat4<T> l_m;

		l_m.m[0][0] = (one<T> -two<T> * rhs.y * rhs.y - two<T> * rhs.z * rhs.z);
		l_m.m[0][1] = (two<T> * rhs.x * rhs.y - two<T> * rhs.z * rhs.w);
		l_m.m[0][2] = (two<T> * rhs.x * rhs.z + two<T> * rhs.y * rhs.w);
		l_m.m[0][3] = (T());

		l_m.m[1][0] = (two<T> * rhs.x * rhs.y + two<T> * rhs.z * rhs.w);
		l_m.m[1][1] = (one<T> -two<T> * rhs.x * rhs.x - two<T> * rhs.z * rhs.z);
		l_m.m[1][2] = (two<T> * rhs.y * rhs.z - two<T> * rhs.x * rhs.w);
		l_m.m[1][3] = (T());

		l_m.m[2][0] = (two<T> * rhs.x * rhs.z - two<T> * rhs.y * rhs.w);
		l_m.m[2][1] = (two<T> * rhs.y * rhs.z + two<T> * rhs.x * rhs.w);
		l_m.m[2][2] = (one<T> -two<T> * rhs.x * rhs.x - two<T> * rhs.y * rhs.y);
		l_m.m[2][3] = (T());

		l_m.m[3][0] = (T());
		l_m.m[3][1] = (T());
		l_m.m[3][2] = (T());
		l_m.m[3][3] = one<T>;

		return l_m;
	}
#endif

Further more

For the multiplication algorithm of matrix, the naive implementation is $O(n^3)$, and there are lots of optimized algorithms (e.g. Solvay Strassen algorithm $O(n^{2.807})$/Coppersmith-Winograd algorithm $O(n^{2.3737})$/Stochers-Williams algorithm $O(n^{2.3729})$ could bring a better time complexity around $O(n^{2.4})$.

For the transpose algorithm, we could consider about in-situ matrix transposition.

And for the inverse algorithm, Gaussian Elimination is $O(n^3)$, we could choose LU decomposition, and also there are lots of optimized algorithms which I don’t have time to investigate too much.

Hope one day I’ll finally sit down and optimize my super slow matrix inverse operation!

Space, coordinates, and the magic Quaternion

zhangandhang@gmail.com (Hang Zhang) — Wed, 13 Dec 2017 10:24:00 +0000

Where is where?

In our daily life, when someone asked for the help about how to reach the place they want to go, we would answer them like “walk toward north and turn left at the first corner”, and here we use some kind of coordinate system to represent the relativity of position in the world we are. And in our virtual world inside the computer, it would be awesome if we could use the same north and left concepts. But since the computer only knows 0 and 1, there are lots of difficulties to make a north really north.

Because there is no absolute north exists actually (let’s forget about the North Pole for a while, or forever), it’s just one of the four relative directions from the origin of a cycle to its edge, you could always choose a north anywhere on the circumference, and then rotate clockwise 90 degrees and you’ll get the east, and so on the same, you’ll get the south and the west. But when thinking about the left and the right, their definitions would be more complex and tricky because it related to the very basic property (Some physics!) of our universe. Indeed, it would be another article to talk about, let’s forget about it too because, after all, I think most of the people could understand which hand is on the left in most of the daily cases.

As you could see here if we really want to help a virtual tourist (let’s call him Jerry, Tom’s best friend) in our virtual world find the place he wants to arrive, we must decide where the north it is at least. The simplest choice is that just using the direction where our face is facing as the north, then other 3 directions would be defined immediately at the same time. Also, we could extend them to 6 directions, where our feet are toward to down/bottom direction, and head to up/top direction. Then we would have north/forward, south/backward, west/left, east/right, down/bottom and up/top, enough to use in our 3-dimensional world.

How many directions do we need?

Since we have discussed these concepts with our practical daily-life experience and instincts until now, I would want to talk a little bit further more about some interesting things hidden beneath. Why there are only 6 directions available? Why don’t we just use 5 or 3 or 4 or 8 or 12 instead? Let’s assume if we just living in a 1-dimensional world, then we would only care about moving forward or backward because a 1D property only could augment or diminish (like hot/cold, fast/slow, big/small). And when we say move forward or backward, it actually means that with respect to the position where we are currently, if we want to move, we will always get away from the current position and move closer to another position where it is far/augment (the partner of close/diminish) from the current position. And then we must tell our friend Jerry move to one specific direction in two at least. But what we considered about is just an infinite and “straight” 1D world (or better we mathematically call it a space), if our 1D space is a “cycle”, we could just tell Jerry “go anywhere you want you would arrive finally” (ignore the fact that he will be tired and stop sooner or later. But have you ever seen Jerry in “Tom and Jerry” becoming tired once? Send me an e-mail about the episode number if you know!), because theoretically it’s not so important to say forward or backward there, everything is just in front of and in behind of everything at the same time, magic!

And then we could think about that, what it would look like in the 2-dimensional space? The first of the first step is how we would construct a 2D space. As we’ve talked about above, 2D spaces may have more weird structures than 1D spaces, but let’s start with the first simplest one 2D space appeared in your mind, an infinite flat. In 1D space, we would have a kind of unique freedom to move forward or backward without any restriction (just ideally speaking, Friedrich Hayek wrote a thick book about “real freedom” in 20th century), and we name the quantity of this freedom as Degrees of freedom (abbr. DOF, this concept was introduced in statistics first). And now we could say in 1D space the DOF is 1, and in 2D space, the DOF is 2. Then a conclusion coming naturally that there should be 2 different unique freedoms in 2D space which we could move along separately, and it would just give us 2 + 2 = 4 directions at total in our infinite flat (you may wonder “can’t we go any ‘directions’ as we want?”, yes, practically we could, but all the movements could only be decomposed to 4 orthogonal directions or 2 dimensions, that’s the minimum answer, we can’t use fewer directions than 4 to describe the movement in 2D Euclidian space). In the 17th century, one smart guy, the great mathematician and philosopher René Descartes invented a kind of analytical convention to talk about space, which is widely used today as the Descartes coordinate. Briefly speaking it use one character to represent a dimension and +/- to indicate the directions along this dimension. With the help of it, we could say that our infinite flat is an x-y coordinate system and we could do the analytical experiment now.

Let’s take a look at other possible structures of 2D space, what if an infinite flat was folded by some mysterious hand and changed to another structure which looks like the “round” shape 1D world? There would be more complex cases than 1D space in 2D spaces, if we fold the +x and weld it with -x, then we’ll get a cylinder shape 2D space; If we’re continuously folding the +y and merge it with -y, we’ll get a circle-like tube, and it has been named as the torus; and if we merge +x, -x, +y and -y, we’ll get a sphere shape 2D space like the surface of our planet Earth. And in all of these structures, we would get some additional flexibility, that we could travel without considering the direction because this kind of wrapping operation fundamentally changed the infinity of space to a “self-repeated” finite. In 2D space there are two individual quantities, so we could construct 3 different categories of structures which are full infinity (“flat”), partial infinity (“cylinder” or “torus” in the cycled x-axis or y-axis) and full finite (“sphere”). But strictly speaking, if we really lived in 2D space, it would be very hard for us to comprehend the real structure of our world. When we say “flat”, “cylinder” and “sphere”, we are thinking and observing them in our 3D space, and we have an additional dimension to help us understand the shape of the 2D spaces. This is the same struggling that we’ve gotten when trying to understand the real shape of our universe, let’s leave the headaches to cosmologists now!

Furthermore, if you constructed the partial infinity 2D space (“cylinder”) with a flipped finite dimension you’ll get a very interesting structure. Let’s find a paper strip as the “flat” infinite 2D space and draw an x-y coordinate marker on it, then we stick the shorter edge together and will get a normal “cylinder” partial infinity 2D space like a bracelet. And let’s make another one, but this time we flip the shorter edge upside down before we stick them together, and we’ll just get a strange bracelet which only has one “edge” and one surface! If you don’t believe, just use a pen to draw a line along the direction which the strip is longer as Jerry’s footprint, you’ll find he could travel back where he is as same as the situation on a regular “cylinder”! This is called “Möbius strip” and it’s still a 2D space. Actually, the knot in 1D space is a similar structure as Möbius strip in 2D space, they both have the ability to intersect with themselves.

Finally, we could turn our head to the real 3-dimensional (simply speaking) space. We’ll have 3 DOFs, and it means we have 2 x 3 = 6 directions at total, and of course, we could use our folding tricks that we’ve talked before to construct some more complex and amazing space structures. These are the domains which the Topology mathematicians, Differential Geometry mathematicians, and cosmologists researching in, and they would use some terminologies like “manifold”, “set”, and “group” often to discuss the topics we’ve talked about. Let’s focus on the game development now, a simple infinite x-y-z Euclidean 3D space is all we need to make Jerry appearing on our screen. But as soon as we start to declare some classes structures and functions in our codebase to represent the position in 3D space, an inevitable problem appears immediately.

In 2D space, if we’ve already decided how long 1 distance unit it is, we could just use an x-y pair to represent any positions in space. It is written like $P(x_p, y_p)$ as the coordinate of a point $P$. In 3D space it’s the same, $P(x_p, y_p, z_p)$. And in the 2D space y-axis is always vertical with the x-axis, that means when you defined any one of the axes then you’ll get another one at the same time, the direction of the axis is trivial because you could simply rotate one version of them to get another one. But let’s see what would happen in 3D space, you first may find an x-axis, and use it to construct another y-axis by adding a vertical dimension, and then the problem came, the only one left, the z-axis, its positive direction could be placed in two opposite ways, and it’s impossible to rotate the two versions of the coordinate system to be identical! Like the picture shown below, both of them represent 3 unique orthogonal dimensions and theoretically correct, which means we don’t have any absolute reasons to choose one of them beside another! In practice, we need to choose once and always remember our choice! What a marriage:).

This specialty of 3D space is both fancy and messy, somehow it relates to one of the basic physical quantities what I mentioned before, and it is called chirality in the context of chemistry, which is used to describe one geometric property of those little ions and molecules if they’re existing in both mirrored versions. We usually use the Left-Handed Coordinate System (abbr. LHS) and the Right-Handed Coordinate System (abbr. RHS) to indicate them.

In the box

So how these pieces of knowledge could help us in the game development domain? As you may understand now, the screen in front of us is a finite 2D space, but what we want is displaying a 3D space “on” it or better to say, project a 3D space to our screen. It’s similar to how photography would perform to this objective world. But before the final “shutter” opens and closes to get our “photo”, we must calculate the correct position of everything inside our virtual 3D space. The translation operation just adds some offsets to the local space position of the object and gets the final world space position, and if we use a linear transformation expression it is just a multiplication between some vectors and some translation matrices. But when we want to deal with rotation, things become a bit complex.

Since there are lots of articles blogs tutorials videos books talking about rotation, I assume you’ve already understood the very basic rotation calculation method in 2D space from your elementary algebra class. But in case your memory faded a little, let’s take a brief review right now. If we know the rotation angle and the initial vector, we could use the trigonometry function to get the result vector, or we could write it in a linear algebra favor like (let’s use $\theta$ as the rotation angle to the counterclockwise direction):

\begin{bmatrix} x_2 \\ y_2 \\ \end{bmatrix} = \begin{bmatrix} x_1 \\ y_1 \\ \end{bmatrix} * \begin{bmatrix} \cos_\theta \sin_\theta \\ -\sin_\theta \cos_\theta \\ \end{bmatrix}

As same as:

\begin{bmatrix} x_2 \\ y_2 \\ \end{bmatrix} = \begin{bmatrix} \cos_\theta * x_1 -\sin_\theta * y_1 \\ \sin_\theta * x_1 + \cos_\theta * y_1 \\ \end{bmatrix}

Actually it is just a rotation on the x-y flat around +z/-z axis in 3D space, similarly, we could have another two rotations around +x/-x and +y/-y axis. And we could combine them together to get a very scary looking generalized 3D rotation matrix. And if you really spent some minutes to deduce it, you’ll see the method we used here is just multiplying 3 individual rotations to get 1 combined rotation. If you think about “3 operations to 1” a little bit deeper, you’ve already gotten close to the limitation of this kind of rotation technique, which we typically called as Euler Angles rotation representation.

The problem lies beneath is, when we say, “rotate”, what we actually want to express is “rotating around one axis”, this is the nature of rotation. And in the Euler Angle rotation technique, the axes are those 6 orthogonal axes that made our coordinates system. Then since the general rotation is always the linear combination of 3 individual axis-dependent rotations, once if we rotated any 90 degrees, we would get that some of our axes overlapped with others, and then we can’t rotate around anymore because we lost a DOF! Do one 90 degrees rotation on paper, and then try to do another one around the axes which aren’t the rotation axis, you will find you’re just stuck there. This is the most serious problem Euler Angles rotation brought us, it even has a name called Gimbal Lock.

The magic comes

So is there any other choices we could have? We need some more elegant solutions that the rotation axis could be arbitrary. Luckily it’s 2018 now, the problems I listed above have already been solved perfectly around 40 years ago by Ken Shoemake (it’s SIGGRAPH ‘85!), and later another great blog post by Nick Bobic 20 years ago also demonstrated the solution, the Quaternion it is.

Since it’s easy to find lots of detailed information online about it, I’d rather write some concise concepts and properties about Quaternion here first, in case one day I forget and need them again and have to search again around. Quaternion is one kind of Hypercomplex number, just similar to the complex number but extended to 3 imaginary parts, typically written as $\mathrm{Q} = a + bi + cj + dk$, while $i^0 = j^0 = k^0 = 1$, $i^2 = j^2 = k^2 = ijk = -1$, exactly similar with complex number if you knew it. Another kind of representation is $\mathrm{Q} = {(}\mathrm{Q}{s, } \mathrm{Qv)}$, a scalar real part with a 3D vector imaginary part. But if you never ever occurred to the complex number before, it’s better to introduce it first because really we don’t use it in our daily life, at least not directly.

In a mathematical point of view, if we calculate the square root of $9$, we may immediately know it is $3$, because we do reverse engineering a little bit then found $3^2 = 9$, and there is no hesitation that we could answer $\sqrt{9} = 3$ or $9^{1/2} = 3$. But if we ask what the square root of $-9$ it is, we would be stuck here because there is no way to do some reverse engineering anymore, we don’t know whose square is $-9$, all numbers we’ve known yet all have positive square (or $0$). As usual, if there are some exceptions in our math, that simply means we need to extend the theory to include them. Mathematicians invented a new number element $i$ as the solution for the problem like $\sqrt{-9}$, and it’s called “imaginary number (set $\mathbb{I}$)”, which is the square root of $-1$. Then $\sqrt{-9} = \sqrt{-1 * 9} = \sqrt{-1} * \sqrt{9} = i * 3 = 3i$, and all the numbers whose square is positive now all belong to a set called “real number (set $\mathbb{R}$)”. But the imaginary number only solves a mathematical problem, we couldn’t find any of it in reality because it is really something “imaginary”! Then is it just some decoration for the completion of number theory?

When we talk about something in the time domain, for example, the sound you heard from your headphone or electric current passing through the wire or how often we go to a kebabs restaurant, they all have a property called frequency, which represents the number of occurrences based on some measurements. And we’ve found that using a sinusoidal model to express frequency is a good initial approach because it satisfies the requirement of the frequency definition. But if we just wrote the sinusoid as $y = \sin(x)$ and tried to implement it in our codebase with the original definition it would be quite inefficient and non-intuitive. We would always need to get $y$ from the basic definition of sinusoidal trigonometry, it means constructing a unit triangle then measure the lengths of edges then calculate, and I don’t even understand how to implement it because it’s such a chicken-and-egg problem (or if some people know how to utilize Taylor Series then could calculate by it, this is a nice and practical approximation in some math libraries)! Luckily great mathematician Euler found a very beautiful formula as Euler’s formula, which expressed the relationship of the trigonometric functions and the complex exponential function (also the relationship of geometry and the algebra), use it we could simplify the calculation because we don’t need to construct anything for the assistant purpose, just $e^{ix} = \cos(x) + i\sin(x)$, and $\cos(x) = \frac{e^{ix} + e^{-ix}}{2}$, $\sin(x) = \frac{e^{ix} - e^{-ix}}{2i}$ vice versa.

This is one of the most apparent usages of the imaginary number (or with real number together as the complex number (set $\mathbb{C}$)). If you could understand the rotation in 2D space which I talked previously, then you might see some similarities here, we could just change the 2D Descartes coordinate’s axis from 2 real number axes to a “2D” complex number axis where the imaginary number replaces real number on the y-axis. And what we had before all remains the same because we only changed the axis name! A complex number “2D vector” could just translate, rotate, and scale. If we wrote the 2D rotation matrix again, it now becomes:

\begin{bmatrix} \cos_\theta -\sin_\theta i \\ \sin_\theta + \cos_\theta i \\ \end{bmatrix}

(This kinda useless operation actually indicate an important mathematical fact, that a 2-tuples vector space over the real number field is isomorphic (essentially same) as a 1-tuple vector space over the complex number field.)

An extension not extended so well

And then we could try to mimic a similar alternative rotation representation in the 3D space, but before we start let’s rearrange a little our mathematical knowledge first. If we want to use the complex number to represent 2D rotation, we are actually dancing inside an algebra called Clifford algebra, an unital associative (means $(A * B) * C = A * (B * C)$) algebra, not our dear linear “algebra” (strictly speaking, there is no linear algebra, what we’d talk about daily is the linear transformation inside Euclidean space) anymore, which requires us use the quadratic form to construct the element inside the vector space (mathematically written as $Q: V \to K$, where $Q$ is the quadratic form, $V$ is a vector space and $K$ is a field). “2D” complex number vector space, for example, could be expressed as a quadratic form with one basis element $i$ that satisfies $i^2=-1$.

And if we keep extending our complex number system, we may have a guess first that we could use a 3D Hypercomplex number vector space where we would just append another imaginary element to the 2D complex number vector space. But unfortunately, this kind of space can’t exist inside Clifford algebra directly. Let’s have a not strict but easy to understand experiment to prove this. Assume a “3-components Hypercomplex number” would form like $a + bi + cj$, if we define $i^2 = -1$ and $j^2 = -1$, it looks fine now, but we need to abandon the commutativity, because if $ij = ji$ then we can’t distinguish $i$ and $j$ anymore, they would become identical rather than the two unique elements. Only if $i \neq j$, then $ij \neq i^2, ij \neq -1$, and then assume $ij = -ji = 1$ these new invented elements could satisfy our requirement. Let’s verify if it is associative, first we calculate $(i * i) * j = -1 * j = -j$, and $i * (i * j) = i * 1 =i$, so if it was associative (means $(i * i) * j = i * (i * j)$), then we could get $-j = i$, then let’s use this conclusion to calculate $i^2$, now $i^2 = -j * i = -ji$, while previously we assume $-ji = 1$, now $i^2 = 1$, it is conflict with the original definition $i^2 = -1$!

No associativity, no Clifford algebra, we can’t use this system we just built to help us with 3D rotation. Actually we’ve already found that there is no meaningful 3D hypercomplex number space at all (the correct way to construct hypercomplex number is called Cayley-Dickson construction)! But what if we use a 4D hypercomplex number space instead? Anyway, the 3D Euclidean vector space is a sub-space of 4D space. This is how Quaternion came into the game, it is just a Clifford algebra that has 2 basis elements that satisfy $i^2 = j^2 = k^2 = ijk = -1$ (while $k$ is actually the product of $i$ and $j$, so we say 2 basis elements rather than 3). With the abandonment of the commutativity and have an chain-like non-commutativity like $ij = k$, $jk = i$, $ki = j$, $ji = -k$, $kj = -i$, $ik = -j$, it would have the associativity like $(i * i) * j = i * (i * j), -j = ik, j = ki$. And then if we interpret our 3D vector space just as the subset of a quaternion space, then the rotation just becomes an axis-lock-free operation, a simple quaternion multiplication.

But how? Let’s jump back to 2D complex space first. We deduced a rotation matrix before, and the result looks like:

\begin{bmatrix} x_2 = \cos_\theta * x_1 -\sin_\theta * y_1 \\ y_2 = \sin_\theta * x_1 + \cos_\theta * y_1 \\ \end{bmatrix}

Let’s rewrite it with a complex number favor:

\begin{bmatrix} a_2 = \cos_\theta * a_1 + \sin_\theta i * i * b_1 \\ b_2 i = \sin_\theta i * a_1+ \cos_\theta i * b_1 \\ \end{bmatrix}

Same as:

$a_2 + b_2 i = (\cos_\theta + \sin_\theta i) * a_1 + (\sin_\theta i + \cos_\theta) * b_1 i$,

Change the order:

$a_2 + b_2 i = (\cos_\theta + \sin_\theta i) * a_1 + (\cos_\theta + \sin_\theta i) * b_1 i$,

Then:

$a_2 + b_2 i = (\cos_\theta + \sin_\theta i) * (a_1 + b_1 i)$,

We could say this $\cos_\theta + \sin_\theta i$ is our complex number “rotator” $q$, and if we write the original vector $a_1 + b_1 i$ as $p_1$, then we could get $p_2 = qp_1$, or write this equation as a linear transformation like:

\begin{bmatrix} a_2& -b_2 \\ b_2 & a_2 \end{bmatrix}= \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix} \begin{bmatrix}a_1 & -b_1 \\ b_1 & a_1 \end{bmatrix}

And now let’s extend our 2D $p_1$ to 4D as a quaternion $p_1(a_1 + b_1 i + c_1 j + d_1 k)$, then if we rotate it with a “rotator” $q$, we could use exactly what we deduced above, $q = \cos_\theta + \sin_\theta i + \sin_\theta j + \sin_\theta k$, and $p_2 = qp_1$ to calculate the rotation in 4D quaternion space. And since now on we have 3 imaginary parts instead of 1, the 4D rotation must be constructed around an arbitrary axis $v(a_v + b_v i + c_v j + d_v k)$, then the generalized rotator would be $q = \cos_\theta + \sin_\theta i b_v + \sin_\theta j c_v + \sin_\theta k d_v$.

But how to insert our 3D space rotation into it? We could imagine, 2D space is just a flat in 3D space, a 3D space should be a “hyperflat” in 4D space too (the so-called subset), then if we want to make a 3D sub-space $p$ inside 4D quaternion space, we would add a specific restriction like $a = 0$ (make one variable constant) to describe a hyperflat and finally, we would get $p(0, b i, c j, d k)$ (could give them a name like Pure Quaternion). We could just put our 3D vector inside $p$ as the scale of the imaginary part, and reform the rotation axis and the angle into a quaternion rotator $q$.

Runtime

Let’s see if it works or not. For the sake of readability, let’s use another notation, if we write quaternion as $q = (qs, qv)$, then the multiplication could be written as $q1 * q2 = (q_1s · q_2s - q_{1V} · q_{2V}, q_1s · q_{2V} + q_2s · q_{1V} + q_{1V} × q_{2V})$.

Assume:

$p_1(0, p_{1V})$,

And:

$q(q_s, q_V)$,

Then:

$p_2 = qp_1$,

Then (quaternion multiplication):

$p_2 = (q_s, q_V) * (0, p_{1V})$,

Now use associativity (notice the differences between cross product and dot product, I’ve omitted the details):

$p_2 = (0 · q_s - p_{1V} · q_V, 0 · q_V + q_s · p_{1V} + p_{1V} \times q_V)$,

Same as:

$p_2 = ( - p_{1V} · q_V, q_s · p_{1V} + p_{1V} \times q_V)$,

As you could see the scalar part is not 0, means we got a $p_2$ not pure quaternion anymore, and we can’t convert it back to 3D space!

So what could we do now? Let’s think what we did above like, we rotate a 4D (pure) quaternion $p_1$ from a 3D hyperflat away into the 4D space, then if we rotate it back to 3D hyperflat, everything would become sweet again. So how could we achieve this goal? Since we’ve rotated it away, then we could keep rotating until it falls again on the 3D hyperflat. In our virtual world, any translation rotation and scaling operation should be totally revertible, which means we need something like A $operator$ B = B $operator$ A or generally speaking, we need to find a path to reconstruct A through lots of same binary operation like A $operator$ B = C, C $operator$ A = B.

Because the non-commutative nature of the quaternion space multiplication exists, we must find a way to return to the 3D hyperflat without hurting our rotation result. You may feel that there is no way to keep rotating and finally fall back to the 3D hyperflat unless we rotate really “backward” in the invert directions where we came, but hey, we are living in a 3D world, just think about 2D Jerry, he never knows how the additional dimension helps us to construct 4 different 2D space shape for him! Let’s put Jerry to 4D quaternion space, he would see two additional directions than us, he has 8 directions now! And if we asked him “hey Jerry I rotate my forward direction vector in your 4D space and got lost, could you help me to rotate it back in your additional dimension?”, if you didn’t cheat him in that 1D round shape space at the very beginning, he may answer “Ok let’s rotate!” happily to you. But how do we operate it?

If we want to rotate something back in 2D complex space, we must rotate in an invert direction, that means we need to calculate the invert of the rotator, it is defined as $q^{-1} = \frac{\overline{q}}{||q||^2}$, and this $\overline{q}$ is called the conjugation of $q$, defined as $\overline{q}(a - bi)$. If we normalized the rotator before (divided it by its magnitude $||q|| = \sqrt{a^2+ b^2}$) then we could just say for the unit rotator (magnitude is 1 now), its conjugation is its invertion. Now we could just alter the 2D rotator’s imaginary part to the opposite direction to get the inverted rotator, means to change $q(\cos_\theta + \sin_\theta i)$ to $\overline{q}(\cos_\theta - \sin_\theta i)$ (or interpret as $\sin(-x) = -\sin x$, rotate to opposite direction with same angle). Then we just do a $q^{-1} p_2$ or a $p_2 q^{-1}$ and we’ll get $p_1$.

But in 4D quaternion space, we must consider about the difference between $q^{-1} p_2$ and $p_2 q^{-1}$ because the non-commutativity restrains us. Let’s clear our mind a little, now everything in 2D complex space are rotators and vectors at the same time, it’s reasonable because after all, they share the same convention as $p(a + bi)$, then a rotation $p_\alpha p_\beta$ could be interpreted in two ways, $p_\alpha$ rotated by $p_\beta$ in clockwise direction or, $p_\beta$ rotated by $p_\alpha$ in counterclockwise direction. And with this mindset, we could say $p_\alpha p_\beta = p_\beta p_\alpha^{-1}$, as “$p_\beta$ rotated by $p_\alpha$ in counterclockwise direction is as same as $p_\beta$ rotated by $p_\alpha^{-1}$ in clockwise direction”.

Now let’s generalize the conclusion and see what we get with a $qp \overline{q}$ operations. With the help of the deduction what we did before:

$p = (0, p_V)$, $q = (q_s, q_V)$ and $\overline{q}(q_s, -q_V)$

Then we could get:

$qp = (q_s · 0 - q_V · p_V, q_s · p_V + 0 · q_V + q_V \times p_V)$, $p’ = qp = (- q_V · p_V, q_s · p_V + q_V \times p_V) = (p’_s, p’_V)$,

then $p’’ = p’q = qp \overline{q} = (- q_V · p_V, q_s · p_V + q_V \times p_V) * (q_s, -q_V) = (p’_s, p’_V) * (q_s, -q_V) = (p’’_s, p’’_V)$

Let’s expand it step by step (be careful about the negative sign of the conjugated $\overline{q}$):

$(p’’_s, p’’_V) = (p’_s · q_s + p’_V · q_V, - p’_s · q_V + q_s · p’_V - p’_V \times q_V)$

Replace with the original $p$:

$(p’’_s, p’’_V) = [(- q_V · p_V) · q_s + (q_s · p_V + q_V \times p_V) · q_V, - (- q_V · p_V) · q_V + q_s · (q_s · p_V + q_V \times p_V) - (q_s · p_V + q_V \times p_V) \times q_V]$

Use distribution law and commutativity of dot products, and omit the vector part since we don’t care about it now:

$(p’’_s, p’’_V) = [- q_s · q_V · p_V + q_s · q_V · p_V + q_V \times p_V · q_V, p’’_V]$

And finally:

$(p’’_s, p’’_V) = [q_V \times p_V · q_V, p’’_V]$

Because the result vector of $q_V \times p_V$ is orthogonal with $q_V$ and $p_V$, so the dot product between it and $q_V$ or $p_V$ would be always 0, and what we finally deduced is a pure quaternion because the $p’’s$ is perfectly canceled by each component inside, and this result means we safely arrive at our 3D world again. If you compare the result of quaternion rotation method with the Euler angle rotation method, you would find an interesting fact that if we rotate any 3D vector by quaternion rotation method with a certain angle, the result is exactly the same like if we rotate it by the Euler angle rotation method twice. It caused simply by the truth that we actually rotate twice, once in 4D “clockwise” direction and another time in 4D “counterclockwise” direction. It’s very easy to deal with this problem, just simply change the rotator to $q = \cos\frac{\theta}{2} + \sin_\frac{\theta}{2} i b_v + \sin\frac{\theta}{2} j c_v + \sin_\frac{\theta}{2} k d_v$ would make everything look perfect.

Another implicit problem is, which we discussed above should better be calculated with normalized quaternion, otherwise, the magnitude of the vectors would influence our rotation results. Because the quaternion we used here is just the combination of rotation axis and angle, the coupling relation of them exists, if you didn’t normalize it after any multiplication, then the growth of the precision error would pollute your results and bring the overflow problem. Also, since we need the invert of the quaternion rotator, if we first normalized them like $q = \frac{q}{||q||}$, then we could use the conjugated quaternion to replace the inverted quaternion directly like $p_2 = qp_1q^{-1}$.

For further detailed formulae works I advise to read this Wikipedia, since I’m more focusing on practical code than theory (although already talked too many theoretical things), with the help of quaternion, we could just use a 4D vector class or structure to build the almost entire math library in our codebase, I would show a concise example about one optimized quaternion rotation from the Lemma 4 of Geometric Skinning with Approximate Dual Quaternion Blending as $p_2v = p_2v+2qv × (qv × p_1v+qs · p_1v)$.

// V' = QVQ^-1, for unit quaternion, the conjugated quaternion is as same as the inverse quaternion

 // naive version
 // get Q * V by hand
 //vec4 l_hiddenRotatedQuat;
 //l_hiddenRotatedQuat.w = -m_rot.x * l_directionvec4.x - m_rot.y * l_directionvec4.y - m_rot.z * l_directionvec4.z;
 //l_hiddenRotatedQuat.x = m_rot.w * l_directionvec4.x + m_rot.y * l_directionvec4.z - m_rot.z * l_directionvec4.y;
 //l_hiddenRotatedQuat.y = m_rot.w * l_directionvec4.y + m_rot.z * l_directionvec4.x - m_rot.x * l_directionvec4.z;
 //l_hiddenRotatedQuat.z = m_rot.w * l_directionvec4.z + m_rot.x * l_directionvec4.y - m_rot.y * l_directionvec4.x;

 // get conjugated quaternion
 //vec4 l_conjugatedQuat;
 //l_conjugatedQuat = conjugate(m_rot);

 // then QV * Q^-1
 //vec4 l_directionQuat;
 //l_directionQuat = l_hiddenRotatedQuat * l_conjugatedQuat;
 //l_directionvec4.x = l_directionQuat.x;
 //l_directionvec4.y = l_directionQuat.y;
 //l_directionvec4.z = l_directionQuat.z;

 // traditional version, change direction vector to quaternion representation

 //vec4 l_directionQuat = vec4(0.0, l_directionvec4);
 //l_directionQuat = m_rot * l_directionQuat * conjugate(m_rot);
 //l_directionvec4.x = l_directionQuat.x;
 //l_directionvec4.y = l_directionQuat.y;
 //l_directionvec4.z = l_directionQuat.z;

 // optimized version ([Kavan et al. ] Lemma 4)
 //V' = V + 2 * Qv x (Qv x V + Qs * V)
 vec4 l_Qv = vec4(m_rot.x, m_rot.y, m_rot.z, m_rot.w);
 l_directionVec4 = l_directionVec4 + l_Qv.cross((l_Qv.cross(l_directionVec4) + l_directionVec4.scale(m_rot.w))).scale(2.0f);

Crossfade sometimes matters

The interpolation of the quaternion is more special than the typical vectors, we have three common interpolation methods, Linear Interpolation (abbr. LERP), Spherical Linear Interpolation (abbr. SLERP) and Normalized Linear Interpolation (abbr. NLERP).

LERP is a special case of polynomial interpolation, defined as $R={\alpha}A + (1-\alpha)B$, which is commonly used with the translation or scaling. If we use LERP to interpolate two quaternion rotators and let the $\alpha$ changes constantly, then when it arrives near the middle point 0.5, the changing speed of quaternion would become slower. It’s because that the rotation of the unit quaternion actually could be imagined as a point which moves on the surface of a 4D hypersphere, LERP is just interpolating on the secant line between the beginning point and the end point, the speed on the secant line is constant, then on the arc, the speed must be variant.

In order to optimize this phenomenon, SLERP was invented, defined as $R=\frac{\sin(\alpha\Omega)}{\sin(\Omega)}A + \frac{\sin((1 - \alpha)\Omega)}{\sin(\Omega)}B$, and it depends on the angle between two quaternions. We could get it by calculating the arccos shown in the code examples below. As you could see, it’s more computationally expensive, and we have to cover some extreme conditions in practical, but it could provide us a way more smooth interpolation result.

The NLERP is a balanced or compromised solution between LERP and SLERP, basically, it does LERP first, but then normalized the result to compensate the error. Jonathan Blow wrote a nice blog about them, I highly recommend reading it.

vec4 vec4::lerp(const vec4 & a, const vec4 & b, double alpha) const
{
 return a * alpha + b * (1.0 - alpha);
}

vec4 vec4::slerp(const vec4 & a, const vec4 & b, double alpha) const
{
 double cosOfAngle = a * b;
 // use nlerp for quaternions which are too close
 if (cosOfAngle > 0.9995) {
 return (a * alpha + b * (1.0 - alpha)).normalize();
 }
 // for shorter path
 if (cosOfAngle < 0.0) {
 double theta_0 = acos(-cosOfAngle);
 double theta = theta_0 * alpha;
 double sin_theta = sin(theta);
 double sin_theta_0 = sin(theta_0);

 double s0 = sin_theta / sin_theta_0;
 double s1 = cos(theta) + cosOfAngle * sin_theta / sin_theta_0;

 return (a * -1.0 * s0) + (b * s1);
 }
 else
 {
 double theta_0 = acos(cosOfAngle);
 double theta = theta_0 * alpha;
 double sin_theta = sin(theta);
 double sin_theta_0 = sin(theta_0);

 double s0 = sin_theta / sin_theta_0;
 double s1 = cos(theta) - cosOfAngle * sin_theta / sin_theta_0;

 return (a * s0) + (b * s1);
 }
}

vec4 vec4::nlerp(const vec4 & a, const vec4 & b, double alpha) const
{
 return (a * alpha + b * (1.0 - alpha)).normalize();
}

Tech on zhangdoa

Rendering analysis - Cyberpunk 2077

Overview

Pre-geometry passes

Billboard and GUI

Sky visibility/top-view shadow/mini map

Unknown compute pass 01

Unknown compute pass 02

Terrain

Unknown color pass 01

Ocean wave

Geometry passes

Depth-pre pass

Base material passes

DS convert passes

Motion-stencil pass

Mask and LUT passes

Reflection mask

Compute pass to noise normal

Some passes to mask the sky out

(HB)AO (?)

Color-grading LUT

Ocean wave noise

Light and shadow passes

CSM passes for direct light

Omni shadow maps for point/area lights

Cloud distribution

Sky cubemaps

Coat mask (?)

Clustered light index (?)

Direct light shadow

Local lights mask (?)

Unconfirmed compute dispatches to update some StructuredBuffers

Environment Radiance capture

A lot GI-related (?) compute passes

Landscape maps

Local volumetric fog

Head mask

Direct and emissive lights

Sky

SSR

Indirect light

All lights composition pass

Transparent LUT

Skin and eyes

AO(?)/GI shadow

Volumetric fog

Water reflection

Post-processing passes

TAA

Blur

Unknown transparent

Cloud

Holo and transparent objects

Post-TAA

HDR LUT(?)

Bloom

Camera lens effects

Color grading

Gamma correction

GUI elements

Film Grain

Swap chain image

Undrawnable conclusions

How expensive it is to create your own AAA-level game audio solution?

Don’t be greedy, unless it’s really cheap

What’s the matter?

The architecture

The implementation

API style

The lowest low-level details

The Object

WAV instance

Audio Player

Graph Editor

Graph Compiler

Plug-in Hook

Robust is the king

Reflection in C++... could we nail it now?

All we need is just a little patience