AR Occlusion
30/10/2025
One of the simplest but most effective pieces of AR tech I developed was a custom occlusion shader. When producing AR visuals, typically all that's happening is an image is rendered on the screen over the top of a camera feed. If the object is tracked well, it can look like it is sitting in the room with you — until a real-world object passes in front of it. Since the AR object is being rendered on top of the camera feed, it will always appear on top of whatever the camera is displaying, even if the virtual object should be behind it. This completely breaks the illusion and pulls the user out of the world you're trying to create for them. The solution to this is called occlusion.
Occlusion is the effect of covering up or hiding parts of an image to create the illusion that an object is passing in front of it. To create this effect, I wrote an HLSL shader that took depth map data from the device and compared it against the depth data from the virtual scene camera. For each pixel, the shader rendered whichever surface was closer to the camera. The result is pretty cool.
With occlusion in place, the illusion is much stronger. When you hold your hand up in front of the camera, you can cover up any game objects behind it. A small detail, but the result is powerful. It had its problems, though. The device depth map was noisy and relatively low resolution, hence the pixelated edges around my hand. The depth map provided by the ARKit framework is actually upscaled from the raw data with some clever algorithms. This led to some visual artifacts in certain conditions and left a flickery effect near the edges. I came up with an idea of how to clean this up and create clean occlusion borders that match the resolution of the image, but I never got the chance to try it.
The effect was good enough for what we needed at the time, and so we moved on to other challenges. Here's a video with the turret placed in a wall and me passing my hand through the barrels. The sense of presence this feature adds to the AR experience is immense.
It was a focused piece of rendering work, but it made the whole interaction feel significantly more grounded and tactile.