3D Glass Portal Card Effect with React Three Fiber and Gaussian Splatting

You’ve probably seen this supercool demo by Max @Aximoris created with a new Spline feature:

This new @splinetool feature (importing Gaussian Splatting) and portal card effect is a perfect match! You should try it!!https://t.co/H3AdPNItb2 pic.twitter.com/xi973iGMXG

— Max (@Aximoris) November 14, 2023

There is a nice video tutorial by Max on how to do that with Spline. But being a developer myself I would like to have more control over the scene, so I decided to replicate it using React Three Fiber (R3F) and Three.js.

A big shoutout to the folks at Poimandres, whose work makes all this possible!

Let’s dive into this effect right away!

Basic scene

Thanks to the amazing R3F community there is a MeshTransmissionMaterial in the Drei repository now. I also took a model to put behind that glass object. Keep in mind that for glass and such materials it is super important to add environment lighting (usually that’s a .hdr or .exr file), so that we’ll be able to see all those nice reflections.

View the live demo/code.

Now, there is our first issue with this. We shouldn’t be seeing the object behind the glass. Only through the glass. The way this material works, it renders the sculpture twice: first as a texture for the glass, and a second time as an actual object.

Luckily, there is an easy way to fix this, just by hiding the object for the final render.

useFrame((state) => {
	sculpture.current.visible = true;
	glass.current.visible = false;
	// this buffer will be used by MeshTransmissionMaterial
	state.gl.setRenderTarget(buffer);
	state.gl.render(state.scene, state.camera);
	// we need this to switch rendering back to "on screen"
	state.gl.setRenderTarget(null);
	sculpture.current.visible = false; // hiding sculpture
	glass.current.visible = true;
  });

And now using the Transmission material props we can set the “behind-the-glass scene” for it manually, like this:

<MeshTransmissionMaterial
      buffer={buffer.texture}
  >

Not only do we achieve the portal effect now, we also optimize the whole scene rendering significantly! That’s because we only render the object once.

This is what we have now:

View the live demo/code.

Splats

3D portal is interesting, but let’s go further and use Gaussian Splatting!

If you haven’t heard about Gaussian Splatting before, I really recommend watching this video, explaining it in detail. TLDW: It’s a new awesome way to render 3D scenes.

In our case it actually opens up a whole real world to be integrated into our 3D scenes. In Max’s original Spline scene, a human scan was used. I saw this as a great opportunity to feature one of the symbols of my beloved Kyiv—the Ukraine Motherland Monument.

The problem is, the sculpture is over 100 meters tall. And you can’t fly civil drones in Ukraine, we have regular air attacks in here, even this very moment I’m writing these words.

So, I went to YouTube and found 4 year old footage of the monument. I uploaded the video directly to the Luma app (but you could also use Polycam). And here is the result:

Honestly, I didn’t expect it to be that good! It amazes me that some old video can be used to recreate objects pretty accurately. But our quest is not finished here. There is lots of unneeded city landscape in the scan, too. And if we download this “splat” file, it is 260MB! Not very usable in a web environment.

Editing the scene

That’s where the editor from Playcanvas developers comes to the rescue. You can just remove the splats you don’t need.

So after cleaning it up the splat file is only 800kb, which is pretty normal for such a 3D model.

Now we can finally export this as .splat format, which can then be used with Three.js!

Back to R3F

Recently, there has been significant work in adapting this rendering technique for use in Three.js. Excitingly, just last week, Paul Henschel successfully ported it to React Three Fiber! If you’re not familiar with Paul, I highly recommend following him to stay at the forefront of the latest developments in 3D web technology!

So, I got a chance and used his recent example to easily add my edited .splat file to my scene. Coding it couldn’t be simpler:

  <Splat
    	scale={2}
    	rotation={[0, 0, 0]}
    	position={[0, 0, 0]}
    	src="my.splat"
  />

Remember, this code is an early version, so there might be changes in the future. It’s likely to become a part of the fantastic Drei collection as a module.

Here’s what it looks like:

Check out the code and live demo in this CodeSandbox.

And we are done! I also added an additional box with stripes, to add some depth. It has side="backside" enabled, so we only see the insides of it, that adds to the illusion.

Here’s another object, a sculpture by Oleksii Zolotariov. I snapped a couple of photos at an exhibition and effortlessly incorporated it into my scene:

View the live demo/code.

I’d love to see what you come up with and how you expand on these demos. Feel free to let me know on Twitter!

In closing, it’s truly fascinating to see the involvement of so many amazing individuals in the rapid spread of this new technology. While I can’t mention everyone, I encourage you to continue supporting all the contributors and open source in general, as it’s the driving force behind these advancements. I also encourage you to sponsor amazing authors of Drei and React Three Fiber.

It’s also mind-blowing that I can record a random video on my phone and have those real-life objects in my Three.js scene within an hour, looking fantastic! Isn’t that incredible?