Scroll, Refraction and Shader Effects in Three.js and React

In this tutorial I will show you how to take a couple of established techniques (like tying things to the scroll-offset), and cast them into re-usable components. Composition will be our primary focus.

In this tutorial we will:

• build a declarative scroll rig
• mix HTML and canvas
• handle async assets and loading screens via React.Suspense
• add shader effects and tie them to scroll
• and as a bonus: add an instanced variant of Jesper Vos multiside refraction shader

Setting up

We are using React, hooks, Three.js and react-three-fiber. The latter is a renderer for Three.js which allows us to declare the scene graph by breaking up tasks into self-contained components. However, you still need to know a bit of Three.js. All there is to know about react-three-fiber you can find on the GitHub repo’s readme. Check out the tutorial on alligator.io, which goes into the why and how.

We don’t emulate a scroll bar, which would take away browser semantics. A real scroll-area in front of the canvas with a set height and a listener is all we need.

I decided to divide the content into:

• virtual content sections
• and pages, each 100vh long, this defines how long the scroll area is

function App() {
  const scrollArea = useRef()
  const onScroll = e => (state.top.current = e.target.scrollTop)
  useEffect(() => void onScroll({ target: scrollArea.current }), [])
  return (
    <>
      <Canvas orthographic>{/* Contents ... */}</Canvas>
      <div ref={scrollArea} onScroll={onScroll}>
        <div style={{ height: `${state.pages * 100}vh` }} />
      </div>

scrollTop is written into a reference because it will be picked up by the render-loop, which is carrying out the animations. Re-rendering for often occurring state doesn’t make sense.

A first-run effect synchronizes the local scrollTop with the actual one, which may not be zero.

Building a declarative scroll rig

There are many ways to go about it, but generally it would be nice if we could distribute content across the number of sections in a declarative way while the number of pages defines how long we have to scroll. Each content-block should have:

• an offset, which is the section index, given 3 sections, 0 means start, 2 means end, 1 means in between
• a factor, which gets added to the offset position and subtracted using scrollTop, it will control the blocks speed and direction

Blocks should also be nestable, so that sub-blocks know their parents’ offset and can scroll along.

const offsetContext = createContext(0)

function Block({ children, offset, factor, ...props }) {
  const ref = useRef()
  // Fetch parent offset and the height of a single section
  const { offset: parentOffset, sectionHeight } = useBlock()
  offset = offset !== undefined ? offset : parentOffset
  // Runs every frame and lerps the inner block into its place
  useFrame(() => {
    const curY = ref.current.position.y
    const curTop = state.top.current
    ref.current.position.y = lerp(curY, (curTop / state.zoom) * factor, 0.1)
  })
  return (
    <offsetContext.Provider value={offset}>
      <group {...props} position={[0, -sectionHeight * offset * factor, 0]}>
        <group ref={ref}>{children}</group>
      </group>
    </offsetContext.Provider>
  )
}

This is a block-component. Above all, it wraps the offset that it is given into a context provider so that nested blocks and components can read it out. Without an offset it falls back to the parent offset.

It defines two groups. The first is for the target position, which is the height of one section multiplied by the offset and the factor. The second, inner group is animated and cancels out the factor. When the user scrolls to the given section offset, the block will be centered.

We use that along with a custom hook which allows any component to access block-specific data. This is how any component gets to react to scroll.

function useBlock() {
  const { viewport } = useThree()
  const offset = useContext(offsetContext)
  const canvasWidth = viewport.width / zoom
  const canvasHeight = viewport.height / zoom
  const sectionHeight = canvasHeight * ((pages - 1) / (sections - 1))
  // ...
  return { offset, canvasWidth, canvasHeight, sectionHeight }
}

We can now compose and nest blocks conveniently:

<Block offset={2} factor={1.5}>
  <Content>
    <Block factor={-0.5}>
      <SubContent />
    </Block>
  </Content>
</Block>

Anything can read from block-data and react to it (like that spinning cross):

function Cross() {
  const ref = useRef()
  const { viewportHeight } = useBlock()
  useFrame(() => {
    const curTop = state.top.current
    const nextY = (curTop / ((state.pages - 1) * viewportHeight)) * Math.PI
    ref.current.rotation.z = lerp(ref.current.rotation.z, nextY, 0.1)
  })
  return (
    <group ref={ref}>

Mixing HTML and canvas, and dealing with assets

Keeping HTML in sync with the 3D world

We want to keep layout and text-related things in the DOM. However, keeping it in sync is a bit of a bummer in Three.js, messing with createElement and camera calculations is no fun.

In three-fiber all you need is the <Dom /> helper (@beta atm). Throw this into the canvas and add declarative HTML. This is all it takes for it to move along with its parents’ world-matrix.

<group position={[10, 0, 0]}>
  <Dom><h1>hello</h1></Dom>
</group>

Accessibility

If we strictly divide between layout and visuals, supporting a11y is possible. Dom elements can be behind the canvas (via the prepend prop), or in front of it. Make sure to place them in front if you need them to be accessible.

Responsiveness, media-queries, etc.

While the DOM fragments can rely on CSS, their positioning overall relies on the scene graph. Canvas elements on the other hand know nothing of the sort, so making it all work on smaller screens can be a bit of a challenge.

Fortunately, three-fiber has auto-resize inbuilt. Any component requesting size data will be automatically informed of changes.

You get:

• viewport, the size of the canvas in its own units, must be divided by camera.zoom for orthographic cameras
• size, the size of the screen in pixels

const { viewport, size } = useThree()

Most of the relevant calculations for margins, maxWidth and so on have been made in useBlock.

Handling async assets and loading screens via React.Suspense

Concerning assets, Reacts Suspense allows us to control loading and caching, when components should show up, in what order, fallbacks, and how errors are handled. It makes something like a loading screen, or a start-up animation almost too easy.

The following will suspend all contents until each and every component, even nested ones, have their async data ready. Meanwhile it will show a fallback. When everything is there, the <Startup /> component will render along with everything else.

<Suspense fallback={<Fallback />}>
  <AsyncContent />
  <Startup />
</Suspense>

In three-fiber you can suspend a component with the useLoader hook, which takes any Three.js loader, then loads (and caches) assets with it.

function Image() {
  const texture = useLoader(THREE.TextureLoader, "/texture.png")
  // It will only get here if the texture has been loaded
  return (
    <mesh>
      <meshBasicMaterial attach="material" map={texture} />

Adding shader effects and tying them to scroll

The custom shader in this demo is a Frankenstein based on the Three.js MeshBasicMaterial, plus:

• the RGB-shift portion from DigitalGlitch
• a warping effect taken from Jesper Landberg
• and a basic UV-coordinate zoom

The relevant portion of code in which we feed the shader block-specific scroll data is this one:

material.current.scale =
  lerp(material.current.scale, offsetFactor - top / ((pages - 1) * viewportHeight), 0.1)
material.current.shift =
  lerp(material.current.shift, (top - last) / 150, 0.1)

Adding Diamonds

The technique is explained in full detail in the article Real-time Multiside Refraction in Three Steps by Jesper Vos. I placed Jesper’s code into a re-usable component, so that it can be mounted and unmounted, taking care of all the render logic. I also changed the shader slightly to enable instancing, which now allows us to draw dozens of these onto the screen without hitting a performance snag anytime soon.

The component reads out block-data like everything else. The diamonds are put into place according to the scroll offset by distributing the instanced meshes. This is a relatively new feature in Three.js.

Wrapping up

This tutorial may give you a general idea, but there are many things that are possible beyond the generic parallax; you can tie anything to scroll. Above all, being able to compose and re-use components goes a long way and is so much easier than dealing with a soup of code fragments whose implicit contracts span the codebase.