Audio-based Image Distortion Effects with WebGL

We’ve covered in the past how we can read data from Audio, using the p5.sound library and how we can use that data, to draw things in the canvas, using p5.js.

Well, what if instead of drawing a sketch, we used audio to distort an image? Today we want to show you some demos that play around that idea.

We’ve created some experiments using the theme of movie trailers where the background image of the movie poster is being distorted using a sound sample. It kind of adds some drama to an otherwise static image in this case.

Here’s a short video of the beginning of one of the effects:

How it works

We analyze the sound and map the range of frequencies, to some uniforms we pass in our fragment shader. Then depending on the effect/distortion we have, we can tweak different parameters, using the audio frequencies which constantly change overtime.

In our first demo, we create a simple sinewave in our fragment shader, by using the bass frequencies of the audio track to control its frequency and the mid frequencies to control its amplitude. Then we add the distortion in both axes (x & y) of our uv and add that distortion to the initial texture coordinates.

It looks like this:

  float wave = sin(uv.y * u_bass + u_time) * u_mid;
  vec2 d = vec2(wave); // could be vec2(wave, 0.0) or vec2(0.0, wave) for distortion only in 1 axis.
  vec4 image = texture2D(u_texture, uv + d);
  gl_FragColor = image;

The possibilities are endless if you want to play around that idea, it’s just a matter of what effect you’re after. Make sure you’re mapping values to your uniforms, that are within a range that can distort your visual, and you can always use some generic uniforms like u_time, that can put some ‘overdrive’ to your distortion.

Head over to the demos and check out the variations we’ve made.

Hope you’ll have fun with this one and be sure to share any of your own versions!

Reference & Credits

• p5.js
• p5.sound
• p5jsShaderExamples