Audio-based Image Distortion Effects with WebGL

Some experiments that show how to make audio-based distortion effects on images with p5.js and its sound library.
Soundbasedimagedistortion_featured

We’ve covered in the past how we can read data from Audio, using the p5.sound library and how we can use that data, to draw things in the canvas, using p5.js.

Well, what if instead of drawing a sketch, we used audio to distort an image? Today we want to show you some demos that play around that idea.

We’ve created some experiments using the theme of movie trailers where the background image of the movie poster is being distorted using a sound sample. It kind of adds some drama to an otherwise static image in this case.

Here’s a short video of the beginning of one of the effects:

How it works

We analyze the sound and map the range of frequencies, to some uniforms we pass in our fragment shader. Then depending on the effect/distortion we have, we can tweak different parameters, using the audio frequencies which constantly change overtime.

In our first demo, we create a simple sinewave in our fragment shader, by using the bass frequencies of the audio track to control its frequency and the mid frequencies to control its amplitude. Then we add the distortion in both axes (x & y) of our uv and add that distortion to the initial texture coordinates.

It looks like this:


  float wave = sin(uv.y * u_bass + u_time) * u_mid;
  vec2 d = vec2(wave); // could be vec2(wave, 0.0) or vec2(0.0, wave) for distortion only in 1 axis.
  vec4 image = texture2D(u_texture, uv + d);
  gl_FragColor = image;

The possibilities are endless if you want to play around that idea, it’s just a matter of what effect you’re after. Make sure you’re mapping values to your uniforms, that are within a range that can distort your visual, and you can always use some generic uniforms like u_time, that can put some ‘overdrive’ to your distortion.

Head over to the demos and check out the variations we’ve made.

Hope you’ll have fun with this one and be sure to share any of your own versions!

Reference & Credits

Tagged with:

Yannis Yannakopoulos

Interactive Developer. Currently exploring Generative Art, WebGL, GLSL & Web Audio. Musicing at The Blimp!.

https://neundex.com/

Stay up to date with the latest web design and development news and relevant updates from Codrops.

CSS Reference

Learn about all important CSS properties from the basics with our extensive and easy-to-read CSS Reference.

It doesn't matter if you are a beginner or intermediate, start learning CSS now.

Feedback 9

Comments are closed.
    • Do you have any error messages? It’s very difficult to troubleshoot something when you only say it doesn’t work…

  1. I downloaded the zip file, clicking on index but it just flashes the loading, why doesn’t it work?

  2. Love this! How would I go about using the “Two Pass Blur” shader instead of this sinewave distort? So, the audio triggers a blurring of the image that goes away as the sound dissipates… is that even possible?