Working with sound , Audacity , ffmpeg and sox

 

Day 4

Adding sound – what sort of sound do we want to add to our video? Creative commons options, reusing sound from video, ways to treat that using audacity or cd scratching and recording methods. Turning sound into video operating on that then reversing that process to retrieve sound , using audacity and the gimp to modify sound based on Letsglitchits’ reverse sonification methodology optimised for open source software.

Today we will be working with sound

We will mainly be working with ffmpeg, and sox today, I only added sox to the software list recently so if you haven't got it installed already do this – open powershell as administrator and do ‘choco install sox.portable’ on Linux use your package manager to install it or from the terminal in Debian based distros do sudo apt install sox or on arch based systems do sudo pacman -S sox

We are also going to be using audacity today and there is a known problem with audacity in windows ( but not Linux) in that to import certain files like ogg audio or m4a we need to have ffmpeg installed , unfortunately the version it requires is older and slightly different to the one we installed using chocolatey so we will have to download and install a different version for audacity to reference . Issue and links here https://manual.audacityteam.org/man/installing_ffmpeg_for_windows.html. Installer is here https://lame.buanzo.org/ffmpeg64audacity.php

one of the biggest influences on my attitude towards sound is probably this music by the caretaker , and im going to play it back so we can kind of get in the mood, not too much of it but some https://www.youtube.com/watch?v=wJWksPWDKOc its the atmosphere that I most like about it and too a great degree when I’m making sound to go with my videos that’s what I’m thinking of, atmosphere.

My general attitude towards sound is if it doesn’t need it don’t add it, sometimes what is going on visually in a video will be enough in itself, and sound that happens through the process of databending or hex editing a video should stay in or be the basis for what is added or re-added. Failing that we can take the original audio from the video, if it has any and play about with it in various ways. So I’m going to run through a few of those and then let you loose on breaking some sound.

We could find creative commons licenced music to add or even public domain sources, (there are a lot on archive.org ) if I use those I generally data-bend them in various ways. One of the most interesting ways I’ve found is to transcode to a codec that is very very compressed such as codec2, info on codec2 here https://www.rowetel.com/?page_id=452 unfortunately the windows version off ffmpeg we are using doesn’t include c2 so ill have to encode this on linux and bring it back that way ill show you that in a second.

Now we’ve talked about extracting audio from video using ffmpeg

( find a video file with audio open git-bash or bash terminal in that folder )

and do this ffmpeg -i somefile.extension prisoners.wav

Now we can listen to this but how would we begin to glitch it ?  One effective way is to transcode it to a codec which uses a lower bit rate , like speex 

On windows use libspeex at a really low bit rate ( this is 8.2kbits per second)like this

ffmpeg -i somefile.wav -c:a libspeex -q 0 somefile.spx

if you play that back in vlc you can see we get a more crunchy distorted texture.

or if we encode to gsm codec using sox (install using ‘choco install sox.portable’) (as our version of ffmpeg wont encode to gsm)

sox somefile.wav somefile.gsm

Then play that back vlc wont play this type of file to my knowledge but another video player like mpv ( choco install mpv ) will or if you dont want to install yet another media player just transcode that file back to wav using ffmpeg or use ffplay as ffplay will play audio files as well.

ie ffplay somefile.gsm 

and weird things really happen if we do this with sox

sox −r 1k −e signed −b 8 −c 1 somefile.wav somefiledownsample.wav

playback that file using vlc or ffplay 

but if we encode it to c2 on Linux we get this

using this 

ffmpeg -i somefile.wav -c:a codec2 somefile.c2

we get something far more interesting. I've used this codec a lot myself .

but cross-platform on windows and Linux is a newer codec called opus ( which is also fully open source ) which acts very much like c2 indeed its even crunchier

so if we take our test file we can do this

ffmpeg -i somefile.wav -c:a libopus -b:a 500 somefileopus.ogg

Vlc will play opus files , but ffplay will as well

so we can mess with sound without even glitching it by using sox and ffmpeg to encode to different formats unsuited to the material at low bit rates ( the final file was encoded at 500 bits persec whereas a low bit rate mp3 would be 128kbps and the lowest you can go with mp3 is 32kbps so if we found a creative commons licenced or public domain music source it would sound a little like this

( music from here https://archive.org/details/07-rare-and-hot-1925-1930-historical-vol-12) Download that.

lets play memphisjazzers.mp3 first as it is as mp3

Now encode the same file to opus at 500bps

ffmpeg -i memphisjazzers.mp3 -c:a libopus -b:a 500 memphisjazzers.ogg

play memphisjazzers.ogg

now if we open that in audacity we can play with it a bit more . Open audacity add some reverb , playback its starting to sound a little odd lets add some tempo changes and normalise it to adjust the volume . I tend not to speed up, more speed down . Its like a nightmare comb bouncing off instruments the beeps and bops caused by the low bit rate are marvellous. Of course we could edit this more later on or chop and reverse etc or take sections out and reuse them . But I like that pretty much as it is.

We can save that and then use it later on , I'd tend to save as wav if I want to work on it more later.

so with a complete soundtrack for a video we could also do the same thing .

Hex editing audio

Interestingly it is possible to hex edit the opus codec , which surprised me , as I haven’t had much luck with compressed audio files , mp3 in particular is very difficult ( there are online guides By Nic Briz as sky Goodman pointed out in her recent workshops) but this did surprise me .

doing this 

xxd -p somefile.ogg | sed 's/ff/07/g'| xxd -p -r > somefilehex.ogg

then imported into audacity , vlc might recognise that file now I've hex edited it ffplay might but importing it into audacity makes it playable and thus bakable if we export it

speex encoded files  aren't openable in VLC after hex editing but will import into audacity – speex doesn’t do anything overly interesting compared to opus

so going further with the same track but hex edited slightly differently

  try thisand import that into audacity

xxd -p somefileopus.ogg | sed 's/ff/077/g'| xxd -p -r > somefileopushex2.ogg

If at first it doesn’t sound promising , dig around in the files timeline listening for somethingt hat catches your ear . often i found by slowing something down by changing tempo downwards more interesting sonics can be revealed.

sometimes I’ll have a collection of files that I’ve worked on and kept by just in case and I'll go through a process of adding them to the video timeline I'm working on and seeing if they match , maybe incorporating some of the original audio if its been damaged during either the hex edit process or more often through tomato which can give quite interesting files.

so maybe for the next ten minutes pick a short file that you want to work on extract the sound in a codec of your choice and try hex editing it to a different codec. To get a list of codecs just issue this command in git-bash 'ffmpeg -codecs' any which are marked as D.E.A mean that ffmpeg can decode and encode that and its audio. 

Can we edit sound as video and back to sound

What if we want to edit it as video ? 

so lets export that as a wav file then change the file extension to .yuv

then do this

Using this ffmpeg -f rawvideo -s 640x480 -r 25 -pix_fmt yuv420p -i yourfile.yuv -c:v rawvideo output.avi

open that in kdenlive now use rgb splitor , then add pixelate filter and maybe waves as well , change to whatever looks good , export that as huffyuv with flac , open the resulting file ( mkv extension ) in audacity as a stereo track   !

Its an interesting one to play around with, I’m not sure if its a dead end or not sometimes you get complete noise sometimes not. Try different video effects on a shorter file as larger files can be quite difficult to handle   taking a lot of time to load and render and saving raw files takes up a lot of hard drive space.

Dawnia Darkstones reverse sonification modified for gimp.

This is possibly a more useful method for manipulating sound as image based on Dawnia Darkstones reverse sonification methodology that she gave a workshop on for Hacknet last year but modified for use with purely open source tools and specifically the gimp.

The basic methodology goes like this :

1) import an audio file to audacity, import a photograph underneath that file (to get an idea of size (for when you import it to Photoshop as we need to know the size i.e. width and height), delete the photo track before saving  save it as a raw file (ie uncompressed with .raw extension)

2) open that raw file in Photoshop (as photoshop supports raw import) with dimensions close to dimensions found during step 1

3)Add some effects cut and paste etc then export again from photoshop as raw

4) open that file In Audacity and play it back and see what we’ve got

Essentially it treats audio as image and allows us to add image effects to audio. So far so simple. But it becomes more complicated if we only want to use open source software (as we have been doing in all the sessions) as I won’t use closed source software and Adobes business model of software as a service that you don’t own and have to keep on paying a subscription for is deeply suspect (other than using windows ten in these sessions for pragmatism realizing most people are going to be using windows 10) we can do something similar with sox on still images, treating images as sound and I’ve written about that in blog posts – if you are interested the write up is here https://crash-stop.blogspot.com/2021/05/bash-script-for-sonification-images.html

and here https://crash-stop.blogspot.com/2021/05/quick-and-dirty-guide-to-using-shell.html

To use this approach with only open source software we have to take a slightly different path, for one thing the Gimp which we will be using doesn’t import raw files , so to get around that once our wav file is exported ( and I’d advise you to shorten any file you are working on) we will change the file extension to .data which the gimp will open .

So the revised methodology will be

1 ) Open audio in Audacity shorten if needed , possibly only try 5 to 10 minutes in length to begin with depending on the speed and age of your machine .

2)Export that audio as raw data file > export > export audio > specify

save as type ‘other uncompressed files’ >choose raw > unsigned 8bit

3) hit save and remember where the file is being save too. Find that file with file explorer change extension from raw to .data

4) exit audacity . Open gimp find the file we just made and open it as .data but now we have to give dimensions for that file otherwise we will lose information or change it beyond recognition.

In Dawnias tutorial she speaks about placing an image file below the audio you are working on in audacity as a reference for size and dimensions and advises cutting one to suit the other . There is a cheat around this in that if we know the size of standard images we can guess what dimensions we will need to import the raw data into the gimp .

This is a cheat sheet to help you based on standard digital camera dimensions.

MB of  audio          Resolution

4                             2464x1632

6                             3008x2000

8                             4264x2448

10                           3872x2592

12                           4290x2800

16                           4927x3264

18                           5184x3456

19                            5380x3620

21                            5616x3744

24                            6048x4032 ( 6648x4032 ?)

31                            6496x4872

39                            7216x5412

51                            8754x5836

or we could use an online calculator like this - https://www.scantips.com/mpixels.html and feed in the size of our raw file which will give us dimensions to feed into gimp as width and height

The following is based on the information found here https://www.wikihow.com/Calculate-a-Digital-Camera%27s-Resolution-from-its-Pixel-Count

But we could also calculate it by imagining the size of the file we want to input to the gimp is in fact a cameras ccd , the file size gives the first number we need so 137mb we could imagine to be a 3:2 ratio 137megapixel camera ( 3:2 is the most common ratio of dslr and the same ratio as 35mm film)

If we then multiply that by 1million ( 137 megapixels ) we get 137,000,000

with that number we can get our width by height by doing this -

1) Get a horizontal-to-vertical and vertical-to-horizontal ratio. You get the horizontal-to-vertical ratio by dividing the first part of your aspect ratio by the second; you get the vertical-to-horizontal ratio by dividing the second part of your aspect ratio by the third

So with a ratio of 3:2

width = 3/2 =1.5

height = 2/3 = 0.666

Multiply your pixel count by the horizontal-to-vertical ratio, then separately, by your vertical-to-horizontal ratio. Then take the square root of that number 

So 137000000 x 1.5 = 205,500,000 = 14,335 = width

137000000x0.666 = 91,242,000 = 9,552 = height

so with this method for any given file size we can get an appropriate width x height

## I wrote a small basic shell script to run this calculation , requires that you have bc installed – on windows you need to have mysys2 installed so that first you can do pacman -S bc to install bc then navigate to the mysys2 folder , look in usr/bin for bc.exe , copy that file then navigate to git folder in programs and paste bc.exe into usr/bin/ otherwise git-bash will throw a wobbler ( bc allows floating point operations in bash as bash only allows whole numbers otherwise ) on linux do Debian based sudo apt install bc , Arch based Sudo pacman -S bc .

script as follows

#!/bin/bash

#automated calculator for obtaining dimensions to use in gimp

#for reverse sonification

#assuming a 3:2 ratio for pictures

w=1.5

h=0.666

echo -n "Size of file in megabytes (give whole number only) ? : "

#get width

read n

a=$(($n * 1000000 ))

b=$(echo "$a * $w" | bc)

echo $b

# get square root to give width in pixels

width=$(echo "sqrt($b)" | bc)

#get height

c=$(echo "$a * $h" | bc)

echo $c

height=$(echo "sqrt($c)" | bc)

echo " Width is $width Height is $height"

so open the gimp then go to open file select your renamed file with .data extension change the image type to indexed ( it stops a weird flaw with gimp that doesn’t resize picture correctly if its rgb even if you select the right file dimensions) in width and height type in either a rough guess by matching size to megapixel above or work them out given the formula above then press open , we should have a correctly dimensioned greyish looking image . Now we can get to work .

Just wander around the image , zoom in have a look at detail , we could apply effects such as edge ( which works really well ) or we could just start randomly cutting and pasting , I try and stay away from the edges doing this . You could even spin the whole file around by 180dgrees which as the effect of reversing the audio .

Once we’ve added a few effects and cut and pasted, export that file giving it a new name as .data again , to a place you will remember , minimise the gimp , open audacity then import the file as raw data , 8bit unsigned , little endian ( for some reason big endian just results in a mush of noise) stereo and import , then flick through the track and see what it sounds like .

If you don’t like it exit and go back to the file open in the gimp and either undo the changes and restart or go for broke and add more export again as .data and import it again to audacity . If you do like it export it as an mp3 or wav from audacity.

And that's essentially it . Some files work well others don’t , but sometimes you hit on something really interesting . Sometimes if you add effects here , like normalise , time stretch change tempo , delay etc you can get something really nice If you have a slower computer I wouldnt advidse going much beyond 50mb in size though .

So now we have some sound what to do with it . Obviously we want to put it back onto the videos we have been making so fire up kdenlive and open the project you’ve been working and import the file and chuck it on the timeline like so

open kdenlive and demonstrate this .

Often Ill import multiple files and try each or combinations sometimes overlaying , shortening etc . And we can open the audio mixer and adjust levels, often files made using hex editing reverse sonification etc will be very loud , bear this in mind , its a mistake ive made myself before and its been complained about by people using headphones .

Notice with kdenlive we can also add video over the top of other video ie alpha transparency and we can add audio and video on the timeline wherever we wish , there is no snap too where all the clips reorganise themselves, this can be handy or a curse if you have lots of small edits

Burning and scratching cds

The use of glitch in music probably predates visual glitch art , and our very first experiences of glitch may be the sound of a skipping cd – in fact whole albums have been made using this method , especially by the group Oval and this seminal work from 1994 - Diskont , and they have influenced my approach to sound in my own work using their methodology ie take a cd , mark on it with felt tip pens then record the stuttering sounds the cd creates.

Extract the sound from a video file using this command 'ffmpeg -i your.mp4 -vn rippedsound.wav' ( I use wav as I want to retain the highest quality file I can for burning to cd ).Burn that file to cd, mark the cd with felt-tip pen. Record that playback in the software of your choice ( for me audacity )

Adjust the sound and edit, add to your newly glitched film.

I use this technique on a lot of the black and white film noir that I sourced from archive.org .

Open source burning software for windows can be found here https://cdrtfe.sourceforge.io/cdrtfe/download_en.html

and for Linux I generally use k3b