The Ethics of sources Day 2 - FFmpeg and datamoshing

The Ethics of sources Day 2 - FFmpeg and datamoshing

So this is part two of the Ethics of sources, in part one  I talked about what kind of sources I like working with, why and how to download them . Today we are going to start working with those sources and I'm going to be running through some of the basic tools that I use.

The original talk can be found here Ethics of sources day 2

Lets have a quick look at what we downloaded in part one 

 

I want to convert them to different formats, one of the most important parts of glitch art is working out how each format reacts to different methods and the different textures and artifacts you can achieve, and working out what format suits the material you are working with using the foundations of glitch art which are to me hex-editing , data-moshing and misinterpretation.

Lets go ahead and convert one of the files we found yesterday, to do that I am going to be using the command-line, ffmpeg and handbrake.

The command line( specifically the bash shell )is probably the tool that I use the most in all of my work, common to must Linux distributions and Mac OS and even within the Windows 10 these days via the windows subsystem for linux .

More information on the bash shell here https://en.wikipedia.org/wiki/Bash_%28Unix_shell%29 

More information on Windows subsystem for Linux here  https://docs.microsoft.com/en-us/windows/wsl/install-win10

Find releases of ffmpeg here https://ffmpeg.org/

Find Handbrake here https://handbrake.fr/

There are file conversion programs based on GUI’S like Handbrake but the command-line is quicker ( these methods with a little alteration and research should be common to windows and mac, I use Linux almost exclusively as I believe software and hardware should be free to use and change as you wish, just like video and content ) . So lets look at the files we downloaded yesterday and open up a command-line .

What do I want to do with this file as that determines what I will convert it too , do I want to datamosh , or hex-edit if I want to datamosh the best format for that is libxvid,  mpeg4 or h261 , though equally zmbv works really nicely ( though takes an age to encode to) and for all methodsits best to use an avi container. One of the programs I'm going to demonstrate, Aviglitch, really only works with libxvid the other two can use multiple formats as long as they are in an .avi container. So anyway lets go ahead and run this command in the directory containing the file .

And we can see ffmpeg doing its stuff – just to break down the command quickly ffmpeg -i means input this file to ffmpeg, filename -an means leave the file with no sound ( its often easier to delete the sound as some programs wont work with it intact , then the -c:v part means use this codec libxvid the -q = quality I set it to 9 here – experiment though as sometimes setting a file to a low quality can give interesting results later, output file is last which is our freshly transcoded file in an avi container . 

Another handy function of FFmpeg, especially if you want to get detailed information about a file such as what codecs its using for sound and video, bitrate , size ie is 640x480, 1280x720, colourspace and metadata is FFprobe, so lets check the file we just created with ffprobe as a quick illustration .

I'm going to datamosh this first then hexedit it but to hexedit it ill probably convert it to a different format .

Now there are a couple of different ways of datamoshing , the long and hard but most effective , way using Avidemux ( theres loads of tutorials on you-tube so I'm not going to cover that ) or command-line ways which rely on brute force so my favorite method is using a ruby script called aviglitch find that here http://ucnv.github.io/aviglitch/ ( on debian based linux you just have to do sudo gem install aviglitch , so simple )

It works by taking out all the key frames – now depending on how much and what type of movement you have in the video this will be more or less effective – you have to learn how to judge that and what type of movement works best so lets play with the video we just transcoded and run aviglitch on it on. Unlike some of the later techniques I will show you this works across any folder ie its universally available in the same way that ffmpeg is on linux if installed via package manager like apt or if you have compiled it yourself from source, you just navigate to the folder your file is in open a terminal in that folder  and issue the command. It goes like this :

And this is what that looks like.

As you can see frames start to bleed into each other and images break through , the colours also take on that blended bleeding texture that we love so much, as if the colours have become liquid.

Lets do another one, if we look at the  video by the Russian witchouse band Icesp3ak  we downloaded yesterday we can see a similar effect but with different movement and points of breakage .

If we want to upload this or work on it further we will need to bake or fix it , to do that I usually use handbrake as it seems to work the best of anything I've found so far, even with quite broken files.

There is a way of doing it through mencoder on the command-line which does work very well , a lot better than ffmpeg in the same situation. So we could open a terminal in the folder with the file and do this 'mencoder <input_movie> -oac mp3lame -ovc lavc -o output_movie.avi' ( for when you want sound ) or 'mencoder out.avi -nosound -ovc lavc -o output_movie.avi' ( with no sound ) , I don't know why this works better than ffmpeg you'll have to take my experience on trust - find mencoder via your package manager in linux or otherwise here -( mplayer is built onto mencoder but get one you get the other) http://www.mplayerhq.hu/design7/news.html

But I’ve recently found the the easiest way is just to use handbrake. While I'm here I'll convert to h265 as we will use that as an example when we come to hex-editing (I'll show you why then). H265 takes a long while to encode compared to other formats but its worth it.

so lets see if that worked . Okay so now we could upload this or we could just flick through and see if there are any decent stills in there ( I must say use vlc because it really is the best tool for video playback, related tasks and taking screenshots ). Compare this to playback in the original output from aviglitch shown earlier

If Handbrake doesn't work and occasionally a file will be so broken it's impossible to put it through any baking program we may have to use a screencapture software – generally I use gtk-recordmydesktop if the file will play in vlc , if not try mpv ( mpv will often play files vlc wont , though it does try to render datamoshed files without the datamoshing) or even gnome-mplayer  also compare how different media players play the same file, sometimes the difference can be astonishing - for instance VLC can often refuse to play H261 in an avi wrapper , and when it does the playback will leap all over the screen ( I usually have vlc play a file on a loop ) .

Find VLC here - https://www.videolan.org/vlc/

Find MPV here - https://mpv.io/

Find recordmydesktop here - https://sourceforge.net/projects/recordmydesktop/

Gnome-mplayer can be installed through apt ( sudo apt install gnome-mplayer)or your package manager on linux .

Now for datamoshing there are two other tools we can use, both command-line and I’ll cover Tomato first . Find tomato here  https://github.com/itsKaspar/tomato

Tomato was written by Kaspar ravel  (find him here https://www.kaspar.wtf/ )and quite simply it is one of the best tools there is for datamoshing . A simple python script used on the commandline ( the video file must be in the same folder as the python script )

The command python 'tomato.py -i adbreak.avi -m pulse -c 7 -n' on one of the files we collected yesterday ( icespeak.avi ) gives us this ( playback in vlc ) - what I hadn't realized was that in the new version it retains the sound , in previous versions you had to remove the sound otherwise the script would throw a spanner ( and when converting the file with ffmpeg I'd forgotten to us the -an option so the sound was left in ).

And here is the full version:

There is one other tool thats handy for datamoshing , and thats ffglitch find ffglitch here -
http://ffglitch.org/

The same provisios as aviglitch hold true – get your file transcoded to xvid, without sound, though it will work with mpeg and mjpeg libxvid is just easier for our purposes here . The true master of this is Thomas Collet and his work really is quite astonishing. Find it here https://vimeo.com/chepertom

FFglitch does has a lot of advantages unlike the other approachs ffglitch doesn’t require you to bake the file after datamoshing , on the other hand it is way more complex to use and quite poorly documented ( though it was pointed out to me by Vedran Gligo that if you issue the command ./ffedit -h it will bring up a handy help text ) see below .

How does it work ? First we have to parse the file we are working on to find movement values ( which are then saved to a .json file) in our video using this command (you have to move the file you are working on into the same folder as ffglitch so all commands you see from now on will be issued from a terminal in that folder) and were going to take the ballerina again and work on thatwith this command

./ffedit pirouette.avi -f mv -e test.json

This creates a .json file which we can then examine and modify then re-apply back onto the original video. You have to move the file you are working on into the same folder as ffglitch so all command you see from now on will be in that folder.

If we look at that file we can see the values and think hmmm – we could change one value at a time but in this case im going to do a global search and replace with values I know thru trial and error will work , and that command looks like this

sed 's/0/7/g' test.json > testhex.json

This is a type of  command-line text editing which I'm going to cover in the next blog. Here I'm using sed which is short for stream editor ( part of the basic GNU coreutils that most linux distros come with)) .

Get gnu-coreutils here https://www.gnu.org/software/coreutils/coreutils.html

slide – so having changed those values  we have to apply those back to the original file with this command.

./ffedit pirouette.avi -f mv -a testhex.json pirouettehex.avi

So as you can see thats a lot more controlled but we could change the initial start values so lets do that and see how it changes the video . We'll change the initial start values to these :

sed 's/-1/-2/g;s/0/32/g;' test.json > testhex.json

Then reapply the values as before

./ffedit pirouette.avi -f mv -a testhex.json pirouettehex.avi

And this is the complete thing.

Notice also notice how quick it is once you have the commands in the command   line to just repeat them or alter values the trick is too look through the initial json file that ffedit produces and spot values that are changeable that will have an effect.

Now I touched in the above on sed and editing a file on the command line and this is the basis for what I’ll be talking about in the next blog post , ffmpeg and hex editing .