Editing video with open source tools

* Note I will probably add to this in the next few days after the workshops have finished with illustrations so check back.

If you haven't installed kdenlive  you can download it for windows here https://kdenlive.org/en/ or on Debian or Ubuntu based distros do sudo apt install kdenlive. On arch based distros sudo pacman -S kdenlive .

So lets open kdenlive and look at it !

 

 

Kdenlive is pretty much the same on Windows and Linux, on the left you can see a window which has tabs named project bin , effects undo history clip properties etc. then next to that you have the clip monitor which just displays the clip you are working on at the time and next to that the  project monitor which displays the whole thing that's on the timelines below , bottom right you have the audio mixer and effects stack window , more of that later , and below that there's a small slider which took me a while to find which gives us the zoom level , not important until we have a video on the time line , pretty straightforward so far

This is my approach , I don’t present it as the last word or a method to use yourself , its what works for me .

Personally when I’m editing I’m not that interested in narrative as I said in the first session

Naturalism in Film is a construct and a betrayal of possibility

Narrative is unimportant.

The moving image can be treated as a material like paint

In editing seek what is interesting visually rather than a story

The viewer is allowed to move backwards or forwards through the ‘finished’ piece creating it for themselves

With that in mind

My approach to editing reflects this, I look for what I find visually interesting and what ‘flows’ though what I make ostensibly occupies a linear time frame I’m not that concerned if people can pause or rewind to catch something again , its also one of the advantages of making work for the web . Though I do make long work Some of the most amazing pieces of glitch art I’ve seen are relatively short in duration , such as the work of Rui martins aka feathersmakepplattractive https://www.tumblr.com/tagged/Rui+Martins?sort=recent a lot of the time these are very very short , just a moment or a repeating gif of a moment and I will cover making gifs quickly today . Why do these short videos have such impact ? What length should your video be ? In Rui martins case and a lot of datamoshing they are leveraging the moment of surprise where one thing becomes unexpectedly other so the video only has to represent that moment .

Consider these things before editing

1)Preparing material beforehand baking broken files, getting everything the right dimensions , remembering to save periodically and backing up files .

2)Cut out sections that don’t do much, unless that's what you want, avoiding repeating sections which can be especially important in datamoshing. Scanning to find interesting sections deleting boring stuff. unless thats what you want , adding in new material if needed.

3)effects or not . I'm fairly on the no effects side of editing , the glitches are what I’ve already made , having said that , sometimes a little bit of sharpness is good etc,I will run through some of the more interesting effects you can do with kdenlive .

Lets get into it

But what if after having used tomato or ffgac I have a longer video say 5 , 10 or even 30 minutes to an hour long , essentially my approach would be to look through the video finding those moments of surprise and editing out the fluff in between as if you aren't using avidemux to make datamoshes automated scripts will produce a lot of stuff that I don’t want

For instance this video ‘prisoners of the lost universe’ https://archive.org/details/PrisonersOfTheLostUniverse1983 run through Kaspar ravels tomato v2 using this command ‘python tomato.py -i prisoners.avi -m pulse -c 20 -n 7’ which ended up being 30 hours long from an initial 1 hour and 34 minutes

To import a  video into kdenlive – go to the left and open the project bin, right click and add clip to import video. Sometimes when importing you will get a warning which asks if you want to adjust to the profile of the clip , if you are just working on this one video then click to do that , the otherwise we will talk about a little later . You might have to wait a while for it to import your video depending on size and other factors . Now that has actually imported okay , drag and drop that onto your time line like so , and then well try and play it back ( I’ve had problems sometimes with videos manipulated using other methods , if it wont import your video you may have to bake it with handbrake or try re-encoding it with ffmpeg , or capture the glitched video using ob-studio via playback in ffplay or videoplayer of choice , (which may take some time if the video is long ). Now we can see what the zoom control does , handy for zooming in and out of the timeline and finding your way about. We can also scrub back and forwards on the timeline which gives a wonderful weird sound effect as well ( which might be possible to use as an audio scratcher if we had say audacity opn in the background to record what we are doing) . Now as you can see we are getting stuttering and seek problems because the video is datamoshed and it needs to be baked , We could bake it from here but its probably easier to exit and re-encode it using handbrake or ffmpeg .

Open handbrake  run through dimensions , altering , making sure crop is right ( you can check in summary tab handy if you’ve screen grabbed the video sometimes) the video tab encoder ie h264 or not , uhmm choose quality , at this stage we’ll leave it at 25 which is fairly high , lower gives bigger file size though this file is already 8gb ! Audio , important to check this tab out especially if you're resizing or finishing to upload to Instagram or Tumblr as they won’t accept anything but aac audio . Also check out where the file is going to be written , it will warn you if its going to overwrite an existing file but I’m always making different versions so make sure you stick to some kind of naming plan be organised !!

Naming plans side note

When I’m working on a project I can create a lot of files, so its important to know where those files are so be consistent , either save directly to your home directory in the video folder or create a project folder to work from, this is very important if you are going to spend a couple of days putting together a longer project and will be saving an editing timeline constantly as the video editor will need to know where the files are to be able to reopen the project , there is a save project in the file menu of kdenlive , and that function pretty much exists in most video editors but it means you shouldn't move any file you have imported to a project around or the editor won’t be able to find it and any edits you have made will have to either be redone or you will have to locate the media for the video editor ( sometimes they will give a menu for relinking ) .

Organisation is key . Know where your files are , keep them there and don’t move them around until you've finished with them , after that I have backup hardrives where I store them in dated folders by year.

When I’m saving files or versions I tend to put in a shorthand for whatever technique I’ve used so something like somefilehextomatofinalshort.mp4 ( somefile hex edited, sometimes I'll write int the values I've used, Datamoshed using tomato , final being an already baked or finished file, short means I've shortened it , possibly to upload to Tumblr which has a 100mb file limit upload or for Instagram which prefers video under one minute on the main timeline)   would be a typical example. Or maybe version 1 2 3 , I tend not to leave gaps in file names as its just easier and quicker when typing file names into the terminal than if they have spaces in them.

All that being checked out and changed for preference lets encode . Bear in mind encoding can take a while depending on your hardware , tailor the output to your machines capabilities and proposed destination .

So using handbrake would have taken about four hours to render this file . Why so long welllll tomato increased the file size to 8gb but it also stretched the duration of the file to something like 31 hours , from 1hr and 37 minutes , obviously because we are adding extra frames so I then used ffmpeg to re-bake instead using

ffmpeg -i prisoners-pulse-c20-n7.avi -c:v copy -c:a copy prisonersbaked.avi

which left me with a usable but large and long file can we still import that into kdenlive ?

Yes we can , but it started to struggle because its a long file  . But also notice the sound , we could extract that now and store it for later by going back to that directory and doing

ffmpeg -i prisoners-pulse-c20-n7.avi -c:a libmp3lame -abr 1 -b:a 256k prisonerpulsesound.mp3

Which gives me a largeish and reasonable quality mp3 file, usually I'd save to wav format but this file would be too big .

so even though kdenlive can import the 30 hour long video lets cut it down to an hour for manageability ( the bigger the file the more the editor will struggle especially on lower end or older machines or machines lacking in ram ( hopefully you will have 4gb or more ) .

but we will do that using ffmpeg ( obviously if your clip is shorter you wont be having these problems ! )

ffmpeg -ss 00:05:00 -t 01:00:00 -i prisonersbaked.avi -c:a copy -c:v copy prisonersbakedshort.avi

so back with the clip imported into kdenlive so I’m just going to find an area I like and delete the rest and then work on that  , on a small screen , say on a laptop, you can un-dock or delete windows you aren't using and it makes more sense to have the viewports larger.

If you click the scissor icon move it to where you want it then click on the timeline , then delete the excess, now we can stretch the clip on the timeline so we can get a better feel for what’s there using the slider in the bottom right hand corner and we can home in on what we want to keep generally I want to get rid of original references and keep the datamoshed part so something like this ( show searching bracketing and deleting )

now I’m kind of happy with that lets export it to h264 with aac audio by pressing this button here (red button that says render ) and this window opens up giving us various output formats , well stay simple and go for h264 aac , if you flick thru the options you can see which your computer can handle , if they aren’t in red you can use them, and various options just to export audio or whatever . Remember to name the file as well ! And where its going ( here videos in my home folder) you can alter profiles as well if you click on the more options icon at the bottom middle of that window a new window pops open to the side showing you the various parameters kdenlive will use , there in kind of ffmpeg format so you can adjust them as you would with ffmpeg . You can also make adjustments to these scripts by opening the settings menu on the top bar and go to configure kdenlive and click on transcode button and that gives all the formats and encoding setting for each format you can add or subtract to those .

The format for these is basically that used by ffmpeg but without the input and output parts ie it takes off the ffmpeg -i yourfile.extension and the output.extension at the end so if you have an ffmpeg profile you like using you can add it in and then use that for rendering with.

Hit red button wait and

lets look at that now okay lets reload that as a new file in kdenlive ( or we could save that edit as a project before we start again) and run through a few of the filters so go to the window in the top left and click on filters and lets add something like rgbsplittor , if we drag that onto the time line then we can control it from the effects stack on the bottom right im not a big fan of adding effects other than sharpening but some of these are quite fun , rgbsplittor especially , so ima going to render that to a new file again

and play back …… nice !

And if you don’t like the effect you can just click on the red bin icon in the top right hand corner of that window and it disappears !

Try out different effects and see what you like , or none . ( rgbsplittor with lens correction !)

The boring bit

so what if we have lots of different files that we want to make a long form video from ?

I have an external hardrive with videos of different lengths different codecs and different sizes , I generate a lot of files ! But also remember when working with found files you will have the same problem , archive.org is not consistent in the dimensions of videos it hosts and often dimensions will vary depending on era i.e. black and white vs colour or the way the file was encoded .. So what if I want to import them edit them into kdenlive and have a single output size ? Lets see what happens if we try that .If we try and add the whole folder at once what happens is that kdenlive tries to re render all of the videos at once , now as I’d already imported two videos to begin with and the first video was 1280x720 I think that its going to be import everything as 1280x720 no matter what size they are if you look at the top here's a render button and thats giving a time until finished and a slowly moving progress bar nope thats going to take to long to import that many videos ( 18 ) lets do one at a time .

(We could use handbrake on the files we want to work with first which in the end will be quicker but we will do this first in kdenlive) to do the size conversions do this  import one at a time first a 1280x720 ( 2021-04-30_16-48-26.flv) then treestorm.mp4 which is 4:3 show that edges aren’t stretched so that clip will have black borders. If we click on that in the project bin it opens in the clip editor we can then go to the effects menu and use edgecrop to bring that into the same dimensions as the first clip , we can do that for each subsequent clip if needed , so lets import a few more ( vertigosnow.avi) thats 4:3 as well but we have to use edgecrop slightly different , then add that to the timeline . One more this one doesn’t need any altering . Notice the first two have no audio but the third one does , Ill click on the audio icon to mute the audio in that clip)

If we want to add effects we will have to do each clip individually. Open the effects menu select the one for video drag and drop the effect onto the clip on the timeline then alter to taste in the right hand bottom where the effect menu opens now if we look at the top of the kdenlive window we can see while we’ve been doing this kdenlive is rendering stuff in the background so now we cant export the video till that’s finished . Is there a simpler way around resizing video to prepare for this ? Yes there is handbrake !

Which also brings up some of my reasoning behind the way I work when editing – i.e. I don’t just use on program , its possible , but each program is a tool which works more efficiently than the other , so I use handbrake a lot for resizing and cropping and getting stuff to the right dimensions , and ffmpeg for re-encoding or baking , and finally the editor itself for finish and proper cutting and adding and deleting , its not one thing its a process . And a lot of the time I’ll think aah I need that clip but its in this format so ill exit from the editor , saving an edit , goo off and find the clip I want get it to where I want it to be with ffmpeg and handbrake , and then reload the final editor and the edit I was working on

But back to handbrake

Ill show you how quickly now with the windows version of handbrake , its slightly different on Linux . In the windows version we can ( if our hardware allows) turn on intel qsv go to preferences before loading a file to encode and tick the box which says allow qsv ( it might already be ticked for allowing qsv on lower power hardware ) ill talk about qsv in a while cos it saves a lot of time when encoding files.

Decide what dimensions you want to work at ie 1920x1080 , 1280x720, 960x540 , 640x360 are all 16:9 or 4:3 which is generally 640x480 or pal 756x568 open handbrake , open the first video , check dimensions are what you want and codec , then press add to queue then open the next file in your list if its smaller or you want to change its dimensions ( in the dimensions section in the windows verison  remember to tick allow upscaling , turn off anamorphic scaling and at the bottom change where it says display size to the length dimension of your desired size here I'm using 1280x720 so 1280 go back in the summary window and check you don’t need to crop anything , if you do do that then add that video to the queue then add your next video and so on until you have the videos you want ready , then press the start queue , if you’ve queued up a lot of videos to re-encode and resize it could take some while. One of them vertigosnow I had to re-encode using ffmpeg first as it was in a weird codec that handbrake can handle but does so very slowly and ffmpeg was way quicker so its worth playing around with encoding with different applications just to make sure especially on a slow pc or if you are time constrained ( handbrake gave an estimate of 2and a half hours to encode that one file as it was whereas first re-encoding to h264 took maybe 15 minutes in ffmpeg – faster with qsv ! But it will save time when feeding it back into handbrake

interesting discussions on what sar dar and far are in ffmpeg with aspect ratios https://forum.videohelp.com/threads/402501-ffmpeg-and-SAR-and-DAR for further reading and this page on ffmpeg resizing video and changing aspect ratios as well https://ottverse.com/change-resolution-resize-scale-video-using-ffmpeg/

so that took about an hour and a half , compared to 2 and a half hours plus . Could we do it faster ? Obviously if you are only working on a small file this isnt going to be a problem overly much, but if you are working on a longform video ( as I often do ) this will be a concern

Welllll there is a way if your computer has a newer intel processor on windows 10 , linux maybe , using newer versions of ffmpeg we can render faster using a built in function of the gpu called qsv or intel quick sync video more info on that here https://www.intel.com/content/www/us/en/architecture-and-technology/quick-sync-video/quick-sync-video-general.html

not only is it quicker but because it shunts the encoding from the cpu to the gpu and its hardware based it reduces cpu load , heat and general wear and tear

some applications have that enabled including ffmpeg and handbrake ( depending on version) and in fact re-encoding with ffmpeg from the command line can work out faster than using handbrake

simple re-encoding using qsv

ffmpeg -i somefile.extension -c:v h264_qsv outfile.extension

we could use the other opensource video editor openshot which does have qsv enabled by default if available, but openshot on windows and linux isnt very stable on longer files, it used to be my editor of choice but Kdenlive works much better and seems to be more stable ( though I have managed to crash it more than once)  but we could also reconfigure kdenlive to use the latest ffmpeg that we installed with chocolatey ( which does have qsv encoding built in) by clicking settings , then configure kdenlive then click on environment then point each line where it asks for where the exe file is to C:/ProgramData/chocolatey/bin/ and each binary asked for in turn then restart kdenlive , export then becomes a lot faster . But I couldn’t find a way to get qsv encoding to work from kdenlive so far

To alter an encoding script in Kdenlive click on the codec in the main window you want to use after you press render and alter the script in the window that opens replacing libx264 with h264_qsv if you can find a way for it work (?) or whichever codec you choose then give that profile a name and apply it will appear in the render options in the main window now.

but its probably simpler and quicker to do this from the command line using ffmpeg

But as I found out today while helping someone with a problem where they had two videos with the same dimensions ie 1280x720 they didnt have the same pixel ratios which wouldnt normally be a problem but as they were trying to do displacement mapping ffmpeg threw a fit and decided it didn't want to play so we had to run one of the files through handbrake to get the correct pixel ratio and final display size so its probably better to use ffprobe to find the file that is off ( ffmpeg will tell you which one when it exits with an error message ) and use handbrake to change the file thats bad to match the file thats good , its easier to do this until I've found an ffmpeg formula that works .

more info on sar/dar and par here and why that would be an issue - 

https://en.wikipedia.org/wiki/Pixel_aspect_ratio

We can use intel qsv with ffmpeg on four codecs , and actually you can get by

with the two ie vp8 ( vp8 webm which googles' YouTube and Facebook will accept and

it also gives nice results with hex editing but its quite fragile as well so

experiment with values there ) and h264 but also mpeg2 , which is an interesting 

codec for datamoshing webm vp8 encoder is  vp8_qsv mpeg2 is mpeg2_qsv and also

 h265  hevc_qsv  ( hevc is quite an interesting codec for hex editing , you get 

some really nice artefacts with it . ) 


so now we have all our material back in the right aspect ratio back to kdenlive ( 

as I say you could re-aspect everything in kdenlive but then it might take hours 

for it to render especially with bigger files as it has to re-aspect each clip then 

add effects then re-encode and render ) . Thinking ahead and re-encoding to how you

 want the file size wise is often better and quicker before you edit .

Reload the files that are now correct aspect ratios and dimensions and edit a bit

 then add effects then render as final video ( gives a chance to reiterate the way

 effects can applied on timeline and too separate clips where the windows appear etc ) 

also talk a little about what im looking for and how I look for flow and rhythm –

 let the material guide you , don’t use assumptions from having watched narrative

 videos use the untutored eye i.e. datamoshing I’m cutting out references to the 

original real source as is and just trying to isolate the flow of the pixels and 

excluding repetition if possible making something new with the flow of the pixels

 as they move .

With hex editing I’m trying to bring out the artefacts or any delicious pieces of breakage would be different to my approach to datamoshing as im trying to show things like colour movement flow , or a deliberate jaggedness letting the way the file has broken guide me , if there’s no information there I remove it , if I like a texture ill keep that .

homogeneity is the enemy of glitch art

But remember

Speed of rendering is  dictated by codec chosen and machine – tailoring output resolution to machine ie 1920x1080 is useless to encode on a core 2 duo especially if its h264 , mpeg 4 and libxvid work better on older computers , ogv , vp8 and newer are more intensive on anything less than a core i3 second gen ( unless you have a decent graphics card or the processor has qsv extensions ) Get to know the capabilities of your machine What processor do you have ? What gpu . Consider shorter videos on older hardware, or lower resolutions longer videos take a lot of time to render depending on format and hardware.

What platform are you posting too What formats do YouTube etc allow for and what length and size ? Most sites need h264 but YouTube will take a wider range including webm Tumblr wants h264 with aac audio but restricts size to 100mb max ( so you could encode to the size but have a 20 minute video ) consider that facebook and instagram destroy video quality and Instagram on mobile works best with a squarish ratio and on the main feed again less than a minute of video and under 100mb.

The final thing before gifs which will help us in the next session with sound

In the second session I showed you how you could take audio and transform that into video . Using kdenlive we can then transform that video by adding effects and then turn it back into audio for use later . Caveat it doesn’t work with all effects and sometimes all you get is noise but its worth it for the times it does come up with something interesting .

So as a quick run through take some audio as wav and change the file extension to .yuv as before . Make a video out of that using :

extract audio from file ffmpeg -i somefile.extension somefile.wav

ffmpeg -f rawvideo -s 640x480 -r 25 -pix_fmt yuv420p -i somefile.yuv -c:v rawvideo output.avi

import that into kdenlive like so add effects then export as lossless huffyYUV

then open that file in audacity like this open audacity , go to file , import raw data . Locate the file with the file explorer click on it and specify unsigned 8bit byte order little endian , 2 channel ( stereo ) press import and then playback and see what it sounds like , if you like it maybe add a few effects change speed and pitch etc export it as wav and then repeat. But ill cover techniques like this in the next session .

Gifs

Making gifs 

to make gif from video select the time segment you want and cut it out of the video 

into a new file so repeating what we did before ie 

ffmpeg -ss 00:00:00 -t 00:0:10 -i somefile.avi -an -c:v copy somefilesegment.avi

( change the -ss 00:00:00 to the point you want to start from in your video ) 

(remember -ss gives us the point to start from and -t gives the amount to cut out 

so sort through the video you have already glitched for a part you want to make 

into a gif note down the time start point and how many seconds after that you want

to include 

note the -an which tells ffmpeg to remove the sound as we don’t need sound in gifs 

so then we do 

ffmpeg -i somefilesegment.avi  somefilesegment.gif

this is a good guide for making gifs into videos and videos into gifs

 https://engineering.giphy.com/how-to-make-gifs-with-ffmpeg/

aaand it goes into better depth than we can cover here  

 Using Imagemagick and FFmpeg to create a gif from a set video 

get video from the work you’ve done so far and isolate a 10 second clip .

 The we will divide that clip into stills using ffmpeg . You might want to put

 the video into a new folder the navigate to that folder and open git bash or 

terminal there . 
 

Windows 10 
ffmpeg -i somefile.avi -vf fps=1 image-%05d.png

Linux
ffmpeg -i somefile.avi -vf fps=1 image-%05d.png

then use imagemagick to make into gif via 

win 10 

magick convert -delay 10 -loop 5 -dispose previous *.png iamanimating.gif


Linux
convert -delay 10 -loop 5 -dispose previous *.png iamanimating.gif

.

Working with found sources

What to do with video once you have it - bake it break it remix it , ffmpeg from the command line, on the fly hex editing using ffmpeg and ffplay from the command line, capturing ffplay output in real time using obs-studio or simplescreenrecorder, ffglitch on Linux and windows, extracting sound from video for later, tomato, tomato sound, best formats for tomato , ffglitch etc , advantages disadvantages , basic scripting, black and white dithering .

I want to first carry on from what we were doing yesterday and introduce a few new ideas then we will be getting into using ffmpeg to prepare and re-encode videos , extract sound etc. 

Windows users if you don’t already have it turned on open windows file explorer, click on view and tick the box that says file name extensions , it means its easier to see what files are rather than just a list of names and thumbnails.

Cutting files to size

If you don’t know how to do this already first we are going to cut a video you downloaded yesterday down to a minute or two. Using a file explorer go to the place you downloaded your file , open a terminal ( right click git-bash here in windows , open in terminal in Linux ) and use ffprobe to find out its length ie ffprobe yourfile look for the part of the output which says duration . Now open the file you want to work on with vlc or mpv , or any media player , then find a section you like, make a note of the time where it starts then we are going to enter that into ffmpeg and slice out a segment of that file to a new smaller file with this the basic command :

ffmpeg -ss 00:00:00 -t 00:00:00 -i yourfile.extension -c:v copy -c:a copy yourfile.extension

-ss gives us the start point (the point in the file you found a moment ago) -t gives us how many minutes after that we want to copy

so doing this

ffmpeg -ss 00:19:00 -t 00:01:00 -i yourfile.extension -c:v copy -c:a copy yourfileshort.extension

will give us one minute of video starting from 19 minutes in ( or whatever point you choose to start and end with)

Do that and now we have a short section of video to play with we can live hex edit it quickly so stay in the folder where your shortened video is.

Live Hex editing (again)

Using a slightly different method than yesterday , we are going to do it this way because although its easier to hex edit raw-video live we don’t get any of the artefacts that different file formats would give us , which is what gives us the lovely crunchy textures. It’s a slightly less forgiving method though and you have to choose values quite carefully , and as we are using ffplay we can also press the left arrow on the keyboard to flick forwards thru the film if it gets stuck ( but not if it stubbornly refuses to play)

where I have below somefile.extension , replace that with the short file you just created

xxd -p somefile.extension | sed 's/cade/6ebe/g' | xxd -r -p | ffplay -i -

the values to change are these sed 's/cade/6ebe/g' (numbers and letters in bold) change them to any combination of hexadecimal values ie 0 to f , experiment with different values on different formats 

so with that all entered into the terminal hit enter and wait for the video to appear, check the output in the terminal for signs of life , sometimes it will just show no movement, and you get no video output , if thats the case click on the terminal and press ctrl+c to kill that process, wait for the terminal to give you back control then up arrow to recall that command 

some formats are easier than others to find interesting values , some files just wont play we can try different combinations of values until we find something that will work , though the advantage of doing it this way is that we don’t have to write to file to view results and we can tune it as we go along . And we don’t have to worry about matching the output input and output dimensions of the file to the script .

Compare this to the method we used yesterday which would give us this

ffmpeg -i somefile.extension -f rawvideo -vcodec rawvideo -pix_fmt yuv420p -s 640x480 - | xxd -p | sed 's/76/888/g' | xxd -p -r | ffplay -f rawvideo -vcodec rawvideo -pix_fmt yuv420p -s 640x480 –

Unlike the new method this doesn’t allow us to flick through sections using the left arrow and its slower.

 

How will we capture this ? Either via obs-studio or if we wanted to output to a file after testing values which give good results we could do this.

xxd -p somefile.extension | sed 's/cade/6ebe/g' | xxd -r -p > somefilehex.extension

Which will take the output and write to a new file, how you name it is up to you but name it differently to the input file otherwise you will overwrite that file and end up with nothing . Also note that the terminal will not display any output during this process just a blinking cursor after you press enter.

Now we could change the codec of the video to something more interesting like magicyuv using ffmpeg ( your ffmpeg may not have magicyuv , if not choose msvideo1 and substitute that in the script where I have magicyuv ) this script if you are not familiar with ffmpeg takes yourvideo as input ( the ffmpeg -i where -i stands for input , changes the codec used from its original codec via the -c:v part and keeps the original audio , via the c:a copy part )

so do this on your shortened file 

ffmpeg -i yourfile.extension -c:v magicyuv -c:a copy yourfilenewname.avi 

( magicyuv needs an avi extension )

Once that has encoded do the live hex editing script on that file

experiment with different codecs such as zmbv or dirac or snow ( though zmbv takes a long time to encode )

Baking 

If you output to a file sometimes that file will be unplayable or might be partially playable so you might want to bake it so its usable later on.

There’s a couple of different ways of doing that my main choice these days is just to use handbrake as it seems to be able to handle even heavily damaged files . Though there is an advantage to using a live hex editing script via ffplay and capturing that with obs-studio , especially if you are using the previous days method of using rawvideo as a format as that is damn near unbreakable. Try using handbrake on files hex edited using other methods though . Sometimes handbrake will over fix and remove errors that we want , codecs like snow are more susceptible to that , but its easy enough to use ffmpeg on a file you have hex edited and saved to file in this way.

ffmpeg -i yourhexedfile.extension -c:v formatofchoice yourhexedfilefixed.extension

Displacement mapping 

From the simple script we used yesterday to activate and glitch our webcams there is something else we can do with a similar script using a displacement map - if you’ve seen the work of Ras Alhague ( see here www.curatedbygirls.com/ras-alhagues-glitch-world and on Instagram here https://www.instagram.com/ras.alhagve/ or Yebu tonu on Instagram here https://www.instagram.com/yebutonu/ you will know that ( apart from being some of the most amazing work you will see on the inter-webs ) a displacement map is the process of taking two inputs still or video and using the texture of one to move the other against itself so that elements from one bleed into and effect the other , see Wikipedia article here https://en.wikipedia.org/wiki/Displacement_mapping doing a similar thing on windows and Linux with video is also possible using ffmpeg which has a filter that can be used to achieve this in real time. 

On linux/ windows we can do this by using a video against itself but displacement mapping is very sensitive to input sizes so use ffprobe first to check the dimensions of your video  , the first and second files must match in dimension or this won’t work if you are using different sources . Also note that if using longer videos  the lengths of each must match so you will need to cut the longest files length to match the shortest . ( using the handy combination of ffprobe then the script illustrated earlier )

Displacing a film against itself

So first ffprobe yourfile.extension and then change the output size at the end of this script to your result ie the bit that says 640x360 , alter that to your file dimensions and then replace the appropriate parts in the script below where we are displaceing the same film against itself.

ffmpeg -i yourfile.extension -i yourfile.extension -lavfi '[1]split[x][y],[0][x][y]displace' -f rawvideo -pix_fmt yuv420p - | ffplay -f rawvideo -pix_fmt yuv420p -s 640x360 -

now that's pretty simple but you can also use two different sources first two different films of the same dimensions . but how to change different dimensions to match each other ?

Changing dimensions with handbrake

If the films aren't the same dimensions we can use Handbrake to change that – also when using films with different lengths this will sometimes fail depending on the order of the film so first try changing round the order the films are processed in , ( I don’t think it really matters if we are using a live webcam feed as one of the inputs ) to avoid that we can alter the length of the films by using the command I showed earlier

So with handbrake open and having obtained the dimensions of your first film with ffprobe encode your second film with the same dimensions . Handbrake on linux and windows is pretty similar but the dimensions are in a slightly different arrangement on windows demo this . Especially the ‘allow upscaling tick box ‘ otherwise it won't work. Not the same on linux at all.

So to give an idea of what this would work like, practice with different films.

Displace one video against another

I prepared two films earlier using handbrake  , first by matching the aspect ratios (3:2) and dimensions 480x320 using handbrake

The first ‘Horror hotel’ from 1960 you can find here https://archive.org/details/HorrorHotel1960 which is published under this creative commons licence which allows reuse , adaptation and sharing with attribution more info here https://creativecommons.org/licenses/by/3.0/

The second source is called ‘prisoners of the lost universe’ found here https://archive.org/details/PrisonersOfTheLostUniverse1983_201511 which according to its licence is completely within the public domain so I can do what I like pretty much with it ( taking into consideration the things we discussed yesterday ) https://creativecommons.org/publicdomain/mark/1.0/

do this on windows and linux ( its the same )

ffmpeg -i Horror_Express.mp4 -i prisoners.mp4 -lavfi '[1]split[x][y],[0][x][y]displace' -f rawvideo -pix_fmt yuv420p - | ffplay -f rawvideo -pix_fmt yuv420p -s 480x320 -

or with the video shortened and the order swapped

ffmpeg -i prisonersshort.mp4 -i Horror_expressshort.mp4 -lavfi '[1]split[x][y],[0][x][y]displace' -f rawvideo -pix_fmt yuv420p - | ffplay -f rawvideo -pix_fmt yuv420p -s 480x320 -

Depending on the source we can achieve different textures , this technique works really well if you have already glitched video to work with , say from glitched to non-glitched bear in mind colour as well , some strong colours like bright red/orange or blue can come through exceptionally well depending on the material you start with and if you have a particular video with a particular texture it can work really well.

If you dont want to watch this happen live or are happy that you are getting a file you can output to a file like this but be aware that these files can be quite large and may need re-encoding via handbrake to make them manageable for later editing, especially on resource limited machines.

ffmpeg -i prisonersshort.mp4 -i Horror_expressshort.mp4 -lavfi '[1]split[x][y],[0][x][y]displace' filename.mp4

And then maybe start adding back over that by using that file as input for other videos

Because this is rawvideo format we are working live we can hex edit it as well by modifying our earlier script.

ffmpeg -i prisonersshort.mp4 -i Horror_expressshort.mp4 -lavfi '[1]split[x][y],[0][x][y]displace' -f rawvideo -pix_fmt yuv420p - | xxd -p | sed 's/cade/6eb/g' | xxd -r -p | ffplay -f rawvideo -pix_fmt yuv420p -s 480x320 -

and as we are using ffplay if we hit the spacebar we can pause or resume video so we can screenshot stills we also like along the way.

so we are starting  to chain ideas and techniques together .

To capture with hex editing use obs-studio to capture the piped ffplay output or use your screen recorder software of choice.

Displacing live video over found video

We could also can also use live sources like webcams to mix with video if the output of the camera matches the dimensions of the video . How would we do that ? On windows like this

ffmpeg -i prisoners640x360.mp4 -f dshow -i video="Live! Cam Sync HD VF0770" -lavfi '[1]split[x][y],[0][x][y]displace' -f rawvideo -pix_fmt yuv420p - | ffplay -f rawvideo -pix_fmt yuv420p -s 640x360 -

or reversing the order ie camera first and video input after notice I’ve slowed the frame rate of the camera as it seems to stutter , but the first method seems to work better

ffmpeg -f dshow -i video="Live! Cam Sync HD VF0770" -r 5 -i prisoners640x360.mp4 -lavfi '[1]split[x][y],[0][x][y]displace' -f rawvideo -pix_fmt yuv420p - | ffplay -f rawvideo -pix_fmt yuv420p -s 640x360 -

the equivalent on linux would be

ffmpeg -i prisoners640x360.mp4 -i /dev/video0 -lavfi '[1]split[x][y],[0][x][y]displace' -f rawvideo -pix_fmt yuv420p - | ffplay -f rawvideo -pix_fmt yuv420p -s 640x360 -

and reverse the input

ffmpeg -i /dev/video0 -i prisoners640x360.mp4 -lavfi '[1]split[x][y],[0][x][y]displace' -f rawvideo -pix_fmt yuv420p - | ffplay -f rawvideo -pix_fmt yuv420p -s 640x360 -

( again alter the input and output dimensions to reflect your webcam resolution and the dimensions of the video )

Grabbing the desktop and using that to displace video 

Grab a part of the desktop as a virtual camera and use that to displace the video playing

on Linux

ffmpeg -f x11grab -follow_mouse centered -framerate 25 -video_size 640x360 -i :0.0 -i prisoners640x360.mp4 -lavfi '[1]split[x][y],[0][x][y]displace' -f rawvideo -pix_fmt yuv420p - | xxd -p | sed 's/808/181/g'| xxd -p -r | ffplay -f rawvideo -pix_fmt yuv420p -s 640x360 -

on Windows

ffmpeg -f gdigrab -framerate 6 -offset_x 10 -offset_y 20 -video_size 640x360 -i desktop -i prisoners640x360.mp4 -lavfi '[1]split[x][y],[0][x][y]displace' -f rawvideo -pix_fmt yuv420p - | xxd -p | sed 's/808/181/g'| xxd -p -r | ffplay -f rawvideo -pix_fmt yuv420p -s 640x360 -

On windows 10 to make this work well open another video in vlc or with ffplay and place that in the upper left hand corner of your screen the window opened previously uses that as a displacement map which in itself can be quite interesting , but it doesn’t have the flexibility of the linux option, but I’m going to be dealing with that in the fifth and final session ‘What is the desktop anyway’ .

Before we get to ffglitch extracting sound from video

This is kind of a by the by as well be dealing with sound in another session but its a good idea to cover this now so you have files to play with later

So in the same directory as the files you are working with ( on linux and windows)

just type in

ffmpeg -i yourfile.extension yourfile.wav

it really is that simple why wav and not mp3 or something else ? Wav is near raw , and so we can use it in more interesting ways and it tends to break less .

For instance if we wanted to visualize sound we could just do this with the wav we just created .

Find your wav file change the extension to .yuv

ffmpeg -f rawvideo -s 640x480 -r 25 -pix_fmt yuv420p -i yourfile.yuv -c:v libx264 -preset ultrafast -qp 0 output.mp4

which will save it to a file or we could do it live using ffplay like this

ffplay -f rawvideo -pix_fmt yuv420p -s 640x480 -qp 0 -i yourfile.yuv

If you want to remove sound from video which is handy for some kinds of datamoshing like tomato (old version which I still prefer ) do this

ffmpeg -i yourfile.extension -an -c:v copy newfile.extension

Datamoshing with ffgac ( aka ffglitch)

we’ve downloaded some content , hex edited it and maybe run a few displacement maps over it

If you havent used it before What is FFglitch? Ramiro Polla's FFglitch is basically the best tool for datamoshing there is, Kaspar Ravels tomato v2 is a beautiful and simple tool but it has limits , FFglitch is more complicated to use but gives much more fine grained control of the process. Where to get it - here https://ffglitch.org/ it comes in Linux , Mac and Windows flavors Download it and unpack it, there's no need to install it, as its a stand-alone binary, but to use it we will need to take a few steps first.

FFglitch is essentially a command line tool , and this is where windows sucks , powershell is okay but we need a bit more power which is why we have git-bash installed I’m not going to run through using python scripts with ffgac as Ramiro Polla states he is moving away from python to using javascript . Now I’m presuming everyone in windows land has either notepad ++ or geany installed – check . Okay

All of that out of the way I'm going to walk through the example I was actually asked about - the move sink and rise example that Ramiro Polla has on his website . First things first we need to change the format of any file we are using so it suits ffgac ( the new version is called ffgac not ffglitch ) which handily there is an example script to do just that - now one of the misconceptions people have on using ffglitch / ffgac is that they can use standard ffmpeg to prep their files and this leads to a further misconception which leads to trying and failing to use ffglitch / ffgac - and I think that explains the renaming - ffgac is a standalone version of ffmpeg designed to be used with this set of tools , all the scripts reference it and not any other version you may have installed - this is why we work within the ffgac / ffglitch directory .

We are also going to be using shellscripts today , if you don’t know what a shellscript is well think of it as a list of commands put into a text file that the computer will execute in order unless it finds an error , we run them from the terminal ( thus git-bash ) and they enable us to put together long and complex sequences of commands it would otherwise be tedious to type in or call up constantly from the terminal interface. Its job automation similar in some ways to dos .bat scripts but way more powerful

so prep your file with the script Ramiro Polla has on his website but modified for use with windows - open notepad++ and copy pasta what I’m about to paste into the shared notes from #!/bin/bash to done , save that script in the folder where you have ffgac as ffgacprep.sh and also copy the shortened file you made earlier into this folder for simplicity's sake as well

#!/bin/bash
mkdir mpeg2

#change mp4 below to whatever the file extension your source file has
for I in *.mp4;

do

#on linux change the line below to ./ffgac -i "$I" \

./ffgac.exe -i "$I" \

-an -vcodec mpeg2video -f rawvideo \

-mpv_flags +nopimb \

-qscale:v 6 \

-r 30 \

-g 90 \

-s 1280x720 \

-y mpeg2/"${I/.mp4/.mpg}";

done

open git bash terminal in that folder ie right click and click on Git bash here - the terminal window will appear - before we can use that script we have to make it executable so in the terminal type in exactly this ( without quote marks) ' chmod u+x ffgacprep.sh ' - if all goes well there will be no error messages and the command line returns . so to prep your file now type in exactly in the terminal :

./ffgacprep.sh

The script will create a folder within that folder called mpeg2 and output your file all ready for the next stage move it into the main ffglitch folder .

You could also use the command line commands shown on the website. So in git bash terminal and with your video in the ffglitch folder copy and paste that command into git bash terminal ( to paste into git bash terminal press shift and ins ) - change where it says input file to the name of your file - leave temp_file.mpg as your output file name unless your working on a lot of files and alter the first part of the script from ffgac to ./ffgac.exe ( cos we are working on windows and not linux ) so it will look like this ( but as one long line) - using the left and right arrow keys to navigate through the command and change it . ( read these commands as being on one line )

./ffgac.exe -i input_file -an -mpv_flags +nopimb+forcemv -qscale:v 0 -g 1000 -vcodec mpeg2video -f rawvideo -y temp_file.mpg

for linux just remove the .exe so it reads 

 
./ffgac -i input_file -an -mpv_flags +nopimb+forcemv -qscale:v 0 -g 1000 -vcodec mpeg2video -f rawvideo -y temp_file.mpg

On the ffgac website there is a script called ‘mv_sink_and_rise.js’ here https://ffglitch.org/2020/07/mv.html look for the section about halfways down where he says ' With all that being said, here’s the script that we will be using' open notepad++ and copy the script and save it in the ffglitch folder as 'mv_sink_and_rise.js' (without the quotes ) .

function glitch_frame(frame)
{
    // bail out if we have no motion vectors
    let mvs = frame["mv"];
    if ( !mvs )
        return;
    // bail out if we have no forward motion vectors
    let fwd_mvs = mvs["forward"];
    if ( !fwd_mvs )
        return;

    // clear horizontal element of all motion vectors
    for ( let i = 0; i < fwd_mvs.length; i++ )
    {
        // loop through all rows
        let row = fwd_mvs[i];
        for ( let j = 0; j < row.length; j++ )
        {
            // loop through all macroblocks
            let mv = row[j];

            // THIS IS WHERE THE MAGIC HAPPENS

            mv[0] = 0; // this sets the horizontal motion vector to zero
            // mv[1] = 0; // you could also change the vertical motion vector
        }
    }
}

Now for the magic run the command he has on his website but for windows we change it to ;

./ffedit.exe -i temp_file.mpg -f mv -s mv_sink_and_rise.js -o glitched_file.mpg
and linux again just remove the .exe so it reads 
./ffedit -i temp_file.mpg -f mv -s mv_sink_and_rise.js -o glitched_file.mpg

copy and paste ( or type !) the above into git bash terminal and press enter :

Once that has run succesfully look for the file in the ffglitch folder and play it!

Once we have this running and you are happy with this process go back to Ramiro Polla's tutorials and start messing about with values in the mv_sink_and_rise.js script.

We can do more complex things if we mess with this bit

mv[0] = 0; // this sets the horizontal motion vector to zero

            // mv[1] = 0; // you could also change the vertical motion vector

to something like this 

//mv[0] = 0; // this sets the horizontal motion vector to zero

            mv[1] = -1; // you could also change the vertical motion vector

Mess about with these values until you have something you like and then go and read

 more on  Ramiro’s website. Ive used this in part to replicate some of the effects 

I can get using the avance logic card that I have that gives the slidng

 horizontals effect.

Black and white dithering  

Sometimes we want to convert colour to black and white , there are a number of ways

 to do that , some simple some complicated ffmpeg gives us a way to do that simply 

( it looks complex but it isn’t) 

ffmpeg -i yourfile.extension -f lavfi -i color=gray:s=1920x1080 -f lavfi -i color=black:s=1920x1080

-f lavfi -i color=white:s=1920x1080 -filter_complex threshold -sws_dither a_dither yourfile.extension

change where I have 1920x1080 to the dimensions of your file , again find that 

using ffprobe 

If we take the file we just generated using ffgac

or we could do that more simply instead of it being just black and white it could be to desaturate the file

ffmpeg -i yourfile.extension -vf hue=s=0 output.extension

true greyscale would be

ffmpeg -i input -vf format=gray output