Simple motion vector tutorial

Introduction

One of the most used features of ffglitch is the ability to tinker around with motion vectors in MPEG-2 files.

I’m not going to go in full details explaining what motion vectors are but basically they describe movements between frames in video files. If we alter those values, we can make all sorts of funky glitches.

In this post I will show how to do this with a very simple script.

0 - Generate input file

First and foremost, we want to be playing around with a file that does have motion vectors. Usually the encoder will not write motion vectors to the bitstream if there are no motion vectors to write (duh), but we want to force them everywhere. We also want to force that no I macroblocks will be written to the bitstream in P frames (no need to understand this now). We also want to make sure we have a long run of P frames, because I frames effectively deglitch the video, and we don’t want that to happen.

All of this preparation can be done with the modified ffmpeg build that comes with ffglitch (called ffgac). Run the following command line, replacing input_file with the original video file you want to glitch, and temp_file.mpg with a filename that will be used as your input to glitch.

$ ffgac -i input_file -an -mpv_flags +nopimb+forcemv -qscale:v 0 -g max -sc_threshold max -vcodec mpeg2video -f rawvideo -y temp_file.mpg

Great! Now you have an MPEG-2 file ready to be glitched (it’s temp_file.mpg).

For this tutorial, I will use the Rush Hour Traffic Circa 1956 public domain video.

1 - Glitch file with Javascript

To glitch the file using ffglitch’s native scripting capabilities (made possible by integrating quickjs), you will have to write code in Javascript.

This file should have a function called glitch_frame, which will be called once for each frame.

The frame object passed to glitch_frame will have an object for each feature that was requested (in our case, mv for motion vectors), and we modify them directly.

The motion vector object might have a backward object if we are dealing with B frames (ignore this for now), and might have a forward object for P frames (this is what we will use).

In the backward or forward objects, there will be one array for each row of macroblocks, and these arrays will each have one array of 2 values (for horizontal and vertical motion vectors), for each macroblock.

The structure for motion vectors in frame looks like this:

"mv": {
    "forward": [
        [ [ x, y ], [ x, y ], ..., [ x, y ] ],
        [ [ x, y ], [ x, y ], ..., [ x, y ] ],
        [ [ x, y ], [ x, y ], ..., [ x, y ] ],
        ...
        [ [ x, y ], [ x, y ], ..., [ x, y ] ]
    ],
    "backward": [
        [ [ x, y ], [ x, y ], ..., [ x, y ] ],
        [ [ x, y ], [ x, y ], ..., [ x, y ] ],
        [ [ x, y ], [ x, y ], ..., [ x, y ] ],
        ...
        [ [ x, y ], [ x, y ], ..., [ x, y ] ]
    ]
}

With all that being said, here’s the script that we will be using. Save it as a file named mv_sink_and_rise.js:

function glitch_frame(frame)
{
    // bail out if we have no motion vectors
    let mvs = frame["mv"];
    if ( !mvs )
        return;
    // bail out if we have no forward motion vectors
    let fwd_mvs = mvs["forward"];
    if ( !fwd_mvs )
        return;

    // clear horizontal element of all motion vectors
    for ( let i = 0; i < fwd_mvs.length; i++ )
    {
        // loop through all rows
        let row = fwd_mvs[i];
        for ( let j = 0; j < row.length; j++ )
        {
            // loop through all macroblocks
            let mv = row[j];

            // THIS IS WHERE THE MAGIC HAPPENS

            mv[0] = 0; // this sets the horizontal motion vector to zero
            // mv[1] = 0; // you could also change the vertical motion vector
        }
    }
}

Now run ffedit with mv_sink_and_rise.js to glitch the file:

$ ./ffedit -i temp_file.mpg -f mv -s mv_sink_and_rise.js -o glitched_file.mpg

And this is the resulting glitched_file.mpg:

Conclusion

I don’t like writing posts with introductions and conclusions, but since there was an introduction section, there is also a conclusion section.

Go wild with your scripts and glitch the fuck out of those motion vectors…