Trying to Copy Instagram Face Filters

Since we’ve learned about using media in p5.js, including image, video, and sound, I’d like to explore about using web camera. I didn’t know that using a web camera in javascript would be kinda easy (thought it would be hard to code).

Inspired by Snapchat at Instagram face filters, I wanted to make a similar thing. What came up in my mind was to use a tool for face recognition. I had no idea what was it and how to use it, but thank God I found a library that was developed under MIT license called clmtrackr.js. It is a javascript library to track precise positions of face in images and videos. Using this library, I was able to track the face movement in camera.

TD;LR Here’s what it looks like in the end.

In p5.js, to use a web camera, first we need to set it up. In setup function, I wrote this code:

let videoInput; // video input

function setup() {

    // setup camera capture
    videoInput = createCapture();
    videoInput.size(800, 600);
    videoInput.position(200, 110);
    videoInput.loop();
}

Talking about clmtrackr.js, it is not hard to be used. First, I just downloaded it from its site, then I put clmtrackr.js in my addons folder, then linked it in my index.html. I also put the model inside models folder.

 <img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" data-wp-preserve="%3Cscript%20src%3D%22addons%2Fclmtrackr.js%22%3E%3C%2Fscript%3E" data-mce-resize="false" data-mce-placeholder="1" class="mce-object" width="20" height="20" alt="<script>" title="<script>" />
 <img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" data-wp-preserve="%3Cscript%20src%3D%22models%2Fmodel_pca_20_svm.js%22%3E%3C%2Fscript%3E" data-mce-resize="false" data-mce-placeholder="1" class="mce-object" width="20" height="20" alt="<script>" title="<script>" />

In setup function, create the tracker and initialize the model.

function setup() {
    // setup tracker
    ctracker = new clm.tracker();
    ctracker.init(pModel);
    ctracker.start(videoInput.elt);
}

To get the current position of the face, I used ctracker.getCurrentPosition(). Here’s an example how to draw a hat on the head:

function drawHat() {
    // get array of face marker positions [x, y] format
    var positions = ctracker.getCurrentPosition();
    for (var i=0; i<positions.length; i++) {

        // draw shapes based on marker positions
        if (i == 20) {
            push();
            translate(-70, -60);
            fill(100, 50, 60);
            rect(positions[i][0], positions[i][1], 280, 25);
            pop();

            push();
            translate(-30, -140);
            fill(100, 50, 60);
            rect(positions[i][0], positions[i][1], 200, 100);
            pop();
        }
    }
}

To be honest, I didn’t know what exactly was the array [x, y]. But looking at the reference of the model. I assumed that the index of array is based on this image.

Array Positions

Making this face filters was kinda fun actually. I had a lot of fun making shapes, using some images, and using Video Pixels. I tried to make a simple drawing using basic shapes such as rectangle and ellipse. Also, since Snapchat and Instagram have this doggy face filter, I inserted images of dog’s nose, and ears. Other than that, I also uploaded an image that look like a mask.

Mask Filter
Cool Filter

Besides that, I also playing around with Video Pixels, how to manipulate the pixels in video. I looked at an example here, so I tried to play around with it.

Daftpunk Filter
Fog Filter

After all, I had fun making these filters using media in p5.js. Here’s my code.

And try this filters here!

Leave a Reply

Your email address will not be published. Required fields are marked *