Server side images and animations generation with node

On my spare time, I work on several projects. One of them is a twitter game on which I work with other people. This is a thinking game consisting in solving a problem with an instruction sequence. The bug must reach the goal with fewest instructions and displacements possible.

Currently, this is a twitter game. Every day, a new challenge is sent in a tweet. People can reply to the tweet with their instructions; then the bot reply to them with their score. One of the interest of the game is to have a feedback when someone tries to resolve the challenge.

This article explains a simple way to generate an image (png) or an animation (gif or webm) on the server side, with a Node.js server.

Node.js web server

We want to build a simple web service in order to send generated medias. T he server receive a request with the url, and generate the media.

As example, on my project, we want to generate a 2D view of a map. We can specify inside the URL the resolution of the map, the squares, a theme, the instructions and other things.

The format of the url is here /api followed of the parameters:

• /res/[x]:[y]
• /theme/[value]
• /cmd/[instructions]
• /map/[squares]
• /type/[png|gif|webm]

Here, I use the express package as the web server to define the route and extract parameters.

var express = require('express'),
              ... ;

var app = express();

// /api/[parameters] route
app.get(/^\/api\/(.*)/, function(req, res) {

  // Default configuration
  var config = {
    res: new Point(6, 6),
    theme: 13,
    squares: '',
    command: '',
    type: 'png'
  };

  // Parse URL parameters and set right configurtion
  var params = req.params[0].toUpperCase().split('/');
  params = (params === null)? [] : params;
  ...

  handleRequest(config, res);

});

Node canvas

Node canvas is a Canvas implementation for Node.js: https://github.com/Automattic/node-canvas.

It has dependencies on native libraries. A wiki explains how to install everything on each OS: https://github.com/Automattic/node-canvas/wiki.

This package adds some useful features as loading image from the disk or handle streams.

Usage

We use this implementation almost the same way we could use a HTML5 canvas on a browser. This is so the best choice to generate an image in javascript if you already know how to draw on a canvas.

var Canvas = require('canvas');

var canvas = new Canvas(200, 150);
var context = canvas.getContext("2d");
context.beginPath();
context.arc(100, 75, 50, 0, 2 * Math.PI);
context.stroke();

.toBuffer() and .toDataURL()

One of the main features is the possibility to get the canvas image as a raw buffer or as an image format. The .toDataUrl() method is only able to return the image representation as a PNG file.

These methods can be used to manipulate pixels in order to render images or animations.

Send as a PNG file

It is really easy to send the result as a PNG file because it is native in the node-canvas package.

function sendAsPNG(response, canvas) {

  var stream = canvas.createPNGStream();
  response.type("png");
  stream.pipe(response);

};

Performances

On a standard server, it takes around 10ms to generate a standard PNG image on my application.

Send as a GIF file

Here, we need to use a package. But it remains relatively easy to generate a GIF.

We only need to use the gifencoder package: https://github.com/eugeneware/gifencoder.

var GIFEncoder = require('gifencoder');

function createGifEncoder(resolution, response) {

  var encoder = new GIFEncoder(resolution.x * 32, resolution.y * 32);

  var stream = encoder.createReadStream();
  response.type("gif");
  stream.pipe(response);

  encoder.start();
  encoder.setRepeat(0);   // 0 for repeat, -1 for no-repeat
  encoder.setDelay(150);  // frame delay in ms
  encoder.setQuality(15); // image quality. 10 is default.

  return encoder;

}

function sendAsGIF(response, canvas) {

  var encoder = createGifEncoder({x: canvas.width, y: canvas.height}, response);

  var context = canvas.getContext("2d");

  // Add 3 frames
  encoder.addFrame(context);
  encoder.addFrame(context);
  encoder.addFrame(context);

  encoder.finish();

};

Performances

On a standard server, it takes around 500ms to generate a standard GIF of 10 frames and 500ko on my application.

Send as a WebM file

A GIF is great, but it is really heavy. WebM is a format supported by Google. It is an open video format which encapsulate WebP compressed images.

Handle WebP picture

On some browsers (as Chrome), you can ask toDataUrl() in a WebP format. It is then really easy to embed these pictures inside a WebM video. But in the node-canvas implementation, we can’t use it. We need an other way.

Sharp is a node package allowing to handle JPEG, PNG, and TIFF pictures, but also WebP: https://github.com/lovell/sharp.

It has a dependency on libvips. Here is a wiki on how to install vips: http://www.vips.ecs.soton.ac.uk/index.php?title=Supported.

var sharp = require('sharp');

function canvasToWebp(canvas, callback) {

  sharp(canvas.toBuffer()).toFormat(sharp.format.webp).toBuffer(function(e, webpbuffer) {
    var webpDataURL = 'data:image/webp;base64,' + webpbuffer.toString('base64');
    callback(webpDataURL);
  });

}

Handle WebM video

Whammy is a real time javascript webm encoder: https://github.com/jbouny/whammy.

To have the latest versio working on node.js, you need to add the git repo inside your package.json (if npm used).

"dependencies": {
    "node-whammy": "git://github.com/jbouny/whammy.git",
},

The problem with sharp is that the callback is asynchronous. We need to control that frames are added is the right order.

var Whammy = require('node-whammy');

function sendAsWEBP(response, canvas) {

  var encoder = new Whammy.Video(7);

  var currentId = 0,
      time = 0,
      timeout = 20000,
      delay = 20
      addedFrame = -1,
      totalFrames = 3,
      tmpFrames = Array.apply(null, Array(totalFrames));

  var addFrame = function addFrame(context) {
    var id = currentId++;
    canvasToWebp(context.canvas, function(webmData) {
      tmpFrames[id] = webmData;
      for(var i = addedFrame + 1; i < totalFrames; ++i) {
        if(tmpFrames[i] !== undefined) {
          encoder.add(tmpFrames[i]);
          addedFrame = i;
        }
        else {
          break;
        }
      }
    });
  };

  var checkReady = function checkReady() {
    if(totalFrames <= addedFrame + 1) {
      try {
        var output = encoder.compile(true);
        response.type('webm');
        response.send(new Buffer(output));
        console.log('Webm compilation: ' + time + 'ms');
      }
      catch(err) {
        response.send(err.toString());
      }
    }
    else if((time += delay) < timeout) {
      setTimeout(checkReady, delay);
    }
    else {
      response.send('Timeout of ' + timeout + 'ms exceed');
    }
  };

  var context = canvas.getContext("2d");

  // Add 3 frames
  addFrame(context);
  addFrame(context);
  addFrame(context);

  setTimeout(checkReady, delay);

};

Performances

On a standard server, it takes around 500ms to generate a standard WebM of 10 frames and 150ko on my application.