Pushing the limits of video encoding with WebCodecs

JSConf India 2023

👋 Hey there!
I'm Akash Hamirwasia

Writing code @Razorpay. Love building products on the web, trying out new technologies and contributing to open-source.

⚡️ Blaze

🔎 UnTab

🔌 Diode

Slant it

Let's connect

Twitter: @blenderskool

GitHub: @blenderskool

Website: akashhamirwasia.com

akashhamirwasia.com

JSConf India 2023

Little back story...

I was building a video editor on the browser to make interesting product videos.
Everything was going fine until I started writing code for the export process.
Got stuck at bunch of places... video formats, audio, encoding, muxing, containers, etc.
Finally figured it out! 🙌

akashhamirwasia.com

JSConf India 2023

Drawing a video on the web

Video is a set of image frames changing at a constant rate.
Each frame can be rendered in a variety of ways:

CSS Animation
Modifying DOM directly with JavaScript
SVGs
HTML5 Canvas

The frames rendered on screen can be recorded and saved as a video file.

akashhamirwasia.com

JSConf India 2023

Record something on screen using JavaScript?

akashhamirwasia.com

JSConf India 2023

Canvas allows video capture!

Through it's captureStream() API, a part of the MediaStream Recording API.

1
const canvas = document.querySelector('canvas');
2
const record = (duration) => {
3
  const stream = canvas.captureStream(30);
4
  const recorder = new MediaRecorder(stream);
5
  const chunks = [];
6
7
  recorder.ondataavailable = (event) => chunks.push(event.data);
8
  recorder.onstop = () => {
9
    const videoBlob = new Blob(chunks, { type: 'video/webm' });
10
    download(videoBlob);
11
  };
12
13
  recorder.start();
14
  setTimeout(() => recorder.stop(), duration);
15
}

akashhamirwasia.com

JSConf India 2023

Sadly, it's not well-suited for this use case

No constant framerate.
```
canvas.captureStream(framerate)
```
framerate – maximum framerate in which the video is supposed to be captured.
No option to change video file format.
Low reliablity.
Ok quality.

akashhamirwasia.com

JSConf India 2023

MediaRecording APIs are not designed to be used for precise and high quality video exports

akashhamirwasia.com

JSConf India 2023

Can we capture and encode each frame of the canvas ourself?

akashhamirwasia.com

JSConf India 2023

1
const images = [];
2
3
while(frame < totalFrames) {
4
  seekCanvas(frame);
5
  const image = canvas.toDataURL('image/webp');
6
  images.push(image);
7
  ++frame;
8
}
9
10
const video = encode(images);
11
download(video);

images = []

Canvas

akashhamirwasia.com

JSConf India 2023

1
const images = [];
2
3
while(frame < totalFrames) {
4
  seekCanvas(frame);
5
  const image = canvas.toDataURL('image/webp');
6
  images.push(image);
7
  ++frame;
8
}
9
10
const video = encode(images);
11
download(video);

images = []

Canvas

Frame 1

akashhamirwasia.com

JSConf India 2023

1
const images = [];
2
3
while(frame < totalFrames) {
4
  seekCanvas(frame);
5
  const image = canvas.toDataURL('image/webp');
6
  images.push(image);
7
  ++frame;
8
}
9
10
const video = encode(images);
11
download(video);

images = [

📸
Frame 1

]

Canvas

Frame 1

akashhamirwasia.com

JSConf India 2023

1
const images = [];
2
3
while(frame < totalFrames) {
4
  seekCanvas(frame);
5
  const image = canvas.toDataURL('image/webp');
6
  images.push(image);
7
  ++frame;
8
}
9
10
const video = encode(images);
11
download(video);

images = [

📸
Frame 1

]

Canvas

Frame 2

akashhamirwasia.com

JSConf India 2023

1
const images = [];
2
3
while(frame < totalFrames) {
4
  seekCanvas(frame);
5
  const image = canvas.toDataURL('image/webp');
6
  images.push(image);
7
  ++frame;
8
}
9
10
const video = encode(images);
11
download(video);

images = [

📸
Frame 1

📸
Frame 2

]

Canvas

Frame 2

akashhamirwasia.com

JSConf India 2023

1
const images = [];
2
3
while(frame < totalFrames) {
4
  seekCanvas(frame);
5
  const image = canvas.toDataURL('image/webp');
6
  images.push(image);
7
  ++frame;
8
}
9
10
const video = encode(images);
11
download(video);

images = [

📸
Frame 1

📸
Frame 2

...]

Canvas

Frame ...

akashhamirwasia.com

JSConf India 2023

1
const images = [];
2
3
while(frame < totalFrames) {
4
  seekCanvas(frame);
5
  const image = canvas.toDataURL('image/webp');
6
  images.push(image);
7
  ++frame;
8
}
9
10
const video = encode(images);
11
download(video);

How to encode an array of images to a video?

images = [

📸
Frame 1

📸
Frame 2

...]

Canvas

Frame ...

akashhamirwasia.com

JSConf India 2023

Whammy.js

A realtime JavaScript WebM Encoder.
Appeared on Hacker News 11 years ago!
Takes snapshots of canvas as images and encodes them to a video file.

function encode(images) {
  return window.Whammy.fromImageArray(images, 30);
}

github.com/antimatter15/whammy

akashhamirwasia.com

JSConf India 2023

Pros & Cons of these JS libraries

✅ High quality output

✅ Constant framerate

✅ Relatively easy

❌ Very slow

❌ Synchronous

❌ No hardware-acceleration

❌ No choice of video formats

❌ No support for adding audio

akashhamirwasia.com

JSConf India 2023

Now what?

WebCodecs

akashhamirwasia.com

JSConf India 2023

What is WebCodecs?

Set of APIs that give low-level access to individual frames of a video stream.
Supports encoding / decoding of each frame in various formats.
It's asynchronous and hardware accelerated!
It's more than just video, supports audio too!

akashhamirwasia.com

JSConf India 2023

From MDN – developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API

akashhamirwasia.com

JSConf India 2023

Video encoding process

akashhamirwasia.com

JSConf India 2023

Video encoding process

akashhamirwasia.com

JSConf India 2023

Video encoding process

akashhamirwasia.com

JSConf India 2023

Let's look at the code to encode videos using WebCodecs

akashhamirwasia.com

JSConf India 2023

Creating a `VideoEncoder`


1
const encoder = new VideoEncoder({
2
  output: (chunk, metadata) => {
3
    // Do something with encoded chunk
4
  },
5
  error: (err) => {
6
    // Handle error
7
  }
8
});

akashhamirwasia.com

JSConf India 2023

Creating a `VideoEncoder`


1
const chunks = [];
2
const encoder = new VideoEncoder({
3
  output: (chunk, metadata) => {
4
    chunks.push(chunk);
5
  },
6
  error: (err) => {
7
    console.log(err);
8
  }
9
});

akashhamirwasia.com

JSConf India 2023

Configuring the `VideoEncoder`

1
const encoder = new VideoEncoder(...);
2
3
encoder.configure({
4
  codec: 'vp8',
5
  hardwareAcceleration: 'prefer-hardware',
6
  framerate: 30,
7
  latencyMode: 'quality',
8
  width: 1920,
9
  height: 1080,
10
  bitrate: 2_000_000,
11
  ...
12
});

akashhamirwasia.com

JSConf India 2023

How to find the right `codec`?

It's not as easy as .webm, .mp4, .avi, etc.
They are seemingly random strings!
Eg. vp8, avc1.420034, av01.0.12M.10

video codec != video file format

akashhamirwasia.com

JSConf India 2023

Video Encoding 101

akashhamirwasia.com

JSConf India 2023

Individual frames

Frame 1

Frame 2

Frame 3

Encoding algorithm

Defines how to convert, compress and store frames of the video

Video stream

Frame 1

Frame 2

Frame 3

akashhamirwasia.com

JSConf India 2023

Video stream

Frame 1

Frame 2

Frame 3

Muxer

Video container

video.mp4

Video stream

akashhamirwasia.com

JSConf India 2023

Video stream

Audio stream

Subtitles

"Muxer"

Combines(multiplexes) multiple encoded streams to a single container

Video container

video.mp4

Video stream

Audio stream

Subtitles

akashhamirwasia.com

JSConf India 2023

What's a `codec` again?

Encoding algorithm to convert, compress the individual frames of the video.
A codec and container must be compatible with each other.

container	codec	example
WebM	VP8, AV1	vp8
MP4	AVC, AV1	avc1.420034, av01.0.12M.10

Every browser supports different set of codecs.

developer.mozilla.org/en-US/docs/Web/Media/Formats/codecs_parameter#codec_options_by_container developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs#common_codecs

akashhamirwasia.com

JSConf India 2023

Back to code

1
const chunks = [];
2
const encoder = new VideoEncoder(...);
3
const FPS = 30;
4
5
encoder.configure({
6
  codec: 'avc1.420034',
7
  avc: { format: 'annexb' },
8
  hardwareAcceleration: 'prefer-hardware',
9
  framerate: FPS,
10
  latencyMode: 'quality',
11
  width: 1920,
12
  height: 1080,
13
  bitrate: 2_000_000,
14
  ...
15
});

akashhamirwasia.com

JSConf India 2023

Back to code

1
const chunks = [];
2
const encoder = new VideoEncoder(...);
3
const FPS = 30;
4
5
const config = {
6
  codec: 'avc1.420034',
7
  ...
8
};
9
10
const support = await VideoEncoder.isConfigSupported(config);
11
if (!support.supported) {
12
  console.log("Oh noes, config not supported! 😢");
13
  return;
14
}
15
16
encoder.configure(config);

akashhamirwasia.com

JSConf India 2023

Back to code

1
const chunks = [];
2
const encoder = new VideoEncoder(...);
3
const FPS = 30;
4
encoder.configure({ ... });
5
6
while(frame < totalFrames) {
7
  seekCanvas(frame);
8
9
  const videoFrame = new VideoFrame(canvas, {
10
    timestamp: frame / FPS * 1_000_000,
11
  });
12
  encoder.encode(videoFrame);
13
  videoFrame.close();
14
15
  ++frame;
16
}

akashhamirwasia.com

JSConf India 2023

Back to code

1
const chunks = [];
2
const encoder = new VideoEncoder(...);
3
const FPS = 30;
4
encoder.configure({ ... });
5
6
while(frame < totalFrames) {
7
  ...
8
}
9
10
await encoder.flush();
11
encoder.close();
12
13
const videoBuffer = concatBuffers(chunks);

download videoBuffer?

akashhamirwasia.com

JSConf India 2023

Recall the video encoding process

Individual frames

Frame 1

Frame 2

Frame 3

codec

Video stream

Muxer

Video container

video.mp4

Handled by WebCodecs

How do we do this?

akashhamirwasia.com

JSConf India 2023

Using ffmpeg.wasm as a muxer

1
...
2
await encoder.flush();
3
encoder.close();
4
5
const videoBuffer = concatBuffers(chunks);
6
7
const ffmpeg = createFFmpeg();
8
await ffmpeg.load();

akashhamirwasia.com

JSConf India 2023

Using ffmpeg.wasm as a muxer

1
...
2
await encoder.flush();
3
encoder.close();
4
5
const videoBuffer = concatBuffers(chunks);
6
7
const ffmpeg = createFFmpeg();
8
await ffmpeg.load();
9
10
ffmpeg.FS('writeFile', 'raw.h264', videoBuffer);
11
ffmpeg.run('-i', 'raw.h264', '-map', '0:v:0', '-c:v', 'copy', 'output.mp4');

akashhamirwasia.com

JSConf India 2023

Using ffmpeg.wasm as a muxer

1
...
2
await encoder.flush();
3
encoder.close();
4
5
const videoBuffer = concatBuffers(chunks);
6
7
const ffmpeg = createFFmpeg();
8
await ffmpeg.load();
9
10
ffmpeg.FS('writeFile', 'raw.h264', videoBuffer);
11
ffmpeg.run('-i', 'raw.h264', '-map', '0:v:0', '-c:v', 'copy', 'output.mp4');
12
13
const output = ffmpeg.FS('readFile', 'output.mp4');
14
ffmpeg.FS('unlink', 'raw.h264');
15
ffmpeg.FS('unlink', 'output.mp4');

akashhamirwasia.com

JSConf India 2023

Using ffmpeg.wasm as a muxer

1
...
2
await encoder.flush();
3
encoder.close();
4
5
const videoBuffer = concatBuffers(chunks);
6
7
const ffmpeg = createFFmpeg();
8
await ffmpeg.load();
9
10
ffmpeg.FS('writeFile', 'raw.h264', videoBuffer);
11
ffmpeg.run('-i', 'raw.h264', '-map', '0:v:0', '-c:v', 'copy', 'output.mp4');
12
13
const output = ffmpeg.FS('readFile', 'output.mp4');
14
ffmpeg.FS('unlink', 'raw.h264');
15
ffmpeg.FS('unlink', 'output.mp4');
16
17
const video = new Blob([output.buffer]);
18
download(video);

akashhamirwasia.com

JSConf India 2023

Isn't ffmpeg.wasm heavy?

After all, it's just 24.4 MB
FFmpeg has encoders, decoders, muxers, demuxers, etc.
We are only interested in the muxers (aka libavformat)
It should be possible to isolate and compile libavformat as a WebAssembly binary.
Inspiration: github.com/Yahweasel/libav.js
A great read: medium.com/p/20a06d36ece1