Pushing the limits of video encoding with WebCodecs
JSNation 2023
👋 Hey there!
I'm Akash Hamirwasia
Writing code @Razorpay. Love building products on the web, trying out new technologies and contributing to open-source.
⚡️ Blaze
🔎 UnTab
🔌 Diode
Slant itLet's connect
akashhamirwasia.comJSNation 2023
Little back story...
- I was building a video editor on the browser to make interesting product videos.
- Everything was going fine until I started writing code for the export process.
- Got stuck at bunch of places... video formats, audio, encoding, muxing, containers, etc.
- Finally figured it out! 🙌
akashhamirwasia.comJSNation 2023
Drawing a video on the web
- Video is a set of image frames changing at a constant rate.
- Each frame can be rendered in a variety of ways:
- CSS Animation
- Modifying DOM directly with JavaScript
- SVGs
- HTML5 Canvas
- The frames rendered on screen can be recorded and saved as a video file.
akashhamirwasia.comJSNation 2023
Record something on screen using JavaScript?
akashhamirwasia.comJSNation 2023
Canvas allows video capture!
Through it's captureStream() API, a part of the MediaStream Recording API.
0123
1const canvas = document.querySelector('canvas');2const record = (duration) => {3const stream = canvas.captureStream(30);4const recorder = new MediaRecorder(stream);5const chunks = [];67recorder.ondataavailable = (event) => chunks.push(event.data);8recorder.onstop = () => {9const videoBlob = new Blob(chunks, { type: 'video/webm' });10download(videoBlob);11};1213recorder.start();14setTimeout(() => recorder.stop(), duration);15}
akashhamirwasia.comJSNation 2023
Sadly, it's not well-suited for this use case
- No constant framerate. canvas.captureStream(framerate)
framerate– maximum framerate in which the video is supposed to be captured. - No option to change video file format.
- Low reliablity.
- Ok quality.
akashhamirwasia.comJSNation 2023
MediaRecording APIs are not designed to be used for precise and high quality video exports
akashhamirwasia.comJSNation 2023
Can we capture and encode each frame of the canvas ourself?
akashhamirwasia.comJSNation 2023
1const images = [];23while(frame < totalFrames) {4seekCanvas(frame);5const image = canvas.toDataURL('image/webp');6images.push(image);7++frame;8}910const video = encode(images);11download(video);
images = []Canvas
akashhamirwasia.comJSNation 2023
1const images = [];23while(frame < totalFrames) {4seekCanvas(frame);5const image = canvas.toDataURL('image/webp');6images.push(image);7++frame;8}910const video = encode(images);11download(video);
images = []Canvas
Frame 1
akashhamirwasia.comJSNation 2023
1const images = [];23while(frame < totalFrames) {4seekCanvas(frame);5const image = canvas.toDataURL('image/webp');6images.push(image);7++frame;8}910const video = encode(images);11download(video);
images = [📸
Frame 1
Frame 1
]Canvas
Frame 1
akashhamirwasia.comJSNation 2023
1const images = [];23while(frame < totalFrames) {4seekCanvas(frame);5const image = canvas.toDataURL('image/webp');6images.push(image);7++frame;8}910const video = encode(images);11download(video);
images = [📸
Frame 1
Frame 1
]Canvas
Frame 2
akashhamirwasia.comJSNation 2023
1const images = [];23while(frame < totalFrames) {4seekCanvas(frame);5const image = canvas.toDataURL('image/webp');6images.push(image);7++frame;8}910const video = encode(images);11download(video);
images = [📸
Frame 1
Frame 1
📸
Frame 2
Frame 2
]Canvas
Frame 2
akashhamirwasia.comJSNation 2023
1const images = [];23while(frame < totalFrames) {4seekCanvas(frame);5const image = canvas.toDataURL('image/webp');6images.push(image);7++frame;8}910const video = encode(images);11download(video);
images = [📸
Frame 1
Frame 1
📸
Frame 2
Frame 2
...]Canvas
Frame ...
akashhamirwasia.comJSNation 2023
1const images = [];23while(frame < totalFrames) {4seekCanvas(frame);5const image = canvas.toDataURL('image/webp');6images.push(image);7++frame;8}910const video = encode(images);11download(video);
How to encode an array of images to a video?
images = [📸
Frame 1
Frame 1
📸
Frame 2
Frame 2
...]Canvas
Frame ...
akashhamirwasia.comJSNation 2023
Whammy.js
- A realtime JavaScript WebM Encoder.
- Appeared on Hacker News 11 years ago!
- Takes snapshots of canvas as images and encodes them to a video file.
function encode(images) {return window.Whammy.fromImageArray(images, 30);}
akashhamirwasia.comJSNation 2023
Pros & Cons of these JS libraries
✅ High quality output
✅ Constant framerate
✅ Relatively easy
❌ Very slow
❌ Synchronous
❌ No hardware-acceleration
❌ No choice of video formats
❌ No support for adding audio
akashhamirwasia.comJSNation 2023
Now what?
WebCodecs
akashhamirwasia.comJSNation 2023
What is WebCodecs?
- Set of APIs that give low-level access to individual frames of a video stream.
- Supports encoding / decoding of each frame in various formats.
- It's asynchronous and hardware accelerated!
- It's more than just video, supports audio too!
akashhamirwasia.comJSNation 2023
Video encoding process
akashhamirwasia.comJSNation 2023
Video encoding process
akashhamirwasia.comJSNation 2023
Video encoding process
akashhamirwasia.comJSNation 2023
Let's look at the code to encode videos using WebCodecs
akashhamirwasia.comJSNation 2023
Creating a VideoEncoder
01
1const encoder = new VideoEncoder({2output: (chunk, metadata) => {3// Do something with encoded chunk4},5error: (err) => {6// Handle error7}8});
akashhamirwasia.comJSNation 2023
Creating a VideoEncoder
01
1const chunks = [];2const encoder = new VideoEncoder({3output: (chunk, metadata) => {4chunks.push(chunk);5},6error: (err) => {7console.log(err);8}9});
akashhamirwasia.comJSNation 2023
Configuring the VideoEncoder
01234
1const encoder = new VideoEncoder(...);23encoder.configure({4codec: 'vp8',5hardwareAcceleration: 'prefer-hardware',6framerate: 30,7latencyMode: 'quality',8width: 1920,9height: 1080,10bitrate: 2_000_000,11...12});
akashhamirwasia.comJSNation 2023
How to find the right codec?
- It's not as easy as .webm, .mp4, .avi, etc.
- They are seemingly random strings!
Eg. vp8, avc1.420034, av01.0.12M.10
video codec != video file format
akashhamirwasia.comJSNation 2023
Video Encoding 101
akashhamirwasia.comJSNation 2023
Individual frames
Frame 1
Frame 2
Frame 3
Encoding algorithm
Defines how to convert, compress and store frames of the video
Video stream
Frame 1
Frame 2
Frame 3
akashhamirwasia.comJSNation 2023
Video stream
Frame 1
Frame 2
Frame 3
Muxer
Video container
video.mp4
Video stream
akashhamirwasia.comJSNation 2023
Video stream
Audio stream
Subtitles
"Muxer"
Combines(multiplexes) multiple encoded streams to a single container
Video container
video.mp4
Video stream
Audio stream
Subtitles
akashhamirwasia.comJSNation 2023
What's a codec again?
- Encoding algorithm to convert, compress the individual frames of the video.
- A codec and container must be compatible with each other.
| container | codec | example |
|---|---|---|
| WebM | VP8, AV1 | vp8 |
| MP4 | AVC, AV1 | avc1.420034, av01.0.12M.10 |
- Every browser supports different set of codecs.
akashhamirwasia.comJSConf India 2023
Back to code
0
1const chunks = [];2const encoder = new VideoEncoder(...);3const FPS = 30;45encoder.configure({6codec: 'avc1.420034',7avc: { format: 'annexb' },8hardwareAcceleration: 'prefer-hardware',9framerate: FPS,10latencyMode: 'quality',11width: 1920,12height: 1080,13bitrate: 2_000_000,14...15});
akashhamirwasia.comJSConf India 2023
Back to code
012
1const chunks = [];2const encoder = new VideoEncoder(...);3const FPS = 30;4encoder.configure({ ... });56while(frame < totalFrames) {7seekCanvas(frame);89const videoFrame = new VideoFrame(canvas, {10timestamp: frame / FPS * 1_000_000,11});12encoder.encode(videoFrame);13videoFrame.close();1415++frame;16}
akashhamirwasia.comJSConf India 2023
Back to code
012
1const chunks = [];2const encoder = new VideoEncoder(...);3const FPS = 30;4encoder.configure({ ... });56while(frame < totalFrames) {7...8}910await encoder.flush();11encoder.close();1213const videoBuffer = concatBuffers(chunks);
download
videoBuffer?akashhamirwasia.comJSNation 2023
Recall the video encoding process
Individual frames
Frame 1
Frame 2
Frame 3
codec
Video stream
Muxer
Video container
video.mp4
Handled by WebCodecs
How do we do this?
akashhamirwasia.comJSConf India 2023
Using ffmpeg.wasm as a muxer
1...2await encoder.flush();3encoder.close();45const videoBuffer = concatBuffers(chunks);67const ffmpeg = createFFmpeg();8await ffmpeg.load();
akashhamirwasia.comJSConf India 2023
Using ffmpeg.wasm as a muxer
1...2await encoder.flush();3encoder.close();45const videoBuffer = concatBuffers(chunks);67const ffmpeg = createFFmpeg();8await ffmpeg.load();910ffmpeg.FS('writeFile', 'raw.h264', videoBuffer);11ffmpeg.run('-i', 'raw.h264', '-map', '0:v:0', '-c:v', 'copy', 'output.mp4');
akashhamirwasia.comJSConf India 2023
Using ffmpeg.wasm as a muxer
1...2await encoder.flush();3encoder.close();45const videoBuffer = concatBuffers(chunks);67const ffmpeg = createFFmpeg();8await ffmpeg.load();910ffmpeg.FS('writeFile', 'raw.h264', videoBuffer);11ffmpeg.run('-i', 'raw.h264', '-map', '0:v:0', '-c:v', 'copy', 'output.mp4');1213const output = ffmpeg.FS('readFile', 'output.mp4');14ffmpeg.FS('unlink', 'raw.h264');15ffmpeg.FS('unlink', 'output.mp4');
akashhamirwasia.comJSConf India 2023
Using ffmpeg.wasm as a muxer
1...2await encoder.flush();3encoder.close();45const videoBuffer = concatBuffers(chunks);67const ffmpeg = createFFmpeg();8await ffmpeg.load();910ffmpeg.FS('writeFile', 'raw.h264', videoBuffer);11ffmpeg.run('-i', 'raw.h264', '-map', '0:v:0', '-c:v', 'copy', 'output.mp4');1213const output = ffmpeg.FS('readFile', 'output.mp4');14ffmpeg.FS('unlink', 'raw.h264');15ffmpeg.FS('unlink', 'output.mp4');1617const video = new Blob([output.buffer]);18download(video);
akashhamirwasia.comJSNation 2023
And that's how you encode a video in the browser!
✅ High quality output
✅ Constant framerate
✅ Fast
✅ Asynchronous
✅ Hardware accelerated
✅ Use any video format
✅ Support for adding audio
❌ Relatively hard
✅ Now easy! 🙌
akashhamirwasia.comJSNation 2023
Speaking of browser support...
WebCodecs
WebAssembly
Supported in all browsers 🎉
Remember I was building a tool to make product videos?
Make catchy product videos in just few minutes
Thank you! 💛
Let's connect
Pushing the limits of video encoding with WebCodecs JSNation 2023