Bloggy.

NodeJS streams and its applications

date

Aug 07, 2024.

Why do we need node.js streams?

When dealing with large files from the server to our clients, efficiently handling large amounts of data is a key requirement. Node.js streams offer a powerful way to process data piece by piece, making them an ideal choice for video streaming applications, file upload and download services, Real time data analysis of large datasets. In this blog, we'll explore how Node.js streams work and build a video streaming application that showcases their potential.

What are streams?

Imagine you want to binge-watch on netflix and for a high resolution video, an episode in a series is 2GB in size. You don't necessarily have to download to disk right.

So, let's assume the netflix server sends the video file to your device, the whole 2GB!.

If it was never downloaded, then it goes straight to your memory(RAM) and depending on your device, It could slow down you device or literally fry it.

Stream allows us to send the video file in batches(chunks). Node.js streams are objects that allow you to read or write data in chunks, rather than loading it all into memory at once. Streams are especially useful for processing large files like videos, where reading or writing the entire file at once could lead to performance issues. With streams, you can process data efficiently and smoothly.

Categories of node.js streams.

Streams in Node.js are categorized into four types: Readable, Writable, Duplex, and Transform streams. Each serves a unique purpose. [[Readable streams allow data to be read in chunks., Writable streams allow data to be written in chunks., Duplex streams combine the functionalities of both readable and writable streams., Transform streams apply modifications to data as it's read or written.]]

Applying streams in solving a problem

Imagine a video streaming application that will rely heavily on Node.js readable and writable streams to handle large video files efficiently. The core idea is to stream the video data directly to the user’s device in chunks, without overwhelming the server or the user’s memory.

javascript

const fs = require('fs');
const http = require('http');

const server = http.createServer((req, res) => {
    const videoPath = './videos/sample.mp4';
    const stat = fs.statSync(videoPath);
    const fileSize = stat.size;
    const range = req.headers.range;

    if (range) {
        const parts = range.replace(/bytes=/, '').split('-');
        const start = parseInt(parts[0], 10);
        const end = parts[1] ? parseInt(parts[1], 10) : fileSize - 1;

        const chunkSize = end - start + 1;
        const file = fs.createReadStream(videoPath, { start, end });
        res.writeHead(206, {
            'Content-Range': `bytes ${start}-${end}/${fileSize}`,
            'Accept-Ranges': 'bytes',
            'Content-Length': chunkSize,
            'Content-Type': 'video/mp4'
        });
        file.pipe(res);
    } else {
        res.writeHead(200, { 'Content-Length': fileSize, 'Content-Type': 'video/mp4' });
        fs.createReadStream(videoPath).pipe(res);
    }
});

server.listen(8000, () => console.log('Server running on port 8000'));

Read streams

The above code forms the foundation of our video streaming application. Here, we create a simple HTTP server that listens for requests to stream video files. Using the Node.js `fs` module, we open a readable stream from the video file and pipe it to the response. This allows us to serve video content efficiently without loading the entire file into memory.

Range requests

One of the key features of streaming is handling range requests. Range requests allow clients to request specific portions of a video file rather than downloading the entire file at once. This is particularly useful for large video files, where users may want to skip ahead or resume watching from a specific point.

javascript

const range = req.headers.range;
if (range) {
    const parts = range.replace(/bytes=/, '').split('-');
    const start = parseInt(parts[0], 10);
    const end = parts[1] ? parseInt(parts[1], 10) : fileSize - 1;
    const chunkSize = end - start + 1;
    const file = fs.createReadStream(videoPath, { start, end });
    res.writeHead(206, {
        'Content-Range': `bytes ${start}-${end}/${fileSize}`,
        'Accept-Ranges': 'bytes',
        'Content-Length': chunkSize,
        'Content-Type': 'video/mp4'
    });
    file.pipe(res);
}

With range requests, we enable our video streaming server to deliver specific parts of the video file based on the client's request. This is particularly useful for skipping through video content and ensures that users don’t have to download unnecessary data.

Writable streams

Writable streams in Node.js allow us to send data in chunks to an output destination, like writing the streamed video data to a file. This is useful for recording or archiving video streams on the server side.

javascript

const writable = fs.createWriteStream('./output/recorded_video.mp4');

fs.createReadStream('./videos/sample.mp4').pipe(writable);

writable.on('finish', () => {
    console.log('Video has been recorded!');
});

In this example, we are using a writable stream to record the video stream into a new file. As the video data is read in chunks, it is also written in chunks to the output file. This allows us to handle large video files efficiently without running into memory issues.

Duplex streams

Duplex streams in Node.js combine both readable and writable streams. This means they can read and write data simultaneously, which is useful in situations where you need to process data as it is being read, such as compressing video files while streaming.

In our video streaming application, we can use a Duplex stream to compress video files on the fly as they are being streamed to the client. ((https://i.imgur.com/xKqr0Yc.png,Duplex streams allow you set your video quality on the go.)) This not only reduces the bandwidth required but also ensures that users with slower connections can still access the video content smoothly.

javascript

const zlib = require('zlib');
const duplex = zlib.createGzip();

fs.createReadStream('./videos/sample.mp4').pipe(duplex).pipe(fs.createWriteStream('./output/compressed_video.gz'));

In this example, we are using a Duplex stream to compress the video file as it is being read from the disk. The compressed video file is then written to a new file using a writable stream. This approach helps save disk space and bandwidth when streaming large video files.

Transform streams

Transform streams are special types of Duplex streams that modify the data as it is being read or written. In a video streaming application, you might use a Transform stream to apply video effects, such as adjusting the brightness or contrast of the video in real-time.

By using a Transform stream, you can apply these video effects without having to load the entire video into memory. Instead, you process the video data in chunks as it is being streamed, ensuring smooth performance even with large files.

javascript

const { Transform } = require('stream');

class VideoTransform extends Transform {
    _transform(chunk, encoding, callback) {
        // Apply transformation to video data
        this.push(chunk); // For simplicity, we are not modifying the data
        callback();
    }
}

const videoTransform = new VideoTransform();

fs.createReadStream('./videos/sample.mp4').pipe(videoTransform).pipe(fs.createWriteStream('./output/transformed_video.mp4'));

In this example, we are using a custom Transform stream to process the video data. The transformation could include various operations, such as adding filters or effects to the video. The processed video is then written to a new file, ready for streaming to the client.

Error Handling

Now that we've covered the different types of streams and their applications, let's discuss how to handle errors during video streaming. Streams can emit errors during the read, write, or transformation process, so it's important to handle these errors gracefully to ensure the smooth functioning of your application.

javascript

fs.createReadStream('./videos/sample.mp4')
  .on('error', (err) => console.error('Read Error:', err))
  .pipe(res)
  .on('error', (err) => console.error('Write Error:', err));

Proper error handling ensures that your video streaming application can recover from unexpected issues, such as a missing video file or a network failure. By listening to the 'error' event on streams, you can catch these errors and respond accordingly, perhaps by sending an error message to the client or retrying the operation.

Conclusion

In conclusion, Node.js streams provide a powerful and efficient way to handle video streaming in web applications. By using readable, writable, duplex, and transform streams, you can build a scalable video streaming service that processes large files smoothly and efficiently, without overwhelming the server or client resources.

Incorporating streams into your Node.js applications opens up a world of possibilities, from live video streaming to real-time video processing. By mastering streams, you can build more responsive and scalable applications that offer better performance and user experience.