Streams

In Node.js, streams are objects that enable us to read data from a source or write data to a destination in a continuous manner. They are particularly useful for handling large amounts of data efficiently by processing it piece by piece, rather than loading everything into memory at once. Streams are an essential part of Node.js and provide a powerful way to work with data.

Types of Streams

1. Readable Streams:
   • Definition: Streams from which data can be read.
   • Examples: fs.createReadStream(), HTTP responses, process.stdin.
   • Use Case: Reading a large file from the filesystem.

     const fs = require('fs');
     const readStream = fs.createReadStream('largefile.txt');
     readStream.on('data', (chunk) => {
       console.log(`Received ${chunk.length} bytes of data.`);
     });
     

2. Writable Streams:
   • Definition: Streams to which data can be written.
   • Examples: fs.createWriteStream(), HTTP requests, process.stdout.
   • Use Case: Writing a large file to the filesystem.

     const fs = require('fs');
     const writeStream = fs.createWriteStream('output.txt');
     writeStream.write('Some data to write');
     writeStream.end('Finished writing data');
     

3. Duplex Streams:
   • Definition: Streams that are both readable and writable.
   • Examples: TCP sockets, zlib streams.
   • Use Case: Implementing a network protocol.

     const net = require('net');
     const server = net.createServer((socket) => {
       socket.write('Hello from server');
       socket.on('data', (data) => {
         console.log(`Received: ${data}`);
       });
     });
     server.listen(8080, () => {
       console.log('Server listening on port 8080');
     });
     

4. Transform Streams:
   • Definition: Duplex streams that can modify or transform the data as it is read or written.
   • Examples: zlib.createGzip(), crypto.createCipher().
   • Use Case: Compressing a file using gzip.

     const fs = require('fs');
     const zlib = require('zlib');
     const readStream = fs.createReadStream('input.txt');
     const writeStream = fs.createWriteStream('input.txt.gz');
     const gzip = zlib.createGzip();
     readStream.pipe(gzip).pipe(writeStream);
     

Example Use Case

Reading a Large File and Sending It Over HTTP

Here’s a practical example of using streams to read a large file and send it as a response to an HTTP request:

const http = require('http');
const fs = require('fs');

const server = http.createServer((req, res) => {
  const readStream = fs.createReadStream('largefile.txt');
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  readStream.pipe(res); // Pipes the read stream to the HTTP response
});

server.listen(3000, () => {
  console.log('Server listening on port 3000');
});

In this example:

• Readable Stream: The file stream created by fs.createReadStream reads the large file in chunks.
• Writable Stream: The HTTP response is a writable stream.
• Pipe: The pipe method is used to direct the data from the readable stream to the writable stream.

Advantages of Using Streams

• Efficiency: Streams handle data piece by piece, reducing memory consumption and improving performance.
• Scalability: Streams enable applications to handle large volumes of data and high concurrency more effectively.
• Composability: Streams can be piped together to form a chain of operations, making complex data processing tasks simpler to implement.

Streams are a powerful feature in Node.js, making it easy to handle large files, real-time data, and network communications efficiently.