Significant memory usage increase starting from 20.10.0 for a simple HTTP server (potentially stream related)

[poolik]

Version

v20.10.0

Platform

Darwin arm64

Subsystem

stream

What steps will reproduce the bug?

Simple HTTP server that returns a ~1mb JSON file and reports memory usage every 5s. Starting with node 20.10.0 memory consumption jumps significantly when servicing requests (see below).

const http = require('http');
const process = require('process');
const port = 3000;

// Use require to load the JSON data once at startup
const jsonData = require('./data.json');

setInterval(() => {
    const memoryUsage = process.memoryUsage();
    console.log(`Memory Usage: 
        RSS: ${memoryUsage.rss / 1024 / 1024}MB,
        Heap Total: ${memoryUsage.heapTotal / 1024 / 1024}MB,
        Heap Used: ${memoryUsage.heapUsed / 1024 / 1024}MB,
        External: ${memoryUsage.external / 1024 / 1024}MB,
        Array Buffers: ${memoryUsage.arrayBuffers / 1024 / 1024}MB`);
}, 5000); // Log memory usage every 5 seconds

http.createServer((req, res) => {
  // Check for a GET request to the root path
  if (req.method === 'GET') {
    // Set the content type to application/json
    res.setHeader('Content-Type', 'application/json');
    res.writeHead(200);

    // Use JSON.stringify to convert the JSON object to a string
    res.end(JSON.stringify(jsonData));
  } else {
    // Handle any requests that are not to the root path or not GET requests
    res.writeHead(404);
    res.end('Not Found');
  }
}).listen(port, () => {
  console.log(`Server running at http://localhost:${port}/`);
});

The random generated JSON I used for this test:
data.json

The k6 load test script (named load_test.js) that sends requests to show the increase (run with k6 run load_test.js):

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  // A number specifying the number of VUs to run concurrently.
  vus: 333,
  // A string specifying the total duration of the test run.
  duration: '900s',
};

export default function() {
  const res = http.get('http://localhost:3000/');
  sleep(1);
}

How often does it reproduce? Is there a required condition?

Reproduces always.

What is the expected behavior? Why is that the expected behavior?

Using node 20.9.0 (last good version) I see the following memory usage reported:

Memory Usage:
        RSS: 122.34375MB,
        Heap Total: 50.34375MB,
        Heap Used: 20.336807250976562MB,
        External: 1.5704345703125MB,
        Array Buffers: 0.009982109069824219MB
Memory Usage:
        RSS: 170.53125MB,
        Heap Total: 97.90625MB,
        Heap Used: 71.03762817382812MB,
        External: 1.5704345703125MB,
        Array Buffers: 0.009982109069824219MB
Memory Usage:
        RSS: 139.125MB,
        Heap Total: 68.015625MB,
        Heap Used: 40.76661682128906MB,
        External: 1.5704345703125MB,
        Array Buffers: 0.009982109069824219MB
Memory Usage:
        RSS: 126.8125MB,
        Heap Total: 52.875MB,
        Heap Used: 23.89117431640625MB,
        External: 1.5704345703125MB,
        Array Buffers: 0.009982109069824219MB
Memory Usage:
        RSS: 129.109375MB,
        Heap Total: 56.578125MB,
        Heap Used: 33.01460266113281MB,
        External: 1.5704345703125MB,
        Array Buffers: 0.009982109069824219MB
Memory Usage:
        RSS: 156.171875MB,
        Heap Total: 83.65625MB,
        Heap Used: 55.899253845214844MB,
        External: 1.5704345703125MB,
        Array Buffers: 0.009982109069824219MB

What do you see instead?

Starting from node 20.10.0 memory usage jumps considerably:

Memory Usage:
        RSS: 453MB,
        Heap Total: 401.453125MB,
        Heap Used: 374.45359802246094MB,
        External: 1.6024627685546875MB,
        Array Buffers: 0.009982109069824219MB
Memory Usage:
        RSS: 466.4375MB,
        Heap Total: 416.8125MB,
        Heap Used: 386.71961975097656MB,
        External: 1.6024627685546875MB,
        Array Buffers: 0.009982109069824219MB
Memory Usage:
        RSS: 986.40625MB,
        Heap Total: 936.75MB,
        Heap Used: 894.882453918457MB,
        External: 1.6024627685546875MB,
        Array Buffers: 0.009982109069824219MB
Memory Usage:
        RSS: 992.125MB,
        Heap Total: 941.171875MB,
        Heap Used: 900.2207260131836MB,
        External: 1.6024627685546875MB,
        Array Buffers: 0.009982109069824219MB
Memory Usage:
        RSS: 1316.484375MB,
        Heap Total: 1266MB,
        Heap Used: 1222.880615234375MB,
        External: 1.6024627685546875MB,
        Array Buffers: 0.009982109069824219MB

Additional information

We discovered this jump in memory consumption when we upgrade one of our microservices in production. Service works fine with 20.9.0, but starting with 20.10.0 started going over the allotted K8 memory limits.

Taking a look at the heap snapshot and comparing between old and good I saw that 20.10.0 had a lot of string objects in heap that were the unsent responses. Screenshot from the chrome debugger:

Looks like 20.10.0 started to buffer responses a bit longer which under a normal request load manifested as increased memory consumption for the server.

After briefly going over the changelog for 20.10.0 I suspect it might be related to this change - #50014 (but haven't verified and I don't know internals of nodejs that well).

[orgads]

@poolik #50014 is unrelated to this. I bisected to:

• stream: writable state bitmap #49899

[orgads]

PR ready:

• stream: fix memory usage regression in writable #53188