Proposal: Making performance team a working group

[anonrig]

When I first proposed the performance team, I needed clarification about the attendance and the impact to the Node.js project. The past 3 months proved that people are interested in joining and contributing to the Node.js project and improving its performance. We already had 5 meetings, with an average of ~7 members per meeting. Referencing: https://github.com/nodejs/performance/tree/main/meetings

I propose making the performance team a working group. I don't know what the next steps are but happy to make it official.

cc @nodejs/performance

[Trott]

What's the reason for wanting to make it a working group? What would that make it possible to do that you can't do as a team?

In my view at least, it's best to be a team if you can. There's less overhead. Once you're a working group, there are charters and delegated responsibilities etc., and most of the time, no real benefit. I've kind of been hoping that the whole "working group" vs. "team" distinction would fade out over time.

[anonrig]

I'm okay with keeping the performance team as it is, but to keep things going, the following concerns need to be resolved:

1. There needs to be a clear goal regarding Node.js' performance. The only WG mentioning performance is Build WG with Running performance testing and comparisons.
2. TSC's strategic initiatives and performance team goals align from time to time. For example, the fantastic work done on startup performance by @joyeecheung. Keeping track of these changes in a single place would be beneficial for outside collaborators and companies focused on tracking the work done by the Node.js performance team.
3. Setting clear goals for the performance team will allow us to take more responsibility and set a course for planning the long-term bottlenecks of Node.js, which no other WG is primarily focusing on.
4. To resolve the conflicts with the Building Working group responsibilities, including: Running performance testing and comparisons.
5. [If needed and if there are volunteers:] taking responsibility for the benchmark CI and updating it regularly to keep up to date with more realistic benchmark results. More context: Current performance CI is pretty old and does not support AVX instructions (and is 6 years old - Dedicated machine for benchmarks build#791) - Crossing it due to Proposal: Making performance team a working group #1343 (comment)
6. If number 5 is not a maintainable solution, then I recommend increasing the priority of benchmark CI in the @nodejs/build wg.
7. Giving the option to have permanent access to benchmark CI or at least offering them CI credits through the sponsors of Node.js or through OpenJSF to performance team members to be used in local benchmarking (referencing: Request access to benchmark CI build#3118)

[anonrig]

It would be beneficial to ping @nodejs/build to share and receive feedback about my concerns regarding benchmarks, performance, and benchmark CI machine.

[richardlau]

5. [If needed and if there are volunteers:] taking responsibility for the benchmark CI and updating it regularly to keep up to date with more realistic benchmark results. More context: Current performance CI is pretty old and does not support AVX instructions (and is 6 years old - [Dedicated machine for benchmarks build#791](https://github.com/nodejs/build/issues/791))

Lack of AVX instructions is a hardware issue, isn't it? Our two benchmark machines are physical machines that were donated by Intel.

It would be beneficial to ping @nodejs/build to share and receive feedback about my concerns regarding benchmarks, performance, and benchmark CI machine.

The important thing to keep in mind re. build resources is that almost all of it has been donated to the Node.js project, either in the form of physical hardware (such as the two existing benchmark machines and several of the now retired Raspberry Pi's) or resources on cloud providers such as Digital Ocean. For the latter we do not have infinite capacity to spin up servers -- in some cases we have a set amount of credits (and for those we're pretty much at capacity).

[anonrig]

For the latter we do not have infinite capacity to spin up servers -- in some cases we have a set amount of credits (and for those we're pretty much at capacity).

Thanks @richardlau. I can take this to OpenJSF if it would help? (I'm a member of cross-project council)

[joyeecheung]

I share the same concerns as @Trott, and I was glad that we didn't make the automation team a working group so that we didn't have the overhead of dechartering it. There was a lot of momentum in the beginning and we did produce the node-core-utils out of it which is still useful today, but the momentum eventually faded away, which is natural, and what we have left is node-core-utils which we still maintain because we actively use it all the time. I think this was basically the same thing with the old benchmarking WG - there was a lot of momentum, we had benchmarking.nodejs.org that tracked the performance (which I still miss from time to time), and we got a benchmark CI that's still actively used today, but the momentum eventually faded, which is also normal. I don't think there is really any special benefit of making a group a proper WG, and I think most of the things mentioned in #1343 (comment) can also be solved by having a team that has its own repo already? What really matters in the momentum is just the people, not whether this group is a team or a WG - a WG can still be in limbo when the momentum fades away.

[richardlau]

For the latter we do not have infinite capacity to spin up servers -- in some cases we have a set amount of credits (and for those we're pretty much at capacity).

Thanks @richardlau. I can take this to OpenJSF if it would help? (I'm a member of cross-project council)

My suggestion is that you first need to be more specific in what is being asked for. If it's extra hardware/cloud resources, what exactly is the performance team after -- that will help estimate costs and potential options. Those proposals should then be given to the build and/or TSC -- we may still go back to the OpenJS Foundation but we would want such requests to be generally agreed if it's on behalf of the Node.js project.

[anonrig]

My suggestion is that you first need to be more specific in what is being asked for.

That is an excellent suggestion and is the right way of moving forward.

[mcollina]

The fundamental reason to have a Working Group is to have some responsibility delegated from the TSC to that group. I don't see what could be delegated here.

All long term/successful working groups exists because they perform an important function for the project and they have the autonomy to get it done, independently of their meeting schedule and attendance. (As an example, we don't hold regular meetings for undici - yet we got a lot done).

At this point I don't see performance as a working group.

[mhdawson]

Having been involved in both working groups and teams I'll echo what people have shared already.

In terms of day-to-day getting work done there is no difference. There is also no effective difference in getting help/resources. That all comes down to getting other volunteers to spend their time to help move your ask forward.

Only in cases where you need different level of access.rights for different members of the team, then some goverance may be needed and the WG may provide a way to document/capture that.

[mcollina]

Given the recent spark of interest in the topic, I think we should make performance a strategic initiative instead.

What would you all think?

[anonrig]

Thank you all for your responses and feedback. I'm all in for making performance a strategic initiative.

[joyeecheung]

Making it a strategic initiative makes sense to me (more than making it a WG). What about the startup performance strategic initiative though? Do we just merge them, or make the startup performance strategic initiative more specific, like just startup snapshot? (I think that's mostly what's left at this point, and user-land startup snapshots is more like a feature on its own than performance improvements in core).

[mhdawson]

I thinking leaving them as separate initiatives is fine as you mention user-land snapshorts is a more specific feature.

[BridgeAR]

I think it makes sense to keep the two initiatives separate.

[mcollina]

@anonrig usually the strategic initiatives have an end goal. what would be the "end goal" of the performance one?

[anonrig]

@anonrig usually the strategic initiatives have an end goal. what would be the "end goal" of the performance one?

I don't have a clear answer to this. To the best of my ability, I can say: "Be within the range of execution speeds for fundamental APIs compared to the ecosystem and the alternative runtimes"

[mhdawson]

@mcollina if you meant an "end" as a key part of the end goal, one of our strategic initiatives is keeping up to date on v8. So clearly has an important goal, but not necessarily and "end". I think it has been useful having that in our strategic initiatives as a way to highlight when there are issues/help is needed.

[mhdawson]

I think there is agreement to make the performance work a strategic initiative and the next step is just a PR to do that. @anonrig would you like to do that or would you like me to create one?

[anonrig]

@mhdawson I opened a PR to the Node.js repository. Is this the only place we mention the strategic initiatives?

[anonrig]

I'd like to thank all the TSC members for making me a champion of this initiative.

[mhdawson]

@anonrig yes, that looks good.

[mhdawson]

@anonrig you can post updates on the issue opened for the TSC meetings as we try to go through initiatives in each meeting to share progress or identify if there are any blockers. For cases were you want more input/discussion you can also ask to be invited to a TSC meeting for that discussion.