Testing other projects in CI

[gibfahn]

There seem to be lots of new projects coming into the nodejs org, CitGM is an established one, but we also have nodereport and llnode (and maybe nvm at some point).

There are two distinct use-cases.

• Does a Node version (or PR) break module tests for existing modules?
• Does a new module version (or PR) break module tests on existing Node versions?

The first use case is covered by CitGM (for everything that has a package.json with an npm test).

For the second use-case it'd be good to have a standard job people could copy from. I don't think we really need to build node every time we want to test nodereport (for example). Maybe we could test with the latest nightly master build, and then with supported versions (v7, v6, v4).

We also wouldn't need a job for each architecture, 1 job covering all architectures should be fine.

I'd like to work on getting something set up if people agree.

Also (slight tangent), thinking about the test-npm job, once that's stable I guess it should be a standard test job? It would be good to have a generic test job where you could pass different parameters to make rather than test-ci. Not sure if that should be a separate job or an option in test-pull-request.

[jbergstroem]

There's a similar discussion in #448.

[gibfahn]

@jbergstroem True, although I was thinking more about the details of Jenkins job configurations than the higher-level access stuff.

[mhdawson]

I'd agree with @gibfahn . I think #448 would be fore projects not under github/nodejs while this issue to is to address those that are.

I agree not having to build node would be nice. The nightlies and latest supported versions sound like the right answer to me.

I don't think omitting coverage across platforms is a good idea, at least for all cases. In cases like nodereport and llnode there are definitely components that are os specific so a change could break one but not another. We already have all of the platforms in the nightlies and I'd be thinking we'd limit ourselves to the platforms that we produce binaries for on the other releases. This would be all for 6.x and a subset on 4.x and 0.12.x.

[mhdawson]

One other consideration is whether we'd want to run these tests as part of the release validation process. In that case the nightly and last supported release might not be what we want. In that case the option to build node would be useful. Having said that it might be that we normally want to use the nightly/latest but make the job flexible enough so we can build a specific version of node when that is required.

[gibfahn]

@mhdawson Sorry, when I said we wouldn't need 1 job per Arch, I meant you could put all the Archs into 1 multi-config job. Not that we wouldn't test all Archs.

For the release validation process we'd be talking about testing node, in which case (assuming everything can be added to CitGM) a CitGM run would be sufficient.

[Fishrock123]

I think additional projects probably need to not be running jobs very frequently or else we'll end up needing a bunch more hardware? If getting hardware is a problem (seems to be for e.g. OS X)...

[Trott]

Should this be closed at this point?