Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert iteration based STF load test playlist targets to time-based #2203

Merged
merged 1 commit into from
Jan 29, 2021
Merged

Convert iteration based STF load test playlist targets to time-based #2203

merged 1 commit into from
Jan 29, 2021

Conversation

Mesbah-Alam
Copy link
Contributor

@Mesbah-Alam Mesbah-Alam commented Jan 24, 2021 •
edited
Loading

Note 1: The following System tests have not been converted to time-based in this pass, as they are not really load tests:

  1. LockingLoadTest (Not really a load test; it runs a minimix load and runs lock contention tests).
  2. HCRLateAttachWorkload (Not a load test It launches process 'A' with a mini-mix load and then runs process 'B' that attaches an agent to 'A' and runs various unit tests repeatedly).
  3. JDITest (Not a load test; it starts and monitors various relevant attaching or listening processes).
  4. All tests under jlm (Not really load tests. They do stuff like launching sub-process A with a load, while launching sub-process B that issues JLM based commands to test JLM API).
  5. All tests under sharedClasses (Not really load tests. These tests run a mini-mix load in a sub-process and runs various sharedclasses chache verification tests, softmx tests, etc).
  6. All tests under modularity (not really load tests; these tests are mostly for unit-testing various modularity features).

Note 2: This PR contains deletion of the old MixedLoadTest target, as we now have the time-based equivalent of this target (e.g. MiniMix_5m) that was added in an earlier PR.

FYI @pshipton

Signed-off-by: [email protected] [email protected]

@Mesbah-Alam
Copy link
Contributor Author

Tests:

@Mesbah-Alam Mesbah-Alam self-assigned this Jan 24, 2021
@Mesbah-Alam
Copy link
Contributor Author

This PR is to be delivered After the following 3 PRs have been merged:

@Mesbah-Alam Mesbah-Alam changed the title Convert iteration based STF load tests to time-based tests Convert iteration based STF load test playlist targets to time-based Jan 24, 2021
@llxia
Copy link
Contributor

llxia commented Jan 24, 2021

We have the path limitation issue on windows. Does it make sense to assume the default is 5mins and avoid adding _5min into every test name?

@smlambert
Copy link
Contributor

re: #2203 (comment) when someone wants to add a fleet of test targets for 30min or 2hrs you won't have a symmetric naming strategy (which may be okay, but then automation to parse names for some other purpose must know the default case). Just want to mention it now, as I expect to see some longer running targets showing up in the special level at some point.

@Mesbah-Alam
Copy link
Contributor Author

I agree with Shelley, we would most probably be introducing some other time variants of these tests in future.

About the path limit, please note that this PR does contain a few test name updates with that in mind as well - I've converted "OpenJ9" to "J9", "Hotspot" to "HS", ConcurrentScavenge" to "CS", etc. in test names in this PR.

@llxia
Copy link
Contributor

llxia commented Jan 24, 2021 •
edited
Loading

Thanks @ShelleyLambert . That is a good point. We may want to use other programs to parse and triage it.
In this case, instead of adding the time in the middle of the test name, can we have it at the end and make it shorter? For example, DaaLoadTest_daa1_5m, DaaLoadTest_daa1_2h, DaaLoadTest_daa1_30m

@Mesbah-Alam Mesbah-Alam force-pushed the add-timebased-tests branch 2 times, most recently from 4218529 to 3073f42 Compare January 24, 2021 18:52
@Mesbah-Alam
Copy link
Contributor Author

can we have it at the end and make it shorter?

That's a great suggestion! Done!

@Mesbah-Alam Mesbah-Alam marked this pull request as ready for review January 25, 2021 06:32
llxia
llxia reviewed Jan 25, 2021
View reviewed changes
system/otherLoadTest/playlist.xml
@@ -314,24 +314,4 @@
</impls>
<platformRequirements>bits.64,^arch.arm</platformRequirements>
</test>
<!-- Temporarily excluded from z/Os due to : jdk11-zos/issues/342-->
<test>
<testCaseName>MixedLoadTest</testCaseName>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MixedLoadTest is removed. Is this intended?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, That target was redundant since we are now running the time-based version of the same test.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to update the issue as the test is replaced by MiniMix_5m.

Copy link
Contributor Author

@Mesbah-Alam Mesbah-Alam Jan 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of "replacement", we can look at it as as "conversion to time-based equivalent".

So, Just like many other load tests are being converted to their 5m equivalents in this PR, so has MixedLoadTest. The only difference being, for MixedLoadTest, conversion to its time-based equivalent(e.g. MiniMix_5m) happened in an earlier PR. This PR only contains the deletion of the old MixedLoadTest target. I will add this explanation in the description of this PR.

@smlambert smlambert mentioned this pull request Jan 26, 2021
Closed
@llxia
Copy link
Contributor

llxia commented Jan 27, 2021

From the Grinder result, this PR will increase the system tests execution time. For example, https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_sanity.system_x86-64_linux/ takes ~2hrs and the above Grinder took ~3hrs.

We will monitor the test builds and tune the tests if needed.

@pshipton
Copy link
Contributor

We could also increase parallelization from 2 to 3.

llxia
llxia approved these changes Jan 28, 2021
View reviewed changes
Copy link
Contributor

@llxia llxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@llxia
Copy link
Contributor

llxia commented Jan 28, 2021

For hotspot testing, please see adoptium/aqa-systemtest#396 (comment)

@karianna karianna added this to the January 2021 milestone Jan 29, 2021
smlambert
smlambert approved these changes Jan 29, 2021
View reviewed changes
Copy link
Contributor

@smlambert smlambert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks @Mesbah-Alam !

@smlambert smlambert merged commit b3664d8 into adoptium:master Jan 29, 2021
@smlambert smlambert mentioned this pull request Feb 10, 2021
Closed
patkarns added a commit to patkarns/openjdk-tests that referenced this pull request Mar 4, 2021
@patkarns @LongyuZhang
@Mesbah-Alam @github-actions @smlambert @sxa @babsingh @pshipton @KKES335 @llxia @lumpfish @JasonFengJ9 @AdamBrousseau @renfeiw @knn-k @Icohedron @xichen1 @pxc-ddd @adamfarley @yanshan24 @xfxcwynlc @sophia-guo
Sync original repo (#5)
fca88f9 
* Add add-exports to HCRLateAttachWorkload_previewEnabled (adoptium#2208)

- Add add-exports to HCRLateAttachWorkload_previewEnabled

Signed-off-by: Longyu Zhang <[email protected]>

* Convert iteration based STF based load tests to time-based tests (adoptium#2203)

Signed-off-by: [email protected] <[email protected]>

* AUTO: auto exclude test jdk_foreign_native impl=openj9 ver=16+ (adoptium#2219)

- related: adoptium#2218 (comment)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: smlambert <[email protected]>

* get.sh: stop mandating GNU tar --strip when extracting debug images (adoptium#2184)

Signed-off-by: Stewart X Addison <[email protected]>

* Remove references to eclipse-openj9/openj9#11135 (adoptium#2215)

eclipse-openj9/openj9#11135 refers to an uber issue, which will be closed soon.
So, the references to eclipse-openj9/openj9#11135 have been replaced with
stand-alone trackers for the respective issues.

Also, a duplicate for SuperMethodTest.java has been removed.

Signed-off-by: Babneet Singh <[email protected]>

* Exclude ChaCha20/ChaCha20Poly1305ParamTest.java (adoptium#2221)

Issue eclipse-openj9/openj9#11689

Signed-off-by: Peter Shipton <[email protected]>

* Update OpenJ9 JDK17 exclude list (adoptium#2217)

* Apply adoptium#2188 to OpenJ9 JDK17 exclude list

Apply adoptium#2188 to OpenJ9 JDK17 exclude list.

Signed-off-by: Babneet Singh <[email protected]>

* Apply adoptium#2192 to OpenJ9 JDK17 exclude list

Apply adoptium#2192 to OpenJ9 JDK17 exclude list.

Signed-off-by: Babneet Singh <[email protected]>

* Apply adoptium#2215 to OpenJ9 JDK17 exclude list

Apply adoptium#2215 to OpenJ9 JDK17 exclude list.

Signed-off-by: Babneet Singh <[email protected]>

* Fix charsets,i18n,language_tag and locale_matching for 16 and Create new 4 test cases (adoptium#2214)

Co-authored-by: Kentaro Kuramochi <[email protected]>

* Allow users to set lightweight option (adoptium#2216)

The benefit of having lightweight checkout is to save space on Jenkins master.
However, Jenkins does not support using Repository URL and Branch build parameters with lightweight checkout together (see https://issues.jenkins.io/browse/JENKINS-48431).

In this PR, we set LIGHT_WEIGHT_CEHCKOUT to false and use Repository URL and Branch as build parameters by default. If LIGHT_WEIGHT_CEHCKOUT is set to true, set Repository URL and Branch to its explicit value

Signed-off-by: lanxia <[email protected]>

* Create ParallelStreamsLoadTest (adoptium#2229)

Signed-off-by: Simon Rushton <[email protected]>

* Remove _5m from ParallelStreamsLoadTest targets (adoptium#2231)

Signed-off-by: Simon Rushton <[email protected]>

* Enable java/lang/reflect/DefaultMethodMembers/FilterNotMostSpecific.java (adoptium#2230)

This was forgot after resolving openj9 issue #7623

Signed-off-by: Jason Feng <[email protected]>

* Merge openj9_custom into jdk_custom (adoptium#2232)

- Remove openj9_custom to merge it into jdk_custom

Signed-off-by: Longyu Zhang <[email protected]>

* Move JdiTest playlist entry from lambdaLoadTest to otherLoadTest (adoptium#2233)

Signed-off-by: Simon Rushton <[email protected]>

* Reinstate ChaCha20Poly1305ParamTest (adoptium#2236)

Issue eclipse-openj9/openj9#11689

Signed-off-by: Peter Shipton <[email protected]>

* Replace deprecated triggers with new syntax (adoptium#2237)

See https://github.com/jenkinsci/job-dsl-plugin/blob/master/docs/Migration.md#migrating-to-177

Signed-off-by: Adam Brousseau <[email protected]>

* Update userGuide to include vendor disable feature (adoptium#2239)

Signed-off-by: renfeiw <[email protected]>

* Use the new format of disable in playlist file (adoptium#2240)

Signed-off-by: renfeiw <[email protected]>

* Add a custom playlist target to run system tests (adoptium#2234)

Signed-off-by: [email protected] <[email protected]>

* OpenJ9 AArch64: Enable a BigInteger test back again (adoptium#2242)

This commit enables the following test, which used to fail with timeouts
on AArch64:

- java/math/BigInteger/LargeValueExceptions.java (eclipse-openj9/openj9#9082)

Signed-off-by: KONNO Kazuhiro <[email protected]>

* OpenJ9 AArch64: excluding stream/CountLargeTest (adoptium#2243)

The following test is known to timeout with OpenJ9 AArch64.
This commit excludes it.

- java/util/stream/test/org/openjdk/tests/java/util/stream/CountLargeTest.java eclipse-openj9/openj9#9040

Signed-off-by: KONNO Kazuhiro <[email protected]>

* Allow test autogen user to specify parallel params (adoptium#2248)

- Param PARALLEL_DEFAULT will be set as default
  PARALLEL option
- Param NUM_MACHINES will be set if passed

Signed-off-by: Adam Brousseau <[email protected]>

* Exclude com/sun/crypto AEAD ByteBuffer tests (adoptium#2249)

Issue eclipse-openj9/openj9#11390

Signed-off-by: Peter Shipton <[email protected]>

* Fix command injection vulnerability in autoTestPR (adoptium#2244)

Fixes a command injection vulnerability with the auto test PR GitHub Workflow.

See https://securitylab.github.com/research/github-actions-untrusted-input

Also removes the unecessary `&& github.event_name != 'pull_request'` condition from the autoTestPR job because it will always be satisfied. (This workflow only triggers upon issue_comment, so `github.event_name` will never be 'pull_request')

* Update the userGuide (adoptium#2259)

- add a one line to cover the special case of auto disable

Signed-off-by: renfeiw <[email protected]>

* Add certain directories trigger for PR/push testing (adoptium#2241)

* Add certain directories trigger for PR/push testing

* use run-aqa with the target specified by the changed dirs

* remove unnecessary push trigger, leave only PR trigger

* merge openj9 and hotspot, deal with duplication situation

* Combine getChangeLocation and getBuildLists into one getBuildLists, remove unnecessary code

Co-authored-by: xichen1 <[email protected]>

* Add 1 hour timeouts to all copyArtifacts steps (adoptium#2261)

Avoid long hanging builds when copyArtifacts hangs

Signed-off-by: Adam Brousseau <[email protected]>

* Set Lightweight checkout on child jobs (adoptium#2251)

Pass along the LIGHT_WEIGHT_CHECKOUT parameter
to the generater in order to have children
with the same value. Generally this will be false
for Ginders and true eveywhere else

Fixes adoptium#2250

Signed-off-by: Adam Brousseau <[email protected]>

* Revert "Exclude com/sun/crypto AEAD ByteBuffer tests (adoptium#2249)" (adoptium#2266)

This reverts commit 2227228.

* AUTO: auto exclude test dacapo-jython impl=openj9 ver=16+ (adoptium#2269)

- related: adoptium#2256 (comment)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: smlambert <[email protected]>

* Only pass LIGHTWEIGHT to DSL if defined (adoptium#2270)

- Currently it will put NULL which will cause
  toBoolean() to fail

Related adoptium#2251

Signed-off-by: Adam Brousseau <[email protected]>

* Add jdk_vector for jdk16+ (adoptium#2267)

Signed-off-by: lanxia <[email protected]>

* AUTO: auto exclude test jdk_foreign_native plat=.*(ppc|arm|390).* (adoptium#2280)

- related: adoptium#2265 (comment)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: adamfarley <[email protected]>

* temporarily fix bug of get changed files (adoptium#2277)

* Exclude jdk_net related failed tests temporarily (adoptium#2282)

- exclude jdk_net failed tests temporarily
- for jdk 11, 15, and 16 on related  platforms

Signed-off-by: Longyu Zhang <[email protected]>

* add suffix to ignore native tests (adoptium#2283)

* add suffix to ignore native tests

* remove unnecessary matrix.bulid_list in target list

* Remove maketest.sh (adoptium#2285)

Signed-off-by: [email protected] <[email protected]>

* Add comment-triggered PR build action (adoptium#2222)

* Add comment-triggered PR build action (adoptium#2199)

Add workflow `action.yml` that triggers on a PR comment starting with the word `action` then runs AQA tests based on the parameters.
Add `action_argparse.py` that parses PR comments to determine the parameters for AQA tests in the `action.yml` workflow.

Co-authored-by: Xichen Pan <[email protected]>
Co-authored-by: patkarns <[email protected]>

* Update indentation for exclusion user guide (adoptium#2288)

Signed-off-by: renfeiw <[email protected]>

* Fix reportStatus job running on every comment (adoptium#2290)

Prevents reportStatus from running on every comment by checking for the same conditions as parseComment:
- The comment must begin with `run aqa` keyword
- The comment must be a PR comment

* Use any node for copying TAP files in parallel test builds (adoptium#2295)

Resolve: adoptium#2262

Signed-off-by: lanxia <[email protected]>

* Set ulimit for sharedclasscache multithread tests (adoptium#2274)

Signed-off-by: Simon Rushton <[email protected]>

* Add test excluded label for auto exclusion (adoptium#2300)

Signed-off-by: renfeiw <[email protected]>

* Change subset to version (adoptium#2301)

Signed-off-by: renfeiw <[email protected]>

* Fix GitHub head ref error (adoptium#2308)

* test error

* test error

* remove parentheses

* use github.head_ref

* remove test file

* Update userGuide to change subset to version (adoptium#2307)

Signed-off-by: renfeiw <[email protected]>

* remove PushOrPRtestbuild.yml (adoptium#2309)

* Add alpine-linux and x86-64_solaris platforms (adoptium#2304)

* Trial external_custom target

Signed-off-by: Shelley Lambert <[email protected]>

* Update build.xml

Signed-off-by: Shelley Lambert <[email protected]>

* Add alpine-linux platform label

Signed-off-by: Shelley Lambert <[email protected]>

* add bits

Signed-off-by: Shelley Lambert <[email protected]>

* Add x86_solaris

Signed-off-by: Shelley Lambert <[email protected]>

* Remove waitForANodeToBecomeActive post-parallel (adoptium#2312)

This step is unnecessary now that we run
on any node. Also remove function as it
was only used for this step.

Follow up to adoptium#2295

Signed-off-by: Adam Brousseau <[email protected]>

* Remove ParallelStreamsLoadTest_special_J9 long running modes 113, 614 (adoptium#2315)

Modes 113 and 614 take too long to run across all platforms, and
contribute to special.system timeout problems on Windows.

(5) 113: -Xgcpolicy:gencon
-Xjit:count=0,optlevel=warm,gcOnResolve,rtResolve -Xmn512k
-Xnocompressedrefs
aix 8: 23min, 11: 31min
osx 8: 10min, 11: 20min
plinux 8: 18min, 11: 23min
xlinux 8: 25min, 11: 40min
zlinux 8: 12min, 11: 14min
Windows 11: 30min

(20) 614: -Xcompressedrefs -Xgcpolicy:gencon -Xjit:counts="- - - - - - 1
1 1 1000 250 250 - - - 10000 100000
10000",gcOnResolve,rtResolve,sampleInterval=2,scorchingSampleThreshold=10000,quickProfile
-Xmn512k -Xcheck:gc:vmthreads:all:quiet
aix 8: 19min, 11: 27min
osx 8: 17min, 11: 17min
plinux 8: 18min, 11: 22min
xlinux 8: 24min, 11: 33min
zlinux 8: 10min, 11: 14min
Windows 11: 27min

Signed-off-by: Peter Shipton <[email protected]>

* Change Renaissance 0.9.0 to 0.11.0 in build.xml (adoptium#2314)

Fixes adoptium#2298

* Disable ParallelStreamsLoadTest slow OSRG modes on Windows (adoptium#2316)

The OSRG modes run slow only on Windows, and contribute to Windows
special.system tests timing out. The Mode Mode107-OSRG (24) can timeout
the 60min limit and fail.

Issue eclipse-openj9/openj9#11904

variation: (24) Mode107-OSRG
JVM_OPTIONS:  -Xgcpolicy:optthruput -Xdebug
-Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
Windows 11: 67min
aix 8: 9min, 11: 4min
osx 8: 3min, 11: 5min
plinux 8: 2min, 11: 4min
xlinux 8: 4min, 11: 10min
zlinux 8: 4min, 11: 6min

variation: (25) Mode110-OSRG
JVM_OPTIONS:
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
-Xgcpolicy:gencon
Windows 11: 32min
aix 8: 4min, 11: 5min
osx 8: 3min, 11: 2min
plinux 8: 4min, 11: 5min
xlinux 8: 4min, 11: 6min
zlinux 8: 1min, 11: 2min

variation: (26) Mode610-OSRG
JVM_OPTIONS:  -Xcompressedrefs
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
-Xgcpolicy:gencon
Windows 11: 25min
aix 8: 4min, 11: 5min
osx 8: 3min, 11: 2min
plinux 8: 3min, 11: 4min
xlinux 8: 5min, 11: 6min
zlinux 8: 2min, 11: 2min

variation: (27) Mode612-OSRG
JVM_OPTIONS:  -Xcompressedrefs -Xgcpolicy:gencon
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
Windows 11: 33min
aix 8: 4min, 11: 5min
osx 8: 2min, 11: 4min
plinux 8: 3min, 11: 5min
xlinux 8: 4min, 11: 6min
zlinux 8: 1min, 11: 2min

Signed-off-by: Peter Shipton <[email protected]>

* AUTO: auto exclude test renaissance-dec-tree ver=16+ (adoptium#2319)

- related: adoptium#2255 (comment)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: smlambert <[email protected]>

* Disable ParallelStreamsLoadTest slow OSRG modes on Windows (adoptium#2325)

Fixes adoptium#2316 (comment)

The OSRG modes run slow only on Windows, and contribute to Windows
special.system tests timing out. The Mode Mode107-OSRG (24) can timeout
the 60min limit and fail.

Issue eclipse-openj9/openj9#11904

variation: (24) Mode107-OSRG
JVM_OPTIONS:  -Xgcpolicy:optthruput -Xdebug
-Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
Windows 11: 67min
aix 8: 9min, 11: 4min
osx 8: 3min, 11: 5min
plinux 8: 2min, 11: 4min
xlinux 8: 4min, 11: 10min
zlinux 8: 4min, 11: 6min

variation: (25) Mode110-OSRG
JVM_OPTIONS:
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
-Xgcpolicy:gencon
Windows 11: 32min
aix 8: 4min, 11: 5min
osx 8: 3min, 11: 2min
plinux 8: 4min, 11: 5min
xlinux 8: 4min, 11: 6min
zlinux 8: 1min, 11: 2min

variation: (26) Mode610-OSRG
JVM_OPTIONS:  -Xcompressedrefs
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
-Xgcpolicy:gencon
Windows 11: 25min
aix 8: 4min, 11: 5min
osx 8: 3min, 11: 2min
plinux 8: 3min, 11: 4min
xlinux 8: 5min, 11: 6min
zlinux 8: 2min, 11: 2min

variation: (27) Mode612-OSRG
JVM_OPTIONS:  -Xcompressedrefs -Xgcpolicy:gencon
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
Windows 11: 33min
aix 8: 4min, 11: 5min
osx 8: 2min, 11: 4min
plinux 8: 3min, 11: 5min
xlinux 8: 4min, 11: 6min
zlinux 8: 1min, 11: 2min

Signed-off-by: Peter Shipton <[email protected]>

* Set parallel machine limit to the total number of machines (adoptium#2330)

Resolve: eclipse-openj9/openj9#12088
Signed-off-by: lanxia <[email protected]>

* Convert subset to version (adoptium#2329)

- using version in playlists

Signed-off-by: renfeiw <[email protected]>

* Handle a git command's request for a username on Windows (adoptium#2287)

If we call git ls-remote on Window, when the repo doesn't exist,
the command asks for a username and hangs.

This change disables the request for a username and provides more
useful error messages. It also handles the problem appropriately,
so instead of failing the build, it correctly sets the repo location
to the non-'u' version.

Added some timeout-handling code as well, in case someone's running
this on a version of git older than 2.3, which may ignore the env
var we're using to suppress the username request.

Signed-off-by: Adam Farley <[email protected]>

* OpenJ9 JDK17 Exclude java/lang/Thread/UncaughtExceptionsTest.java (adoptium#2333)

Signed-off-by: Jason Feng <[email protected]>

* Only run foreign native test on x86_64 or aarch64 platforms (adoptium#2334)

This test requires CLinker, and the upstream community has no
current intention to port it to new platforms. So we need to exclude
the test on platforms that aren't x86-64 or aarch64.

Signed-off-by: Adam Farley <[email protected]>

* Enable download testimage for jdk8 (adoptium#2320)

Signed-off-by: Sophia Guo <[email protected]>

* Ignore leading and trailing whitespace in parsing (adoptium#2318)

Also simplifies the mapping of comment/command arguments to outputs.

* os upate to latest (adoptium#2331)

* use sw.os.alpine-linux label instead of sw.os.musl for Alpine (adoptium#2336)

Signed-off-by: Stewart X Addison <[email protected]>

Co-authored-by: LongyuZhang <[email protected]>
Co-authored-by: Mesbah Alam <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: smlambert <[email protected]>
Co-authored-by: Stewart X Addison <[email protected]>
Co-authored-by: Babneet Singh <[email protected]>
Co-authored-by: Peter Shipton <[email protected]>
Co-authored-by: KKES335 <[email protected]>
Co-authored-by: Kentaro Kuramochi <[email protected]>
Co-authored-by: Lan Xia <[email protected]>
Co-authored-by: Simon Rushton <[email protected]>
Co-authored-by: Jason Feng <[email protected]>
Co-authored-by: AdamBrousseau <[email protected]>
Co-authored-by: Renfei Wang <[email protected]>
Co-authored-by: KONNO Kazuhiro <[email protected]>
Co-authored-by: Deric Cheung <[email protected]>
Co-authored-by: XichenPan <[email protected]>
Co-authored-by: xichen1 <[email protected]>
Co-authored-by: adamfarley <[email protected]>
Co-authored-by: yanshan24 <[email protected]>
Co-authored-by: Shelley Lambert <[email protected]>
Co-authored-by: YananW <[email protected]>
Co-authored-by: Adam Farley <[email protected]>
Co-authored-by: sophia-guo <[email protected]>
smlambert added a commit that referenced this pull request Apr 28, 2021
@patkarns @LongyuZhang
@Mesbah-Alam @github-actions @smlambert @sxa @babsingh @pshipton @KKES335 @llxia @lumpfish @JasonFengJ9 @AdamBrousseau @renfeiw @knn-k @Icohedron @xichen1 @pxc-ddd @adamfarley @yanshan24 @xfxcwynlc @sophia-guo
Added system_custom example in testJobTemplate (#2488)
88f21ff 
* Create createTestList.py

* Delete createTestList.py

* Sync original repo (#5)

* Add add-exports to HCRLateAttachWorkload_previewEnabled (#2208)

- Add add-exports to HCRLateAttachWorkload_previewEnabled

Signed-off-by: Longyu Zhang <[email protected]>

* Convert iteration based STF based load tests to time-based tests (#2203)

Signed-off-by: [email protected] <[email protected]>

* AUTO: auto exclude test jdk_foreign_native impl=openj9 ver=16+ (#2219)

- related: #2218 (comment)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: smlambert <[email protected]>

* get.sh: stop mandating GNU tar --strip when extracting debug images (#2184)

Signed-off-by: Stewart X Addison <[email protected]>

* Remove references to eclipse-openj9/openj9#11135 (#2215)

eclipse-openj9/openj9#11135 refers to an uber issue, which will be closed soon.
So, the references to eclipse-openj9/openj9#11135 have been replaced with
stand-alone trackers for the respective issues.

Also, a duplicate for SuperMethodTest.java has been removed.

Signed-off-by: Babneet Singh <[email protected]>

* Exclude ChaCha20/ChaCha20Poly1305ParamTest.java (#2221)

Issue eclipse-openj9/openj9#11689

Signed-off-by: Peter Shipton <[email protected]>

* Update OpenJ9 JDK17 exclude list (#2217)

* Apply #2188 to OpenJ9 JDK17 exclude list

Apply #2188 to OpenJ9 JDK17 exclude list.

Signed-off-by: Babneet Singh <[email protected]>

* Apply #2192 to OpenJ9 JDK17 exclude list

Apply #2192 to OpenJ9 JDK17 exclude list.

Signed-off-by: Babneet Singh <[email protected]>

* Apply #2215 to OpenJ9 JDK17 exclude list

Apply #2215 to OpenJ9 JDK17 exclude list.

Signed-off-by: Babneet Singh <[email protected]>

* Fix charsets,i18n,language_tag and locale_matching for 16 and Create new 4 test cases (#2214)

Co-authored-by: Kentaro Kuramochi <[email protected]>

* Allow users to set lightweight option (#2216)

The benefit of having lightweight checkout is to save space on Jenkins master.
However, Jenkins does not support using Repository URL and Branch build parameters with lightweight checkout together (see https://issues.jenkins.io/browse/JENKINS-48431).

In this PR, we set LIGHT_WEIGHT_CEHCKOUT to false and use Repository URL and Branch as build parameters by default. If LIGHT_WEIGHT_CEHCKOUT is set to true, set Repository URL and Branch to its explicit value

Signed-off-by: lanxia <[email protected]>

* Create ParallelStreamsLoadTest (#2229)

Signed-off-by: Simon Rushton <[email protected]>

* Remove _5m from ParallelStreamsLoadTest targets (#2231)

Signed-off-by: Simon Rushton <[email protected]>

* Enable java/lang/reflect/DefaultMethodMembers/FilterNotMostSpecific.java (#2230)

This was forgot after resolving openj9 issue #7623

Signed-off-by: Jason Feng <[email protected]>

* Merge openj9_custom into jdk_custom (#2232)

- Remove openj9_custom to merge it into jdk_custom

Signed-off-by: Longyu Zhang <[email protected]>

* Move JdiTest playlist entry from lambdaLoadTest to otherLoadTest (#2233)

Signed-off-by: Simon Rushton <[email protected]>

* Reinstate ChaCha20Poly1305ParamTest (#2236)

Issue eclipse-openj9/openj9#11689

Signed-off-by: Peter Shipton <[email protected]>

* Replace deprecated triggers with new syntax (#2237)

See https://github.com/jenkinsci/job-dsl-plugin/blob/master/docs/Migration.md#migrating-to-177

Signed-off-by: Adam Brousseau <[email protected]>

* Update userGuide to include vendor disable feature (#2239)

Signed-off-by: renfeiw <[email protected]>

* Use the new format of disable in playlist file (#2240)

Signed-off-by: renfeiw <[email protected]>

* Add a custom playlist target to run system tests (#2234)

Signed-off-by: [email protected] <[email protected]>

* OpenJ9 AArch64: Enable a BigInteger test back again (#2242)

This commit enables the following test, which used to fail with timeouts
on AArch64:

- java/math/BigInteger/LargeValueExceptions.java (eclipse-openj9/openj9#9082)

Signed-off-by: KONNO Kazuhiro <[email protected]>

* OpenJ9 AArch64: excluding stream/CountLargeTest (#2243)

The following test is known to timeout with OpenJ9 AArch64.
This commit excludes it.

- java/util/stream/test/org/openjdk/tests/java/util/stream/CountLargeTest.java eclipse-openj9/openj9#9040

Signed-off-by: KONNO Kazuhiro <[email protected]>

* Allow test autogen user to specify parallel params (#2248)

- Param PARALLEL_DEFAULT will be set as default
  PARALLEL option
- Param NUM_MACHINES will be set if passed

Signed-off-by: Adam Brousseau <[email protected]>

* Exclude com/sun/crypto AEAD ByteBuffer tests (#2249)

Issue eclipse-openj9/openj9#11390

Signed-off-by: Peter Shipton <[email protected]>

* Fix command injection vulnerability in autoTestPR (#2244)

Fixes a command injection vulnerability with the auto test PR GitHub Workflow.

See https://securitylab.github.com/research/github-actions-untrusted-input

Also removes the unecessary `&& github.event_name != 'pull_request'` condition from the autoTestPR job because it will always be satisfied. (This workflow only triggers upon issue_comment, so `github.event_name` will never be 'pull_request')

* Update the userGuide (#2259)

- add a one line to cover the special case of auto disable

Signed-off-by: renfeiw <[email protected]>

* Add certain directories trigger for PR/push testing (#2241)

* Add certain directories trigger for PR/push testing

* use run-aqa with the target specified by the changed dirs

* remove unnecessary push trigger, leave only PR trigger

* merge openj9 and hotspot, deal with duplication situation

* Combine getChangeLocation and getBuildLists into one getBuildLists, remove unnecessary code

Co-authored-by: xichen1 <[email protected]>

* Add 1 hour timeouts to all copyArtifacts steps (#2261)

Avoid long hanging builds when copyArtifacts hangs

Signed-off-by: Adam Brousseau <[email protected]>

* Set Lightweight checkout on child jobs (#2251)

Pass along the LIGHT_WEIGHT_CHECKOUT parameter
to the generater in order to have children
with the same value. Generally this will be false
for Ginders and true eveywhere else

Fixes #2250

Signed-off-by: Adam Brousseau <[email protected]>

* Revert "Exclude com/sun/crypto AEAD ByteBuffer tests (#2249)" (#2266)

This reverts commit 2227228.

* AUTO: auto exclude test dacapo-jython impl=openj9 ver=16+ (#2269)

- related: #2256 (comment)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: smlambert <[email protected]>

* Only pass LIGHTWEIGHT to DSL if defined (#2270)

- Currently it will put NULL which will cause
  toBoolean() to fail

Related #2251

Signed-off-by: Adam Brousseau <[email protected]>

* Add jdk_vector for jdk16+ (#2267)

Signed-off-by: lanxia <[email protected]>

* AUTO: auto exclude test jdk_foreign_native plat=.*(ppc|arm|390).* (#2280)

- related: #2265 (comment)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: adamfarley <[email protected]>

* temporarily fix bug of get changed files (#2277)

* Exclude jdk_net related failed tests temporarily (#2282)

- exclude jdk_net failed tests temporarily
- for jdk 11, 15, and 16 on related  platforms

Signed-off-by: Longyu Zhang <[email protected]>

* add suffix to ignore native tests (#2283)

* add suffix to ignore native tests

* remove unnecessary matrix.bulid_list in target list

* Remove maketest.sh (#2285)

Signed-off-by: [email protected] <[email protected]>

* Add comment-triggered PR build action (#2222)

* Add comment-triggered PR build action (#2199)

Add workflow `action.yml` that triggers on a PR comment starting with the word `action` then runs AQA tests based on the parameters.
Add `action_argparse.py` that parses PR comments to determine the parameters for AQA tests in the `action.yml` workflow.

Co-authored-by: Xichen Pan <[email protected]>
Co-authored-by: patkarns <[email protected]>

* Update indentation for exclusion user guide (#2288)

Signed-off-by: renfeiw <[email protected]>

* Fix reportStatus job running on every comment (#2290)

Prevents reportStatus from running on every comment by checking for the same conditions as parseComment:
- The comment must begin with `run aqa` keyword
- The comment must be a PR comment

* Use any node for copying TAP files in parallel test builds (#2295)

Resolve: #2262

Signed-off-by: lanxia <[email protected]>

* Set ulimit for sharedclasscache multithread tests (#2274)

Signed-off-by: Simon Rushton <[email protected]>

* Add test excluded label for auto exclusion (#2300)

Signed-off-by: renfeiw <[email protected]>

* Change subset to version (#2301)

Signed-off-by: renfeiw <[email protected]>

* Fix GitHub head ref error (#2308)

* test error

* test error

* remove parentheses

* use github.head_ref

* remove test file

* Update userGuide to change subset to version (#2307)

Signed-off-by: renfeiw <[email protected]>

* remove PushOrPRtestbuild.yml (#2309)

* Add alpine-linux and x86-64_solaris platforms (#2304)

* Trial external_custom target

Signed-off-by: Shelley Lambert <[email protected]>

* Update build.xml

Signed-off-by: Shelley Lambert <[email protected]>

* Add alpine-linux platform label

Signed-off-by: Shelley Lambert <[email protected]>

* add bits

Signed-off-by: Shelley Lambert <[email protected]>

* Add x86_solaris

Signed-off-by: Shelley Lambert <[email protected]>

* Remove waitForANodeToBecomeActive post-parallel (#2312)

This step is unnecessary now that we run
on any node. Also remove function as it
was only used for this step.

Follow up to #2295

Signed-off-by: Adam Brousseau <[email protected]>

* Remove ParallelStreamsLoadTest_special_J9 long running modes 113, 614 (#2315)

Modes 113 and 614 take too long to run across all platforms, and
contribute to special.system timeout problems on Windows.

(5) 113: -Xgcpolicy:gencon
-Xjit:count=0,optlevel=warm,gcOnResolve,rtResolve -Xmn512k
-Xnocompressedrefs
aix 8: 23min, 11: 31min
osx 8: 10min, 11: 20min
plinux 8: 18min, 11: 23min
xlinux 8: 25min, 11: 40min
zlinux 8: 12min, 11: 14min
Windows 11: 30min

(20) 614: -Xcompressedrefs -Xgcpolicy:gencon -Xjit:counts="- - - - - - 1
1 1 1000 250 250 - - - 10000 100000
10000",gcOnResolve,rtResolve,sampleInterval=2,scorchingSampleThreshold=10000,quickProfile
-Xmn512k -Xcheck:gc:vmthreads:all:quiet
aix 8: 19min, 11: 27min
osx 8: 17min, 11: 17min
plinux 8: 18min, 11: 22min
xlinux 8: 24min, 11: 33min
zlinux 8: 10min, 11: 14min
Windows 11: 27min

Signed-off-by: Peter Shipton <[email protected]>

* Change Renaissance 0.9.0 to 0.11.0 in build.xml (#2314)

Fixes #2298

* Disable ParallelStreamsLoadTest slow OSRG modes on Windows (#2316)

The OSRG modes run slow only on Windows, and contribute to Windows
special.system tests timing out. The Mode Mode107-OSRG (24) can timeout
the 60min limit and fail.

Issue eclipse-openj9/openj9#11904

variation: (24) Mode107-OSRG
JVM_OPTIONS:  -Xgcpolicy:optthruput -Xdebug
-Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
Windows 11: 67min
aix 8: 9min, 11: 4min
osx 8: 3min, 11: 5min
plinux 8: 2min, 11: 4min
xlinux 8: 4min, 11: 10min
zlinux 8: 4min, 11: 6min

variation: (25) Mode110-OSRG
JVM_OPTIONS:
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
-Xgcpolicy:gencon
Windows 11: 32min
aix 8: 4min, 11: 5min
osx 8: 3min, 11: 2min
plinux 8: 4min, 11: 5min
xlinux 8: 4min, 11: 6min
zlinux 8: 1min, 11: 2min

variation: (26) Mode610-OSRG
JVM_OPTIONS:  -Xcompressedrefs
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
-Xgcpolicy:gencon
Windows 11: 25min
aix 8: 4min, 11: 5min
osx 8: 3min, 11: 2min
plinux 8: 3min, 11: 4min
xlinux 8: 5min, 11: 6min
zlinux 8: 2min, 11: 2min

variation: (27) Mode612-OSRG
JVM_OPTIONS:  -Xcompressedrefs -Xgcpolicy:gencon
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
Windows 11: 33min
aix 8: 4min, 11: 5min
osx 8: 2min, 11: 4min
plinux 8: 3min, 11: 5min
xlinux 8: 4min, 11: 6min
zlinux 8: 1min, 11: 2min

Signed-off-by: Peter Shipton <[email protected]>

* AUTO: auto exclude test renaissance-dec-tree ver=16+ (#2319)

- related: #2255 (comment)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: smlambert <[email protected]>

* Disable ParallelStreamsLoadTest slow OSRG modes on Windows (#2325)

Fixes #2316 (comment)

The OSRG modes run slow only on Windows, and contribute to Windows
special.system tests timing out. The Mode Mode107-OSRG (24) can timeout
the 60min limit and fail.

Issue eclipse-openj9/openj9#11904

variation: (24) Mode107-OSRG
JVM_OPTIONS:  -Xgcpolicy:optthruput -Xdebug
-Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
Windows 11: 67min
aix 8: 9min, 11: 4min
osx 8: 3min, 11: 5min
plinux 8: 2min, 11: 4min
xlinux 8: 4min, 11: 10min
zlinux 8: 4min, 11: 6min

variation: (25) Mode110-OSRG
JVM_OPTIONS:
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
-Xgcpolicy:gencon
Windows 11: 32min
aix 8: 4min, 11: 5min
osx 8: 3min, 11: 2min
plinux 8: 4min, 11: 5min
xlinux 8: 4min, 11: 6min
zlinux 8: 1min, 11: 2min

variation: (26) Mode610-OSRG
JVM_OPTIONS:  -Xcompressedrefs
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
-Xgcpolicy:gencon
Windows 11: 25min
aix 8: 4min, 11: 5min
osx 8: 3min, 11: 2min
plinux 8: 3min, 11: 4min
xlinux 8: 5min, 11: 6min
zlinux 8: 2min, 11: 2min

variation: (27) Mode612-OSRG
JVM_OPTIONS:  -Xcompressedrefs -Xgcpolicy:gencon
-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation
Windows 11: 33min
aix 8: 4min, 11: 5min
osx 8: 2min, 11: 4min
plinux 8: 3min, 11: 5min
xlinux 8: 4min, 11: 6min
zlinux 8: 1min, 11: 2min

Signed-off-by: Peter Shipton <[email protected]>

* Set parallel machine limit to the total number of machines (#2330)

Resolve: eclipse-openj9/openj9#12088
Signed-off-by: lanxia <[email protected]>

* Convert subset to version (#2329)

- using version in playlists

Signed-off-by: renfeiw <[email protected]>

* Handle a git command's request for a username on Windows (#2287)

If we call git ls-remote on Window, when the repo doesn't exist,
the command asks for a username and hangs.

This change disables the request for a username and provides more
useful error messages. It also handles the problem appropriately,
so instead of failing the build, it correctly sets the repo location
to the non-'u' version.

Added some timeout-handling code as well, in case someone's running
this on a version of git older than 2.3, which may ignore the env
var we're using to suppress the username request.

Signed-off-by: Adam Farley <[email protected]>

* OpenJ9 JDK17 Exclude java/lang/Thread/UncaughtExceptionsTest.java (#2333)

Signed-off-by: Jason Feng <[email protected]>

* Only run foreign native test on x86_64 or aarch64 platforms (#2334)

This test requires CLinker, and the upstream community has no
current intention to port it to new platforms. So we need to exclude
the test on platforms that aren't x86-64 or aarch64.

Signed-off-by: Adam Farley <[email protected]>

* Enable download testimage for jdk8 (#2320)

Signed-off-by: Sophia Guo <[email protected]>

* Ignore leading and trailing whitespace in parsing (#2318)

Also simplifies the mapping of comment/command arguments to outputs.

* os upate to latest (#2331)

* use sw.os.alpine-linux label instead of sw.os.musl for Alpine (#2336)

Signed-off-by: Stewart X Addison <[email protected]>

Co-authored-by: LongyuZhang <[email protected]>
Co-authored-by: Mesbah Alam <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: smlambert <[email protected]>
Co-authored-by: Stewart X Addison <[email protected]>
Co-authored-by: Babneet Singh <[email protected]>
Co-authored-by: Peter Shipton <[email protected]>
Co-authored-by: KKES335 <[email protected]>
Co-authored-by: Kentaro Kuramochi <[email protected]>
Co-authored-by: Lan Xia <[email protected]>
Co-authored-by: Simon Rushton <[email protected]>
Co-authored-by: Jason Feng <[email protected]>
Co-authored-by: AdamBrousseau <[email protected]>
Co-authored-by: Renfei Wang <[email protected]>
Co-authored-by: KONNO Kazuhiro <[email protected]>
Co-authored-by: Deric Cheung <[email protected]>
Co-authored-by: XichenPan <[email protected]>
Co-authored-by: xichen1 <[email protected]>
Co-authored-by: adamfarley <[email protected]>
Co-authored-by: yanshan24 <[email protected]>
Co-authored-by: Shelley Lambert <[email protected]>
Co-authored-by: YananW <[email protected]>
Co-authored-by: Adam Farley <[email protected]>
Co-authored-by: sophia-guo <[email protected]>

* Update testJobTemplate

Testing param separator

* Update testJobTemplate

adding styles to header

* Update testJobTemplate

Categorized all parameters

* Update testJobTemplate

updated CSS variables

* Update testJobTemplate

* Update testJobTemplate

* Update testJobTemplate

* Update testJobTemplate

* Update testJobTemplate

* Update testJobTemplate

* Update testJobTemplate

* Update testJobTemplate

* Update testJobTemplate

* Update testJobTemplate

* Update testJobTemplate

Testing parameter separator

* Update testJobTemplate

added CSS styles to parameter separators

* Update testJobTemplate to better reflect mockup

* Update testJobTemplate

* update testJobTemplate, added subtitle styling

* fixed typo

* Update testJobTemplate

* Update testJobTemplate

* Update testJobTemplate

* Style parameters in testJobTemplate

* Add link to Grinder Wiki

* edited styling for Grinder wiki link

* edited 'grinder wiki' positioning

* Added padding to 'Grinder Wiki' link

* Updated JDK_BRANCH parameter order

* added spaces back to copyright

* Update some parameters' default values

* Removed extra tab

* Removed Grinder wiki link and separator bg color

* removed extra spaces in testJobTemplate

* added examples in CUSTOM_TARGET

* fixed /n error

* changed multiline string declaration

* added triple quotations around description

* moved CUSTOM_TARGET examples to the right spot

Co-authored-by: LongyuZhang <[email protected]>
Co-authored-by: Mesbah Alam <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: smlambert <[email protected]>
Co-authored-by: Stewart X Addison <[email protected]>
Co-authored-by: Babneet Singh <[email protected]>
Co-authored-by: Peter Shipton <[email protected]>
Co-authored-by: KKES335 <[email protected]>
Co-authored-by: Kentaro Kuramochi <[email protected]>
Co-authored-by: Lan Xia <[email protected]>
Co-authored-by: Simon Rushton <[email protected]>
Co-authored-by: Jason Feng <[email protected]>
Co-authored-by: AdamBrousseau <[email protected]>
Co-authored-by: Renfei Wang <[email protected]>
Co-authored-by: KONNO Kazuhiro <[email protected]>
Co-authored-by: Deric Cheung <[email protected]>
Co-authored-by: XichenPan <[email protected]>
Co-authored-by: xichen1 <[email protected]>
Co-authored-by: adamfarley <[email protected]>
Co-authored-by: yanshan24 <[email protected]>
Co-authored-by: Shelley Lambert <[email protected]>
Co-authored-by: YananW <[email protected]>
Co-authored-by: Adam Farley <[email protected]>
Co-authored-by: sophia-guo <[email protected]>
@Mesbah-Alam Mesbah-Alam deleted the add-timebased-tests branch May 12, 2021 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@smlambert smlambert smlambert approved these changes

@llxia llxia llxia approved these changes

Assignees

@Mesbah-Alam Mesbah-Alam

Labels
enhancement
Projects
None yet
Milestone
January 2021
Development

Successfully merging this pull request may close these issues.
5 participants
@Mesbah-Alam @llxia @smlambert @pshipton @karianna