We are in the process of adding more microbenchmarks (both bumblebench and jmh formats) and full open-source benchmarks to this directory. New perftest jobs are added to the Test_perf view. While we will run these benchmarks regularly at adoptium, the intention is to make it easy for developers to run performance testing locally.
You can follow the same manual testing instructions to run these tests, as you do for all of the other tests we run. The top-level make target for tests contained in this directory (aqa-tests/perf) is "perf". So, once you compile test material, running "make perf" will run all of the testcases defined in playlist.xml files in this directory and its subdirectories. Each unique benchmark is housed in a subdirectory and given a meaningful name. Once the reorganization of this directory is complete, the directory structure will look like:
//./
//├── bumbleBench
//├── idle_micro
//├── dacapo
//├── renaissance
//├── liberty
Each subdirectory requires a build.xml file describing where to pull the benchmark suite from, and how to build and run it. Each subdirectory also requires a playlist.xml file which describes 1 or more benchmarks and what commands to run in order to execute a particular benchmark run.
Currently we run a single micro-benchmark called idle_micro against OpenJDK8 builds (both hotspot and openj9 variants).
Microbenchmarks found in the bumblebench repo. There are already a good variety of microbenchmarks for evaluating performance of various aspects of code, such as string, lambda, gpu, math, crypto, etc. Microbenches include bumbleBench-ArrayListForEachConsumerAnonymousBench, bumbleBench-ArrayListForEachConsumerFinalBench, bumbleBench-ArrayListForEachLambdaBench, bumbleBench-ArrayListForEachTradBench, bumbleBench-ArrayListRemoveIfLambdaBench, bumbleBench-ArrayListRemoveIfTradBench, bumbleBench-ArrayListSortCollectionsBench, bumbleBench-ArrayListSortComparatorBench, bumbleBench-ArrayListSortLambdaBench, bumbleBench-HashMapForEachBiConsumerAnonymousBench, bumbleBench-HashMapForEachBiConsumerFinalBench, bumbleBench-HashMapForEachLambdaBench, bumbleBench-HashMapForEachTradBench, bumbleBench-HashMapReplaceAllLambdaBench, bumbleBench-HashMapReplaceAllTradBench, bumbleBench-CipherBench, bumbleBench-DigestBench, bumbleBench-EllipticCurveBench, bumbleBench-GCMBench, bumbleBench-HMACBench, bumbleBench-KeyExchangeBench, bumbleBench-RSABench, bumbleBench-SignatureBench, bumbleBench-SSLSocketBench, bumbleBench-BitonicBench-CPU, bumbleBench-BitonicBench-GPULambda, bumbleBench-DoitgenBench-CPU, bumbleBench-DoitgenBench-GPULambda, bumbleBench-KmeansBench-CPU, bumbleBench-KmeansBench-GPULambda, bumbleBench-MatMultBench-CPU, bumbleBench-MatMultBench-GPULambda, bumbleBench-DistinctStringsEqualsBench, bumbleBench-NewStringBufferWithCapacityBench, bumbleBench-NewStringBuilderWithCapacityBench, bumbleBench-SameStringsEqualsBench, bumbleBench-DispatchBench-InnerClasses, bumbleBench-FibBench-Vanilla, bumbleBench-FibBench-InnerClass, bumbleBench-FibBench-Lambda, bumbleBench-FibBench-LocalLambda, bumbleBench-FibBench-DynamicLambda, bumbleBench-FibBench-LocalMethodReferences, bumbleBench-GroupingBench-Serial, bumbleBench-GroupingBench-Parallel, bumbleBench-SieveBench, bumbleBench-ExactBench, bumbleBench-SIMDDoubleMaxMinBench, bumbleBench-StringConversionBench, bumbleBench-StringHashBench, bumbleBench-StringIndexOfBench and bumbleBench-StringIndexOfStringBench.
Transparency and the ability to see how the binaries are being exercised is important to us. We will focus on running fully open-sourced benchmarks at Adoptium so that developers have full-access to see the benchmarking code.
Dacapo benchmarks from https://github.com/dacapobench/dacapobench - including dacopo-eclipse, dacapo-avrora, dacapo-fop, dacapo-h2, dacapo-jython, dacapo-luindex, dacapo-lusearch-fix, dacapo-pmd, dacapo-sunflow, dacapo-tomcat and dacapo-xalan.
Liberty benchmarks from https://github.com/OpenLiberty - including liberty-dt7-startup and liberty-dt7-throughput
Renaissance benchmarks from https://github.com/renaissance-benchmarks/renaissance - including renaissance-akka-uct, renaissance-als, renaissance-chi-square, renaissance-db-shootout, renaissance-dec-tree, renaissance-finagle-chirper, renaissance-finagle-http, renaissance-fj-kmeans, renaissance-future-genetic, renaissance-gauss-mix, renaissance-log-regression, renaissance-mnemonics, renaissance-movie-lens, renaissance-naive-bayes, renaissance-par-mnemonics, renaissance-philosophers and renaissance-scala-kmeans
SPECjbb 2015 benchmark from https://www.spec.org/jbb2015/
Additional benchmarks are being reviewed for addition and if you wish to include more, please comment in the open performance benchmarks issue 1112.