Exception java.lang.NoSuchMethodError: uk.ac.manchester.tornado.api.exceptions.Debug.tprintf(Ljava/lang/String;[Ljava/lang/Object;)

[andrii0lomakin]

Hi.

I am working on implementation of the LLM inference engine using TornadoVM, and of course, I had to create a test configuration that runs tests inside the Gradle.

I have generated JVM parameters according to the source code of the tornado Python script, which is listed in this gist: https://gist.github.com/andrii0lomakin/d4e8893e46d4ee1a66820cba25e5358f.

And though the showDevices task works perfectly well.
During the creation of the task graph I have got the following:

java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchMethodError: uk.ac.manchester.tornado.api.exceptions.Debug.tprintf(Ljava/lang/String;[Ljava/lang/Object;)

Which seems logical because this method does absent in provided class.
But it also means that I miss something could you help me with this problem ?

[jjfumero]

Hi @andrii0lomakin . I have little experience with Gradle, but just a few things that can help:

1. Are you using all Java flags printed by the tornado script?

For example, for my configuration using the OpenCL backend, I got all these flags:

$ tornado --printJavaFlags 

/home/juan/repos/tornadovm/etc/dependencies/TornadoVM-graalvm-jdk-21/graalvm-community-openjdk-21.0.1+12.1//bin/java -server -XX:-UseCompressedOops -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI -XX:-UseCompressedClassPointers --enable-preview -Djava.library.path=/home/juan/repos/tornadovm/bin/sdk/lib  --module-path .:/home/juan/repos/tornadovm/bin/sdk/share/java/tornado -Dtornado.load.api.implementation=uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph -Dtornado.load.runtime.implementation=uk.ac.manchester.tornado.runtime.TornadoCoreRuntime -Dtornado.load.tornado.implementation=uk.ac.manchester.tornado.runtime.common.Tornado -Dtornado.load.annotation.implementation=uk.ac.manchester.tornado.annotation.ASMClassVisitor -Dtornado.load.annotation.parallel=uk.ac.manchester.tornado.api.annotations.Parallel  -XX:+UseParallelGC  @/home/juan/repos/tornadovm/bin/sdk/etc/exportLists/common-exports @/home/juan/repos/tornadovm/bin/sdk/etc/exportLists/opencl-exports --add-modules ALL-SYSTEM,tornado.runtime,tornado.annotation,tornado.drivers.common,tornado.drivers.opencl

2. In addition, use the same JAVA_HOME as tornado. After the TornadoVM installation, the installer generates a conf file called (setvars.sh). You can copy your PATH and JAVA_HOME from this source file.

Let us know if this helps.

[andrii0lomakin]

My preliminar impression at the moment is that you have some plugins registered incorrectly. Will be happy to be disproved.

[andrii0lomakin]

If you agree, I already started to work on the modified version because that is a show stopper for me, and I will be glad to provide PR.

[andrii0lomakin]

I understand all of this. My question is different. If you provide a plugin for Math.pow(float, float) and this code does not exist (the method is absent from the user's perspective), the user will never write it. How will the plugin be used and called?

The plugin should be for TornadoMath.pow, otherwise we have a bug.Let me check.

I do not remember exact place (this concrete one related to PTX driver), there are several ones, probably better for you do not waste time, information is not very concrete I will provide PR soon and you will be able to check code.

[jjfumero]

Can you run with these flags?

# We do not satisfy the Graal compiler assertions because we only support a subset of the Java specification.
# This allows us to have the GraalIR in states which normally would be illegal.
__GRAAL_ENABLE_ASSERTIONS__ = " -ea -da:org.graalvm.compiler... "

tornado -ea enables assertions and disables the ones for the Graal Compiler.

So you can use:

--jvm="-ea -da:org.graalvm.compiler..."

[andrii0lomakin]

I see, I suppose it will work in such case (I will check). In such case you also skip valid assertions as the ones related to creation of plugins to the methods that existed before but disappeared not valid anymore. Like ones that I listed above. Let me try anyway.

[andrii0lomakin]

Yes, it works but as I have noticed it will skip valid cases too. I will try to gather such cases and create separate issue they are not critical obviously. I have another issue, TornadoVM does not work with Kotlin intrisincs (does not take into account that for them trusted stamp can be null during parsing of byte code) . That is small fix I will provide it soon.

[andrii0lomakin]

IMHO would be good to create small a guideline how to run test if TornadoVM is part of bigger application. If I have time I will help with that.

[jjfumero]

Thanks. That will be great.

[jjfumero]

I forgot to mention that the flags depend on the backend you have installed. The ones I showed are specific for the OpenCL backend and my configuration. If you compiled with the PTX or SPIR-V, you will see a couple of different flags. The way to query is via

$ tornado --printJavaFlags 

[andrii0lomakin]

@jjfumero, thank you for your help. I am aware of this. I converted the tornado Python script into Gradle script line-by-line.

[jjfumero]

For reference, @mikepapadim , core member of the TornadoVM team, extended LLM Llama2.java with TornadoVM:

https://github.com/mikepapadim/llama2.tornadovm.java

This project uses Maven, but it can serve as reference.

[andrii0lomakin]

I appreciate the reference. It is useful. But in general, I find the JLama design, not GPU performance-friendly. I hope to achieve a better speed-up. 1.15x speedup is not motivating users to move to quite a specific library implementation.

[jjfumero]

Indeed. So 15% improvements is just because 1 of the compute kernels was rewritten to use TornadoVM (the matrix-vector kernel). What Michalis found is that the data transfers back and forth (host <-> device) are now the bottleneck, and we are working on a new version to improve TornadoVM with automatic shared memory.

[andrii0lomakin]

Yeah, I expected that host-device transfer would be a bottleneck. I am working on a design that avoids that from scratch. I also work on other optimizations like Flashed Attention 2.0 and so on. That is why I raised the question of support of tensor cores in TornadoVM.

[andrii0lomakin]

Hopefully, we will provide this contribution later after the initial inference implementation is ready.

[jjfumero]

Yeah, I expected that host-device transfer would be a bottleneck. I am working on a design that avoids that from scratch. I also work on other optimizations like Flashed Attention 2.0 and so on. That is why I raised the question of support of tensor cores in TornadoVM.

That would be awesome. Keep us posted about your progress.

[andrii0lomakin]

BTW you disable compressed pointers. Is this option really needed? Imagine an application that uses TornadoVM as an embedded inference engine; would it not cause an explosion in heap consumption?

[andrii0lomakin]

I mean native memory consumption of course.

[jjfumero]

Good catch. I think we do not need this in TornadoVM anymore. So you can disable it.

[andrii0lomakin]

I will try it once I have more or less a working implementation and will provide my first meaningful PR :-)