Repository navigation
Pubsub: ClassNotFoundException org/eclipse/jetty/alpn/ALPN with versions 0.32, 0.35, ..., 0.38 #3044
Description
Activity
1) for me I could solve similar issue using
jetty-alpn-agentjar - see my reproducable codebase here - https://github.com/upd-machine/nlu-pipeline/issues/1 (I am using against google-dialogflow though)wget http://central.maven.org/maven2/org/mortbay/jetty/alpn/jetty-alpn-agent/2.0.7/jetty-alpn-agent-2.0.7.jar java -jar -javaagent:jetty-alpn-agent-2.0.7.jar nlu-intent-toolkit.jar agent1 "where is porcupine tree playing?" credentials.jsonBut that is not a prod ready code so i m not sure if that's the best way i will fix it. This might be helpful - https://github.com/grpc/grpc-java/blob/master/SECURITY.md#tls-with-jdk-jetty-alpnnpn
2) ALSO, their
google-cloud-compat-checkermight be useful. Its bit painful to use it because they use parent versions. I don't understand why dont make it a jar that can be used anywhere.the compat-checker tells if you have ALPN supported in your environment, example,
$ mvn exec:java [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Google Cloud Java Compatibility Checker 0.39.0-alpha [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker --- OS details: os.detected.name: osx os.detected.arch: x86_64 os.detected.classifier: osx-x86_64 os.detected.release: null os.detected.release.version: null JVM details: Java version: 1.8.0_151 Java specification version: 1.8 JVM bit mode: 64 OpenSSL details: open ssl is available: false ALPN is supported: false Checking compatibility... [PASS] This OS + architecture is supported. [PASS] 64-bit JVM is supported. [FAIL] Open SSL is NOT available Open SSL Unavailability cause: java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-osx-x86_64, netty-tcnative] at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178) at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403) at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85) at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58) at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282) at java.lang.Thread.run(Thread.java:748) [FAIL] Open SSL ALPN is NOT supported Result: FAIL Your environment is not supported by Forked Tomcat Native. See http://netty.io/wiki/forked-tomcat-native.html for details. This means that you won't be able to use grpc-based APIs, but http1-based APIs should still work.
That didn't seem to work for me. Neither
jetty-alpn-agentnorjetty-alpn-client. The problem is inio.grpc.netty.JettyTlsUtilwhich is doing aClass.forName("org.eclipse.jetty.alpn.ALPN", true, null);in theisJettyAlpnConfigured()method.@j256 I'm glad you figured this out, but this is rather surprising. We should get to the bottom of this.
- addedtype: questionRequest for information or clarification. Not an issue.Request for information or clarification. Not an issue.priority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.
on Mar 14, 2018 garrettjonesgoogle commented
on Mar 20, 2018 ContributorMore actionsALPN is a red herring of sorts. The ssl layer that's supposed to be used is netty-tcnative-boringssl-static, which should be built into grpc-netty-shaded. If you are facing issues with ssl not being available, it's more likely a platform issue. Could you run
google-cloud-compat-checkerlike @prayagupd suggested?Hi @j256 I took your Test Subscriber code and tested in my MacOS with java 7/java 8. Its working fine - Here's my testcode for both emitting and consuming events from google pubsub -
https://github.com/duwamish-os/eventstream-skipper-ii.java (runmvn exec:java. It will show you google-cloud logs as well). Please let me know if you have different settings than how I tested.Dependency jars same as yours;
mvn dependency:tree [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building eventstream-skipper-java 1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ eventstream-skipper-java --- [INFO] com.eventstream.googlepubsub:eventstream-skipper-java:jar:1.0-SNAPSHOT [INFO] +- com.google.cloud:google-cloud-pubsub:jar:0.39.0-beta:compile [INFO] | +- com.google.cloud:google-cloud-core:jar:1.21.0:compile [INFO] | | +- com.google.guava:guava:jar:20.0:compile [INFO] | | +- joda-time:joda-time:jar:2.9.2:compile [INFO] | | +- com.google.http-client:google-http-client:jar:1.23.0:compile [INFO] | | | \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile [INFO] | | | +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile [INFO] | | | +- commons-logging:commons-logging:jar:1.1.1:compile [INFO] | | | \- commons-codec:commons-codec:jar:1.3:compile [INFO] | | +- com.google.code.findbugs:jsr305:jar:3.0.1:compile [INFO] | | +- com.google.api:api-common:jar:1.4.0:compile [INFO] | | +- com.google.api:gax:jar:1.19.0:compile [INFO] | | | +- org.threeten:threetenbp:jar:1.3.3:compile [INFO] | | | \- com.google.auth:google-auth-library-oauth2-http:jar:0.9.0:compile [INFO] | | | \- com.google.http-client:google-http-client-jackson2:jar:1.19.0:compile [INFO] | | | \- com.fasterxml.jackson.core:jackson-core:jar:2.1.3:compile [INFO] | | +- com.google.protobuf:protobuf-java-util:jar:3.5.1:compile [INFO] | | | \- com.google.code.gson:gson:jar:2.7:compile [INFO] | | +- com.google.api.grpc:proto-google-common-protos:jar:1.3.0:compile [INFO] | | \- com.google.api.grpc:proto-google-iam-v1:jar:0.4.0:compile [INFO] | +- com.google.cloud:google-cloud-core-grpc:jar:1.21.0:compile [INFO] | | +- com.google.auth:google-auth-library-credentials:jar:0.9.0:compile [INFO] | | +- com.google.protobuf:protobuf-java:jar:3.5.1:compile [INFO] | | +- io.grpc:grpc-protobuf:jar:1.9.0:compile [INFO] | | | \- io.grpc:grpc-protobuf-lite:jar:1.9.0:compile [INFO] | | +- io.grpc:grpc-context:jar:1.9.0:compile [INFO] | | \- com.google.api:gax-grpc:jar:1.19.0:compile [INFO] | +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.4.0:compile [INFO] | +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.4.0:compile [INFO] | +- io.grpc:grpc-netty-shaded:jar:1.9.0:compile [INFO] | | \- io.grpc:grpc-core:jar:1.9.0:compile (version selected from constraint [1.9.0,1.9.0]) [INFO] | | +- com.google.errorprone:error_prone_annotations:jar:2.1.2:compile [INFO] | | +- com.google.instrumentation:instrumentation-api:jar:0.4.3:compile [INFO] | | +- io.opencensus:opencensus-api:jar:0.10.0:compile [INFO] | | \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.10.0:compile [INFO] | +- io.grpc:grpc-stub:jar:1.9.0:compile [INFO] | +- io.grpc:grpc-auth:jar:1.9.0:compile [INFO] | \- com.google.auto.value:auto-value:jar:1.4:compile [INFO] \- com.google.cloud:google-cloud-logging-logback:jar:0.39.0-alpha:compile [INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile [INFO] | +- ch.qos.logback:logback-core:jar:1.2.3:compile [INFO] | \- org.slf4j:slf4j-api:jar:1.7.25:compile [INFO] \- com.google.cloud:google-cloud-logging:jar:1.21.0:compile [INFO] \- com.google.api.grpc:proto-google-cloud-logging-v2:jar:0.4.0:compile [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.335 s [INFO] Finished at: 2018-03-20T23:13:01-07:00 [INFO] Final Memory: 15M/309M [INFO] ------------------------------------------------------------------------
platform config is as below
$ mvn exec:java [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Google Cloud Java Compatibility Checker 0.39.0-alpha [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker --- OS details: os.detected.name: osx os.detected.arch: x86_64 os.detected.classifier: osx-x86_64 os.detected.release: null os.detected.release.version: null JVM details: Java version: 1.7.0_80 Java specification version: 1.7 JVM bit mode: 64 OpenSSL details: open ssl is available: false ALPN is supported: false Checking compatibility... [PASS] This OS + architecture is supported. [PASS] 64-bit JVM is supported. [FAIL] Open SSL is NOT available Open SSL Unavailability cause: java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-osx-x86_64, netty-tcnative] at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178) at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403) at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85) at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58) at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282) at java.lang.Thread.run(Thread.java:745) [FAIL] Open SSL ALPN is NOT supported [WARN] gRPC doesn't work on Google App Engine Standard under Java 1.7 Result: FAIL Your environment is not supported by Forked Tomcat Native. See http://netty.io/wiki/forked-tomcat-native.html for details. This means that you won't be able to use grpc-based APIs, but http1-based APIs should still work.
Ok. Swapping this back in. Thanks for the help @prayagupd . I'm getting the following from the compatibility checker which seems to match your's.
google-cloud-compat-checker {localhost} mvn exec:java [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [INFO] Installing Nexus Staging features: [INFO] ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin [INFO] [INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder with a thread count of 1 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Google Cloud Java Compatibility Checker 0.40.1-alpha-SNAPSHOT [INFO] ------------------------------------------------------------------------ ```[INFO] [INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker --- OS details: os.detected.name: linux os.detected.arch: x86_64 os.detected.classifier: linux-x86_64 os.detected.release: "rhel" os.detected.release.version: 7.3 JVM details: Java version: 1.7.0_79 Java specification version: 1.7 JVM bit mode: 64 OpenSSL details: open ssl is available: false ALPN is supported: false Checking compatibility... [PASS] This OS + architecture is supported. [PASS] 64-bit JVM is supported. [FAIL] Open SSL is NOT available Open SSL Unavailability cause: java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora, netty-tcnative] at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178) at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403) at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85) at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58) at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282) at java.lang.Thread.run(Thread.java:745) [FAIL] Open SSL ALPN is NOT supported [WARN] gRPC doesn't work on Google App Engine Standard under Java 1.7 Result: FAIL Your environment is not supported by Forked Tomcat Native. See http://netty.io/wiki/forked-tomcat-native.html for details. This means that you won't be able to use grpc-based APIs, but http1-based APIs should still work.However, when I go to run your program I get similar errors to mine and I'm seeing the following in my logs.
17:18:03.502 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless the application has already loaded the symbols by some other means. See http://netty.io/wiki/forked-tomcat-native.html for more information. java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_32, netty_tcnative_linux_x86_32_fedora, netty_tcnative_x86_32, netty_tcnative]Ultimately I get the followign with the -javaagent:jetty-alpn-agent-2.0.7.jar argument on the commandline.
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)I'm running on a:
> uname -a Linux gwatson-l 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 13 10:46:25 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux > cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.3 (Maipo)Looks like none of the shared libraries load for me. I see a bunch of log messages like:
17:18:03.492 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_linux_x86_32', trying other loading mechanism. java.lang.UnsatisfiedLinkError: no io_grpc_netty_shaded_netty_tcnative_linux_x86_32 in java.library.path 17:18:03.493 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_linux_x86_32 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp java.lang.UnsatisfiedLinkError: no io_grpc_netty_shaded_netty_tcnative_linux_x86_32 in java.library.path 17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_32', trying next name... 17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_linux_x86_32_fedora', trying other loading mechanism. 17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_linux_x86_32_fedora cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp 17:18:03.495 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_32_fedora', trying next name... 17:18:03.495 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_x86_32', trying other loading mechanism. 17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_x86_32 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp 17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_x86_32', trying next name... 17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative', trying other loading mechanism. 17:18:03.499 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp 17:18:03.499 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative', trying next name... 17:18:03.502 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless the application has already loaded the symbols by some other means. See http://netty.io/wiki/forked-tomcat-native.html for more information. 17:18:03.503 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to initialize netty-tcnative; OpenSslEngine will be unavailable. See http://netty.io/wiki/forked-tomcat-native.html for more information.That's interesting because I'm on a non-fedora 64-bit machine but the messages seem to be trying to load a 32-bit version. Let me see if I can force that selection. I've tried the static linked stuff and forcing the native classifier
<classifier>linux-x86_64</classifier>.Wow, this just came up again on another developer's box. Related to grpc/grpc-java#3025
garrettjonesgoogle commented
on Apr 18, 2018 ContributorMore actions@j256 , the real error that is relevant is "gRPC doesn't work on Google App Engine Standard under Java 1.7". The easiest way to fix this is to upgrade to Java 1.8 on App Engine, which should work fine.
garrettjonesgoogle commented
on Apr 18, 2018 ContributorMore actionsThis applies to @prayagupd 's case too.
Moving to Java 8 just isn't possible given my corporate software requirements.
garrettjonesgoogle commented
on Apr 23, 2018 ContributorMore actionsThat is very unfortunate - the Java 7 runtime of App Engine will be entirely shut down in January next year (2019): https://cloud.google.com/appengine/docs/deprecations/java7
Ok. I think I've finally figured this out. We run with a lot of different JREs on our servers supporting various Java variants and I believe this is a 32bit/64bit issue and not a Java7 versus 8 issue. I'm going to mark this closed but I'll add some details here as I get them. Thanks for all of the help.
Reacted by PrayagFor the record, I continue to have problems with this on older build boxes running older variants of LibC, mismatched 32-bit and 64-bit JREs, and mismatched JRE versions. It's tricky. Older servers are especially susceptible to this.
Also, older servers with glibc < version 2.7 will not work because of netty requirements. Yes, sigh, in 2018 my company has to support our code on these older servers.
- added a commit that references this issue
on Feb 20, 2026 - added a commit that references this issue
on Mar 9, 2026 - added a commit that references this issue
on Mar 20, 2026
Although I've seen similar issues, they all seem to be closed, I'm still seeing the following error with recent versions of this API. Related to: #2740 #3036
My test program is doing the following:
Here's my maven pom.xml dependency. Dependency tree attached:
Any idea what I'm doing wrong? Can I even run this outside of the Google network? Please excuse the obvious ignorance of this stuff.
dependencies.txt