Skip to content

Pubsub: ClassNotFoundException org/eclipse/jetty/alpn/ALPN with versions 0.32, 0.35, ..., 0.38 #3044

Description

@j256

Although I've seen similar issues, they all seem to be closed, I'm still seeing the following error with recent versions of this API. Related to: #2740 #3036

Caused by: java.lang.ClassNotFoundException: org/eclipse/jetty/alpn/ALPN

My test program is doing the following:

ProjectSubscriptionName subName = ProjectSubscriptionName.of("project", "subscriber");
Builder builder = Subscriber.newBuilder(subName, new OurReceiver());
// also tried adding the following
// GoogleCredentials cred=GoogleCredentials.fromStream(new FileInputStream("creds.json"));
// builder.setCredentialsProvider(FixedCredentialsProvider.create(cred));
Subscriber subscriber = builder.build();
subscriber.startAsync().awaitRunning();

Here's my maven pom.xml dependency. Dependency tree attached:

<dependency>
	<groupId>com.google.cloud</groupId>
	<artifactId>google-cloud-pubsub</artifactId>
	<version>0.38.0-beta</version> <!-- also tried 32, 35, 36, 37 -->
</dependency>

Any idea what I'm doing wrong? Can I even run this outside of the Google network? Please excuse the obvious ignorance of this stuff.

dependencies.txt

Activity

  1. prayagupa commented on Mar 14, 2018

    @prayagupa

    1) for me I could solve similar issue using jetty-alpn-agent jar - see my reproducable codebase here - https://github.com/upd-machine/nlu-pipeline/issues/1 (I am using against google-dialogflow though)

    wget http://central.maven.org/maven2/org/mortbay/jetty/alpn/jetty-alpn-agent/2.0.7/jetty-alpn-agent-2.0.7.jar
    
    java -jar -javaagent:jetty-alpn-agent-2.0.7.jar nlu-intent-toolkit.jar agent1 "where is porcupine tree playing?" credentials.json 
    

    But that is not a prod ready code so i m not sure if that's the best way i will fix it. This might be helpful - https://github.com/grpc/grpc-java/blob/master/SECURITY.md#tls-with-jdk-jetty-alpnnpn

    2) ALSO, their google-cloud-compat-checker might be useful. Its bit painful to use it because they use parent versions. I don't understand why dont make it a jar that can be used anywhere.

    the compat-checker tells if you have ALPN supported in your environment, example,

    $ mvn exec:java
    [INFO] Scanning for projects...
    [INFO] 
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Google Cloud Java Compatibility Checker 0.39.0-alpha
    [INFO] ------------------------------------------------------------------------
    [INFO] 
    [INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker ---
    OS details:
      os.detected.name: osx
      os.detected.arch: x86_64
      os.detected.classifier: osx-x86_64
      os.detected.release: null
      os.detected.release.version: null
    JVM details:
      Java version: 1.8.0_151
      Java specification version: 1.8
      JVM bit mode: 64
    OpenSSL details:
      open ssl is available: false
      ALPN is supported: false
    Checking compatibility...
      [PASS] This OS + architecture is supported.
      [PASS] 64-bit JVM is supported.
      [FAIL] Open SSL is NOT available
             Open SSL Unavailability cause:
    java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-osx-x86_64, netty-tcnative]
    	at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178)
    	at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403)
    	at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85)
    	at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58)
    	at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
    	at java.lang.Thread.run(Thread.java:748)
      [FAIL] Open SSL ALPN is NOT supported
    Result: FAIL
      Your environment is not supported by Forked Tomcat Native.
      See http://netty.io/wiki/forked-tomcat-native.html for details.
      This means that you won't be able to use grpc-based APIs, but
      http1-based APIs should still work.
  2. j256 commented on Mar 14, 2018

    @j256
    Author

    That didn't seem to work for me. Neither jetty-alpn-agent nor jetty-alpn-client. The problem is in io.grpc.netty.JettyTlsUtil which is doing a Class.forName("org.eclipse.jetty.alpn.ALPN", true, null); in the isJettyAlpnConfigured() method.

  3. pongad commented on Mar 14, 2018

    @pongad
    Contributor

    @j256 I'm glad you figured this out, but this is rather surprising. We should get to the bottom of this.

  4. added
    type: questionRequest for information or clarification. Not an issue.
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on Mar 14, 2018
  5. garrettjonesgoogle commented on Mar 20, 2018

    @garrettjonesgoogle
    Contributor

    ALPN is a red herring of sorts. The ssl layer that's supposed to be used is netty-tcnative-boringssl-static, which should be built into grpc-netty-shaded. If you are facing issues with ssl not being available, it's more likely a platform issue. Could you run google-cloud-compat-checker like @prayagupd suggested?

  6. prayagupa commented on Mar 21, 2018

    @prayagupa

    Hi @j256 I took your Test Subscriber code and tested in my MacOS with java 7/java 8. Its working fine - Here's my testcode for both emitting and consuming events from google pubsub -
    https://github.com/duwamish-os/eventstream-skipper-ii.java (run mvn exec:java. It will show you google-cloud logs as well). Please let me know if you have different settings than how I tested.

    Dependency jars same as yours;

    mvn dependency:tree
    [INFO] Scanning for projects...
    [INFO] 
    [INFO] ------------------------------------------------------------------------
    [INFO] Building eventstream-skipper-java 1.0-SNAPSHOT
    [INFO] ------------------------------------------------------------------------
    [INFO] 
    [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ eventstream-skipper-java ---
    [INFO] com.eventstream.googlepubsub:eventstream-skipper-java:jar:1.0-SNAPSHOT
    [INFO] +- com.google.cloud:google-cloud-pubsub:jar:0.39.0-beta:compile
    [INFO] |  +- com.google.cloud:google-cloud-core:jar:1.21.0:compile
    [INFO] |  |  +- com.google.guava:guava:jar:20.0:compile
    [INFO] |  |  +- joda-time:joda-time:jar:2.9.2:compile
    [INFO] |  |  +- com.google.http-client:google-http-client:jar:1.23.0:compile
    [INFO] |  |  |  \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
    [INFO] |  |  |     +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
    [INFO] |  |  |     +- commons-logging:commons-logging:jar:1.1.1:compile
    [INFO] |  |  |     \- commons-codec:commons-codec:jar:1.3:compile
    [INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.1:compile
    [INFO] |  |  +- com.google.api:api-common:jar:1.4.0:compile
    [INFO] |  |  +- com.google.api:gax:jar:1.19.0:compile
    [INFO] |  |  |  +- org.threeten:threetenbp:jar:1.3.3:compile
    [INFO] |  |  |  \- com.google.auth:google-auth-library-oauth2-http:jar:0.9.0:compile
    [INFO] |  |  |     \- com.google.http-client:google-http-client-jackson2:jar:1.19.0:compile
    [INFO] |  |  |        \- com.fasterxml.jackson.core:jackson-core:jar:2.1.3:compile
    [INFO] |  |  +- com.google.protobuf:protobuf-java-util:jar:3.5.1:compile
    [INFO] |  |  |  \- com.google.code.gson:gson:jar:2.7:compile
    [INFO] |  |  +- com.google.api.grpc:proto-google-common-protos:jar:1.3.0:compile
    [INFO] |  |  \- com.google.api.grpc:proto-google-iam-v1:jar:0.4.0:compile
    [INFO] |  +- com.google.cloud:google-cloud-core-grpc:jar:1.21.0:compile
    [INFO] |  |  +- com.google.auth:google-auth-library-credentials:jar:0.9.0:compile
    [INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.5.1:compile
    [INFO] |  |  +- io.grpc:grpc-protobuf:jar:1.9.0:compile
    [INFO] |  |  |  \- io.grpc:grpc-protobuf-lite:jar:1.9.0:compile
    [INFO] |  |  +- io.grpc:grpc-context:jar:1.9.0:compile
    [INFO] |  |  \- com.google.api:gax-grpc:jar:1.19.0:compile
    [INFO] |  +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.4.0:compile
    [INFO] |  +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.4.0:compile
    [INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.9.0:compile
    [INFO] |  |  \- io.grpc:grpc-core:jar:1.9.0:compile (version selected from constraint [1.9.0,1.9.0])
    [INFO] |  |     +- com.google.errorprone:error_prone_annotations:jar:2.1.2:compile
    [INFO] |  |     +- com.google.instrumentation:instrumentation-api:jar:0.4.3:compile
    [INFO] |  |     +- io.opencensus:opencensus-api:jar:0.10.0:compile
    [INFO] |  |     \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.10.0:compile
    [INFO] |  +- io.grpc:grpc-stub:jar:1.9.0:compile
    [INFO] |  +- io.grpc:grpc-auth:jar:1.9.0:compile
    [INFO] |  \- com.google.auto.value:auto-value:jar:1.4:compile
    [INFO] \- com.google.cloud:google-cloud-logging-logback:jar:0.39.0-alpha:compile
    [INFO]    +- ch.qos.logback:logback-classic:jar:1.2.3:compile
    [INFO]    |  +- ch.qos.logback:logback-core:jar:1.2.3:compile
    [INFO]    |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
    [INFO]    \- com.google.cloud:google-cloud-logging:jar:1.21.0:compile
    [INFO]       \- com.google.api.grpc:proto-google-cloud-logging-v2:jar:0.4.0:compile
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 1.335 s
    [INFO] Finished at: 2018-03-20T23:13:01-07:00
    [INFO] Final Memory: 15M/309M
    [INFO] ------------------------------------------------------------------------

    platform config is as below

    $ mvn exec:java
    [INFO] Scanning for projects...
    [INFO] 
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Google Cloud Java Compatibility Checker 0.39.0-alpha
    [INFO] ------------------------------------------------------------------------
    [INFO] 
    [INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker ---
    OS details:
      os.detected.name: osx
      os.detected.arch: x86_64
      os.detected.classifier: osx-x86_64
      os.detected.release: null
      os.detected.release.version: null
    JVM details:
      Java version: 1.7.0_80
      Java specification version: 1.7
      JVM bit mode: 64
    OpenSSL details:
      open ssl is available: false
      ALPN is supported: false
    Checking compatibility...
      [PASS] This OS + architecture is supported.
      [PASS] 64-bit JVM is supported.
      [FAIL] Open SSL is NOT available
             Open SSL Unavailability cause:
    java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-osx-x86_64, netty-tcnative]
    	at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178)
    	at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403)
    	at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85)
    	at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58)
    	at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:606)
    	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
    	at java.lang.Thread.run(Thread.java:745)
      [FAIL] Open SSL ALPN is NOT supported
      [WARN] gRPC doesn't work on Google App Engine Standard under Java 1.7
    Result: FAIL
      Your environment is not supported by Forked Tomcat Native.
      See http://netty.io/wiki/forked-tomcat-native.html for details.
      This means that you won't be able to use grpc-based APIs, but
      http1-based APIs should still work.
  7. j256 commented on Mar 26, 2018

    @j256
    Author

    Ok. Swapping this back in. Thanks for the help @prayagupd . I'm getting the following from the compatibility checker which seems to match your's.

    google-cloud-compat-checker {localhost}  mvn exec:java
    [INFO] Scanning for projects...
    [INFO] Inspecting build with total of 1 modules...
    [INFO] Installing Nexus Staging features:
    [INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
    [INFO] 
    [INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder with a thread count of 1
    [INFO]                                                                         
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Google Cloud Java Compatibility Checker 0.40.1-alpha-SNAPSHOT
    [INFO] ------------------------------------------------------------------------
    
    ```[INFO] 
    [INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker ---
    OS details:
      os.detected.name: linux
      os.detected.arch: x86_64
      os.detected.classifier: linux-x86_64
      os.detected.release: "rhel"
      os.detected.release.version: 7.3
    JVM details:
      Java version: 1.7.0_79
      Java specification version: 1.7
      JVM bit mode: 64
    OpenSSL details:
      open ssl is available: false
      ALPN is supported: false
    Checking compatibility...
      [PASS] This OS + architecture is supported.
      [PASS] 64-bit JVM is supported.
      [FAIL] Open SSL is NOT available
             Open SSL Unavailability cause:
    java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora, netty-tcnative]
            at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178)
            at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403)
            at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85)
            at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58)
            at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
            at java.lang.Thread.run(Thread.java:745)
      [FAIL] Open SSL ALPN is NOT supported
      [WARN] gRPC doesn't work on Google App Engine Standard under Java 1.7
    Result: FAIL
      Your environment is not supported by Forked Tomcat Native.
      See http://netty.io/wiki/forked-tomcat-native.html for details.
      This means that you won't be able to use grpc-based APIs, but
      http1-based APIs should still work.
    

    However, when I go to run your program I get similar errors to mine and I'm seeing the following in my logs.

    17:18:03.502 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless the application has already loaded the symbols by some other means. See http://netty.io/wiki/forked-tomcat-native.html for more information.
    java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_32, netty_tcnative_linux_x86_32_fedora, netty_tcnative_x86_32, netty_tcnative]
    
    

    Ultimately I get the followign with the -javaagent:jetty-alpn-agent-2.0.7.jar argument on the commandline.

    Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
    

    I'm running on a:

    > uname -a
    Linux gwatson-l 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 13 10:46:25 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
    
    > cat /etc/redhat-release 
    Red Hat Enterprise Linux Server release 7.3 (Maipo)
    
    

    Looks like none of the shared libraries load for me. I see a bunch of log messages like:

    17:18:03.492 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_linux_x86_32', trying other loading mechanism.
    java.lang.UnsatisfiedLinkError: no io_grpc_netty_shaded_netty_tcnative_linux_x86_32 in java.library.path
    17:18:03.493 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_linux_x86_32 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
    java.lang.UnsatisfiedLinkError: no io_grpc_netty_shaded_netty_tcnative_linux_x86_32 in java.library.path
    17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_32', trying next name...
    17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_linux_x86_32_fedora', trying other loading mechanism.
    17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_linux_x86_32_fedora cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
    17:18:03.495 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_32_fedora', trying next name...
    17:18:03.495 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_x86_32', trying other loading mechanism.
    17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_x86_32 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
    17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_x86_32', trying next name...
    17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative', trying other loading mechanism.
    17:18:03.499 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
    17:18:03.499 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative', trying next name...
    17:18:03.502 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless the application has already loaded the symbols by some other means. See http://netty.io/wiki/forked-tomcat-native.html for more information.
    17:18:03.503 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to initialize netty-tcnative; OpenSslEngine will be unavailable. See http://netty.io/wiki/forked-tomcat-native.html for more information.
    

    That's interesting because I'm on a non-fedora 64-bit machine but the messages seem to be trying to load a 32-bit version. Let me see if I can force that selection. I've tried the static linked stuff and forcing the native classifier <classifier>linux-x86_64</classifier>.

  8. j256 commented on Apr 13, 2018

    @j256
    Author

    Wow, this just came up again on another developer's box. Related to grpc/grpc-java#3025

  9. garrettjonesgoogle commented on Apr 18, 2018

    @garrettjonesgoogle
    Contributor

    @j256 , the real error that is relevant is "gRPC doesn't work on Google App Engine Standard under Java 1.7". The easiest way to fix this is to upgrade to Java 1.8 on App Engine, which should work fine.

  10. garrettjonesgoogle commented on Apr 18, 2018

    @garrettjonesgoogle
    Contributor

    This applies to @prayagupd 's case too.

  11. j256 commented on Apr 23, 2018

    @j256
    Author

    Moving to Java 8 just isn't possible given my corporate software requirements.

  12. garrettjonesgoogle commented on Apr 23, 2018

    @garrettjonesgoogle
    Contributor

    That is very unfortunate - the Java 7 runtime of App Engine will be entirely shut down in January next year (2019): https://cloud.google.com/appengine/docs/deprecations/java7

  13. j256 commented on Apr 25, 2018

    @j256
    Author

    Ok. I think I've finally figured this out. We run with a lot of different JREs on our servers supporting various Java variants and I believe this is a 32bit/64bit issue and not a Java7 versus 8 issue. I'm going to mark this closed but I'll add some details here as I get them. Thanks for all of the help.

  14. j256 commented on Jun 8, 2018

    @j256
    Author

    For the record, I continue to have problems with this on older build boxes running older variants of LibC, mismatched 32-bit and 64-bit JREs, and mismatched JRE versions. It's tricky. Older servers are especially susceptible to this.

  15. j256 commented on Jul 16, 2018

    @j256
    Author

    Also, older servers with glibc < version 2.7 will not work because of netty requirements. Yes, sigh, in 2018 my company has to support our code on these older servers.

  16. added a commit that references this issue on Mar 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

dependenciespriority: p2Moderately-important priority. Fix may not be included in next release.type: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions