org.apache.thrift.TApplicationException: Invalid method name: 'get_table'

[wangchanning]

Query engine

spark 3.5.4
iceberg 1.8.1

Question

I'm trying to use Hive 4.0.1 as the metastore service for Iceberg tables, with the metastore backed by a remote PostgreSQL database. My compute engine is Spark 3.5, and the Iceberg runtime version is 1.8.1.

When attempting to write data to an Iceberg table using the Hive catalog, I encounter the following error:

java.lang.RuntimeException: Failed to get table info from metastore db.sample
	at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:171)
	at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:97)
	at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:80)
	at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:47)
	at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908)
	at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404)
	at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387)
	at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
	at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
	at org.apache.iceberg.CachingCatalog.loadTable(CachingCatalog.java:166)
	at org.apache.iceberg.spark.SparkCatalog.load(SparkCatalog.java:643)
	at org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:159)
	at org.apache.spark.sql.connector.catalog.TableCatalog.tableExists(TableCatalog.java:185)
	at org.apache.spark.sql.execution.datasources.v2.CreateTableExec.run(CreateTableExec.scala:42)
	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:107)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:201)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:108)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:66)
	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:107)
	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:461)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:461)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:32)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:437)
	at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:98)
	at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:85)
	at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:83)
	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:220)
	at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:638)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:629)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:659)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table'
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1514)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1500)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1346)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:169)
	at com.sun.proxy.$Proxy30.getTable(Unknown Source)
	at org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:158)
	at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
	at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
	at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122)
	at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:158)
	... 57 more

Here's the code I'm using to reproduce the problem:

from pyspark import SparkConf
from pyspark.sql import SparkSession


conf = SparkConf().setMaster("local[6]") \
                  .set("spark.driver.memory", "4g") \
                  .set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
                  .set("spark.sql.catalog.hive_prod","org.apache.iceberg.spark.SparkCatalog") \
                  .set("spark.sql.catalog.hive_prod.type", "hive") \
                  .set("spark.sql.catalog.hive_prod.uri", "thrift://10.1.22.57:9083")

spark = SparkSession.builder.config(conf=conf).getOrCreate()

# spark.sql("CREATE DATABASE hive_prod.db")

spark.sql("""
    CREATE TABLE IF NOT EXISTS hive_prod.db.sample (
        id bigint NOT NULL COMMENT 'unique id',
        data string)
    USING iceberg
""")

spark.sql("INSERT INTO hive_prod.db.sample VALUES (1, 'a'), (2, 'b'), (3, 'c')")
spark.sql("""
    SELECT count(1) as count, data
    FROM hive_prod.db.sample
    GROUP BY data;
""").show()

[manuzhang]

We don't support hive 4 metastore yet.

[pan3793]

The corresponding API was deleted in HIVE-26537 (apache/hive#3599), while Spark 3.3-3.5 use Hive 2.3.9, Spark 4.0 uses Hive 2.3.10, as Hive 2.3 is marked EOL, it won't work unless Hive brings back these API.

Table get_table(1:string dbname, 2:string tbl_name)
               throws (1:MetaException o1, 2:NoSuchObjectException o2)

cc @pvary @zhangbutao

[zhangbutao]

Refer to this Doc: https://spark.apache.org/docs/3.5.1/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore
This can let Spark work with Hive4.0.1(with HIVE-26537), although this is not an elegant solution. :(

I know HIVE-26537 is a incompatible change. For the api change(get_table -> get_table_req), not only Spark but also Presto has similar troubles. https://github.com/prestodb/presto/pull/24571/files#diff-6aa1466123815b5f316aa052666749a0c8f58d5e12c00dff862a20a00f2ae38bR207

However, this API was already declared as obsolete in an earlier version. Therefore, for Hive, it is a reasonable operation to remove this API in Hive 4.0.1. So, for the downstream OSS, some incompatible upgrade changes should be made.

BTW, if some one raises a PR or email in Hive commmunity to discuss this issue, it would be even better.

[pan3793]

@zhangbutao The IsolatedClassLoader does not work for this case, Iceberg always uses the Spark built-in Hive classes.

[pvary]

CC: @wypoon who is working on the Hive 2,3 and 4 testing.
Based on this we can't really run the same code against Hive 4 and Hive 2 clients.

[wypoon]

@pvary I think this means that when using a Hive 4 HMS, you need to use a Hive 4 HMS client; it doesn't mean that the same Iceberg hive-metastore code cannot be run against both Hive 2 and Hive 4.
This line in Iceberg's HiveTableOperations:

      Table table = metaClients.run(client -> client.getTable(database, tableName));

will use a different client depending on the version of Hive you use, and the client.getTable call will be different depending on the version of Hive, right?

   Table getTable(@NotNull String dbName, @NotNull String tableName) throws TException {
-    return client.get_table(dbName, tableName);
+    GetTableRequest req = new GetTableRequest(dbName, tableName);
+    return client.get_table_req(req).getTable();
   }

If my understanding is correct, the work I'm doing is fine.

[wypoon]

On the other hand, IIUC, this means that the over-the-wire protocol (Thrift) compatibility is broken, which is why a Hive 2 client cannot be used with a Hive 4 HMS. Is this compatibility something Hive is meant to guarantee? @zhangbutao

[zhangbutao]

@wypoon Yes, IMO, Hive 4 is a big major version, which may have some incompatible changes compared with the previous version(Hive2&Hive3). Although personally I think we should do our best to ensure backward compatibility as much as possible, However, there are too many redundant and deprecated codes in Hive. These codes need to be cleaned up to ensure the sustainability of future code development. This is precisely what HIVE-26537 is doing.

@deniskuzZ @dengzhhu653 Any other suggestion about this incompatible change?

On the other hand, IIUC, this means that the over-the-wire protocol (Thrift) compatibility is broken, which is why a Hive 2 client cannot be used with a Hive 4 HMS. Is this compatibility something Hive is meant to guarantee? @zhangbutao

[pvary]

@zhangbutao, @wypoon: We are safe until the IMetaStoreClient keeps the old deprecated methods, and MetastoreUtil.alterTable(IMetaStoreClient client, String databaseName, String tblName, Table table, Map<String, String> extraEnv) is kept. What are the plans there? Some of the IMetaStoreClient methods are deprecated.

[shorrocka]

Hi Everyone, I am catching up on this but it seems as though support for using a hive 4.0.1 metastore with spark 4.0 is not included in iceberg even in the latest release when building from source that includes spark 4.0 support? I can use spark and query regular hive tables in spark but not when using the iceberg extension. Just seeing if there is an update or way to add in that support. Thanks in advance.

[pan3793]

@shorrocka I think the simplest way is to rebuild the latest Hive branch-2.3 with reverting apache/hive@c78ff81, to replace Spark's Hive jars

[shorrocka]

I think the error I am seeing is different than the one that the PR you linked fixes. When I try and query an iceberg table in the hive 4.0.1 metastore I get:
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table'
So I think I would need the opposite to have it be the 'get_table_req' method and use a newer version of the hive-metastore client jars. Would it be as simple as a drop and replace or is iceberg set to use a specific version of those jars?
Thanks

[deniskuzZ]

@zhangbutao, @wypoon: We are safe until the IMetaStoreClient keeps the old deprecated methods, and MetastoreUtil.alterTable(IMetaStoreClient client, String databaseName, String tblName, Table table, Map<String, String> extraEnv) is kept. What are the plans there? Some of the IMetaStoreClient methods are deprecated.

@pvary, we're actively working on modernizing HMS and intend to remove the deprecated methods in an upcoming release - either 4.2 or 5.0
HIVE-27473, HIVE-20189, HIVE-29072, HIVE-12679, HIVE-28879

[deniskuzZ]

@shorrocka try https://mvnrepository.com/artifact/org.apache.hive/hive-iceberg-catalog/4.0.1 instead of iceberg-catalog jar