spark sql with hive

最新推荐文章于 2024-06-01 08:53:20 发布

code_GK

最新推荐文章于 2024-06-01 08:53:20 发布

阅读量885

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.csdn.net/GK_kk/article/details/68924833

本文介绍如何配置Spark 1.5.0版本的spark-env.sh文件，并详细展示了核心参数设置，如SPARK_MASTER_IP、SPARK_WORKER_CORES等。此外，还涉及了如何启动Spark集群及运行Spark Pi示例程序的方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark standalone

spark-1.5.0-bin-hadoop2.6

spark-env.sh配置

export SPARK_MASTER_IP=172.16.31.11

export SPARK_MASTER_PORT=7077

export SPARK_WORKER_CORES=1

export SPARK_WORDER_INSTANCES=1

export SPARK_WORKER_MEMORY=1g

export MASTER=spark://172.16.31.11:7077

slaves

MRguan

启动

./sbin/start-all.sh

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client lib/spark-examples-1.5.0-hadoop2.6.0.jar

spark sql with hive

spark-env.sh添加

export HADOOP_CONF_DIR=/opt/beh/core/hadoop/etc/hadoop

export YARN_CONF_DIR=/opt/beh/core/hadoop/etc/hadoop

export HADOOP_HOME=/opt/beh/core/hadoop

export SPARK_JAR=/opt/beh/core/spark/lib/spark-assembly-1.5.0-hadoop2.6.0.jar

export SPARK_CLASSPATH=/opt/beh/core/spark/lib/mysql-connector-java-5.1.30-bin.jar

将hive-site.xml和hive-log4j.properties复制到$SPARK_HOME/conf下

将 $HIVE_HOME/lib下 mysql-connector-java-5.1.30-bin.jar复制到$SPARK_HOME/lib下

注意版本不然报错

17/02/28 09:12:48 WARN HiveMetaStore: Retrying creating default database after error: Error creating transactional connection factory

javax.jdo.JDOFatalInternalException: Error creating transactional connection factory