SparkSteaming整合kafka和redis 默认偏移量

最新推荐文章于 2022-04-09 14:11:00 发布

JNWsong

最新推荐文章于 2022-04-09 14:11:00 发布

阅读量224

点赞数

CC 4.0 BY-SA版权

分类专栏： sparkstreaming

本文链接：https://blog.csdn.net/java_creatMylief/article/details/113916840

sparkstreaming 专栏收录该内容

6 篇文章

订阅专栏

该博客介绍了如何使用Spark Streaming与Kafka进行数据摄入，并结合Redis进行实时词频统计。通过配置Kafka消费者参数，设置从最早消息开始消费，并启用自动提交偏移量。在Spark Streaming中，数据经过处理后，结果存储到Redis的哈希表中，实现了数据的实时更新和持久化。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

SparkSteaming整合kafka和redis，这里用的是默认偏移量

[root@dream1 ~]# kafka-console-producer.sh --broker-list dream1:9092,dream2:9092,dream3:9092 --topic test
>hello jeery spark tom hello
>hello spark
>hello
>hello tome^H
>heool^H^H
>tom jerry
>jerry
>spark

package com.ws.sparkstreaming

import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.spark.SparkConf
import org.apache.spark.streaming.dstream.{DStream, InputDStream}
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies}
import org.apache.spark.streaming.{Seconds, StreamingContext}
import redis.clients.jedis.Jedis

object WordCountJoinKafkaRedis {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName(this.getClass.getSimpleName).setMaster("local[*]")
    val ssc = new StreamingContext(conf,Seconds(5))
    val kafkaParams: Map[String, Object] = Map[String,Object](
      "bootstrap.servers"->"dream1:9092,dream2:9092,dream3:9092",// kafka地址
      "key.deserializer"->"org.apache.kafka.common.serialization.StringDeserializer", // 设置反序列化组件
      "value.deserializer"->"org.apache.kafka.common.serialization.StringDeserializer",
      "group.id"->"1", // 消费者组
      "auto.offset.reset"->"earliest", // 指定消费者从哪开始消费[latest,earliest]
      "enable.auto.commit"->"true" // 是否自动提交偏移量，默认是true
    )
    val topic: Iterable[String] = Array("test").toIterable
    val rows: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe[String, String](topic, kafkaParams)
    )
    val lines = rows.map(r => r.value())
    val reduce: DStream[(String, Int)] = lines.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _)
    val jedis = new Jedis("dream1")
    reduce.foreachRDD(rdd=>{
      val tuples = rdd.collect()
      for (elem <- tuples) {
        jedis.hincrBy("wordcount",elem._1,elem._2)
      }
    })
    ssc.start()
    ssc.awaitTermination()
  }
}