Using FLiP with influxdb for edgeai iot at scale 2022
1 like502 views
The document discusses the integration of Apache Pulsar, Flink, and NiFi for building scalable edge AI applications and IoT data streams. It highlights the benefits of using Pulsar for unified messaging and streaming, showcasing its features like multi-tenancy, scalability, and geo-replication. Additionally, it presents various use cases and success stories, emphasizing the significant performance improvements achieved by organizations that have adopted this technology.
More Related Content
![[March sn meetup] apache pulsar + apache nifi for cloud data lake](https://cdn.slidesharecdn.com/ss_thumbnails/marchsnmeetupapachepulsarapachenififorclouddatalake-220311125943-thumbnail.jpg?width=560&fit=bounds)
Similar to Using FLiP with influxdb for edgeai iot at scale 2022 (20)
![[AerospikeRoadshow] Apache Pulsar Unifies Streaming and Messaging for Real-Ti...](https://cdn.slidesharecdn.com/ss_thumbnails/aerospikeroadshowapachepulsarunifiesstreamingandmessagingforreal-timedata2-221103174214-18ce3fa2-thumbnail.jpg?width=560&fit=bounds)
Using FLiP with influxdb for edgeai iot at scale 2022
- 1. Using FLiP with InfluxDB for Edge AI IoT at Scale Tim Spann | Developer Advocate +
- 2. streamnative.io + Tim Spann Developer Advocate Tim Spann, Developer Advocate at StreamNative ● FLiP(N) Stack = Flink, Pulsar and NiFI Stack ● Streaming Systems & Data Architecture Expert ● Experience: ○ 15+ years of experience with streaming technologies including Pulsar, Flink, Spark, NiFi, Kafka, Big Data, Cloud, MXNet, IoT and more. ○ Today, he helps to grow the Pulsar community sharing rich technical knowledge and experience at both global conferences and through individual conversations.
- 3. streamnative.io ● Learn how to build an end-to-end streaming edge app ● How to pull messages from Pulsar topics and persists the messages to InfluxDB ● Building a data stream for IoT with NiFi and InfluxDB ● Using Apache Flink + Apache Pulsar ● Using Apache Spark + Apache Pulsar Agenda +
- 4. streamnative.io FLiP Stack Weekly This week in Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark and open source friends. https://bit.ly/32dAJft +
- 5. streamnative.io ● Apache Flink ● Apache Pulsar ● Pulsar Functions ● StreamNative's Flink Connector for Pulsar ● Apache NiFi ● Apache Spark ● Python, Java, Golang FLiP(N)(S) Stack +
- 6. streamnative.io Apache Pulsar is a Cloud-Native Messaging and Event-Streaming Platform. +
- 7. Unified Messaging Model Simplify your data infrastructure and enable new use cases with queuing and streaming capabilities in one platform. Multi-tenancy Enable multiple user groups to share the same cluster, either via access control, or in entirely different namespaces. Scalability Decoupled data computing and storage enable horizontal scaling to handle data scale and management complexity. Geo-replication Support for multi-datacenter replication with both asynchronous and synchronous replication for built-in disaster recovery. Tiered storage Enable historical data to be offloaded to cloud-native storage and store event streams for indefinite periods of time. Pulsar Benefits
- 8. Top Pulsar Use Cases #1 Message Queuing #2 Data Streaming ✓ Not built for the cloud ✓ Single tenant systems ✓ Monolithic architecture couples compute with storage ✓ Lack of geo replication support
- 9. Messaging Ideal for work queues that do not require tasks to be performed in a particular order—for example, sending one email message to many recipients. RabbitMQ and Amazon SQS are examples of popular queue-based message systems. Pulsar: Unified Messaging + Data Streaming
- 10. Messaging Ideal for work queues that do not require tasks to be performed in a particular order—for example, sending one email message to many recipients. RabbitMQ and Amazon SQS are examples of popular queue-based message systems. Pulsar: Unified Messaging + Data Streaming .. and Streaming Works best in situations where the order of messages is important—for example, data ingestion. Kafka and Amazon Kinesis are examples of messaging systems that use streaming semantics for consuming messages.
- 11. Unified Messaging Model Streaming Messaging Producer 1 Producer 2 Pulsar Topic/Partition m0 m1 m2 m3 m4 Consumer D-1 Consumer D-2 Consumer D-3 Subscription D < k 2 , v 1 > < k 2 , v 3 > <k3 ,v2 > < k 1 , v 0 > < k 1 , v 4 > Key-Shared Consumer C-1 Consumer C-2 Consumer C-3 Subscription C m1 m2 m3 m4 m0 Shared Failover Consumer B-1 Consumer B-0 Subscription B m1 m2 m3 m4 m0 In case of failure in Consumer B-0 Consumer A-1 Consumer A-0 Subscription A m1 m2 m3 m4 m0 Exclusive X +
- 12. streamnative.io Use Cases Multi-Tenant Data Infrastructure AdTech Fraud Detection Connected Car IoT Analytics Microservices Development
- 14. Adopted Pulsar to replace Kafka in their DSP (Data Streaming Platform). ● 1.5-2x lower in capex cost ● 5-50x improvement in latency ● 2-3x lower in opex due Adopted Pulsar to power their billing platform, Midas, which processing hundreds of billions of financial transactions daily. Adoption then expanded to Tencent’s Federated Learning Platform and Tencent Gaming. Applied Materials is one of the biggest semiconductor hardware and software supplier in the industry. They adopted Pulsar to enable them to build a message bus to tie all of their data together. They previously used Tibco. Pulsar Adoption Use Cases
- 15. streamnative.io Passionate and dedicated team. Founded by the original developers of Apache Pulsar. StreamNative helps teams to capture, manage, and leverage data using Pulsar’s unified messaging and streaming platform.
- 16. Founded By The Creators Of Apache Pulsar Sijie Guo ASF Member Pulsar/BookKeeper PMC Founder and CEO Jia Zhai Pulsar/BookKeeper PMC Co-Founder ✓ Original creators of Apache Pulsar & BookKeeper ✓ Operated the largest Pulsar/BookKeeper cluster ✓ Data veterans with extensive industry experience Matteo Merli ASF Member Pulsar/BookKeeper PMC CTO
- 17. Powered by Apache Pulsar, StreamNative provides a cloud-native, real-time messaging and streaming platform to support multi-cloud and hybrid cloud strategies. StreamNative Cloud Flink SQL Cloud-Native
- 18. Apache Pulsar ● Serverless computing framework. ● Unbounded storage, multi-tiered architecture, and tiered-storage. ● Streaming & Pub/Sub messaging semantics. ● Multi-protocol support. +
- 19. streamnative.io Why Apache Pulsar? Unified Messaging Platform Guaranteed Message Delivery Resiliency Infinite Scalability +
- 20. streamnative.io Connectivity • Functions - Lightweight Stream Processing (Java, Python, Go) • Connectors - Sources & Sinks (InfluxDB, Kafka, S3, Kinesis, Lambda, …) • Protocol Handlers - AoP (AMQP), KoP (Kafka), MoP (MQTT) • Processing Engines - Flink, Spark, Presto/Trino via Pulsar SQL • Data Offloaders - Tiered Storage - (S3) hub.streamnative.io +
- 21. Use Cases ● Unified Messaging Platform ● AdTech ● Fraud Detection ● Connected Car ● IoT Analytics ● Microservices Development +
- 22. <-> Sensors <-> Streaming Edge Apps StreamNative Hub StreamNative Cloud Unified Batch and Stream COMPUTING Batch (Batch + Stream) Unified Batch and Stream STORAGE Offload (Queuing + Streaming) Tiered Storage Pulsar --- KoP --- MoP --- Websocket Pulsar Sink Streaming Edge Gateway Protocols <-> Sensors <-> Apps +
- 23. StreamNative Hub StreamNative Cloud Unified Batch and Stream COMPUTING Batch (Batch + Stream) Unified Batch and Stream STORAGE Offload (Queuing + Streaming) Apache Flink - Apache Pulsar - Apache NiFi <-> Devices <-> influxdb Tiered Storage Pulsar --- KoP --- MoP --- Websocket --- HTTP Pulsar Sink Pulsar Sink Streaming Edge Gateway Protocols End-to-End Streaming Edge App +
- 24. Why +
- 25. Analytics +
- 27. Analytics +
- 28. Analytics +
- 29. streamnative.io Apache Pulsar InfluxDB Sink https://pulsar.apache.org/docs/en/io-influxdb-sink/ +
- 30. streamnative.io Apache Pulsar InfluxDB Sink pulsar-admin sinks create --archive ./connectors/pulsar-io-influxdb-2.9.1.nar --name inflx-jtsn --sink-config-file conf/influxcloud.yml --inputs jtsninflx pulsar-admin sinks get --name inflx-jtsn pulsar-admin sinks status --name inflx-jtsn +
- 31. streamnative.io Apache Pulsar InfluxDB Sink +
- 32. streamnative.io InfluxDB Telegraf MQTT https://www.influxdata.com/integration/mqtt-monitoring/ https:/ /github.com/streamnative/mop MQTT on Pulsar (MoP) +
- 36. streamnative.io InfluxDB Telegraf RabbitMQ https://www.influxdata.com/integration/mqtt-monitoring/ + https://www.influxdata.com/integration/mqtt-monitoring/ https:/ /github.com/streamnative/aop AMQP on Pulsar (AoP) https:/ /hub.streamnative.io/connectors/amqp-1-0-sink/v2.7.1.1 https:/ /hub.streamnative.io/connectors/amqp-1-0-source/v2.7.1.1
- 37. streamnative.io Why Apache NiFi? https://www.influxdata.com/integration/mqtt-monitoring/ + https://www.influxdata.com/integration/mqtt-monitoring/ • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Hundreds of processors • Visual command and control • Over a 300 sources • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering • Version Control
- 38. streamnative.io Apache NiFi Pulsar Connector https://github.com/david-streamlio/pulsar-nifi-bundle +
- 39. streamnative.io Apache NiFi Pulsar Connector
- 40. streamnative.io Apache NiFi Pulsar Connector +
- 41. streamnative.io Apache NiFi Pulsar Connector +
- 42. streamnative.io InfluxDB via Apache NiFi + https://github.com/influxdata/nifi-influxdb-bundle https://www.influxdata.com/blog/building-a-data-stream-for-iot-with-nifi-and-influxdb/
- 43. streamnative.io ● Unified computing engine ● Batch processing is a special case of stream processing ● Stateful processing ● Massive Scalability ● Flink SQL for queries, inserts against Pulsar Topics ● Streaming Analytics ● Continuous SQL ● Continuous ETL ● Complex Event Processing ● Standard SQL Powered by Apache Calcite Why Apache Flink? +
- 58. ● https://github.com/tspannhw/FLiP-InfluxDB ● https://pulsar.apache.org/docs/en/io-influxdb-sink/ ● https://pulsar.apache.org/docs/en/io-overview/ ● https://github.com/tspannhw/Flip-solr ● https://github.com/tspannhw/Flip-transit ● https://github.com/tspannhw/Flip-iot ● https://www.datainmotion.dev/2020/10/flank-strea ming-edgeai-on-new-nvidia.html ● https://github.com/tspannhw/minifi-jetson-nano ● https://github.com/tspannhw/SpeakerProfile/ ● https://hub.streamnative.io/connectors/influxdb-sin k/2.5.1/ ● https://github.com/tspannhw/minifi-xaviernx/
- 59. Resources Now Available On-Demand Pulsar Training Academy.StreamNative Live 3-day Developers Training Save Your Spot! Free ebook Pulsar in Action Read it now
- 60. Let’s Keep in Touch! Timothy Spann Developer Advocate @PaasDev https://www.linkedin.com/in/timothyspann https://github.com/tspannhw +
- 61. Questions?