Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

BIG IDEAS

F E B R U A RY 2 7 , 2 0 1 9 L A S T U P D AT E D : N O V E M B E R 4 , 2 0 1 9

serverless framework

® mission
critical in 90 percent of companies

1 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

Serverless functions provide a synergistic relationship with event streaming

applications; they behave differently with respect to streaming workloads but are both
event driven

part 1

Overview
What is FaaS?
Event-first FaaS?

2 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

FaaS for streaming processing

FaaS as part of the event-driven, streaming architecture
Next Steps
What is next for FaaS?

Journey to Event Driven – Part 1: Why Event-First Thinking Changes Everything

Journey to Event Driven – Part 2: Programming Models for the Event-Driven Architecture
Journey to Event Driven – Part 4: Four Pillars of Event Streaming Microservices

What is FaaS?

3 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

AWS Step Functions Azure Durable Functions

part 1

Event-first FaaS

connector

4 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

FaaS for stream processing

Attribute Native stream processing FaaS processing

Eventing model Async Function limit (AWS 1,000 soft limit)

Latency Low (<10 ms) High (cold: 5 s), (hot: 25–300 ms)

Throughput High Medium high

Elasticity Externally driven, per node Native, (per function instance)

Stream correctness Yes No

Stateless operation (filter, enrich,

Yes Yes
transform)

Stateful operations (window, aggregate) Yes Bespoke (build it yourself)

Yes, but choose the use case

Stream patterns (fan out, fan in/join) Yes
appropriately

5 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

Consumption based with high residual

Cost Consumption based
(always needing a processor running)

Runtime VM, container, server FaaS provider

Provider dependent: 500 MB storage,

Runtime limits Bespoke
128 MB → 3,008 MB memory

We characterize performance in terms of scalability, cold start latency, and resource

efficiency, with highlights including that AWS Lambda adopts a bin-packing-like
strategy to maximize VM memory utilization, that severe contention between functions
can arise in AWS and Azure, and that Google had bugs that allow customers to use
resources for free.
Peeking Behind the Curtains of Serverless Platforms, 2018

Low latency: for a warm start, AWS: 25

ms, Google: 79 ms and Azure: 320 ms
payload limit payload limit YES
High throughput:

YES
Stateless operation: YES
Stream-oriented stateful operations:
NO
Non-streaming stateful operations:
YES

6 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

Stream correctness (order preservation):

NO
Stream patterns: NO

Serverless Streaming Architectures and Best Practices

AWS Lambda and serverless architectures are well-suited for stream processing
workloads which are often event-driven and have spiky or variable compute
requirements.
Serverless Streaming Architectures and Best Practices, 2018

7 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

A source stream running through concurrent FaaS invocations will break event ordering.

Caveat emptor: FaaS gives us an excellent solution where processing is atomic

(stateless), reliable latency is not a concern (1–100 ms) and the order of processing is
not important. Stateful processing is also suitable provided that it is against an external
resource and stale data concerns are understood.

FaaS event-driven principles:

In band but edge

In band, stateless and not latency sensitive
In band and enriched against external resources
Out of band, but edge:

8 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

Ad hoc requests, but not streaming:

FaaS as part of the event-driven, streaming architecture

The event stream of an auction system: item placement, item bidding and processing

9 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

/auction/items

/auction/bids

inband but edge

10 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

Next steps

What is next for FaaS?

11 of 12 20-04-2020, 12:16
Journey to Event Driven – Part 3: The Affinity Between Events, Streams... file:///C:/Users/GAURAVCHATURVEDI/Desktop/Kafka/Apache Kafka_...

Designing
Data-Intensive Applications Designing Event-Driven Systems The Future of Serverless
and Streaming

download the Confluent Platform

Other articles in this series

Journey to Event Driven – Part 1: Why Event-First Thinking Changes Everything
Journey to Event Driven – Part 2: Programming Models for the Event-Driven Architecture
Journey to Event Driven – Part 4: Four Pillars of Event Streaming Microservices

Twitter

12 of 12 20-04-2020, 12:16