What are Serverless Data Pipelines?

Create a Kinesis Data Stream

Last Updated : 28 Mar, 2023

Pre-requisite: AWS Kinesis

A Kinesis data stream is a collection of data records that are continually and sequentially added to over time, like a stream. It is a real-time data streaming service offered by Amazon Web Services (AWS) that enables us to collect, process, and analyze large streams of data records in real time.

Following are a few key terms related to the Kinesis data stream:

Stream: A stream is a sequence of data records in real-time. A stream can have one or more shards.
Shard: A shard is a unit of capacity in a stream. Each shard can support up to 1,000 PUT records per second and up to 2 MiB of data per second.
Producer: A producer is an application that sends data records to a stream.
Consumer: A consumer is an application that reads data records from a stream.
Checkpoint: A checkpoint is a marker that a consumer application uses to track its progress through a stream.
Streams API: A set of API actions that enable you to create, read, and manage streams and their shards.
Data Retention Period: The amount of time that records are kept in a stream.

Steps to Create a Kinesis Data Stream

To create a Kinesis data stream in AWS (Amazon Web Services), you can use the AWS Management Console, the AWS Command Line Interface (CLI), or the AWS SDKs.

Here are the general steps to create a Kinesis data stream using the AWS Management Console:

Step 1: Open the Amazon Kinesis console in AWS and click on Get Started.

Kinesis console

Step 2: Click on the "Create data stream" button.

Create data stream

Step 3: Enter a name for your data stream in the "Name" field.

Step 4: Choose the number of shards for your data stream in the "Number of shards" field. The number of shards determines the capacity and cost of your data stream. Click on the "Create kinesis stream" button to create the data stream.

kinesis config

What are Serverless Data Pipelines?

Y

yadav3ankit

Improve

Article Tags :

Similar Reads

How to Create AWS Data Lake

In the data-driven world, organizations are flooded with large amounts of data that originate from various sources, ranging from structured databases to unstructured files and logs. In order to effectively make use of this data for insights and decisions, organisations need to have a storage system

Data Lake Architecture - System Design

"Data Lake Architecture" explores the foundational principles and practical steps for building a scalable and efficient data lake. It covers key components such as data ingestion, storage, processing, and governance to ensure effective management and analysis of large-scale, diverse data sets.Data L

What are Serverless Data Pipelines?

A serverless data pipeline is a modern approach to managing and processing large volumes of data without the need for traditional server management. Leveraging cloud services, serverless data pipelines automatically scale to handle data workloads, optimizing cost and performance. These pipelines ena

10 Ways to Create a Stream in Java

The Stream API, introduced in Java 8, it is used to process collections of objects. Stream is a sequence of objects, that supports many different methods which can be pipe lined to produce the desired result. The features of Java stream are â€“ A stream is not a data structure alternatively it takes i

How to Create First DAG in Airflow?

Directed Acyclic Graph (DAG) is a group of all individual tasks that we run in an ordered fashion. In other words, we can say that a DAG is a data pipeline in airflow. In a DAG:There is no loopEdges are directedKey Terminologies:Operator: The task in your DAG is called an operator. In airflow, the n

How to create a new request in Postman?

Postman is a development tool that is used for testing web APIs i.e. Application Programming Interfaces. It allows you to test the functionality of any application's APIs. Almost every developer uses Postman for testing purposes. We can create any type of HTTP request in it such as GET, POST, PUT, D

Create a REST API as an Amazon Kinesis Proxy in API Gateway

Pre-Requisite: AWS Amazon Kinesis is a managed, scalable, cloud-based service that enables real-time processing of enormous amounts of data per second in real-time. It is intended for real-time applications and lets developers collect an unlimited amount of data from many sources, scaling up and dow

Stream Processing System Design Architecture

The ability to process and analyze data streams in real time has become increasingly important for organizations across industries. Stream processing systems offer a powerful solution to handle continuous data streams, enabling organizations to gain valuable insights, make informed decisions, and re

Kubernetes - Creating a ReplicaSet

Pre-requisite: Kubernetes A ReplicaSet is a key component of a Kubernetes application. It is a controller that ensures that a specified number of pod replicas are running at any given time. It is used to automatically replace any pods that fail, get deleted, or are terminated, ensuring the desired n

Handle streaming data in a data engineering pipeline

Streaming data, continuously generated from sources like social media and IoT devices, demands real-time processing. This type of data is characterized by its velocity and volume, making traditional batch processing methods inadequate. As businesses increasingly rely on instant data insights, the im