AWS Kinesis
Edited March 1, 2022 by Suresh Kumar Balasundaramsivaprakash
AWS Kinesis
What is AWS Kinesis and what functionality does it provide for us?
Amazon Kinesis could be a managed, scalable, cloud-based service that enables data processing of streaming large amounts of information per second. you’ll be able to use Amazon Kinesis to securely stream video from camera-equipped devices in homes, offices, factories, and public places to AWS. you’ll then use these video streams for video playback, security monitoring, face detection, machine learning, and other analytics.
Who Should Use AWS Kinesis ?
We recommend Kinesis to be used cases with requirements that are the same as the following: Routing related records to the identical record processor (as in streaming MapReduce). as an example, counting and aggregation are simpler when all records for a given key are routed to the identical record processor.
How Do I Learn About it?
Learn the components and use cases of the Amazon Kinesis service and acquire started using Amazon Kinesis to form solutions and solve business problems. [https://cloudacademy.com/learning-paths/getting-started-with-amazon-kinesis-550/]
Major Components of AWS Kinesis
There are four main components of Kinesis which will be accustomed accomplish different tasks using their AWS services.
- Kinesis Firehose
- Kinesis Data Analytics
- Kinesis Data Streams
- Kinesis Video Streams
Kinesis Firehose
Kinesis Firehose is liable for loading data into Amazon Web Services. It performs a critical function of reworking and loading data into the Cloud services which are primarily used for analytical purposes. a number of these services include Kinesis Analytics, Amazon Redshift, Amazon S3, Amazon Elasticsearch Service etc. This platform requires no continuous administration and automatically scales and manages its functions in step with the information throughput.
Kinesis Data Analytics
Kinesis Data Analytics may be a platform for analysing and processing any Real-Time streaming data using Standard SQL. it’s mainly wont to analyze the info being ingested from Kinesis Firehose and Kinesis Data Streams. It can detect the quality data formats and starts by automatically parsing the information while it recommends a schema which might be customized by the user using the interactive schema editor. The function of the Kinesis Analytics within the data pipeline may be understood using the illustration below.
Kinesis Data Streams
Kinesis Data Streams provides a platform for continuous processing of Real-Time streaming data. It will be accustomed collect log events from servers and other mobile deployments. The platform strongly focuses on security and allows its users to encrypt sensitive data using server-side encryption and AWS KMS master keys. Kinesis Data Streams may be created very easily by using Kinesis Producer Library
Kinesis Video Streams
Kinesis Video Streams provides a use-case specific platform with the flexibility to stream video from Camera-equipped devices to Amazon Web Services. It may be incorporated/implemented to be used cases that involve video streaming over the net or for applications like storage of security footage on Cloud Data Warehouse Deployments. The platform also provides support for WebRTC, which is an open-source project that permits real-time media streaming and interaction between web browsers, mobile applications, and connected devices using simple APIs
Strengths
There are major Strengths of Amazon Kinesis as follows
Real-time Amazon Kinesis enables you to ingest, buffer, and process streaming data in real-time, so you’ll derive insights in seconds or minutes rather than hours or days.
Fully managed Amazon Kinesis is fully managed and runs your streaming applications without requiring you to manage any infrastructure.
Scalable Amazon Kinesis can handle any amount of streaming data and process data from many thousands of sources with very low latencies.
Limitations
- The records of a stream are by default stored and accessible for up to 24 hours. it’s possible to increase this duration by 7 days by enabling extended data retention.
- The maximum size of an information Blob, which refers to the information payload before applying Base64-encoding, for one record is of 1 MB.
- One Shard [Base Throughput unit of Kinesis Data Stream] can support up to 1000 PUT records each second.
Alternative Options
- Confluent
- Google Cloud Dataflow
- Spark Streaminng
- Azure Event Hubs
- TIBCO Spotfire
Video on basics of AWS Kinesis
References
- AWS Kinesis https://aws.amazon.com/kinesis/
- Architecture Image https://docs.aws.amazon.com/streams/latest/dev/images/architecture.png
- https://cloudacademy.com/learning-paths/getting-started-with-amazon-kinesis-550/
- https://www.youtube.com/watch?v=ZjKzGl0O6j8&ab_channel=CloudGuru