Kinesis Data Stream (KDS)

Intro

• Real-time data streaming service
• Used to ingest data in real time directly from source
• Not Serverless
• Data Retention: 1 day (default) to 365 days
• A record consists of a partition key and data blob (max 1MB)
• Once data is inserted in KDS, it can’t be modified or deleted (immutability)
• Records will be ordered in each shard
• Producers use SDK, Kinesis Producer Library (KPL) or Kinesis Agent to publish records
• Consumers use SDK or Kinesis Client Library (KCL) to consume the records
• Ability to re-process (replay) data

Capacity Modes

• Provisioned
  • Publishing: 1MB/sec per shard or 1000 msg/sec per shard
  • Consuming:
    • Shared: 2MB/sec per shard (throughput shared between all consumers)
    • Enhanced Fanout: 2MB/sec per shard per consumer (dedicated throughput for each consumer)
  • Throughput scales with shards (manual scaling)
  • Pay per shard provisioned per hour
• On-demand
  • No need to provision or manage the capacity (shards)
  • Default capacity provisioned - 4 MB/sec or 4000 records/sec
  • Scales automatically based on observed throughput peak during the last 30 days
  • Pay per stream per hour & data in/out per GB

Security

• KDS is present outside the VPC. VPC endpoints can be used to access Kinesis from within the VPC.
• Access control for producing/consuming using IAM
• In-flight encryption using HTTPS
• Server-side at-rest encryption using KMS or client-side encryption