Table of Contents

Class DynamoDbStreamsClient

Namespace
DynaSharp.Streams
Assembly
DynaSharp.dll

A client for the Amazon DynamoDB Streams low-level API: it lists the streams of tables, describes their shards and reads the records of changes to items. It is the counterpart of DynamoDbClient for the change log, and is configured with the same DynamoDbClientOptions.

public sealed class DynamoDbStreamsClient : IDisposable
Inheritance
DynamoDbStreamsClient
Implements
Inherited Members

Remarks

There is no consumer: no checkpointing, no leasing and no sharing of shards among processes. ReadStreamAsync(ReadStreamRequest, CancellationToken) reads every shard of a stream in this process, parents before their children. ReadShardAsync(ReadShardRequest, CancellationToken) reads one shard, which a reader lists with DescribeStreamAsync(DescribeStreamRequest, CancellationToken): it takes the iterator, pages through GetRecordsAsync(GetRecordsRequest, CancellationToken) and renews an expired iterator. GetShardIteratorAsync(GetShardIteratorRequest, CancellationToken) and GetRecordsAsync(GetRecordsRequest, CancellationToken) are there for a loop of its own. A stream is enabled per table, with StreamSpecification when the table is created or later through UpdateTable; LatestStreamArn names the stream.

Streams have their own endpoint, https://streams.dynamodb.<region>.amazonaws.com. The settings follow the same order as for DynamoDbClient, with the endpoint taken from ServiceUrl, then the AWS_ENDPOINT_URL_DYNAMODB_STREAMS environment variable, then AWS_ENDPOINT_URL, and otherwise the public endpoint of the region (DynamoDbStreamsEndpoint). Region, credentials, retries, timeouts, telemetry and logging work as they do for DynamoDbClient, and DynamoDbRequests describes the retries, the errors and the telemetry that both share; requests are signed for the service name dynamodb. A client without Credentials makes its own default credentials provider, so an application that uses this client together with a DynamoDbClient on EC2 or ECS should pass one provider to both, as described there. Note that ServiceUrl applies to both clients: an emulator such as DynamoDB Local serves both APIs on one address, but for AWS itself the two clients need no explicit URL.

Errors surface as DynamoDbException or a derived type. ExpiredIteratorException and TrimmedDataAccessException are not retried: repeating the request cannot fix them. Instances are thread-safe. Create one client per application and dispose it to release its connections; a handler passed in HttpHandler is not disposed.

Each operation emits one CLIENT activity named DynamoDB Streams.<operation> and one measurement of db.client.operation.duration, like the operations of DynamoDbClient. Shard iterators are credentials, and neither they nor the keys and images of records ever appear in telemetry or logs.

Constructors

DynamoDbStreamsClient()

Initializes a new instance of the DynamoDbStreamsClient class that takes all settings from the environment: the region from AWS_REGION (or AWS_DEFAULT_REGION, or the selected profile in the shared AWS config file), the endpoint from AWS_ENDPOINT_URL_DYNAMODB_STREAMS (or AWS_ENDPOINT_URL, otherwise the public endpoint of the region) and the credentials from the default chain of the AWS SDKs, see DefaultCredentialsProvider.

public DynamoDbStreamsClient()

Exceptions

InvalidOperationException

No region is configured, or a region or endpoint environment variable is invalid.

DynamoDbStreamsClient(DynamoDbClientOptions)

Initializes a new instance of the DynamoDbStreamsClient class.

public DynamoDbStreamsClient(DynamoDbClientOptions options)

Parameters

options DynamoDbClientOptions

The settings. Options that are not set fall back to the environment, see DynamoDbClientOptions. The client reads the options once; changing the options object later has no effect.

Remarks

Credentials are not looked up here, but with the first request. See DynamoDbClient for the details of the configuration, which this client shares.

Exceptions

ArgumentNullException

options or its TimeProvider is null.

ArgumentOutOfRangeException

AttemptTimeout is neither positive nor InfiniteTimeSpan, or exceeds about 49.7 days; or MaxAttempts is below 1.

ArgumentException

ServiceUrl is not an absolute http or https URL without user info, path, query and fragment.

InvalidOperationException

No region is configured, or a region or endpoint environment variable is invalid.

Properties

Options

Gets the options as the client resolved them when it was created: what it really uses, so a region from the environment or a profile and an endpoint from the environment are here and not null, and Credentials is the provider in use, also the default chain that the client made itself. The one exception is the public endpoint of the region, see the remarks. Nothing is read again, so the environment of today can say something else.

public DynamoDbClientOptions Options { get; }

Property Value

DynamoDbClientOptions

Remarks

A second client is new DynamoDbStreamsClient(client.Options with { MaxAttempts = 1 }). It shares the provider and with it the cached credentials, and a handler the first client was given. A handler the first client made stays its own, so HttpHandler is null there and the second client makes one too, and each outlives the other.

ServiceUrl holds an endpoint that came from the options or the environment, which a copy keeps. The public endpoint that the client derived from its region is in use and still stays null here, on purpose: it is no choice of anyone's, but a consequence of the region and of the kind of client. So client.Options with { Region = other } goes to the endpoint of the new region, and new DynamoDbClient(client.Options) gets its own endpoint.

The options hold a provider and not credentials, and ToString() prints only its type.

Methods

DescribeStreamAsync(DescribeStreamRequest, CancellationToken)

Reads the settings of a stream and one page of its shards (DynamoDB Streams DescribeStream).

public Task<DescribeStreamResponse> DescribeStreamAsync(DescribeStreamRequest request, CancellationToken cancellationToken = default)

Parameters

request DescribeStreamRequest

The stream, the paging settings and an optional shard filter.

cancellationToken CancellationToken

Cancels the request.

Returns

Task<DescribeStreamResponse>

The description, with at most 100 shards. Continue with LastEvaluatedShardId until it is null.

Remarks

Shards form a tree: a closed shard names its successors as the shards whose ParentShardId points at it. A reader that wants the changes of an item in order finishes a parent before it starts its children. To find the children of a shard that was read to its end, pass a ShardFilter instead of listing everything again. The list of shards changes over time, so a long-running reader describes the stream again from time to time, and whenever a shard ends.

DescribeStream is limited to ten calls per second and stream. The call is safe to repeat.

Exceptions

ArgumentNullException

request is null.

ObjectDisposedException

The client is disposed.

ResourceNotFoundException

The stream does not exist (any more).

ValidationException

DynamoDB rejects the request, for example because the stream ARN is malformed.

AccessDeniedException

The credentials lack permission for dynamodb:DescribeStream on the stream.

DynamoDbException

DynamoDB returns any other error, or a response that is not a DynamoDB response. A throttling or server error (for example InternalServerErrorException) surfaces only when the last attempt failed with it too.

CredentialsUnavailableException

No credentials can be found.

HttpRequestException

The request could not be sent or the response could not be read, on every attempt.

TimeoutException

The last attempt did not complete within AttemptTimeout.

OperationCanceledException

cancellationToken was cancelled.

Dispose()

Releases the HTTP connections. Requests still running may fail. Calling it again has no effect.

public void Dispose()

Remarks

The handler given in HttpHandler is not disposed; its owner does that.

GetRecordsAsync(GetRecordsRequest, CancellationToken)

Reads records from a shard, starting at the position of an iterator (DynamoDB Streams GetRecords).

public Task<GetRecordsResponse> GetRecordsAsync(GetRecordsRequest request, CancellationToken cancellationToken = default)

Parameters

request GetRecordsRequest

The iterator and the maximum number of records.

cancellationToken CancellationToken

Cancels the request.

Returns

Task<GetRecordsResponse>

The records and the iterator for the next call. An empty Records list with a NextShardIterator is normal and does not mean that the shard is finished.

Remarks

Read in a loop, each time with the NextShardIterator of the response, and wait a moment (a second is typical) when a response comes back empty: a shard allows five calls per second, and an empty response is how an open shard says that no change happened. A response can also be empty because DynamoDB looked through a part of the shard without finding a record; keep going.

A null NextShardIterator means that the shard is closed and everything in it has been returned. That ends the loop for this shard; the successors are found with DescribeStreamAsync(DescribeStreamRequest, CancellationToken).

A call returns at most 1 MB and 1,000 records. The call itself changes nothing, so it is safe to repeat with the same iterator within its 15 minutes, which returns the same records. Records stay available for 24 hours.

Exceptions

ArgumentNullException

request is null.

ObjectDisposedException

The client is disposed.

ExpiredIteratorException

The iterator is older than 15 minutes. Get a new one with GetShardIteratorAsync(GetShardIteratorRequest, CancellationToken).

TrimmedDataAccessException

The records at the position of the iterator are older than 24 hours and have been removed.

LimitExceededException

The request rate is too high. The client retries it, so this surfaces only when every attempt failed with it.

ResourceNotFoundException

The stream or shard does not exist (any more).

ValidationException

DynamoDB rejects the request, for example because the iterator is malformed or the limit out of range.

AccessDeniedException

The credentials lack permission for dynamodb:GetRecords on the stream.

DynamoDbException

DynamoDB returns any other error, or a response that is not a DynamoDB response. A throttling or server error (for example InternalServerErrorException) surfaces only when the last attempt failed with it too.

CredentialsUnavailableException

No credentials can be found.

HttpRequestException

The request could not be sent or the response could not be read, on every attempt.

TimeoutException

The last attempt did not complete within AttemptTimeout.

OperationCanceledException

cancellationToken was cancelled.

GetShardIteratorAsync(GetShardIteratorRequest, CancellationToken)

Gets an iterator that marks a position in a shard, to read from with GetRecordsAsync(GetRecordsRequest, CancellationToken) (DynamoDB Streams GetShardIterator).

public Task<GetShardIteratorResponse> GetShardIteratorAsync(GetShardIteratorRequest request, CancellationToken cancellationToken = default)

Parameters

request GetShardIteratorRequest

The stream, the shard and the position.

cancellationToken CancellationToken

Cancels the request.

Returns

Task<GetShardIteratorResponse>

The iterator, or a response without one, which the service model allows: there is nothing to read from the shard then.

Remarks

An iterator is valid for 15 minutes and is meant to be used in a chain: each GetRecordsAsync(GetRecordsRequest, CancellationToken) returns the next one. A reader that wants to survive a restart stores the sequence number of the last record it processed and resumes with AfterSequenceNumber. The iterator is a credential for the shard: do not log it.

Latest skips everything written before the call, so changes between the end of the previous iterator and the new one are lost. Use it to start a new reader, not to recover from an expired iterator. The call is safe to repeat.

Exceptions

ArgumentNullException

request is null.

ObjectDisposedException

The client is disposed.

ArgumentOutOfRangeException

The request has no ShardIteratorType or an undefined value (raised as the faulted task).

ResourceNotFoundException

The stream or the shard does not exist (any more).

TrimmedDataAccessException

The position is older than the 24 hours that records are kept.

ValidationException

DynamoDB rejects the request, for example because a sequence number is missing for AtSequenceNumber.

AccessDeniedException

The credentials lack permission for dynamodb:GetShardIterator on the stream.

DynamoDbException

DynamoDB returns any other error, or a response that is not a DynamoDB response. A throttling or server error (for example InternalServerErrorException) surfaces only when the last attempt failed with it too.

CredentialsUnavailableException

No credentials can be found.

HttpRequestException

The request could not be sent or the response could not be read, on every attempt.

TimeoutException

The last attempt did not complete within AttemptTimeout.

OperationCanceledException

cancellationToken was cancelled.

ListStreamsAsync(ListStreamsRequest, CancellationToken)

Lists the streams of all tables of the account in the region, or of one table (DynamoDB Streams ListStreams).

public Task<ListStreamsResponse> ListStreamsAsync(ListStreamsRequest request, CancellationToken cancellationToken = default)

Parameters

request ListStreamsRequest

The optional table and paging settings.

cancellationToken CancellationToken

Cancels the request.

Returns

Task<ListStreamsResponse>

One page of at most 100 streams. Continue with LastEvaluatedStreamArn until it is null. A table has at most one enabled stream, but streams that were switched off stay listed for 24 hours.

Remarks

To find the stream of a table you know, LatestStreamArn from DescribeTable is simpler. A page can have fewer streams than the limit and still have a successor. The call is safe to repeat.

Exceptions

ArgumentNullException

request is null.

ObjectDisposedException

The client is disposed.

ResourceNotFoundException

The table named in the request does not exist.

ValidationException

DynamoDB rejects the request, for example because Limit is below 1.

AccessDeniedException

The credentials lack permission for dynamodb:ListStreams.

DynamoDbException

DynamoDB returns any other error, or a response that is not a DynamoDB response. A throttling or server error (for example InternalServerErrorException) surfaces only when the last attempt failed with it too.

CredentialsUnavailableException

No credentials can be found.

HttpRequestException

The request could not be sent or the response could not be read, on every attempt.

TimeoutException

The last attempt did not complete within AttemptTimeout.

OperationCanceledException

cancellationToken was cancelled.

ReadShardAsync(ReadShardRequest, CancellationToken)

Reads one shard from a position to its end, as a sequence of records: it takes the iterator, pages through GetRecords, takes a new iterator when one expires, and waits after an empty page.

public IAsyncEnumerable<ChangeRecord> ReadShardAsync(ReadShardRequest request, CancellationToken cancellationToken = default)

Parameters

request ReadShardRequest

The stream, the shard, the position to start at and the paging settings.

cancellationToken CancellationToken

Cancels the reading. A token given to WithCancellation works too.

Returns

IAsyncEnumerable<ChangeRecord>

The records in the order of the shard, which is the order of the changes to each item in it. The sequence ends when the shard is closed and every record of it has been returned, and an open shard has no end: DynamoDB closes a shard after some hours, or when its partition splits.

Remarks

This reads one shard and is no consumer: nothing stores how far it got, and nothing shares the shards among workers. The position to keep is on each record, SequenceNumber of DynamoDb: a reader that stores the one of the last record it handled starts again after it with AfterSequenceNumber. Changes of one item can move to a child shard when a shard ends, so a reader that wants them in order reads a parent to its end before its children, which DescribeStreamAsync(DescribeStreamRequest, CancellationToken) lists with a ShardFilter. ReadStreamAsync(ReadStreamRequest, CancellationToken) does that for every shard of a stream.

Nothing is requested until the first record is asked for. An iterator lives 15 minutes, and a caller that takes longer between two records holds an expired one when the next page is due. The reader then takes a new iterator after the sequence number of the last record it returned, so nothing is skipped and nothing repeats. Before the first record it takes one at the position of the request again, which for Latest skips what was written in between. An iterator that is expired the moment it was taken is not renewed again: the ExpiredIteratorException surfaces.

An open shard answers an empty page when nothing changed, and also when DynamoDB looked at a part of the shard without finding a record, so an empty page is no end: the reader waits EmptyPageDelay and asks again. A shard read to its end answers without a next iterator, and that ends the sequence. A GetShardIteratorAsync(GetShardIteratorRequest, CancellationToken) without an iterator ends it at once.

Each call is a span and a measurement of its own, as with the methods it is made of, and a renewal shows as a failed GetRecords followed by a GetShardIterator. A record without a sequence number does not move the position.

Exceptions

ArgumentNullException

request is null.

ArgumentOutOfRangeException

ShardIteratorType is Unknown or undefined, or EmptyPageDelay is negative or 15 minutes or more.

ArgumentException

SequenceNumber is missing for AtSequenceNumber or AfterSequenceNumber, or set for TrimHorizon or Latest.

TrimmedDataAccessException

The position is older than the 24 hours that records are kept, at the start or while reading, so changes are lost and the reader cannot tell which. It is not handled here: start again at TrimHorizon, and rebuild from the table what the gap matters for.

ExpiredIteratorException

A new iterator was expired before it could be used.

ObjectDisposedException

The client is disposed (raised while reading).

ResourceNotFoundException

The stream or the shard does not exist (any more): DynamoDB removes a shard 24 hours after it closed.

DynamoDbException

DynamoDB returns any other error, for example ValidationException for a malformed ARN or a limit out of range. A throttling or server error surfaces only when the last attempt of a call failed with it too.

CredentialsUnavailableException

No credentials can be found.

HttpRequestException

A request could not be sent or its response could not be read, on every attempt.

TimeoutException

The last attempt of a call did not complete within AttemptTimeout.

OperationCanceledException

cancellationToken was cancelled.

ReadStreamAsync(ReadStreamRequest, CancellationToken)

Reads every shard of a stream in this process, as one sequence of records: it lists the shards, reads them at the same time with ReadShardAsync(ReadShardRequest, CancellationToken), and starts the children of a shard once the shard has been read to its end.

public IAsyncEnumerable<ShardChangeRecord> ReadStreamAsync(ReadStreamRequest request, CancellationToken cancellationToken = default)

Parameters

request ReadStreamRequest

The stream, the position to start at and the wait on an idle shard.

cancellationToken CancellationToken

Cancels the reading. A token given to WithCancellation works too.

Returns

IAsyncEnumerable<ShardChangeRecord>

The records of all shards, each with the shard it came from. The changes of one item come in their order, because a shard is read to its end before its children; records of different shards come in no particular order. The sequence ends only when no shard is left to read, which is when the stream was switched off and every shard has been read; while the stream is enabled it runs until it is cancelled.

Remarks

This reads one stream in one process and is no consumer: nothing stores how far it got, nothing shares the shards with other processes, and there is no position to resume at, no checkpoint store, no lease, no setting of the retries of a shard or of the buffer, and no typed record. An image is an Item: Is<T>() tells the entities apart and To<T>() reads one. The position to keep is on each record: ShardId and SequenceNumber, from which ReadShardAsync(ReadShardRequest, CancellationToken) with AfterSequenceNumber reads that shard on. The shard is on ShardChangeRecord and not on ChangeRecord, because GetRecordsAsync(GetRecordsRequest, CancellationToken) cannot know it.

DescribeStreamAsync(DescribeStreamRequest, CancellationToken) lists the shards, page by page. With TrimHorizon each shard whose parent is not listed is read from its oldest record, closed or open, and the others wait for their parent. With Latest each shard open at the start is read from now on, and a shard closed at the start has nothing to read from now on, but is watched for children as a shard read to its end is, unless its children listed at the start opened more than a minute before: a split in progress at the start can list its second child only later. A shard read to its end starts its children, which DescribeStreamAsync(DescribeStreamRequest, CancellationToken) lists with a ShardFilter of ChildShards. A child whose id says it opened less than 23 hours ago, which is when its records start to be trimmed, is read from the trim horizon; an older one, or one whose id tells no time, with AtSequenceNumber at the StartingSequenceNumber of its range, so that records of it trimmed before its turn came throw instead of being passed over; without a starting sequence number it is read from the trim horizon. A shard counts as one of the children only if its ParentShardId names the parent, also when the service ignores the filter.

DynamoDB can list a child a moment after its parent closed, and the Developer Guide says a shard can split into several. So each child starts as soon as it is listed, and the reader of the parent keeps asking: while the stream is Enabled or Enabling and no child is listed, every EmptyPageDelay but at least every second; after the first child, every five seconds for one more minute. A stream that is no longer enabled opens no more shards, which ends the asking.

Nothing is requested until the first record is asked for. The records go through a small bounded buffer, so a shard waits while the caller is behind, and a reader that keeps a shard busy holds the others back too. Each shard is read as ReadShardAsync(ReadShardRequest, CancellationToken) reads it: an expired iterator is renewed, and an open shard is asked again after EmptyPageDelay.

A shard that fails for a moment is read on from where it stopped. When the last attempt of a call of a shard fails with what IsTransient calls transient, or with a network failure or a timeout of an attempt, the reader of the shard waits, from a second up, doubling to half a minute, with jitter, on TimeProvider, and sends the call again: a page with the same iterator, which returns the same records, and a look for children again. An iterator that expired in the meantime is renewed after the last record the shard handed out, or at the start position of the shard when it has handed out none, so nothing repeats and nothing is skipped, and a position trimmed in the meantime throws TrimmedDataAccessException. The reader tries for 15 minutes from the failure that started the outage, the life of an iterator, within which the position is never lost; a page or a look that succeeds ends the outage. The first listing of the shards is not tried again: nothing has been read then, and a setup that does not work shows at once.

Any other failure, and one that outlasts the 15 minutes, ends the sequence with that exception: the other readers are stopped and awaited, and the records still in the buffer are dropped. Nothing is skipped: a position older than the 24 hours that records are kept throws TrimmedDataAccessException, and so does a closed shard that DynamoDB removed, 24 hours after it closed, before it was read, whose records are lost. Disposing the enumerator or cancelling stops every reader and waits for it, also one that waits to try again, so none outlives the enumeration. OnJoined says when every shard read from the start holds its position.

Each call it makes is a span and a measurement of its own, as with the methods it is made of. Each shard it reads is an Internal span too, named DynamoDB Streams.ReadStream, under the span that was current when the enumeration started, with the calls of the shard under it and a link to the span of its parent shard. It ends when the looks for the children of the shard end, and a shard that DynamoDB keeps open for hours makes a span of hours. The metric dynasharp.client.stream.shards counts the shards being read, dynasharp.client.stream.records the records handed out, and dynasharp.client.stream.retries the retries of a shard, which its span counts in dynasharp.stream.retry_count. A shard read to its end logs its children at the debug level, and so does a child listed only on a later look. A shard that starts to fail transiently logs a warning, once, and an information when it reads again. Neither names a key, an image or the ARN of the stream.

Exceptions

ArgumentNullException

request is null.

ArgumentOutOfRangeException

ShardIteratorType is neither TrimHorizon nor Latest, or EmptyPageDelay is negative or 15 minutes or more.

TrimmedDataAccessException

The position in a shard is older than the 24 hours that records are kept, or a closed shard was removed before it was read, so changes are lost. It is not handled here: start again, and rebuild from the table what the gap matters for.

ExpiredIteratorException

A new iterator was expired before it could be used.

ObjectDisposedException

The client is disposed (raised while reading).

ResourceNotFoundException

The stream does not exist (any more), or a shard that was open when it was listed is gone.

DynamoDbException

DynamoDB returns any other error, for example ValidationException for a malformed ARN. A throttling or server error surfaces only when the first listing failed with it, or when a shard kept failing with it for 15 minutes.

CredentialsUnavailableException

No credentials can be found.

HttpRequestException

A request could not be sent or its response could not be read, on every attempt of the first listing, or for 15 minutes of a shard.

TimeoutException

The last attempt of the first listing, or of a shard for 15 minutes, did not complete within AttemptTimeout.

OperationCanceledException

cancellationToken was cancelled.

SendRawAsync(string, ReadOnlyMemory<byte>, CancellationToken)

Sends a DynamoDB Streams operation by name, with a request body that you write as JSON, and returns the response body as JSON. It is for the operations and parameters that this client does not model.

public Task<byte[]> SendRawAsync(string operation, ReadOnlyMemory<byte> requestJson, CancellationToken cancellationToken = default)

Parameters

operation string

The operation name as DynamoDB Streams knows it, for example GetRecords. It consists of ASCII letters and digits; it is sent as the X-Amz-Target header and is the operation of the span and the metrics of the call.

requestJson ReadOnlyMemory<byte>

The request body: a JSON object in the format of the DynamoDB Streams API (AWS JSON 1.0). The bytes are copied; they are not read or checked, so the service reports a request it cannot parse. The memory can be reused as soon as the returned task has completed.

cancellationToken CancellationToken

Cancels the request.

Returns

Task<byte[]>

The body of the response as UTF-8 JSON, exactly as the service sent it. It is not parsed.

Remarks

The call is signed, repeated, bounded and reported like the other calls of this client; see SendRawAsync(string, ReadOnlyMemory<byte>, CancellationToken), which describes what that means. The span is named DynamoDB Streams.<operation>, and there is no table tag. Shard iterators in a request or response are credentials and never appear in telemetry or logs.

Exceptions

ArgumentNullException

operation is null.

ArgumentException

operation is empty or contains a character other than an ASCII letter or digit.

ObjectDisposedException

The client is disposed.

ValidationException

The service rejects the request.

AccessDeniedException

The credentials lack permission for the operation.

DynamoDbException

The service returns any other error, an operation it does not know, or a response that is not a DynamoDB response. A throttling or server error surfaces only when the last attempt failed with it too.

CredentialsUnavailableException

No credentials can be found.

HttpRequestException

The request could not be sent or the response could not be read, on every attempt.

TimeoutException

The last attempt did not complete within AttemptTimeout.

OperationCanceledException

cancellationToken was cancelled.