Page History

...

The consumer client will initiate the FetchRequest with an extra boolean parameter: RemoteLogSegmentLocationRequested which indicates the client is able to read from remote storage and is interested to get the location of remote log segment file on remote storage. When the leader broker receives FetchRequest from the consumer and it finds out it no longer has the requested fetch offset in the local storage, it can respond with an empty Records and with the extra information about the location of the remote log segment.  This is very similar to how KIP-392 responds with empty Records and preferred read replica in FetchResponse to the consumer.

Authorization check

Security and Access Control

The broker will perform the standard access and authorization check against the fetch request based on Kafka Access Control configuration.  

However since the consumer can later read the object in Object Store from the location information in FetchResponse, some organizations might have concerns on the access in Object Store for the consumer.  Most of the Object Store has its own ACL configurations (for example S3's IAM role/policy configuration) to manage specific client access.  For the organizations who wants a simple way to provide another layer of protection, the broker can protect the location information with a signing mechanism provided by the cloud provider (for example, presigned URL in S3 or SAS token with Azure), this way the client will only be able to access a specific object in object store within a limited time.

 If we choose this design, we will add a method getSignedRemoteLogSegmentLocation(RemoteLogSegmentId, CustomMetadata) in RemoteStorageManager interface to retrieved the signed url for the segment location from the cloud provider and add a field SignedRemoteLogSegmentLocation in the FetchResponse to return to the consumerThe broker will still perform the standard authorization check against the fetch request.

Fetch and Read Remote Log segment file

...

Version compatibility check with broker

Over the time he the broker might evolve the storage format of the log files which makes the earlier version of the client not able to read the log file anymore.  To protect this kind of version incompatibility issue, we can introduce another field in FetchRequest: SupportedStorageFormatVersions, the consumer will fill in this field with the server side storage format versions it understands (we can mark the current storage format as ApachKafkaV1), the server can fallback to the old behavior if it thinks the client version is too low.  And we mark this field as list type to contain a list of versions the client supports, so in the case when the client is newer while the server is older (forward-compatibility case), the server can still respond with the new behavior if server’s old version is still in the list of versions client understands.

...