Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

We can see the one real redundancy example in the following picture. (I just highlight only one segment.)

Image Removed


 Whenthere is no requirement for real-time analytics based on remote storage directly. It has the following drawbacks:

...

Code Block
languagejava
titleTopic Level Configs
    public static final String REMOTE_COPY_LAG_MS_CONFIG = "remote.copy.lag.ms";
    public static final String REMOTE_COPY_LAG_MS_DOC = "Controls how long to delay uploading segments to remote storage. " +
            "When set to 0 (default), segmentsimmediate areupload uploadedwithout asany soondelay ascheck. they" are+
 eligible (no delay check). " +
            "When set to a positive value (ms), a segment can't become eligible for upload utiluntil the time since the latest record in the segment reaches the value. " +
            "The value should not exceed the real local retention ms except the latter is retained save foreverindefinitely (-1). " +
            "When set to -1, resolves to the real local retention ms (maximum delay, or no delay check when the retention is save forever)as maximum delay. " +
            "For how the real local retention time is computed, see <code>local.retention.ms</code>.";

    public static final String REMOTE_COPY_LAG_BYTES_CONFIG = "remote.copy.lag.bytes";
    public static final String REMOTE_COPY_LAG_BYTES_DOC = "Controls size-based delay for uploading segments to remote storage. " +
            "When set to 0 (default), segmentsimmediate areupload uploaded as soon as they are eligible (nowithout any delay check). " +
            "When set to a positive value (bytes), a segment can't become eligible for upload utiluntil the total bytes of log data after the segment reach the value. " +
            "The value should not exceed the real local retention bytes except the latter is retained save foreverindefinitely (-1). " +
            "When set to -1, resolves to the real local retention bytes as (maximum delay, or no delay check when the retention is save forever). " +
            "For how the real local retention size is computed, see <code>local.retention.bytes</code>.";

...

Code Block
languagejava
titleBroker Level Configs
    public static final String LOG_REMOTE_COPY_LAG_MS_PROP = "log.remote.copy.lag.ms";
    public static final String LOG_REMOTE_COPY_LAG_BYTES_PROP = "log.remote.copy.lag.bytes";

...

We can use follow tests to cover the change:

Type\Configure

Case Id

local retention time

local retention size

lag time

lag size

Expect result

Notes

 

 

 

Special cases which can help to understand the basic rule for code review

 

 


 

1

*

 *

-2

0 (default value)

*

No lag
time check but check lag size if need


2

*

 *

*

0 (default value)

*

No any delay check. 
No lag

3

-1

*

-1

*

Invalid configure

*

*

*

-

2 (default value)

1

No
lag
delay check on size. check
but check lag time if need
the time configure as final result

For the case:  4 configures are -1.

The final result is not upload.

4

*

*

*

0

No any delay check. *

-1

*

-1

*

No delay check on time. check the size configure as final result
Invalid configure

 

 

Typical user cases which can test with deployment


 

3 days

5

 7 days


 

-1

 

-2 (default value)

-2 (default value)

No lag

0 (default value)

0 (default value)

No lag

7days + No size limit is
Kafka defalut configure

If you want time lag delay. you need to set time lag to real value and size lag to -1 

1 day

-

2 (default value)

1

lag: 1 day

-1

-

2 (default value)

1

lag:
  3
7 days

6

 

-1

 

3 GB

-2

0 (default value)

-2

0 (default value)

No lag 

-

2 (default value)

1

1 GB

lag: 1 GB 

-

2 (default value)

1

-1

lag: 3 GB 
 

7

3

7 days

 

 

3 GB

-2

0 (default value)

-2

0 (default value)

No lag 

1 day

1 GB

lag: 1 day or 1 GB 

-1

-1

lag: 7 days or 3
days or
GB 

8

7 days

3 GB

1 day

1 GB

lag: 1 days or 1 GB

0 (default value)

No lag

Unit Tests:

  1. Test upload eligibility logic with different configure values
  2. Test configuration validation for topic

...