Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: DSTL merging at TM level

...

  • For shared states, files are merged within each subtask.
  • For private states, files are merged within each TM.


Info
titleMerging DSTL files (changelog state files)

DSTL files are a special case of shared states, since these files will no longer be shared after job restore. So merging at TM level does NOT bring in additional network and space overhead. Currently the batch uploading of GIC(FLIP-158) combines small changes within a TM (across subtasks) into a single file. To prevent performance regression of its merging, they are still merged with in each TM.

  • For DSTL files (changelog state files), they are merged within each TM.


4.2 Rescaling and Physical File Lifecycle

...