Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Current state: Draft

Discussion thread: https://lists.apache.org/thread.html/r9d1c89b871792655cd14ff585980bb0ace639d85d9200e239cc0e1cd%40%3Cdev.kafka.apache.org%3E

Voting threadhttps://lists.apache.org/thread.html/rbfe08bfb15e14db14c54d1ca5c86bfcd17dc952084ad0a4dec8255b6%40%3Cdev.kafka.apache.org%3E

JIRA: 

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyKAFKA-10339

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

MirrorMaker2 is currently implemented on Kafka Connect Framework, more specifically the Source Connector / Task, which do not provide exactly-once semantics (EOS) out-of-the-box, as discussed in https://github.com/confluentinc/kafka-connect-jdbc/issues/461,  https://github.com/apache/kafka/pull/5553

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyKAFKA-6080
 and 
Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyKAFKA-3821
. Therefore MirrorMaker2 currently Therefore current MirrorMaker2 does not provide EOS as well.

This proposal is to provide an option to enable EOS for MirrorMaker 2 if no data loss or duplicate data is preferred. The key idea behind of this proposal is to extend SinkTask with a brand new MirrorSinkTask implementation, which leverages the consumer from WorkerSinkTask and provides which has an option to either deliver messagesmanage the consumer offsets in transactional way (similar high-level idea as HDFS Sink Connector), such that the messages can be delivered across clusters:

(1) Exactly-once: by a transactional producer in MirrorSinkTask and consumer offsets are committed within a transaction by the transactional producer

...

  • as mentioned above, Kafka Connect Source Connector / Task do not provide EOS by nature, mostly because of async and periodic source task offset commit, which in MirorrMaker case, the "offset" is consumer offset. So this is one of the blockers blocker to enable EOS without a lots of changes in WokerSourceTask WorkerSourceTask.
  • MirrorSourceTask explicitly avoided using subscribe() and instead handle rebalances and new topic-partitions explicitly. While MirrorSinkTask that extends SinkTask is initiated by WorkerSinkTask. WorkerSinkTask uses consumer.subscribe() in which the benefits of rebalances and auto-detection of new topics/partitions are out-of-the-box. When the consumer or its rebalance handling is upgraded in WorkerSinkTask as part of Kafka Connect, MirrorSinkTask will take the advantages transparently.
  • Since MirrorSinkTask is a new implementation, on the other end Since MirrorSinkTask (that extends SinkTask) is a new implementation, we can fully control how a producer is created (transactional v.s. non-transactional) and handle various Exception cases (especially in transactional mode), purely in the new implementation, rather than changing the existing producer in WorkerSourceTaskMirrorSinkTask can intentionally return an empty result from preCommit() (defined in SinkTask) by overriding it to disable the consumer offset commit done by WorkerSinkTask, so that the transactional producer is able to commit consumer offset within one transaction
  • HDFS Sink Connector already achieved EOS, we can correctly implement MirrorSinkTask based on the methods of SinkTask by referring the best practices from HDFS Sink Connector.

Public Interfaces

New classes and interfaces include:

...

(1) in MirrorMaker case, there are source and target clusters. The Normally the consumer pulls the data and stores its offsets are typically stored in the source cluster, while then the producer takes over the data from the consumer and sends them to target cluster. However Kafka transaction can not happen across clusters out-of-the-box. What If we want EOS across clusters, what modifications need to be done?

A: The short answer is the consumer group offsets , which are supposed to be stored on source cluster, are maintained / committed by are managed, committed by the transactional producer and are stored on the target cluster instead.

But the However the consumer still has to live on the source cluster in order to pull the data and however the , but “source-of-truth” offsets are not no longer stored in the source cluster (or stored in source cluster, not not accurate). We propose to use the following idea to position rewind the consumer correctly while its when data mirroring task restarts or rebalances, while the “source-of-truth” of consumer offsets are stored in the target cluster: (the pseudocode are shown in below)

  • MirrorSinkTask don't rely on Connect's internal offsets tracking or __consumer_offsets on the source cluster.
  • the offsets are only written by transaction producer to the target cluster.
  • offsets are
  • Consumer offsets are stored on the target cluster using a
  • "fake"
  • “fake” consumer group, that can be created programmatically as long as we know the name of consumer group. The
  • "fake"
  • “fake” means there would be no actual records being consumed by the group, just offsets being stored in __consumer_offsets topic.
  • all records are written in a transaction.
  • when MirrorSourceTask starts, it loads initial offsets from
  • However, the __consumer_offsets topic on the target cluster (managed by the “fake” consumer group) is the “source of truth” offsets.

The outcome of the above idea:

    if the transaction succeeds, the
  • With the “fake” consumer group on target cluster, MirrorSinkTask don't rely on Connect's internal offsets tracking or __consumer_offsets
  • topic
  • on the
  • target
  • source cluster
  • is updated
  • .
  • if the transaction aborts, all data records are dropped, and the __consumer_offsets topic is not updated.
  • when MirrorSinkTask starts/restarts, it resumes at the last committed offsets, as stored in the target cluster.

Some items to pay attention in order to make above idea work correctly:

  • the consumer offsets are only written by the producer evolved in the transaction to the target cluster.
  • all records are written in a transaction, as if in the single cluster
  • when MirrorSinkTask starts or rebalances, it loads initial offsets from __consumer_offsets on the target cluster.

The outcome of the above idea:

  • if the transaction succeeds, the __consumer_offsets topic on the target cluster is updated by following the current protocol of Exactly-Once framework
  • if the transaction aborts, all data records are dropped, and the __consumer_offsets topic on the target cluster is not updated.
  • when MirrorSinkTask starts/restarts, it resumes at the last committed offsets, as stored in the target cluster.

Some items to pay attention in order to make above idea work correctly:

  • If consumer group already exists on source cluster, while the "fake" consumer group (with same Group Id) on the target cluster does not exist or its offsets lower than the high watermark. To avoid duplicate data, it may If consumer group already exists on source cluster, while the "fake" consumer group (with same Group Id) on the target cluster does not exist or its offsets lower than the high watermark. We need to do a one-time offline job to sync the offsets from source cluster to target cluster.

(2) Since the

...

offsets of the consumer group

...

on the source cluster is NOT "source of truth"

...

  and not involved in the transaction. Are they still being updated? Do we still need them in some cases?

A: the offsets of the consumer group on the source cluster are still being updated periodically and independently by the logics in WorkerSInkTask. However they may be lagging behind a little, since (1) they are not involved in transaction, (2) they are periodically committed.

However, they may be still useful in some cases: (1) measure the replication lag between the upstream produce on the source cluster and MirrorMaker's consumption. (2) restore the lost "fake" consumer group with small # of duplicate data

(2) The consumer in WorkerSinkTask periodically commits the offsets that are maintained by itself, how to disable this behavior so that transactional producer in MirrorSinkTask can control when and what to commit?

A: The SinkTask provides a preCommit() method let the implementation of SinkTask (here is MirrorSinkTask) to determine what offsets should be committed in WorkerSinkTask. If in transaction mode, we will intentionally return empty Map in preCommit() (see the code below), so that if the return of preCommit() is empty, the consumer of WorkerSinkTask will ignore committing the offsets that are maintained by itself.



The following is the pseudocode illustrates the high-level key implementation:

Code Block
titleMirrorSinkTaskMirrorSinkTask
    private boolean isTransactional = config.getTransactionalProducer();
    private boolean transactionInProgress = false;
	protected Map<TopicPartition, OffsetAndMetadata> offsetsMap = new HashMap<>();
    private Set<TopicPartition> taskTopicPartitions;
	private KafkaProducer<byte[], byte[]> producer;
    private String connectConsumerGroup;

    @Override
	public void start(Map<String, String> props) {
		config = new MirrorTaskConfig(props);
	    taskTopicPartitions = config.taskTopicPartitions();
        isTransactional = config.transactionalProducer();
	    producer = initProducer(isTransactional);
        connectConsumerGroup = getSourceConsumerGroupId();
		if (isTransactional) {
	        loadContextOffsets();
        }
    }

    @Override
    public void open(Collection<TopicPartition> partitions) {
		if (isTransactional) {
    	    loadContextOffsets();
		}
    }

    private void loadContextOffsets() {
		Map<TopicPartition, OffsetAndMetadata> initOffsetsOnTarget = listTargetConsumerGroupOffsets(connectConsumerGroup);
	
        Set<TopicPartition> assignments = context.assignment();

        // only keep the offsets of the partitions assigned to this task
        Map<TopicPartition, Long> contextOffsets = assignments.stream()
        		                                              .filter(x -> currentOffsets.containsKey(x))
                                                              .collect(Collectors.toMap(
        		    private boolean isTransactional = config.getTransactionalProducer();
    private boolean transactionStart = false;
	protected Map<TopicPartition, OffsetAndMetadata> offsetsMap = new HashMap<>();
    private Set<TopicPartition> taskTopicPartitions;
	private KafkaProducer<byte[], byte[]> producer;
    private String consumerGroupId;

    @Override
	public void start(Map<String, String> props) {
		config = new MirrorTaskConfig(props);
	    taskTopicPartitions = config.taskTopicPartitions();
        isTransactional = config.transactionalProducer();
	    producer = initProducer(isTransactional);
        consumerGroupId = getSourceConsumerGroupId();
		if (isTransactional) {
			Map<TopicPartition, OffsetAndMetadata> offsetsOnTarget = listTargetConsumerGroupOffsets(consumerGroupId);
			Map<TopicPartition, Long> offsets = new HashMap<>();
			
			for (Map.Entry<TopicPartition, OffsetAndMetadata> offset : offsetsOnTarget.entrySet()) {
				offsets.put(offset.getKey(), offset.getValue().offset());
			}
			Map<TopicPartition, Long> taskOffsets = loadOffsets(taskTopicPartitions, offsets);
			context.offset(taskOffsets);
               x -> x, x -> currentOffsets.get(x)));

        }context.offset(contextOffsets);
    }

	protected KafkaProducer<byte[], byte[]> initProducer(boolean isTransactional) {
        Map<String, Object> producerConfig = config.targetProducerConfig();
        if (isTransactional) {
        	log.info("use transactional producer"String transactionId = getTransactionId();
        	log.info("use transactional producer with Id:  producerConfig.put(ProducerConfig.ACKS_CONFIG, "all"{} ", transactionId);
            producerConfig.put(ProducerConfig.ENABLEACKS_IDEMPOTENCE_CONFIG, "trueall");
         	   producerConfig.put(ProducerConfig.TRANSACTIONALENABLE_IDIDEMPOTENCE_CONFIG, genTransactionId()"true");
        	producerConfig.put(ProducerConfig.CLIENTTRANSACTIONAL_ID_CONFIG, "trasactional-producer"transactionId);
        } else {
        	producerConfig.put(ProducerConfig.CLIENT_ID_CONFIG, "non-trasactional-producer"transactionId);
        }
        return MirrorUtils.newProducer(producerConfig);
	}

    @Override
	public void put(Collection<SinkRecord> records) {
        log.info("receive {} messages from consumer", records.size());/**
     * Per some articles, to avoid ProducerFencedException, transaction id is suggested to set application name + hostname
     * Each MirrorSinkTask if (records.size() == 0) {
        	return;
        }
        try {is also assigned with different set of <topic, partition>. To get unique transaction id,
     * one way is to append connector name, hostname and string of each <topic, partition> pair
     */
    protected String  sendBatchgetTransactionId(records, producer);) {
    	return getHostName() + "-" }+ catch getUniquePredictableStr(RebalanceException e) {;
    }

    @Override
	public void put(Collection<SinkRecord>  producer.close();records) {
        log.info("receive {} messages from producer = initProducer(isTransactional);  consumer", records.size());
        } catch (ResendRecordsException eif (records.size() == 0) {
        	return;
    abortTransaction(producer);
    }
        //TODO: add limited retrytry {
            sendBatch(e.getRemainingRecords()records, producer);
        } catch (ThrowableRebalanceException e) {
            log.error(getHostName() + " terminating on exception: {}", e producer.close();
            producer = return;initProducer(isTransactional);  
        }
 catch (ResendRecordsException e) }{

      private void sendBatch(Collection<SinkRecord> records, KafkaProducer<byte[], byte[]> abortTransaction(producer);
  {
        try {
 //TODO: add limited retry
        Map<TopicPartition, List<SinkRecord> remainingRecordsMap = new HashMap<>( sendBatch(e.getRemainingRecords(), producer);
        } catch (Throwable  offsetsMap.clear();e) {
            beginTransactionlog.error(getHostName(producer);
 + " terminating on exception:       SinkRecord record{}", e);
            for ((record = records.peek()) != null) {
return;
        }
    }

    private void sendBatch(Collection<SinkRecord> records, ProducerRecord<byteKafkaProducer<byte[], byte[]> producerRecord = convertToProducerRecord(record);producer) {
        try {
       offsetsMap.compute(new TopicPartition(record.topic(), record.kafkaPartition()),
   Map<TopicPartition, List<SinkRecord> remainingRecordsMap = new HashMap<>();
            offsetsMap.clear();
   (tp, curOffsetMetadata) ->
         beginTransaction(producer);
            SinkRecord record;
         (curOffsetMetadata == null ||for ((record = records.kafkaOffsetpeek()) > curOffsetMetadata.offset())!= null) {
                ProducerRecord<byte[], byte[]> producerRecord = convertToProducerRecord(record);
                offsetsMap.compute(new    ?TopicPartition(record.topic(), record.kafkaPartition()),
                       (tp, curOffsetMetadata) ->
              new OffsetAndMetadata(record.kafkaOffset())
                (curOffsetMetadata == null || record.kafkaOffset() > curOffsetMetadata.offset())
                 : curOffsetMetadata);
                Future<RecordMetadata> future = producer.send(producerRecord, (recordMetadata, e) -> {?
                    if (e != null) {
                       new logOffsetAndMetadata(record.error("{} failed to send record to {}: ", MirrorSinkTask.this, producerRecord.topic(), e);
kafkaOffset())
                                   log.debug("{} Failed record: {}", MirrorSinkTask.this, producerRecord : curOffsetMetadata);
                Future<RecordMetadata> future       throw new KafkaException(e);
= producer.send(producerRecord, (recordMetadata, e) -> {
                    if (e }!= elsenull) {
                        log.infoerror("{} Wrotefailed recordto successfully:send topicrecord {}to partition {}: offset {}", //log.trace", MirrorSinkTask.this, producerRecord.topic(), e);
                        log.debug("{} Failed record: {}", 		MirrorSinkTask.this, producerRecord);
                        throw new KafkaException(e);
      recordMetadata.topic(), recordMetadata.partition(),
             } else {
                 recordMetadata.offset());
       log.info("{} Wrote record successfully: topic {} partition {} offset      {}", //log.trace
   commitRecord(record, recordMetadata);
                    }		MirrorSinkTask.this,
                });
                futures.add(future);recordMetadata.topic(), recordMetadata.partition(),
                records.poll();
            }

        } catch (KafkaException e) {
recordMetadata.offset());
              // Any unsent messages are added to the remaining remainingRecordsMap for re-sendcommitRecord(record, recordMetadata);
            for (SinkRecord record = records.poll(); record != null; record}
 = records.poll()) {
            	addConsumerRecordToTopicPartitionRecordsMap(record, remainingRecordsMap});
          } finally {  //TODO: may futures.add(future);
 more exception handling case
            for (Future<RecordMetadata> future : futures) {
records.poll();
            }

      try {
 } catch (KafkaException e) {
            // Any  future.get();
        unsent messages are added to the remaining remainingRecordsMap for re-send
        } catch (Exception e) {
for (SinkRecord record = records.poll(); record != null; record = records.poll()) {
         SinkRecord record = futureMap.get(future	addConsumerRecordToTopicPartitionRecordsMap(record, remainingRecordsMap);
        } finally {  //TODO: may add more exception handling case
    // Any record failed to send, add to thefor remainingRecordsMap
(Future<RecordMetadata> future : futures) {
                addConsumerRecordToTopicPartitionRecordsMap(record, remainingRecordsMap);try {
                }
    future.get();
        }
         }

 catch  (Exception e) {
    if (isTransactional && remainingRecordsMap.size() == 0) {
          SinkRecord record = producerfutureMap.sendOffsetsToTransaction(offsetsMap, consumerGroupIdget(future);
             commitTransaction(producer);
       // Any record failed to }

send, add to the remainingRecordsMap
    if (remainingRecordsMap.size() > 0) {
            // For transaction case, all records should be put into remainingRecords, as the whole transaction should be redoneaddConsumerRecordToTopicPartitionRecordsMap(record, remainingRecordsMap);
                }
            Collection<SinkRecord> recordsToReSend;}
         }

   if (isTransactional) {
    if (isTransactional && remainingRecordsMap.size() == 0) {
      // transactional: retry all records, as the transaction will cancel all successful and failed records.
 producer.sendOffsetsToTransaction(offsetsMap, consumerGroupId);
             commitTransaction(producer);
       recordsToReSend = records;}

        if (remainingRecordsMap.size() > 0) } else {
            // For transaction  // non-transactional: only retry failed records, others were finished and sent.
case, all records should be put into remainingRecords, as the whole transaction should be redone
            Collection<SinkRecord> recordsToReSend;
       recordsToReSend = remainingRecordsMap;
   if (isTransactional) {
       }
         // transactional: retry throwall new ResendRecordsException(recordsToReSend);
        }records, as the transaction will cancel all successful and failed records.
    }

    @Override
    public Map<TopicPartition, OffsetAndMetadata> preCommit(Map<TopicPartition, OffsetAndMetadata>recordsToReSend offsets)= {records;
    	// if transactional, return empty Map intentionally to disable the offset commit by commitOffsets() in WorkerSinkTask.java
        } else {
                // non-transactional: soonly thatretry thefailed transactionalrecords, producerothers iswere ablefinished toand commitsent.
 the consumer offsets to target cluster in a transaction at its pace
    	if (isTransactional) recordsToReSend = remainingRecordsMap;
    		return new HashMap<TopicPartition, OffsetAndMetadata>();
        }
      	else  // otherwise, return offsetsMap tothrow letnew commitOffsetsResendRecordsException(recordsToReSend);
 in "WorkerSinkTask.java" to commit offsets
    		return offsetsMap;	}
    }

    // This commitRecord() follows the same logics as commitRecord() in MirrorSourceTask, to 
    public void commitRecord(SinkRecord record, RecordMetadata metadata) {
        try {
            if (stopping) {
                return;
            }
            if (!metadata.hasOffset()) {
                log.error("RecordMetadata has no offset -- can't sync offsets for {}.", record.topic());
                return;
            }
            TopicPartition topicPartition = new TopicPartition(record.topic(), record.kafkaPartition());
            long latency = System.currentTimeMillis() - record.timestamp();
            metrics.countRecord(topicPartition);
            metrics.replicationLatency(topicPartition, latency);
            TopicPartition sourceTopicPartition = MirrorUtils.unwrapPartition(record.sourcePartition());
            long upstreamOffset = MirrorUtils.unwrapOffset(record.sourceOffset());
            long downstreamOffset = metadata.offset();
            maybeSyncOffsets(sourceTopicPartition, upstreamOffset, downstreamOffset);
        } catch (Throwable e) {
            log.warn("Failure committing record.", e);
        }
    }

    private void beginTransaction(KafkaProducer<byte[], byte[]> producer) {
        if (isTransactional) {
            producer.beginTransaction();
            transactionStarttransactionInProgress = true;
        }
    }
	
    private void initTransactions(KafkaProducer<byte[], byte[]> producer) {
        if (isTransactional) {
            producer.initTransactions();
        }
    }
    
    private void commitTransaction(KafkaProducer<byte[], byte[]> producer) {
        if (isTransactional) {
            producer.commitTransaction();
            transactionStarttransactionInProgress = false;
        }
    }
    
    private void abortTransaction(KafkaProducer<byte[], byte[]> producer) {
        if (isTransactional && transactionStarttransactionInProgress) {
            producer.abortTransaction();
            transactionStarttransactionInProgress = false;
        }
    }

    public static class ResendRecordsException extends Exception {
        private Collection<SinkRecord> remainingRecords;

        public ResendRecordsException(Collection<SinkRecord> remainingRecords) {
            super(cause);
            this.remainingRecords = remainingRecords;
        }

        public Collection<SinkRecord> getRemainingRecords() {
            return remainingRecords;
        }
    }

...

As SinkTask can only be created by SinkConnector, MirrorSinkConnector will be implemented and follow the most same logics as current MirrorSourceConnector. To minimize the duplicate code, a new class, e.g. "MirrorCommonConnector", will may be proposed to host the common code used by both MirrorSourceConnector and MirrorSinkConnectorcode as a separate code change merged before this KIP.

Migration from MirrorSourceConnector to MirrorSinkConnector /w EOS

...