Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • When producer sends a too large record in non-transactional mode, the producer send() method throws no exception but returns a record metadata that includes the error RecordTooLargeException. Obviously, a failed sent record does not reach the broker.  
  • With the changes made here (because of
    Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyKAFKA-9279
    ) the producer send() method throws a RecordTooLargeException facing too large records in transactions. The user can bring the custom handler to bear to avoid the entire batch failing by dropping the poisoning too large record (SWALLOW the error). This way, this record does not get included in the batch.
  • There is another scenario in which the record size is acceptable by the producer (the record is NOT too large from producer point of view due to setting the "max.request.size" and "buffer.memory" to big numbers in producer config), but it is too large for the broker (The default message size of broker is 1 MB). In such case, the broker throws RecordTooLargeException during commitTransaction(). This scenario is not the focus of this KIP.

Examples

Code Block
languagejava
// Example 1: RecordTooLargeException use case 
public class Example1 {  
	public static void main(String[] args) {
        
		Properties producerProps = new Properties();
        ..... // omitted for brevity
	    producerProps.put("custom.exception.handler", Example1.ContinueTransaction.class.getName());
        KafkaProducer<String, String> producer = new KafkaProducer(producerProps);

        Properties consumerProps = new Properties();
        ..... // omitted for brevity
        KafkaConsumer<String, String> consumer = new KafkaConsumer(consumerProps);

        StringBuilder largeMessageStringBuffer = new StringBuilder();
        for (int i = 0; i < 1000000; i++) { 
            largeMessageStringBuffer.append("0123456789");
        }
        String largeMessge = largeMessageStringBuffer.toString();

        
		producer.initTransactions();
		consumer.subscribe(Collections.singleton("input-topic"));

        while(true) {
            try {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(60));
                if (records.count() > 0) {
                    producer.beginTransaction();
                    Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap();
                    for (ConsumerRecord<String, String> record : records) {
                        ProducerRecord<String, String> largeRecord = new ProducerRecord("output-topic", largeMessge);
						Future<RecordMetadata> send = producer.send(largeRecord);
                        offsets.put(new TopicPartition(record.topic(), record.partition()), new OffsetAndMetadata(record.offset() + 1));
                    }

                    producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
                    producer.commitTransaction();
                }
            } catch (Exception e) {
                producer.abortTransaction();
                throw new RuntimeException(e);
            }
        }
    }
	public static class ContinueTransaction implements ProducerExceptionHandler {
        @Override
        public ProducerExceptionHandler.Response handle(ProducerRecord<byte[], byte[]> producerRecord, Exception e) {
            return ProducerExceptionHandler.Response.SWALLOW;
        }
        @Override
        public void configure(Map<String, ?> map) {

        }
    }
}




Compatibility, Deprecation, and Migration Plan

Changed behaviour: The default behaviour stays as it is, but the user can change the behaviour by implementing the handle() function.

...