You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

In order to use Apache Flink, you need to get data in Flink from a source or send data from Flink to a sink. In order to achieve this, you need connectors. The Flink community has created and maintains multiple Flink connectors, which can be found in multiple locations.

The Flink community wants to improve on the overall connector ecosystem, which includes that we want to moving existing connectors out of Flink's main repository and as a result decouple the release cycle of Flink with the release cycles of the connectors. This should result in:

  • Faster releases of connectors: New features can be added more quickly, bugs can be fixed immediately, and we can have faster security patches in case of direct or indirect (through dependencies) security flaws.
  • Adding newer connector features to older Flink versions: By having stable connector APIs, the same connector artifact may be used with different Flink versions. Thus, new features can also immediately be used with older Flink versions. 
  • More activity and contributions around connectors: By easing the contribution and development process around connectors, we will see faster development and also more connectors.
  • Standardized documentation and user experience for the connectors, regardless of where they are maintained. 
  • A faster Flink CI: By not needing to build and test connectors, the Flink CI pipeline will be faster and Flink developers will experience less build stabilities (which mostly come from connectors). That should speed up Flink development.

The following has been discussed and agreed by the Flink community with regards to connectors:

  • All connector repositories will remain under the ASF, which means that code will still be hosted at https://github.com/apache and all ASF policies will be followed. 
  • Each connector will end up in its own connector repository. For example, https://github.com/apache/flink-connector-kafka for a Kafka connector, https://github.com/apache/flink-connector-elasticsearch for an Elasticsearch connector etc.

  • The following connectors will be moved out from Flink's main repository to an individual repository:

    Kafka
    Upsert-Kafka
    Cassandra
    Elasticsearch
    Kinesis
    RabbitMQ
    Google Cloud PubSub
    NiFi
    Pulsar
    Twitter
    JDBC
    HBase
    Hive

  • Only the following connectors will remain in Flink's main repository:

    Hybrid Source
    FileSystem
    DataGen
    Print
    BlackHole
  • PRs for new connectors to Flink's main repository should not be merged, as these new connectors should also be hosted outside of Flink's main repository. If you have a connector that you would like to build or maintain, please reach out to the Flink Dev mailing list https://flink.apache.org/community.html for more information to get started using the external connector repository setup. 
  • No labels