Background

Apache IoTDB is an open-source IoT-native time-series database designed for high-performance storage, ingestion, and analysis of massive time-series data from IoT devices. It supports deep integration with big data ecosystems like Apache Hadoop, Spark, and Flink, enabling seamless data processing workflows. IoTDB traditionally uses a tree-based data model for organizing time-series data hierarchically (e.g., root.group.device.sensor), which is efficient for device-centric IoT scenarios.
Starting with IoTDB 2.0, a dual-mode SQL architecture was introduced, adding a table mode alongside the tree mode. The table mode allows users to manage time-series data using SQL-like table structures, where each table represents a device type, with columns for timestamps, tags, and fields (e.g., measurements like temperature or humidity). This mode enhances flexibility for data analysis, supports standard SQL queries, and improves interoperability with relational tools. It is particularly useful for scenarios involving heterogeneous devices or advanced analytics, as it supports table-level schema management and retention-related configurations (e.g., TTL).
Apache Flink is a powerful stream and batch processing framework for real-time data analytics. IoTDB already provides a Flink connector (flink-iotdb-connector) for reading from and writing to IoTDB using the tree mode, including IoTDBSource for data ingestion and IoTDBSink for output. There is also a Flink SQL connector (flink-sql-iotdb-connector) for SQL-based interactions and change data capture (CDC). However, these connectors primarily target the tree mode and lack full support for the table mode's features, such as table-specific metadata handling, SQL table mappings in Flink Table API, and optimized read/write operations for table-structured data. As a result, Flink users cannot natively treat IoTDB table-mode data as first-class tables in Flink SQL or the Table API. This gap limits the ability to leverage Flink's processing capabilities with IoTDB's modern table mode, especially in real-time IoT applications like predictive maintenance or anomaly detection.
This project aims to bridge this gap by developing a dedicated Flink connector for IoTDB's 2.X table mode, enabling efficient, real-time integration between Flink and IoTDB tables.

Goal

The primary goal is to create a robust, production-ready Flink connector that supports reading from and writing to IoTDB tables using the 2.X table mode. This will allow Flink users to process IoT time-series data stored in table format, perform transformations, aggregations, and joins in real-time, and sink results back into IoTDB tables. The connector should align with Flink's DataStream and Table APIs, support fault tolerance, and handle table-specific features like tags, fields, and TTL. Ultimately, this will enhance IoTDB's ecosystem integration, making it easier for developers to build scalable IoT data pipelines.
 

Core Tasks (Mandatory)

1. Research and Design: Analyze the existing flink-iotdb-connector and flink-sql-iotdb-connector to identify limitations with the table mode. Design the connector architecture, including schema and type mappings between Flink Table/RowData and IoTDB table-mode concepts (e.g., time column, tags, and fields). Define APIs for source and sink functions compatible with Flink 1.18+.

2. Implement IoTDB Table Source: Develop a Flink source connector (e.g., IoTDBTableSource) that reads data from IoTDB tables. Support filtering by time ranges, tags, and fields using IoTDB's SQL interface. Ensure it handles schema inference and dynamic table changes.

3. Implement IoTDB Table Sink: Create a Flink sink connector (e.g., IoTDBTableSink) for writing processed data back to IoTDB tables. Support batch and streaming modes, automatic schema creation (if enabled in IoTDB), and error handling for constraints like TTL or data types.

4. Testing and Documentation: Write unit and integration tests using Flink's testing utilities and IoTDB test clusters. Document usage examples, configuration options, and deployment guides in the IoTDB repository.

5. Community Contributions: Submit pull requests to upstream repositories for any required changes, and create example Flink jobs demonstrating the use cases.

Advanced Tasks (Optional)

• Performance Optimization: Implement optimizations like parallel reading/writing.

• Benchmarking and Comparison: Develop benchmarks comparing the new connector's performance with the existing tree-mode connector, focusing on throughput, latency, and resource usage in IoT scenarios.

Deliverables

• 
  Source code for the Flink connector for IoTDB table mode, including Maven artifacts (e.g., flink-iotdb-table-connector).
• 
  Comprehensive documentation, including API references, setup guides, and usage examples integrated into the IoTDB website.
• 
  Test suites covering core functionality, edge cases, and integration with Flink.
• 
  A demo application showcasing a complete Flink pipeline reading from/writing to IoTDB tables.
• 
  Optimization reports, benchmarks, and any upstream PRs.

Recommended Skills

• Programming Language: Proficiency in Java, as both Flink and IoTDB are primarily Java-based projects.

• Database Knowledge: Understanding of SQL and fundamental database concepts. Knowledge of time-series data is a plus.

• System Integration: Interest or experience in connecting different systems and understanding data flows.

• Learning and Communication: Ability to quickly understand the codebases of two open-source projects and willingness to actively collaborate with community mentors and members.

Learning Material

• 
  Apache IoTDB Official Website: https://iotdb.apache.org/
• 
  Apache Flink Official Documentation: https://flink.apache.org/
• 
  Integrated Reference: https://github.com/apache/iotdb-extras/tree/master/connectors/flink-iotdb-connector
• 
  IoTDB Table Model Concepts: https://iotdb.apache.org/UserGuide/latest/Background-knowledge/Data-Model-and-Terminology_apache.html
• 
  IoTDB Table Model Query Syntax: https://iotdb.apache.org/UserGuide/latest-Table/SQL-Manual/overview_apache.html

 
Difficulty: medium
Mentor: Haonan Hou (Apache IoTDB PMC member) (haonan@apache.org)
 

Background

Apache IoTDB is a high-performance, open-source time-series database optimized for data management and analysis in Internet of Things (IoT) scenarios, while ThingsBoard is an open-source IoT platform for device management, data visualization, and rule-based automation.
With the release of IoTDB 2.X introducing a dual-mode architecture (tree and table), significant opportunities arise to enhance this integration. The table mode supports standard SQL syntax, JOIN operations, and user-defined functions, enabling more complex queries and analytics. This project proposes to develop an enhanced storage backend for ThingsBoard based on IoTDB's 2.X table mode, providing improved flexibility and performance for IoT data storage and analysis.

Goal

The primary goal of this project is to design and implement a new, enhanced storage backend for ThingsBoard that strategically leverages key features of Apache IoTDB 2.X’s table mode to improve flexibility, query expressiveness, and performance for core IoT telemetry workloads. This enhancement aims to provide ThingsBoard users with more powerful SQL querying capabilities (including complex multi-device joins and time-window aggregations) and improved performance for specific workloads. Furthermore, the project seeks to strengthen the open-source ecosystem by providing a deeper, more capable integration between ThingsBoard and the Apache IoTDB project, resulting in a more robust end-to-end IoT solution for the community.
 

Core Tasks (Mandatory)

1. In-depth Analysis and Design: Conduct a thorough analysis of the existing ThingsBoard-IoTDB integration architecture and ThingsBoard's storage backend interfaces (e.g., TimeseriesDao). Then, design an optimal strategy for mapping the ThingsBoard data model (devices, assets, telemetry, attributes, labels) to the IoTDB 2.X table mode. A key focus will be utilizing IoTDB's TAGS column to efficiently store and manage static device attributes (e.g., location, device type), enabling flexible device filtering and grouping based on these tags .
2. Implementation of Storage Backend Connector:
   1. Data Access Layer: Based on the design, implement the relevant ThingsBoard storage backend interfaces to connect with IoTDB.
   2. Write Path: Develop efficient data writing logic that transforms device telemetry data received by ThingsBoard and performs batch writes to the corresponding tables in IoTDB.
   3. Read/Query Path: Implement query interfaces that translate data requests from ThingsBoard dashboards or the rule engine into efficient SQL queries that take full advantage of IoTDB 2.X table mode features.
3. Performance Benchmarking and Comparison: Design and execute standardized performance test cases (e.g., high-concurrency data ingestion, complex conditional queries, large-scale range queries). Produce a detailed performance comparison report between the new IoTDB 2.X table mode-based backend and ThingsBoard's existing data storage options, This report should quantify improvements in metrics like write throughput and query latency.
4. Testing and Documentation: Write comprehensive integration tests to ensure the correctness and stability of the new functionality. Create detailed user documentation, including installation/configuration instructions, data model explanations, API usage guidelines, and best practices.
5. Community Collaboration and Upstream Contribution: Actively communicate with the ThingsBoard open-source community at key project milestones to discuss designs and gather feedback. Submit high-quality Pull Requests (PRs) to the official ThingsBoard repository, adhering to its coding standards, with the goal of getting the implementation merged.

Advanced Tasks (Optional)

• Leverage IoTDB UDFs: Explore the integration of IoTDB's User-Defined Functions (UDFs) within ThingsBoard's rule engine. This could allow for performing more complex data processing and analysis (e.g., anomaly detection) directly within the database before data is pulled into ThingsBoard.

• Enhanced Data Modeling for Assets: Extend the data mapping design to optimally support ThingsBoard's assets and the relations between entities (devices, assets, customers), exploiting the relational capabilities of the IoTDB table mode for more complex queries.

• Comprehensive Dashboard Demo: Build a detailed ThingsBoard dashboard that showcases the advanced querying capabilities made possible by the new integration, such as visualizations based on multi-device joins or complex aggregations.

Deliverables

1. A fully functional storage backend plugin/implementation, including source code, build scripts, and configuration examples.
2. A detailed design document explaining the data mapping and integration architecture between ThingsBoard and the IoTDB 2.X table mode.
3. A comprehensive performance benchmark report comparing the new solution with existing options.
4. Complete user and developer documentation.
5. A Pull Request submitted to the ThingsBoard community containing the implementation, tests, and relevant documentation.
6. A final project report summarizing work, technical challenges, learnings, and future possibilities.

Recommended Skills

• Programming Language: Proficiency in Java, as both ThingsBoard and IoTDB are primarily Java-based projects.

• Database Knowledge: Understanding of SQL and fundamental database concepts. Knowledge of time-series data is a plus.

• System Integration: Interest or experience in connecting different systems and understanding data flows.

• Learning and Communication: Ability to quickly understand the codebases of two open-source projects and willingness to actively collaborate with community mentors and members.

Learning Material

• Apache IoTDB Official Website: https://iotdb.apache.org/
• ThingsBoard Official Documentation: https://thingsboard.io/docs/
• Integrated Reference: https://github.com/thingsboard/thingsboard/pull/11476
• IoTDB Table Mode Concepts: https://iotdb.apache.org/UserGuide/latest/Background-knowledge/Data-Model-and-Terminology_apache.html
• IoTDB Table Mode Query Syntax: https://iotdb.apache.org/UserGuide/latest-Table/SQL-Manual/overview_apache.html

 
Difficulty: medium
Mentor: Xuan Wang (Apache IoTDB Committer) (critas@apache.org^{Image Added})