THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
March 21, 2021, 2021, Sunday.
Time: 2021-03-21 9:00 PM Beijing Time
WeMeet url: https://meeting.tencent.com/s/xoV6xnkBw5oT
Host: Yuyuan Kang
Status Updates
1. Announce
Apache IoTDB 0.11.13 is still in progress
2. Bug Fixes
- ISSUE-2505 ignore PathNotExistException in recover and change recover error to warn
- IOTDB-1119 Fix C++ SessionDataSet bug when reading value buffer
- Fix SessionPool does not recycle session and can not offer new Session due to RunTimeException
- ISSUE-2588 Fix dead lock between deleting data and querying in parallel
- ISSUE-2546 Fix first chunkmetadata should be consumed first
- IOTDB-1126 Fix unseq tsfile is deleted due to compaction
- IOTDB-1137 MNode.getLeafCount error when existing sub-device
- ISSUE-2624 ISSUE-2625 Avoid OOM if user don't close Statement and Session manually
- ISSUE-2639 Fix possible NPE during end query process
- Alter IT for An error is reported and the system is suspended occasionally
- IOTDB-1149 print error for -e param when set maxPRC<=0
- IOTDB-2648 Last query not right when having multiple devices
- Delete mods files after compaction
- ISSUE-2687 fix insert NaN bug
- ISSUE-2598 Throw explicit exception when time series is unknown in where clause
- Fix timeseriesMetadata cache is not cleared after the TsFile is deleted by a compaction
- ISSUE-2611 An unsequence file that covers too many sequence file causes OOM query
- IOTDB-1135 Fix count timeseries bug when the paths are nested
- ISSUE-2709 IOTDB-1178 Fix cache is not cleared after compaction
- ISSUE-2746 Fix data overlapped bug after the elimination unseq compaction process
- Fix getObject method in JDBC should return an Object
- IOTDB-1188 Fix IoTDB 0.11 unable to delete data bug
- Fix when covering a tsfile resource with HistoricalVersion = null, it鈥檒l throw a NPE
- fix the elimination unseq compaction may loss data bug after a delete operation is executed
3. New Features
- Add explain sql support
4. Improvements
- IOTDB-1140 optimize regular data encoding
- Add more log for better tracing
- Add backgroup exec for cli -e function
- Add max direct memory size parameter to env.sh
5. Closed development
Aligned Timeseries & Device Template
Goal: Eliminate duplicate timestamps, eliminate duplicate definitions of leaf nodes, thereby reducing metadata memory usage and metadata storage
Solutions:
1) By aligning the time series, duplicate timestamps can be eliminated.
2) Add device templates to eliminate duplicate definitions of leaf nodes
Typical Scenarios: To manage 5 billion device * 29 measurements, the measurements on a device are aligned by timestamp
1) SQL statement for aligned timeseries:
create aligned timeseries root.sg.d1.(s1 FLOAT, s2 INT32)
create aligned timeseries root.sg.d1.(s3 FLOAT, s4 INT32) with encoding=(RLE, Grollia), compression=SNAPPY
insert into root.sg.d1(time, s1, s2, s3, s4) values(1, 1, 2.0, 1, 2.0)
insert into root.sg.d1(time, (s1, s2), s5) values(2, 1, 2.0, 3)
select s1, s2 from root.sg.d1
select s1, s3 from root.sg.d1
2) SQL statement for device template:
set storage group root.beijing
create device template temp1(
(s1 INT32 with encoding=Gorilla, compression=SNAPPY),
(s2 FLOAT with encoding=RLE, compression=SNAPPY)
)
set device template temp1 to root.beijing
create device template temp2(
((s1 FLOAT, s2 INT32) aligned with encoding=(Gorilla, RLE), compression=SNAPPY)，
(s3 FLOAT with encoding=RLE, compression=SNAPPY)
show device template
+--------+-------------+----------+----------+-------------+
|template| measurement | datatype | encoding | compression |
+--------+-------------+----------+----------+-------------+
| temp1 | s1 | INT32 | Gorilla | SNAPPY |
| temp1 | s2 | FLOAT | RLE | SNAPPY |
| temp2 | s1 | FLOAT | Gorilla | SNAPPY |
| temp2 | s2 | INT32 | RLE | SNAPPY |
| temp2 | s3 | FLOAT | RLE | SNAPPY |
+--------+-------------+----------+----------+-------------+
Data Management
1) Meta data:
- Implement device template (a group of measurements)
- Eliminate redundant definition of the leaf node
2) Data in Memory:
- New data structure for multiple columns sharing a same timestamp
- Eliminate redundant timestamp
- Reduce memory consumption
3) Data on disk:
- Designed new TsFile format
- Support data of multiple columns sharing a same timestamp
- Reduce disk consumption
Result:
One of the scenarios:
...
Write efficiency (row/sec)
...
Single-state, old strategy
...
190,624
...
Single-state, new strategy
...
487,656
...
Distributed, 3 nodes 1 replica, new strategy
...
1,095,331
6. Open Floor
new ideas, feedback, suggestions.
1) [Discussion] Monthly Contributor Award
February‘s “Contributors of The Month”
- sunjincheng121(Jincheng Sun), with 132571 lines changed and 8 mails sent;
- HTHou(Haonan Hou), with 2116 lines changed and 4 mails sent;
- neuyilan(Houliang Qi), with 2692 lines changed and 3 mails sent.
There are also 22 other contributors who contributed to IoTDB this month.
7 New contributors submitted their first PR in IoTDB
- wuzhaojie(Zhaojie Wu),
- WilliamSong11(Yuxiang Song),
- GLBB,
- THUMarkLau(Xuxin Liu),
- chenjun40,
- 543202718(Haoyu Wang)
- jxlgzwh(Wenhao Zhong).
2) [Proposal] Hackthon (April)
Mar 7st, 2021, Sunday.
Time:
Host:
...
https://github.com/apache/iotdb/blob/0f48ba2de83d2b1c0da00bba8485baa1c60abe28/docs/zh/UserGuide/Operation%20Manual/Triggers.md
...
[I24] so cl ndes a
...
...
...
Future work：
new ideas, feedback, suggestions.
Goal:
Attention:
Preliminary measure:
...
The final equation would be S=0.8Z(code)+0.1Z(doc)+0.1Z(mail)
- Q & A docs
- start up docs
- Sign up : 45 people
https://github.com/apache/iotdb/issues/2715
<= 5
T-shirt, backpack, bottle with IoTDB logo
agenda:
April: get familiar with each other and communicate with mentor
May: coding
Time:
Host
Feb 21st, 2021, Sunday.
Host: Yuan Tian
Online Meeting (Tencent Meeting):
...
meeting room ID:
password:
Time:
Host:
...
nohup ./start-cli.sh -eNotice:
new ideas, feedback, suggestions.
Goal:
Attention:
Preliminary measure:
...
The final equation would be S=0.9Z(code)+0.1Z(mail)
两个方向上的称号：
代码贡献+代码review贡献
- Q & A docs
- start up docs
细分设计
模块
https://github.com/apache/iotdb/issues/2715
Time:
Host
Meeting bi-weekly Sunday at 9:00 pm Beijing Time.
...
https://shimo.im/docs/dt8C9cQtKHGgkH89
Status Updates
Announce
Apache IoTDB 0.11.2 released (01-23)
...
Open Floor
new ideas, feedback, suggestions.
1. [Discussion] Monthly Contributor Award
Goal: Keep community more vibrant and attract new contributors
Attention: Not really an award, just a title that we can recognize with an announcement on the official wechat account
Preliminary measure:
Amount of PR and reviews on GitHub (codes amount)
Activity on mailing lists
Contributions(publicity, tutorial, etc.) on social media (e.g. WeChat official account, CSDN, Stack Overflow)
Being a conference host or event organizer will also count.
2. [Discussion] Community Mentors
Call for mentors and plans
Q & A docs
start up docs
Wechat group
...
- We held the first Apache IoTDB offline meetup successfully in 29 November. We mainly talked about the experience of using IoTDB and the development of it. But as it's the first time, we have many aspects to improve.
- Add free talk
In last meetup, the main activities we held were the lectures about IoTDB using and developing. We got a lot from these lectures. But the offline meetup is also a good chance to help community members get to know each other. So I think, we can add free talk in meetup, maybe academic or not, to offer this chance. - Bring the summary of meetup back to community
For various reasons, some friends in the community can't join meetup offline. Therefore, we need to record the meetup by video and doc, and share the summary of meetup to them by mailing list or IoTDB website.
- Add free talk
- Community gift
We plan to design some community themed little gifts for contributors who has contributions (code, doc or something else) to the community. At present, we have completed the design of mouse pad. Next, we can collect the personal information of the contributors to mail these gifts.
- Call for community mentors
We used to propagate IoTDB in many Chinese universities, and many students are interested in IoTDB. However, the development threshold of IoTDB may be difficult for some newcomers who have no development experience.
So we think if some community members are familiar with some aspects of IoTDB and interested in developing new people. They can be community mentors to help these newcomers by explaining some code and introducing some simple issues etc.
We can start an enrollment program to recruit mentors, and after that start a vote about whether they can be the mentors. - About IoTDB publicity
For a long time, due to the lack of special attention to publicity, the effect of IoTDB publicity is not good. This also leads to the lack of contributors and the unknown situation of IoTDB.
Therefore, next we can try to put IoTDB's promotional materials, documents and conference videos like the meetup video on more platforms, like IoTDB website, WeChat official account, Bilibili or even Tiktok.
...