Page History

Mar 7st, 2021, Sunday.

Time: 2021-03-07 9:00 PM Beijing Time

Host: Kaifeng Xue

Agenda

Status Updates 

Announce

Apache IoTDB 0.11.3 and 0.12.0 is still in progress

Improvement

...

Chinese:

https://github.com/apache/iotdb/blob/0f48ba2de83d2b1c0da00bba8485baa1c60abe28/docs/zh/UserGuide/Operation%20Manual/Triggers.md

...

#2732 [ISSUE-2493] show child nodes path

Image Removed

...

At present, the executeBatch method in JDBC is executed separately for each statement.

...

Now we let execute batch statment in JDBC really execute in batch at server.

...

Currently, only one aggregated measurement is supported in this statement to avoid result ambiguity.

This PR is to support multiple aggregated measurements in group by level statement.

Future work：

1 分布式性能优化

2 单机特殊场景优化

3 社区issue解决

Open Floor 

new ideas, feedback, suggestions.

1. [Discussion] Monthly Contributor Award

Goal: Keep community more vibrant and attract new contributors

Attention: Not really an award, just a title that we can recognize with an announcement on the official wechat account

Preliminary measure:

...

The final equation would be S=0.8Z(code)+0.1Z(doc)+0.1Z(mail)

两个方向上的称号：

代码贡献+代码review贡献

2.[Discussion] Community Mentors

• Call for mentors and plans
• Q & A docs（使用 Github issue 维护，所有小组共享）
• start up docs (导师提供的上手基础资料，如代码仓库、设计文档、如何调试、经验分享、公众号等，导师之间可以共享) —— 组建小组群之前
• Wechat group
• Phase Feedback（阶段性反馈）
• Sign up : 45 people

• To be as commiter : 17
• Users needing professional help : 5
• Both : 23

报名人群可以细分：已经熟悉IoTDB但是对于设计不够熟悉，如何入手修改

感兴趣的模块：查询 / 存储 / 分布式 / 实时计算 / 全部

灵活、不限时间，没有集中的开发时间，与现有社区的committer建立联系

每个导师负责 1-3 人，作为小组

2月10日之前：建群，start up文档，找导师

3.[Proposal] Hackthon (April)

开issue：大家罗列一些想法

https://github.com/apache/iotdb/issues/2715

确定一个issue模板，减少沟通成本

人数：

5人以下

1 committer + 2-4 non-committer

面向人群：校内学生+社区贡献者

奖励：现金奖励形式，总金额10万

第一名3-5万 高

鼓励奖要多

只要参加的都可以给一些带有社区logo的周边，T恤，鼠标垫，背包，电脑包，水杯

流程：

3.8-3.15 指定流程，放出宣传，校内宣传是重点之一

报名（3月底）

提前1个月确定导师与学生，一个月内可以进行线上沟通交流与设计（4月份）

两三天的集中code时间（5月初）

奖励评定（code结束后）

10th Meeting Schedule

Time: 2021-03-21

Host: Yuyuan Kang

Feb 21st, 2021, Sunday.

Host: Yuan Tian

Online Meeting (Tencent Meeting):

...

meeting room ID: 690 200 054

password: 0221

Time: 2021-02-21 9:00 PM Beijing Time

Host: Yuan Tian

Agenda

Status Updates 

Announce

Apache IoTDB 0.11.3 and 0.12.0 is still in progress

Improvement

#2596 Improve the flush pipeline

We used PublicBAOS(which extends ByteArrayOutputStream in java) as container to hold the encoded data. Encoding task call the write method to put data into it and IO task call the read method to read data from it, persist it to disk.

However, the methods we called in ByteArrayOutputStream are mostly synchronized. Although we won't call the method of the same object simultaneously in the stage of encoding and io, the Biased locking(which is added as a feature after JDK1.6 and is announced to be deprecated in JDK15) will cause the encoding stage to be influenced by the io stage.

The detailed analysis and experiments can be seen in confluence(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173085039&preview=/173085039/173085040/Pipeline%E5%88%86%E6%9E%90.pdf)

#2629 [Github Issue 2624, 2625] Avoid OOM if user don't close Statement and Session manually

When reusing IoTDBStatement to do queries. The Set in statementId2QueryId is never removed unless the statment is clsoed.

This will cause OOM if using a Statement is used for too long time.

...

#2658 Enhancement: add backgroup exec for cli -e function

Previously, there were some problems while we use the following command to execute in background. Now, we can successfully use it.

nohup ./start-cli.sh -e

#2684 Improve the quality and efficiency of code formatting

The Spotless with google-java-format now in effect.

Notice:

This change touched most of single java file in the repository(70K+). So

most of PRs will require a rebase, follow the steps below:

1. git checkout YOUR_PR

2. git cherry-pick b1261ecb5893ee0d357fec311b4d92ff9039a989

3. mvn spotless:check

4. mvn spotless:apply

5. git add .

6. git commit -m 'Apply Spotless'

# asf git@github.com:apache/iotdb.git (fetch)

# Pull the latest master branch

7. git checkout -b asf_master asf/master

8. git checkout YOUR_PR

9. git rebase asf_master

# Resolve your conflict:

# NOTE: `Accept Yours` for your PR involves files, and `accept Theirs` for

other files.

10 git rebase --continue

# Double check

11. mvn spotless:check

# If you see "build success" successfully, it is proved that rebase has

been successful.

12. mvn clean package -DskipTests

# If you see "build success" successfully, you can push your changes to you

PR branch.

Open Floor 

new ideas, feedback, suggestions.

[Discussion] Monthly Contributor Award

Goal: Keep community more vibrant and attract new contributors

Attention: Not really an award, just a title that we can recognize with an announcement on the official wechat account

Preliminary measure:

...

The final equation would be S=0.9Z(code)+0.1Z(mail)

两个方向上的称号：

代码贡献+代码review贡献

[Discussion] Community Mentors

• Call for mentors and plans
• Q & A docs（使用 Github issue 维护，所有小组共享）
• start up docs (导师提供的上手基础资料，如代码仓库、设计文档、如何调试、经验分享、公众号等，导师之间可以共享) —— 组建小组群之前
• Wechat group
• Phase Feedback（阶段性反馈）

Sign up : 45 people

• To be as commiter : 17
• Users needing professional help : 5
• Both : 23

报名人群可以细分：已经熟悉IoTDB但是对于设计不够熟悉，如何入手修改

感兴趣的模块：查询 / 存储 / 分布式 / 实时计算 / 全部

灵活、不限时间，没有集中的开发时间，与现有社区的committer建立联系

每个导师负责 1-3 人，作为小组

2月10日之前：建群，start up文档，找导师

[Proposal] Hackthon (April)

开issue：大家罗列一些想法

https://github.com/apache/iotdb/issues/2715

1 committer + 3 non-committer

面向人群：校内学生+社区贡献者

奖励：现金奖励形式

流程：

报名（3月下旬）

提前1个月确定导师与学生，一个月内可以进行线上沟通交流与设计（4月份）

两三天的集中code时间（5月初）

奖励评定（code结束后）

New Committers公众号推送

祥威、禹任 采访

9th Meeting Schedule

Time: 2021-03-07

Host: Kaifeng Xue

Meeting bi-weekly Sunday at 9:00 pm Beijing Time.

Language:

• Meeting Minutes: English
• Speaking: Chinese (if all the attendees are Chinese), English (if there is at least one non-chinese speaker ) 

Online Meeting (Tencent Meeting or VooV Meeting):

• https://meeting.tencent.com/s/sl5Xg1qnqc9V

• meeting room ID：469 7527 6512

• password：2020

January 31st, 2021, Sunday.

Time: 2021-01-31 9:00 PM Beijing Time
Host: Zesong Sun

https://shimo.im/docs/dt8C9cQtKHGgkH89

Status Updates

Announce

Apache IoTDB 0.11.2 released (01-23)

Notice: Prepare to release 0.11.3, and keep maintaining 0.11.x version for a while until v0.12.x is stable.

Currently, the main branch is serving for 0.12 and the version changes the TsFile format once again.

Though we will provide the upgrade tool, it is still a big decision for users to upgrade the version (as they need to upgrade existing data files).

Besides, even 0.12.0 will be released, we may need 1~2 versions (or more) to guarantee the version stable.

Vote

change release tags from release/{full-version} to v{full-version}

eg. release/0.11.2 to v0.11.2

Currently, we use release/{major}.{minor}.{patch} as our tag stale, which may bring some troubles for some languages like GoLang.

New Features

...

Currently out concurrency unit is storage group which can't make full use of cpu cores when user only set 1 or 2 storage group. We need virtual storage group to improve concurrent performance. This PR finish first part of the design

#2488 Add DirectByteBufferPool for WAL

There are two problems in the process of current wal flushing.

Because wal uses FileChannel while flushing, it involves one more memory copy. FileChannel will copy the contents of HeapByteBuffer to DirectByteBuffer before calling native write method, and then call native write with DirectByteBuffer as a parameter, which will cause An extra memory copy operation.

Although 0.12 uses the thread pool to complete asynchronous flushing, to a certain extent, solves the problem of the surge of non-heap memory caused by the wal of 0.11, but when the non-heap memory is released is still depends on the jvm (depends on when gc is performed), jvm cannot know whether non-heap memory should be released in time, so it may also cause non-heap memory not be released in time, but we are clear about that.

So, I want to use DirectByteBufferPool to solve the above two questions.

#2184 [IOTDB-1081] New TsFile Format

The TsFile format version will update to V3.
It will be applied in the next major version (i.e., 0.12).
We find that the former tsfile format waste some disk space, some information was stored duplicately and some legacy fields are no longer useful.

#2603 Accelearate CI by split a CI into several parts

split the main-ci into 3 parts: server, cluster, and others
add two parameters to just test the server or the cluster, e.g.,
-DskipTests -Diotdb.test.only=true
-DskipTests -Dcluster.test.only=true
Split the main-ci into two github action groups to save resource when re-run some actions.
disable enforcer by default, to enable it, use -P enforce
disable generating jar-with-dependencies, to enable it, use -P get-jar-with-dependencies

Open Floor
new ideas, feedback, suggestions.

1. [Discussion] Monthly Contributor Award

Goal: Keep community more vibrant and attract new contributors
Attention: Not really an award, just a title that we can recognize with an announcement on the official wechat account
Preliminary measure:
Amount of PR and reviews on GitHub (codes amount)
Activity on mailing lists
Contributions(publicity, tutorial, etc.) on social media (e.g. WeChat official account, CSDN, Stack Overflow)
Being a conference host or event organizer will also count.

2. [Discussion] Community Mentors

Call for mentors and plans
Q & A docs
start up docs
Wechat group

Sign up : 45 people
To be as commiter : 17
Users needing professional help : 5
Both : 23

3. [Experiment] Compares to influxDB in terms of performance

Need to update to the website:
https://iotdb.apache.org/UserGuide/Master/Comparison/TSDB-Comparison.html

4. [Proposal] Hackthon (April)

1 committer + 3 non-committer

January 3rd, 2021, Sunday.

Time:  9:00 PM Beijing Time

Host：Xinyu Tan

Attendees： 

Agenda: (put what you want to discuss in the meeting here before the meeting starts)

...

1. all committers can be mentors(wechat group)
2. Q & A docs, FAQ docs, start up docs.
3. combine
4. attract new contributors(wechat official accounts)

...

1. https://iotdb.apache.org/UserGuide/Master/Comparison/TSDB-Comparison.html
2. upgrade version 0.8.0→0.11.1 - xiangdong
3. tsbs benchmark: timescaledb、influxdb

...

1. chinese ppt
2. ApacheCon - use cases
3. https://iotdb.apache.org/Community/Materials.html upgrade 2020 - xiangdong

...

1. Houliang Qi
2. Xinyu Tan

...

1. code
   1. PRs(xinyu)
   2. modules - fix significant bug(xinyu)
   3. can maintain a module(xiangdong)
      1. a clear impression of someone
2. doc or others
3. pingcap : https://pingcap.com/blog-cn/tidb-community-upgrade/

...

1. avoid pushing to the repository directly
2. provide relatively detailed commit message.
3. Do not force-update a commit if someone has reviewed your code
4. use squash merge(whether need to disable directly-merge?)(xinyu Tan)
5. list the reproduction procedure when reporting bugs
6. detailed pr/issue title(is Chinese needed?)

...

Minutes:

• discuss1 (proposer): content
• discuss2 (): ...

December 20th, 2020, Sunday.

Time:  9:00 PM Beijing Time

Host：Lingzhe Zhang

Attendees： 

Agenda: (put what you want to discuss in the meeting here before the meeting starts.)

1. review of the conclusions of 3rd meeting
   1. About meet up
   2. Call for community mentors
      1. waiting
   3. About IoTDB publicity
      1. WeChat official account, Bilibili, Tiktok, aliyun(private account)
      2. video is uploaded in Bilibili
      3. self ask, self answer in StackOverFlow - ask jincheng
2. compares to influx in terms of performance
   1. https://iotdb.apache.org/UserGuide/Master/Comparison/TSDB-Comparison.html
   2. upgrade version 0.8.0→0.11.1 - xiangdong
   3. tsbs benchmark: timescaledb、influxdb
3. preproduction site
   1. chinese ppt
   2. ApacheCon - use cases
   3. https://iotdb.apache.org/Community/Materials.html upgrade 2020 - xiangdong
4. interop with ML tools
   1. need roadmap - need discuss
   2. python client - suyurong
      1. doc
      2. readme
      3. use case
5. New committer introduction - qiaojialin
   1. Houliang Qi
   2. Xinyu Tan
6. the label coverage of PR - xiangdongImage Removed
7. github robot - next meeting to be discussed
   1. https://github.com/marketplace
8. contributor weekly (qiaojialin) - need discuss
   1. all important contribution
   2. for all new contributor
9. best contributor monthly (zhouyurun) - need discuss and proposal
   1. PR
   2. review
   3. mailing list
   4. WeChat official account 
   5. doc
   6. event(activity)
10. announce of 0.11.1 (haonan)
    1. WeChat official account 
    2. or just announce 0.11.2 (qiaojialin)
       1. flush
11. to be a committer - need discuss
    1. code
       1. PRs(xinyu)
       2. modules - fix significant bug(xinyu)
       3. can maintain a module(xiangdong)
          1. a clear impression of someone
    2. doc or others

1. (5th presenter) next host
   1. tanxinyu,suyurong

Minutes:

• discuss1 (proposer): content
• discuss2 (): ...

December 6th, 2020, Sunday.

Time:  9:00 PM Beijing Time

Join Tencent Meeting or VooV Meeting:

https://meeting.tencent.com/s/U17cHJmg5T2n

Meeting ID: 335854726

Host：Xiangwei Wei

Attendees： 

Minutes:

...

Image Removed

Image Removed

...

All records before Dec 6th, 2020, can be accessed here : https://shimo.im/docs/rqxJrjHwx6r8tJwd (in Chinese)