FAQ Kylin 4.X

If you did not find the answer of your question, feel free to leave your comment under this wiki.

---

Question List

• How do RowKey affect storage & performance in Kylin 4.0?

To be updated

• What is Sparder(SparderContext)? And how should I take care of it ?

                Sparder is the implemenatation of new distributed query engine which backend by a spark application. If Sparder is dead, all your query will failed. So please check its liveness after Kylin instance(Query Server) was started in application list of Resource Manager Web UI. (We plan to add canary tools for monitor Sparder in 4.0.0-beta)

• Is Hadoop3 supported ?

                Kylin 4.0.0-alpha did not support Hadoop3. It is plan to be supported in 4.0.0-beta.

• If you faced Exception with message like this : "Cannot find hive-site.xml in kylin_hadoop_conf_dir", please:

1. Copy all files under /etc/hadoop/conf to one directory ("/path/to/hadoop_conf").

2. Copy hive-site.xml to "/path/to/hadoop_conf".

3. Edit kylin.properties, modify kylin.env.hadoop-conf-dir=/path/to/hadoop_conf, restart Kylin.

• How to achieve Read/Write Separation Deployment？

                Please refer to Read Write Separation Deployment for Kylin 4.0.

• How to refresh the lookup table snapshot?

                It will be automatically refreshed the next time build.

• How to use the new garbage cleaning tool, which garbage will be cleaned up?

                To be updated

• Can Cube Planner be used?

                Not currently supported.  It is plan to be supported in 4.0.0-beta.

• Where is the dimension dictionary stored?

                Dimension dictionary is removed. The only dictionary remained in Kylin 4.0 is Global Dictionary.

• What are the best practice of optimization for build engine?

                To be updated.

• What are the best practice of optimization for query engine(sparder)?

                Please refer to How to improve cube building and query performance and Improve query performance by setting shard by column .

• Is Kylin 3.x and Kylin 4.x metadata compatible?

                Almost fullly compatible, except please purge segments of your cube because HBase Storage is removed now.  Kylin 4.0 remommend to use RDBMS as Metadata, please refer to Use MySQL as Metastore and How to use HBase metastore in Kylin 4.0.

• Is Kylin 3.x and Kylin 4.x pre-calculated cuboid data compatible? If not, will there be a migration plan? 

                The pre-calculated cuboid data is completely incompatible, and there is no migration plan for the time being, due to relatively large effort in development.

• Is the Spark used by Kylin the community version?

                Spark 2.4.6 is currently supported. Other spark distribution is not supported offically.

• What features will no longer be supported in Kylin 4? And what do Kylin 4 provided ?

                Please refer to Kylin 4.X Feature List.

• What is the performance of query engine and build engine in Kylin 4?

                To be updated

• Will query results in Kylin 4 be consistent with the previous version? 

                To be updated

• "Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown;"
  • https://aws.amazon.com/premiumsupport/knowledge-center/emr-s3-503-slow-down/
  • We can reverse the Kylin's working dir.
• How to develop UDF and UDAF？

                To be updated

• Does Kylin 4 support AWS Glue?

                It is not supported in Kylin 4.0.0-alpha.

• Does query on Spark support Spark Schduler Pool setting(resource isolation)?  

                Use different spark pool for different query

• What is the implementation of the new global dictionary?

                Please refer to Global Dictionary on Spark.