Apache Kylin : Analytical Data Warehouse for Big Data
Welcome to Kylin Wiki.
- What is SparderContext? And how should I take care of it ?
SparderContext is the implemenatation of new distributed query engine which is backend by a spark application. If SparderContext is dead, all your query will failed. So please check its liveness after Kylin instance was started in application list of Resource Manager Web UI.
- Is Hadoop3 supported ?
Kylin 4.0.0-alpha did not support Hadoop3.
- If you faced Exception "Cannot find hive-site.xml in kylin_hadoop_conf_dir", please:
1. Copy all files under /etc/hadoop/conf to one directory ("/path/to/hadoop_conf").
2. Copy hive-site.xml to "/path/to/hadoop_conf".
3. Modify kylin.env.hadoop-conf-dir=/path/to/hadoop_conf, restart Kylin.
- How to achieve Read/Write Separation Deployment?
Please refer to Read Write Separation Deployment for Kylin 4.0.
- How to refresh the lookup table snapshot?
It will be automatically refreshed the next time build.
- How to use the new garbage cleaning tool, which garbage will be cleaned up?
To be updated
- Can Cube Planner be used?
Not currently supported.
- Where is the dimension dictionary stored?
Dimension dictionary is removed.
- What are the optimization methods for build engine?
To be updated.
- What are the optimization methods for query engine?
Please refer to Improve query performance by setting shard by column.
- Is Kylin 3.x and Kylin 4.x metadata compatible?
Fully compatible.
- Is Kylin 3.x and Kylin 4.x cuboid data compatible? If not, will there be a migration plan?
The pre-calculated cuboid data is completely incompatible, and there is no migration plan for the time being, due to relatively large effort in development.
- Is the Spark used by Kylin the community version?
Spark 2.4.6 is currently supported.
- What features will no longer be supported in Kylin 4?
Please refer to Kylin 4.X Feature List.
- What is the performance of query engine and build engine in Kylin 4?
To be updated
- Will query results in Kylin 4 be consistent with the previous version?
To be updated
- "Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown;"
- https://aws.amazon.com/premiumsupport/knowledge-center/emr-s3-503-slow-down/
- We can reverse the Kylin's working dir.
- How to develop UDF and UDAF?
To be updated
- Does Kylin 4 support AWS Glue?
To be updated
- Does query on Spark support Spark Schduler Pool setting(resource isolation)?
Use different spark pool for different query
- What is the implementation of the new global dictionary?
Please refer to Global Dictionary on Spark.
- How to store Metadata after not using HBase?
Kylin supports JDBC to store Metadata, please refer to Use MySQL as Metastore and How to use HBase metastore in Kylin 4.0.