...
This diagram shows how Spark Interpreter can query the table generated from JDBC interpreter.
An interpreter (`A`A) a newly created table result can be registered as a resource.
Since every resource registred registered in a resource pool in an interpreter can be searched via `DisbitrubedResourcePool` and supports remote method invocation, other interpreters (`B`B) can use it.
Let’s say JDBCInterpreter created a table result and keep it (`JDBCTableData`JDBCTableData) into its resource pool.
Then, SparkInterpreter can fetch rows, columns via remote method invocation. if Zeppelin registers the distributed resource pool as Spark Data Source, SparkInterpreter can use all table resources in Zeppelin smoothly. (e.g Querying the table in SparkSQL as like a normal table)
Gliffy Diagram | ||||||
---|---|---|---|---|---|---|
|
3.2. Overview: How an interpreter can handle table resources
Here are is a more detailed view to explain how one interpreter can handle its `TableData` TableData implementation with the resource pool.
Gliffy Diagram | ||||
---|---|---|---|---|
|
4. Public Interfaces
4.1. Interfaces for TableData related classes
TableData interface defines methods to handle a table resource. Each interpreter can implement its own TableData. The reason why we can’t make the global TableData class for all interpreters is that each interpreter uses a different storage and a different mechanism to export/import data.
class | How it can get table data |
---|---|
InterpreterTableDataResult | Contains actual data in memory |
Interpreter specific TableData (e.g SparkTableData, SparkSQLTableData, …) | Knows how to reproduce the original table data. (e.g keep the query in case of JDBC, SparkSQL) |
Gliffy Diagram | ||||
---|---|---|---|---|
|
4.1.1. Additional methods for TableData
Code Block | ||
---|---|---|
| ||
public interface TableData {
…
/**
* filter the input `TableData` based on columns.
*/
public TableData filter(List<String> columnNames);
/**
* Pivot the input `TableData` for visualizations
*/
public TableData pivot(List<String> keyColumns,
List<String> groupColumns,
List<String> valueColumns);
…
} |
Each interpreter can implement its own TableData class. For example,
SparkInterpreter can have SparkTableData which
points RDD to get the table result
filter and pivot can be written by using Spark RDD APIs
JDBCInterpreter can have JDBCTableData which
keeps query to reproduce the table result
filter and pivot can be written using a query that has additional `where` and `group by` statements.
Some interpreters (e.g ShellInterpreter) might not be connected with external storage. In this case, those interpreters can use the InterpreterResultTableData class.
4.2. Example Implementation: ZeppelinResourcePool as Spark Data Source