Page History

...

4.2. Example Implementation: ZeppelinResourcePool as Spark Data Source

(image copied from from https://databricks.com/blog/2015/01/09/spark-sql-data-sources-api-unified-data-access-for-the-spark-platform.html)

Spark supports pluggable data sources. We can use make Zeppelin’s DistributedResourcePool a spark data source using Spark DataSource API. Please refer these articles for more information.

• https://databricks.com/blog/2015/01/09/spark-sql-data-sources-api-unified-data-access-for-the-spark-platform.html

• https://mapr.com/blog/spark-data-source-api-extending-our-spark-sql-query-engine/

4.2.1. BaseRelation Implementation

...

• Watch / Unwatch: for automatic paragraph updating for Streaming Data Representation.

• ZEPPELIN-1494: Bind JDBC result to a dataset on the Zeppelin context

• Ability to construct table result from the resource pool in language interpreters (e.g python)

• Let’s assume that we can build a pandas data frame using TableData

  Code Block
  language py linenumbers true
  # in python interpreter t = z.get("tableResourceName") # will return object that has `hasNext` and `next` p = new PandasTableData(t) # use p.pandasInstance …