Description

Zeppelin is a collaborative data analytics and visualization tool for distributed, general-purpose data processing systems such as Apache Spark and Apache Flink.

It has two main features:

• the data analytic phase
• the data visualization phase.

This project is an improvement or a re-design of the Data Visualization Component.

Zeppelin front-end web application already have rich visualization library based on D3, but is not made to allow other libraries and charts.

The goal of this Google Summer of Code project is to make the visualization module pluggable in order to benefit from the wide range of existing visualization libraries.

Mentors

Unknown User (corneadoug)

BEZZUBOV Alexander

Student

UDANTHA Madhuka

JIRA Issues

Original GSOC Issue Feature Issue

Documentation

Issue Milestones

Milestone-1

Description

This Milestone is to study different charting libraries and understand how we can make a pluggable system.

Since this milestone works as some POC, the source code will be found in the student public repository

Ressources

Feature List

• Reading CSV files from d3
• Support three chart types
  • Google Chart
  • High Chart
  • NVD3 Chart
• Drawing same chat (bar) from 3 charting libraries.
  • car.csv file contains data set
• Switching chart types (bar and line)

Task List

• Update Milestone-1 in Zeppelin Wiki
• Milestone - PoC Web Application(does not depend on Zeppelin codebase as Spring restructuring)
• All charts read one CSV file
• Three different charting library (NVD3, Google Chart, High charts)
• Switching between Line and Barchart
• Must create a Pull Request to have conversation going

Results

Use case:

• User can pick charting library that he preferred
• User have to enter CSV file name. (no need to enter file extension)
• Then user have to load data for particular charting library or all the charting libraries
• Final he can pick chart type he need.
• User can switch the chart type (bar or line) and also the chatting library.

Milestone-2

Description

This Milestone an update of previous Milestone. Instead of making the scope bigger, it was decided that it would be more beneficial to work on code quality.

Since this milestone works as some POC, the source code will be found in the student public repository

Ressources

Feature List

• Improve the UI
• Adding grunt for project and build
• Adding Test
• Refactoring using controller pattern
• Separate the code in smaller functional entities

Task List

Change on UI:

• Have one loading button for each data set
• Make the Navbar .active state using angular
• Make the action steps and status easier to understand (order the steps, add explanations maybe, add css to selected options...)
  
  

Using Zeppelin Tools:

• Use grunt in the project (http://gruntjs.com/)
• Include and use lodash when possible (Resources are Followed)

Improving Code:

• Try to create reusable functions instead of duplicating code
      - Chart Controller Improved on DRY
• Separate the controller into smaller logical elements and files (controllers, services, factory...)
  • Global Chart Factory have generic chart pattern  and each chart library will have it is own factory
• Implement the controller pattern (https://github.com/johnpapa/angular-styleguide#controllers)
  • Services 
  • Controllers (Factory model)

Testing:

• Integrate testing inside the milestone1 (https://github.com/daniellmb/angular-test-patterns)

Git Flow Process

• Make a PR from milestone-2 to milestone-1

Architecture Improvements

• Generic data model for all three chart types
  • Global Chart Factory contains the Global chart model and it is extended by each charting factory.
• Use grunt
  • grunt serve --> start the server
  • grunt test --> run the test
  • grunt build --> build the application  
• Revamp code to "controller pattern"
•  Implement a few tests

Style

• Handling PR with screen shoots and task items
• Use JS 'strict' mode (mainly in controllers)
• Avoid 'global' variables 
• Coding style, follow Google JavaScript

Results

Project is using grunt so you can test application from grunt test (Few test cases are developed)

It contains industrial standard plugins for grunt (clean, wiredep, concurrent, karma. etc.). Grunt serve will start the server

It will start app and open the home page as below

Now you can check milestone in the menu. You will have select the data set from click as button. Data set is store as csv file. D3 is used to read it.

 After you picked the 'Data Set' (Car or Bike), You can pick charting library then chart type as below

Milestone-3

Description

This Milestone will try to make the chart libraries totally pluggable (no hard coded configuration).

We also want to push the project to have more test and be an example when it comes to web testing.

Since this milestone works as some POC, the source code will be found in the student public repository

Ressources

Feature List

• Make library and charts pluggable (Remove hard coded code, and simulate loading libraries)
• Fixes and code improvement
• Bigger Test Cover
• Scenario Tests
• Remove unused files

Task List

Readme:

• Missing the steps: Bower install and npm install

UI:

• No button style (just text) when we run grunt serve
• Need similar width for all charts (NVD3 is smaller than others for example)
• Need to have meaningful legends

Bugs:

• After selecting a dataset, then a chart library, it will draw the chart. However if you click again on the same chart type that the one shown, there won't be any data in the chart.

Code Style:

• Fix trailing spaces
• Make code on multiple lines when html line are too long (~100-120 columns)
• Fix google-chart-factory indentation
• Fix Typos

Html reduction:

• Reduce HTML lines repetitions, use ng-repeat with a configuration object instead
• Remove unused files

Make code scalable for pluggable libraries:

• Remove hard coded library names etc... from javascript code. Only an Array with Library Names should be used as configuration to refer to.
• Include the Chart Html only when needed (ng-include of a {{library}}.html file)

Fix a few bad things:

• Remove the dummy Data from chart factories
• loadData should not be inside of ChartCtrl
• Change hardcoded loadChartLibrary function parameter inside the HTML: loadChartLibrary(2) -> should be loadChartLibrary($index) or (chart.name)

Tests:

• Implement more tests, and change code if necessary
• Implement some scenario type of tests

Results

20 Test Cases are written 'grunt test'

Updated UIs

Milestone-4

Description

Controller should work with any chart library and it should be abstracted. Therefore no library switches in Dev.  Writing the steps how to add new chart type (such as pie chart) and steps to be followed for adding a new chart library to application are main task for documentation. Since this milestone works as some POC, the source code will be found in the student public repository

Ressources

Feature List

• Adding new chart type

• Writing the steps how to add new chart type and library 

Task List

Documentations

• Readme for steps to add a new chart
• Readme on steps to add a new library

Fixes

• Bower dependencies versions need to be persisted in bower.json
• Consistency between rendered graphs: titles, axis, buttons, etc
• Removed debug logs
• Footer position is to be absolute

Deployment

• In every milestone, deploy to repository `gh-pages`

Test Cases:

• Update naming pattern for test cases
• Remove repeating test
• Better separation / mapping for src to test
• Few Test Use cases

Results

• Adding New Chart Type (Read Me)
• Adding New Chart Library (Read Me)
  
  

Milestone-5

Description

This is a temporary roadmap for milestone-5, the features might change in the future

This should be the milestone where we start a PR inside of Zeppelin repository

Ressources

Feature List

• Bring new feature to the chart system (download data, make pdf from graph)

• New features to pivot: ordering, limits

Task List

Results