Page History

Introduction and History
Goals
Issues with Current VSD
Desired features in JVSD v1.0
Future features for post 1.0 versions of JVSD
Documentation
Build and Usage Requirements
Statistics File Format
JVSD Features From VSD
Classic VSD Features
JVSD Functional Differences
Appendix

Introduction and History
The Geode framework software creates statistical archive files logging collected metrics from various subsystems such as the number of puts for a region. The primary application to read the statistics files for over the last 10 years has been the Visual Statistics Display (VSD) tool which can read the sampled statistics data and can produce time-series/value graphs. The original design and usage of VSD was to address a need by consulting and engineering to do performance analysis of the commercial GemStone/S Smalltalk server and later the GemFire product line. The VSD tool has since grown out of its initial role as a consulting tool and is now used by support, users and engineering as a critical post-mortem tool to identify performance issue such as GC and along with other kinds of failures within a distributed system.
Goals
Replace the current TCL based VSD with a new Java based version (JVSD). The initial release will focus on features of VSD used for post-mortem analysis of performance issues and will be essentially a clone of the functionality in the current classic VSD.
Issues with Current VSD (classic VSD)
1. 32 bit only binaries and difficult to setup (install issues)
2. Difficulty managing large files
3. Difficulty managing multiple files
4. Issues running on different platforms (ex. must use Windows XP compatibility mode and additional X11 application is needed on OSX)
5. Not using a modern UI design (ex. Copy/Paste on different menus)
6. UI issues (ex. selecting two points for log delta, selecting two lines to diff)
7. Non-native look and feel (Motif L&F on all platforms, not visually appealing)
8. Requires shipping a TCL interpreter
9. Possibly licensing issues in making VSD open source
10. Issues with finding developer talent with TCL skills
Desired features in JVSD v1.0
1. Meet critical features of legacy VSD
2. Minimize differences in initial usage and GUI from classic VSD
3. Open source and community developed
4. Split out library for users to independently process Statistics files for use by third party software or for monitoring applications
5. Provide test cases and data to validate functionality
6. Make new architecture amenable to extension and future features
7. Manage timezones correctly
8. Use human readable scales ( nanoseconds ->milliseconds/seconds, bytes->megabytes)
  1. Needs to be selectable preference
9. Identify default or common statistics/metrics
  1. Filter stats list displayed by common/default, advanced, no flatlines
Future features for post 1.0 versions of JVSD
1. Enhance Performance
2. Best practices usage guide
3. Snap to sample
4. Increase abilities to manage a large number of files concurrently
5. Increase abilities to manage large files with potentially millions of samples
6. Provide a robust template system to enhance chart reuse
7. Simplify sharing of statistics and charts (email, web uploads…)
8. Annotate charts with notes
9. Have the possibility to merge two charts instead of having to create a third and add the metrics from the two charts
10. Change timezone without having to change the environment timezone to easier sync stats with logs
11. Provide an overview of the loaded gfs files with the gfs file header information and the start and end time of the stats file.
12. Save charts and state of application for fast restarts
13. Provide plug-ins to process statistics data for third party applications
14. Provide export to alternative file formats (csv, excel…)
15. Provide import for alternative file formats (csv, excel...)
16. Provide intelligence/rules to identify and highlight potential issues
  1. Consider Javascript or similar for rules language to make it easy to extend for other users, ideally an interface that permits usage of user prefered rules engine or scripting interface (JRules, Dools...)
17. Interface to live system via JMX
18. Log file analysis (match log file exceptions to times in statistics for events)
19. Add new chart type - timeline, showing when a process is up relative to others
20. Encourage outside development and customization (community building)
21. Mobile version ( iOS, Android) stretch goal to ease system management
22. Web based version (possibly integrate with Pulse )
23. Support for compressed stats. (gz, zip, etc.)
24. Easy uploads to Pivotal support or other third parties.
25. Separate out the statarchiver for use without Geode or a distributed system.

Documentation

	Site
Classic VSD public documentation	http://www.gemstone.com/docs/tools/VSD/index.html
Current VSD public documentation	http://gemfire.docs.pivotal.io/latest/userguide/index.html#gemfire_tools/vsd/chapter_overview.html
VSD analysis tips and tricks	http://gemfirexd.docs.pivotal.io/1.0.0/userguide/manage_guide/Topics/system_performance.html
Analysis advice	http://blogs.vmware.com/vfabric/2012/10/using-visual-statistics-display-to-analyze-gemfire-runtime-configuration-resources-and-performance.html
JavaFX (Interface)	https://docs.oracle.com/javase/8/javase-clienttechnologies.htm
Web based plotting tools (example/ideas from third parties)	https://plot.ly/feed/https://zeppelin.incubator.apache.org/
Vernier tools and plotting (example/ideas from third parties)	http://www.vernier.com/products/software/ga/

Build and Usage Requirements
The build environment should follow common open source practices and hence use tools that support those principles. A secondary goal will be to minimize the usage of third party libraries outside of the JVM to minimize the requirements for users during installation and to reduce the knowledge requirements for new developers to begin contributing to the project.
1. JDK 8 - will target and support only JDK 8 given JDK 7 has been EOL’ed
2. JavaFX as primary UI library - currently ships with JDK and replaces Swing (no additional libraries)
3. Minimal third party libraries including not requiring either GemFire or Geode classes and where required will use the common and well known libraries
4. JUnit will be used for basic testing
5. Maven - build and dependency management
  1. The Geode Gradle should be updated to drive the maven build used by JVSD, the maven build will remain separate to allow JVSD to be forked into a possible future independent project.
6. The development branch will be jvsdfx-mm (temporary during transition to Geode) and other branches will be retired or deleted. The goal will be to move this prototype forward into a production ready release.
  1. Current prototype is hosted at Apache Geode repository https://github.com/apache/incubator-geode.git
  2. Currently in branch feature/GEODE-78
7. No classic VSD source code will be reused in the new JVSD, individuals with licenses for GemFire can use and compare VSD to address functional requirements for JVSD.
Statistics File Format
1. No file or format changes of the statistics will be required or expected for the 1.0 release
  1. Dependency on Geode and tool will need to track any statistics file format changes within it
  2. Will continue to use default gfs file extension for file filters to identify statistics files
  3. Support for gzip files needs to be reviewed and determined if this is critical
2. Provide ability to export data in CSV format for third party usage
JVSD Features From VSD
Fundamentally the initial JVSD release will be a clone of the classic VSD in regards to basic functionality and features. The basic interface should remain similar for most users with a default table or data selection window which will be the primary interface to selecting which data to chart. This data selection window will continue in the new version with only minor updates in form or function to accommodate the change in interface libraries or modern GUI design style. This window will also remain a singleton interface element and home base for user to access other elements such as the charts created. The chart window is still the primary method for visualizing the statistics data. As with the table window, there are no significant changes expected in how users will interact with the chart window and will continue to only draw time-series line graphs. All GUI enhancements in this release will only to simplify usage such as selecting a line or saving a snapshot.
1. GUI
  1. Table Window (Statistics selection) - See Figure 5 prototype.
  2. Chart Window - See Figure 6 prototype.
  3. Log Info Window - a simple scrolling text window where data produced by “Log Delta” operation is displayed - No prototype yet.
2. Analysis tools - There are currently no analysis tools to aid in the processing or understanding of events within statistical files. The first release will not change this basic usage or workflow of correlating events via adding one or more data sets to a time series line graph/chart. But hooks to plug in processing will need to be added for future options.
Classic VSD Features
The following captures in text the basic functional features (menus and actions) of the classic VSD tool and provides a baseline of expected functionality for any replacement tool to meet. As JVSD is initially intended to be a clone of the classic VSD functionality, developers will use and reference the old version for comparison of a feature and to insure it works in a similar and familiar manner.
Features marked with a (*) will be targeted for inclusion in the first JVSD release.
Basic Features of Table Window: See Figure 1.
1. Load Data File * - open file
2. Monitor - Smalltalk usage to connect to a statsmonitor process ( may return via JMX in future release)
3. Append Data File * - Add new data to currently active data set based on PID, concatenate file to currently open files
4. Statistic Info * - Help screen which defines statistics data types, can set specifics about stat such as default sample style/filter (per Second, no filter...) and the statistics level (common, advanced). See Figure 4.
5. New Chart *
6. Close All Charts *
7. Close Chart *
8. Print Chart
9. Add to Chart / Add Line *
10. Sort table data via column (PID, Statistic Type…)*
11. Help *
  1. How To *
  2. Main Window *
  3. Chart Window *
  4. All Topics
  5. All Help text
  6. Choose Text Font
  7. About VSD *
12. Chart Options in Table Window:
  1. Settings for default creation of new charts or adding data/lines to an existing chart.
    1. Show Legend *
    2. Time Format *
      1. Elapsed Time Seconds *
      2. Hour:Minute:Second *
      3. Month/Day Hour:Minute:Second *
    3. Default Line Style
      1. linear *
      2. step
      3. natural, quadratic
    4. Show Time Axis Title *
    5. Show Left Axis Title *
    6. Show Right Axis Title *
    7. Show Current Values *
    8. Show Axis on Left or Right (Graph on Left Axis) *
    9. Show Min/Max *
    10. Show Line Stats *
    11. Show Cross Hairs *
    12. Select Line Color (dialog)
    13. Choose Chart Font (dialog)
    14. Sample Point Symbol
      1. None <default> *
      2. Cross
      3. Plus
      4. Square
      5. Triangle
      6. Circle *
      7. Diamond
      8. Plus
13. Table Selection Options - Context Menu: Using right mouse button(context menu) to select a group of statistics for inclusion in a chart or search for name of statistic type in table.
  1. Search *
  2. Select Statistic *
    1. clear *
    2. by Statistic *
    3. by Type *
    4. All
  3. Chart Select
14. File:
  1. Browse Open Files
  2. Select Single File
  3. Select Recent File
15. General Options:
  1. Statistics Level - Intended to provide a filter for statistics that are useful and commonly used and the more specialized ones that may additional understanding of the product to be useful.
    1. Common
    2. Advanced
    3. All
  2. Absolute Time Stamps
  3. Copy Reference Lines
  4. Single File Mode *
  5. No Flatlines *
  6. Combine Across Files *
  7. Combine *
  8. Change Directory - changes working directory for vsd, effects saving of snapshots or writing of postscripts data used for printing (temp data)
  9. Update - reload data from one file
  10. Update All - reload data from all files
  11. Auto Update - periodically update one one
  12. Templates
    1. Reload Template File
    2. New Template Chart
    3. Templates Use Selection
Chart Display Options: See Figure 2.
1. Information about a selected line or chart including display of values.
  1. X value (time) *
  2. Y value *
  3. Min *
  4. Max *
  5. Standard Deviation *
  6. Sample Style
    1. No Filter * - Displays the raw values for the statistic with no filtering at all.
    2. Per Second * - Displays the difference between two consecutive samples of the statistic, divided by the number of elapsed seconds between the two samples.
    3. Per Sample * - Displays the difference between two consecutive samples of the statistic.
    4. Aggregate * - Displays a running total of per-sample deltas for the statistic. Resets to zero when the delta is zero or changes direction.
  7. Scaler (*,/,+,-) with value
  8. Legend for each statistic *
  9. Delete statistic via legend * - using middle mouse button
  10. Select legend to highlight line in chart *
  11. Select line to highlight by clicking on a point on the line *
2. Chart Functions: Adjusts the visual presentation of the charted data
  1. Update Chart
  2. Zoom/unzoom using middle/right mouse buttons *
  3. Zoom In *
  4. Zoom Out *
  5. Compare Two Points - similar to Log Delta
  6. Compute Scale
  7. Compute Scale All
  8. Unscale
  9. Unscale All
  10. Snapshot - generate a gif file of the chart for sharing *
  11. Add From Template
  12. Save Template
  13. Copy Line * - Copy line from chart to clipboard
  14. Paste Line * - Add line from clipboard to chart
  15. Cut Line * - remove line from chart and store on clipboard
  16. Delete Line * - remove line from chart
  17. Trim/untrim Left * - Ignore older data to the left of trim line
  18. Trim/untrim Right * - Ignore later data to the right of trim line
  19. Normalize
  20. Add Lines * - Add the values of two lines and put result on current chart
  21. Diff Lines * - Subtract second selected line from first and put result on current chart
  22. Divide Lines * - Divide second selected line from first and put result on current chart
  23. Log Delta * - Computes mean, min, max, stddev between two points on a line. Results displayed in Log Info window. See Figure 3.
  24. Log Info * - results from Log delta and Compare two points operations
  25. Select/highlight line/data via Legend *
JVSD Functional Differences
1. Changes
  1. Menus will be significantly rearranged, in particular will follow the common GUI standards for “File”, “Edit”, “View” and “Help” menus capturing specific context related functionality.
  2. Scaling of data/charts will only have limited support in first release, this feature is not critical for many users and multiple independent vertical axis will be supported rather than only two as in classic VSD. For a few users this is a critical feature for certain kinds of performance analysis and aids in comparing two sample runs with different configurations.
  3. Chart display is based on sampled data with refinements when zooming
  4. Future option to allow non-sampled or high resolution view with associated degrading of performance when drawing chart
  5. Multiple vertical axis will be support with one for each data type (bytes, entries…) being the default rather than only two in VSD ( a single left and right)
  6. Many of the charting options should be part of a preferences dialog rather than as menu items of the table window as the defaults once set for a user are rarely changed
  7. Statistics Info window will dropped and its display info added to the help system
  8. Displaying of metadata such as stat file path and machine name that generated file when selecting a legend item needs to addressed to simplify analysis tasks
  9. Chart legend needs to be scrollable or limit the number of items that can on an individual chart (legend can not consume the whole chart window)
  10. Will not support classic VSD preferences stored in the .vsdconfig, .vsdrc and .vsdtempates under the user home directory
  11. Gif output of snapshots will be dropped in favor of PNG output
  12. Legacy features specific to older GemStone products such as GemStone/S will be dropped ( help descriptions, monitor…)
  13. Support library for reading statistics will be provided with examples on usage
Appendix
1. Figure 1 Classic VSD Table Window

2. Figure 2 Classic VSD Chart Window

3. Figure 3 Classic VSD Log Info Window

4. Figure 4 Classic VSD Statistics Info Window

5. Figure 5 JVSD Table Window

6. Figure 6 JVSD Chart Window

PDF File link JVSDFunctionalSpecification.pdf

Space shortcuts

Page tree

Versions Compared

Old Version 1

New Version Current

Key