THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
TODO:
- validate(): use validation method (org.apache.commons.validator.routines.InetAddressValidator) instead of regex check
REST API
DataLakeNoUserResourceV3
org.apache.streampipes.rest.impl.datalakeThis class contains the basic interface definition for setting up a new measurement series in Data Lake and calls the underlying methods of org.apache.streampipes.dataexplorer.DataLakeNoUserManagementV3. Usage of the related API calls does not require any authentification with valid username and password.
| method name | request type | path | description |
|---|---|---|---|
| addDataLake | POST | /{measure} | adds new measurement series with specified measure name and related event properties (column names) in InfluxDB |
TODO:
- [STREAMPIPES-348]: fix issue with special characters in user-defined measure name
- add authentication obligation to addDataLake method
DataLakeResourceV3
org.apache.streampipes.ps
This class contains the extended interface definition including the related paths and calls the underlying methods of org.apache.streampipes.dataexplorer.DataLakeManagementV3 and org.apache.streampipes.dataexplorer.utils.DataExplorerUtils when invoked. Usage of below mentioned API calls requires authentification with valid username and password.
| method name | request type | path | description |
|---|---|---|---|
getPage | GET | /data/{index}/paging | returns pages with predefined number of events per page of a specific measurement series from InfluxDB |
getAllInfos | GET | /info | returns list with ids of all existing measurement series (including event schema) from InfluxDB |
getAllData | GET | /data/{index} /data/{index}/last/{value}/{unit} /data/{index}/{startdate}/{enddate} | returns all stored events of a specific mesurement series from InfluxDB |
aggregated set of all stored events of a specific mesurement series from InfluxDB | |||
| getAllDataGrouping | GET | /data/{index}/{startdate}/{enddate}/grouping/{groupingTag} | returns all events within a specified time frame of a specific mesurement series grouped by a specific tag from InfluxDB |
removeAllData | DELETE | /data/delete/all | removes all stored events from InfluxDB |
| downloadData | GET | /data/{index}/download | downloads all events of a specific mesurement series from InfluxDB in desired format |
getImage | GET | /data/image/{route}/file | returns png image from file route |
saveImageCoco | POST | /data/image/{route}/coco | stores image as file at file route |
getImageCoco | GET | /data/image/{route}/coco | returns image at file route as application/json |
| labelData | POST | /data/{index}/{startdate}/{enddate}/labeling/{column}/{timestampColumn}?label= | updates label in specified column for all events within specified time frame to provided label value |
TODO:
- fix export of data from data lake, which currently returns two timestamp fields
- extend aggregation functionality to support non-numeric values (e.g. strings → majority vote) add the possibility to specify an aggregation function
- in general: alignment of the single endpoint definitions and consideration of the extensions below
Ideas for possible adaptations and extensions of the REST API
In addition to the TODOs listed above in the text, the following adjustments and enhancements might be worth considering. Thereby, it is important that the implementation of the endpoints is as independent as possible from the technology of the data lake (e.g. avoiding InfluxDB-specific methods).
- Extension of the remove endpoint by the capability to
- selectively delete an individual measurement series
- delete measurements of a measurement series within a specific time interval or before a specific date
- Adding an edit endpoint for adjusting data lake specific properties such as retention time.
Both extensions could be included in a kind of data management tool in the UI within an admin view (e.g. in the manner of the pipeline view).
Another possible adaptation would be the comprehensive implementation of an append-only approach for time series data. In particular, the functionality of the labelData method would have to be adapted here, which currently works with updates of existing DB entries.
...
References:
- Apache StreamPipes Documentation: https://streampipes.apache.org/docs/docs/pe/org.apache.streampipes.sinks.internal.jvm.datalake/