Page History

...

• E.g.:
  • MQTT data stream sends {"timestamp": 1660791892, "temperature": 10} … {"timestamp": 11660792892, "temperature": 11.2}
  • CSV File set:

• What is the datatype for temperature of the event schema?
• If the adapter defines integer, then there will be a problem for all float values that come later on
• How can this be resolved?
  • Option 1: As a default select always float for the datatype
  • Option 2: Cast all float value to int
• The problem currently is that the schema guessing uses the values of the first event to derive the data type. Sometimes if the value has only zeros after the decimal point it is not included, so the system can not know it is a float value.

2.  Timestamps

• Timestamps are usually have type long
• They should not be floats

3.  A user defines the data type of the property to integer

• Processing elements with a keep strategy might change the numerical data type
  • E.g. Math Operator
• During runtime the value is transformed to float
• This results in an error in the data lake

Please feel free to extend the problem lists if you find further issues.

Currently Selected Solution Strategy

Schema guessing always uses Float as default data type: 

• If a user wants a different data type, it must be manually changed in the UI 
• When the data type is changed to float or double is selectedThose two value options are possible , there are options during runtime (value=10 or value=10.0).
  • Both should work without any problems.
• When selecting integer for the data type:
  • Then the The adapter must ensure that an integer is sent (value=10.2 must be transformed into rounded to value=10)
• This approach solves Problem 1 & Problem 2
• Solution of Problem 3:
  • This is currently only handled in the in the DataLake sinkSink.
  • If a numerical value can not be casted to integer or long, it is changed to a float before it is stored

...

Only floats are supported as data types for numerical measurements. This means we would lose the other numerical data types.

Also the user would not be A user is not able to manually change the data type. That means the The adapter is responsible to transform the value on ingestion and ensure that all values are floats.

This resultsTherefore, that the processing elements are certain that the numeric value is a float and no special cases must be considered.

...