THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Discussion thread:
JIRA:
Released:
Motivation
More details can be found in the Flink ML Roadmap Document and in the Flink Model Serving effort specific document.
...
This pattern fits well into the overall architecture that we want to implement.
Flink provides 2 ways of implementing low-level joins - key based join based on CoProcessFunction and partitions-based join based on RichCoFlatMapFunction. Although both can be used for required implementation, they provide different SLAs and are applicable for slightly different use cases.
...
- List of available models (per data type) - List[String]
SpeculativeExecutionStats describing statistics on the speculative execution:
final case class SpeculativeExecutionStats(
var name: String,
decider : String,
tmout: Long,
since: Long = System.currentTimeMillis(),
usage: Long = 0,
duration: Double = 0.0,
min: Long = Long.MaxValue,
max: Long = Long.MinValue)
...
- Implementation
An initial implementation and examples are provided for this Flip:
...
, they include just basic model serving implementation without speculative piece. Implementation is provided in both Scala and Java. It Implements both key-base and partition-base joins and Tensorflow support for both "optimized" and saved formats. Implementation is split into 2 part:
Library implementation
This is implementation of the base library independent from the type of the messages that particular solution is using. It is strongly typed, implemented using generics. Library implementation includes 3 modules:
Here Flink Model Serving shared contains protobuf definition (see Pipeline Metadata Definition above) And Flink model serving Java and Flink model serving Scala provides the same implementation in both Java and Scala. The oretically it is possible to combine the two, but Java and Scala syntax is sufficiently different, so the 2 parralel implementations are provided.
Flink Model Serving Scala
The implementation is contained in the namespace org.apache.flink.modelserving.scala, which contains 3 namespaces:
- model - code containing definition of model and its transformation implementation
- query - code containing base implementation for the Flink queryable state query
- server - code containing basic implementation of the Flink specific components of the overall implementation
Model implementation is split into generic and tensorflow based implementation, such implementation allows to add other model types support in the future without disrupting the base code. Generic model implementation includes the following classes:
- DataToServe - a trait defining generic container for data used for model serving and its behavior
- Model - a trait defining generic model and its behavior (see above)
- ModelFactory - a trait defining generic model factory and its behavior (see above)
- ModelFactoryResolver - a trait defining generic model factory resolver and its behavior. The purpose of this trait is to get the model factory based on the model type.
ModelToServe defines additional case classes used for the overall implementation and a set of data transformations used for model manipulation and transforming it between different implementations. Additional classes included here include ServingResult - a container for model serving result; ModelToServeStats - model to serve statistics (see above) and ModelToServe - internal generic representation of the model
- ModelWithType - a combined container for model and its type used by Flink implementation
A tensorflow namespace inside model namespace contains 2 abstract classes:
TensorFlowModel extends Model by adding Tensorflow specific functionality for the case of optimized Tensorflow model
- TensorBundelFlowModel extends Model by adding Tensorflow specific functionality for the case of bundled Tensorflow model
Query namespace contains a single class - ModelStateQuery - implementing Flink Queryable state client for the model state
Server namespace contains 3 namespaces:
- Keyed - contains DataProcessorKeyed - implementation of the model serving using key based join (see above) and based on Flink's CoProcessFunction
- Partitioned - contains DataProcessorMap - implementation of the model serving using partion based join (see above) and based on Flink's RichCoFlatMapFunction
- Typeshema contains support classes used for type manipulation and includes the following:
- ByteArraySchema - deserialization schema for byte arrays used for reading protobuf based data from Kafka
- ModelTypeSerializer - type serializer used for checkpointing
- ModelWithTypeSerializer - type serializer used for checkpointing
Flink Model Serving Java
Provides identical functionality to the above and structured the same way
Example implementation
This is implementation of the a wine quality example based on the above library. Implementation includes 3 modules:
- Flink model serving example shared
- Flink model serving example Java
- Flink model serving example Scala both PMML and Tensorflow.Flink Speculative Model Server https://github.com/FlinkML/flink-speculative-modelServer is am implementation of a speculative model server in both Scala and Java. This is a little more limited - Speculative model server is not supported for partition-based approach. It also uses only TF.