Broadcast Variables

This variant takes a Flink DataSet and makes it available to all function instances that implement RichFunction (JavaDoc).

The following example illustrates that:

 
DataSet<Point> points = env.readCsv(...);
 
DataSet<Centroid> centroids = ... ; // some computation
 
points.map(new RichMapFunction<Point, Integer>() {
 
    private List<Centroid> centroids;

    @Override
    public void open(Configuration parameters) {
        this.centroids = getRuntimeContext().getBroadcastVariable("centroids");
    }
		
    @Override
    public Integer map(Point p) {
        return selectCentroid(centroids, p);
    }
 
}).withBroadcastSet("centroids", centroids);
 

The data flow attaches the centroids data set explicitly to the RichMapFunction. The data will be sent to all RichFunctions in parallel via the Flink network stack that is also used for shuffling the data.

When to use?

Distribute data with a broadcast variable when

• The data is large
• The data has been produced by some form of computation and is already a DataSet (distributed result)
• Typical use case: Redistribute intermediate results, such as trained models