Current Situation

Currently a user does not get any feedback, when an adapter stops sending events. 

Solution

A user should be notified when there are problems with pipeline elements. This should not only be restricted to adapter, but should also work for processing elements.

Idea for first prototype

My suggestion is that adapters (pipeline elements) send status events that can be analyzed with StreamPipes pipelines (e.g. send a email, when adapter stops sending data)

This means, each adapters needs to store statistics about the data streams that are sent regularly on a status topic on the message broker. The main idea here is to reduce the (potential high) frequency of the individual data streams.

Within StreamPipes an (internal) adapter can read those events and an (internal) pipeline can process them.

I'll track the changes in branch and issue STREAMPIPES-445.

Open Questions

• Is there other relevent data that we should track?
• Should it be possible to change the email rules per adapter?
• Should we store the events in the data lake?
• Can we apply this concept for processing elements as well?
• Is this also an approach to collect exception information from the individual services or do we need another solution for that?