Page History

StreamPipes Services

• Core (1, streampipes-backend)
• Pipeline Element Containers (n, streampipes-processors-*, streampipes-sinks-*)
• Adapter Worker (n, streampipes-connect-adapters-**)
• Consul (1)
• Nginx (1) 

Service Discovery

• Should be independent from SD implementation
• Current standard implementation is Consul
• Create new module streampipes-service-discovery
• Move streampipes-config to streampipes-service-discovery-consul for consul-specific implementation

Requirements

• All services register at Consul with an identifying tag
• No fixed hostnames are stored in any graph (DataProcessorInvocation, DataSinkInvocation)
• For each pipeline element, the specific target pipeline element container is resolved at pipeline start
• The hostname of a service is resolved automatically and can be overwritten by an env variable