DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
- "Tailwind" - A virtual / non existing wind park energy company that powers a farm of win-mills to produce clean energy. The company has a strong demand to ETL sensor data from the windmills as well as need to act on data events when base data changes or contracts with customers renew. The company values also the DEI rules and has sustainable targets for clean energy and CO2 reduction.
Event driven case: New data is dropped on a file system (S3 would be great but can not be executed w/o S3, can be added to AWS provider as extension) that triggers data processing for wind energy. Data is loaded and Asset events are generated
Digraph output svg rankdir=LR; subgraph "turbine_report" { label="turbine_report"; bgcolor="yellow"; style=filled; ev [shape=folder, label="File\nEvent", color=grey, bgcolor="yellow"] report [xlabel="with failure / retry"] ev -> ingest -> aggregate -> report }- Asset driven pipeline around reporting, data is split up per city of wind turbine and reports about production are distributed to shareholders. Branching can be used to check if a notification is sent via email or a custom notifier. This can also use branch labels. A third notification channel might be broken and as these shareholders are important we inform the admin in case of any task fails (trigger rule) and start a recovery task.
- Scheduled nightly use case, example reporting is written to file system (where event pipeline is picking up!). As it would be too easy some tasks migth fail and then a custom weight rule is used for retries.
- Manual correction trigger: Correction wind production counters can be submitted which then also are written to file system
- Timetable example for maintenance schedule (selected calendar dates) where maintenance notifications are sent
- Scheduled hourly check for wind turbines state. This requires som special infrastructure to start and stop, using setup+teardown to open a VPN tunnel to the remote machines. This is using a generator pattern and produces the same logic for 3 counties.
...
- Asset driven pipeline around reporting, data is split up per city of wind turbine and reports about production are distributed to shareholders. Branching can be used to check if a notification is sent via email or a custom notifier. This can also use branch labels. A third notification channel might be broken and as these shareholders are important we inform the admin in case of any task fails (trigger rule) and start a recovery task.
- Scheduled nightly use case, example reporting is written to file system (where event pipeline is picking up!). As it would be too easy some tasks migth fail and then a custom weight rule is used for retries.
- Manual correction trigger: Correction wind production counters can be submitted which then also are written to file system
- Timetable example for maintenance schedule (selected calendar dates) where maintenance notifications are sent
- Scheduled hourly check for wind turbines state. This requires som special infrastructure to start and stop, using setup+teardown to open a VPN tunnel to the remote machines. This is using a generator pattern and produces the same logic for 3 counties.
As for demos and examples a lot of functionality is needed in both decorator as well as classic Dag implementation it would be good to have two similar use cases. Or alternatively dscribe that the Tailwind south branch prefers to implement all in Pythonic manner whereas the Tailwind North branch data engineers like the classic implementation?
| Digraph | ||
|---|---|---|
| ||
rankdir = LR;
subgraph cluster_p {
label = "Dag: Turbine Report";
ev [shape = folder;label = "File\nEvent";];
report [xlabel = "with failure / retry";];
ingest;
ev -> ingest -> aggregate -> report;
}
subgraph cluster_c {
label = "Dag: City Reporting";
split [label = "split by city";];
subgraph clusterc2 {
label = "Dynamic Mapped Tasks";
seattle;
portland;
vancouver;
}
split -> seattle -> summary;
split -> portland -> summary;
split -> vancouver -> summary;
}
ingest -> split [color = "#aaaaaa"; constraint = false; label = "asset event";];
subgraph cluster_m {
label = "Dag: Measurement correction";
form [label="Trigger form w/\ncity\nmeasure", shape=component]
notify [shape=diamond]
form -> file -> notify
notify -> email -> completion
notify -> push -> completion
notify -> none -> completion
}
file -> ev [color = "#aaaaaa"; constraint = false; label = "asset event"]
subgraph cluster_f {
label = "Dag: Turbine Monitoring";
cron [shape = doublecircle; label = "cron:hourly"]
inventory [label="get inventory"]
cron -> inventory
subgraph cluster_f1 {
label = "Dynamic Mapped Task Group";
vpn_on [label="vpn connect\n(setup)"]
vpn_off [label="vpn disconnect\n(teardown)"]
health [shape=diamond]
vpn_on -> get_telemetry -> health -> vpn_off
vpn_on -> check_alarms -> health
health -> send_technicial [label="notify if alarm"]
health -> all_ok
}
inventory -> vpn_on
}
subgraph cluster_s {
label = "Dag: Maintenance";
timetable [shape = doublecircle; label = "maintenance\ntimetable"]
form2 [label="Trigger form w/\nturbine\ntime\nduration", shape=component]
maintenance_on [label="maintenance start\nbash script"]
wait [label="wait for maintenance completion\ndeferred"]
maintenance_off [label="maintenance end\nbash script"]
timetable -> maintenance_on [label="planned schedule"]
form2 -> maintenance_on [label="manual trigger"]
maintenance_on -> wait -> maintenance_off
}
subgraph cluster_r2 {
label = "Dag: Shareholder Reporting\nDynamic Dag Generation";
cron2 [shape = doublecircle; label = "cron:monthly"]
cron2 -> load_data
subgraph cluster_g {
label = "Generated by loop";
subgraph cluster_ga {
label="";
color=none;
report_stakeholder_a -> send_report_a
}
subgraph cluster_gb {
label="";
color=none;
report_stakeholder_b -> send_report_b
}
subgraph cluster_gc {
label="";
color=none;
report_stakeholder_c -> send_report_c
}
}
load_data -> report_stakeholder_a
load_data -> report_stakeholder_b
load_data -> report_stakeholder_c
}
|
Technical Work Packages → Features
...