Page History

...

• "Tailwind" - A virtual / non existing wind park energy company that powers a farm of win-mills to produce clean energy. The company has a strong demand to ETL sensor data from the windmills as well as need to act on data events when base data changes or contracts with customers renew. The company values also the DEI rules and has sustainable targets for clean energy and CO2 reduction.
  • Event driven case: New data is dropped on a file system (S3 would be great but can not be executed w/o S3, can be added to AWS provider as extension) that triggers data processing for wind energy. Data is loaded  and Asset events are generated
    Asset driven pipeline around reporting,  data is split up per city of wind turbine and reports about production are distributed to shareholders. Branching can be used to check if a notification is sent via email or a custom notifier. This can also use branch labels. A third notification channel might be broken and as these shareholders are important we inform the admin in case of any task fails (trigger rule) and start a recovery task.
    Scheduled nightly use case, example reporting is written to file system (where event pipeline is picking up!). As it would be too easy some tasks migth fail and then a custom weight rule is used for retries.
    Manual correction trigger: Correction wind production counters can be submitted which then also are written to file system
    Timetable example for maintenance schedule (selected calendar dates) where maintenance notifications are sent
    Scheduled hourly check for wind turbines state. This requires som special infrastructure to start and stop, using setup+teardown to open a VPN tunnel to the remote machines. This is using a generator pattern and produces the same logic for 3 counties.

    As for demos and examples a lot of functionality is needed in both decorator as well as classic Dag implementation it would be good to have two similar use cases. Or alternatively dscribe that the Tailwind south branch prefers to implement all in Pythonic manner whereas the Tailwind North branch data engineers like the classic implementation?

    ...

    output	svg
    template	NodesWithStyles

    ...

    • • Digraph
        output svg template NodesWithStyles
        digraph D { rankdir=LR; subgraph "turbine_report" { label="turbine_report"; bgcolor="yellow"; style=filled; ev [shape=folder, label="File\nEvent", color=grey, bgcolor="yellow"] report [xlabel="with failure / retry"] ev -> ingest -> aggregate -> report } }

        
        

      • Asset driven pipeline around reporting,  data is split up per city of wind turbine and reports about production are distributed to shareholders. Branching can be used to check if a notification is sent via email or a custom notifier. This can also use branch labels. A third notification channel might be broken and as these shareholders are important we inform the admin in case of any task fails (trigger rule) and start a recovery task.
      • Scheduled nightly use case, example reporting is written to file system (where event pipeline is picking up!). As it would be too easy some tasks migth fail and then a custom weight rule is used for retries.
      • Manual correction trigger: Correction wind production counters can be submitted which then also are written to file system
      • Timetable example for maintenance schedule (selected calendar dates) where maintenance notifications are sent
      • Scheduled hourly check for wind turbines state. This requires som special infrastructure to start and stop, using setup+teardown to open a VPN tunnel to the remote machines. This is using a generator pattern and produces the same logic for 3 counties.

    As for demos and examples a lot of functionality is needed in both decorator as well as classic Dag implementation it would be good to have two similar use cases. Or alternatively dscribe that the Tailwind south branch prefers to implement all in Pythonic manner whereas the Tailwind North branch data engineers like the classic implementation?

    Technical Work Packages → Features

    ...