Notes:

  • Brendan is moderating today
  • last meeting was 2018-06-20
  • Attendees:
    • Brendan, Matt, Olivier, Himavant, KeonHee Kim, Martin, Priti, Sam, Dave G., Markus, Dominic, Michele, Dan, Sandeep, James T., Sven, Chetan, TzuCiao, Andy, afield (?), Jeremias, Duy, Michael Marth, 

Introductions of new attendees

  • No one volunteered to intro. themselves


Open comments on status/updates in a few areas:

  • Main/core OpenWhisk (Carlos//Markus/Tyson, etc.)
  • Scheduled topics
    • Markus: highlights proposal on our project CWIKI for Controller→Container Manager→Invoker redesign
      • "OpenWhisk Future Architecture": OpenWhisk future architecture
      • Goal: align many diff. proposals to one common ground that allows a means to
      • wraps itself in idea that crit. path should be quick, and containers are abstracted in a way that they are effectively just an IP and a Port.  The notion of invoker becomes unnecessary
      • Controler talks to user containers directly (/run, /init)
      • writes activates records (and move resp. formerly in Invoker to Controller
      • Introduce a “Container Manager” that can work with diff. pools
        • can be part of controller
        • but recommend it is its own microservice
      • Idea not fully flesshed out, want disc. and use cases (edge cases) discussed on dev list and m
      • Warm. Inv. path:
        • Ctrl: gets request, picks user container, writes act. record, returns response
        • if not free container it requests it from Container Mgr.
        • This abstraction allows “plug ins” for diff needs (e.g., baremetal, Kube, Mesos, etc.)

          • effectively an API abstraction for Invoker/Runtime Containers
        • Ctrl: does NOT know about warm/cold, it knows it has a resource to run requst on or not
      • Marks notes that Tyson had a proposal some while ago on this as well
        • Dom. has has similar proposals to improve perf/sched
        • and discussions on Slack
      • Chetan: ctrl calls init? or just do run
        • Markus: undecided, Container Mgr. could call INIT as well…
      • Chetan: is it manages homogenous or heterogeneous pool of containers?
      • Markus: needs to know about actions of each Container for Load Balancing…
      • Mark: now on Load Balancing
        • 2 purposes:
          • where on cluster container si put (where resoruces are consumeed)
          • do we use a warm container
        • Proposal breask apart these repsonsivlities
      • Ctrl decides if it does a warm start (has resource) or not (cold) and neds to ask for resource
        • Container mgmt. now decides where to “put stuff” or delegate to Mesos or Kube APIs for example
      • Dan: So Ctrl needs to know what pool is avail (warm), but Ctr. Mfr knows complete pool
      • Markus: loosely yes. There are limits to concurrency… currently set to one (1 action per container . Tyson is breaking this up on multi-concurrency now… but still have tight constraints on concurrency inside a container
      • Markus: sharing state is not feasible, share state is too slow to do “live” state mgmt. eventual consistency is possible
      • Dan: Ctrl. 
      • Markus: roughly true. Envision ctr. Mgr. has list of containers for Action A and B, it dist. it across all controllers there, so each Controller may know only a subset of the overall resources…
      • Dan: Ctrl, each has a (subset) pool of resources…
      • Markus: if ctrl. leaves then mgr. redist, the exiting
      • Brendan, there is an Akka cluster manager (could look at this tech to do clustering  what Akka persistence owuld be needed to maintain state
      • Markus: yes
      • Dan: was looking at Akka, but were worried about sync time? Tiem for event. consistency?  for 3-5 nodes for example?
      • Brendan: depends on how you try to use it… would use sharding and sharding manager would “rebalance”… when you request diff entities, it would go back
        • everything broken on chunks based upon nodes avail..
      • Markus: state sharing, can only be effective if
        • if stable controller clusters, then this is a feasible approach
      • Brendan, ctr. mgr is single point of failure?
      • Markus: it is hard to have scalable ctr. mgr.
        • need to decide if it needs “hot standby” or look at horz. scaling
      • Brendan: perhaps active-active?
      • Markus: improves concurrency limits is what we need…
      • Markus: I did mention “edge cases” on wiki in proposal as well (with diagram) to show how it is affected as well
        • i.e., you have less containers to controllers… (not likely, but need to consider)
      • Chetan: old model had “stickiness” (same inv. receiving calls for same action); in tihs model, each controler (rec. requests in a round robin fashion) then it migh cause same request to goto diff. controllers hence diff containers….
      • Markus: may need to look at Nginx and see if this can be improved
      • Chetan: sharding based on hashing was orig. proposal, but we would need some better way to distribute
      • Chetan: kafka overhead?
      • Markus: fairly low… 1-2 msec. each way, not a huge deal… in the model we have today it assists in par, passing
      • Chetan: more impacting web actions?
      • Markus: anything that has params and returns a result…
      • Last section is on Limits
        • today conc. limit … only X number of requests in system at one time
        • with this: we can now have a picture of resources and resource consumption
        • prevent user from allocating an unlim. amount of containers
        • Could limit even on a given action… which is an ongoing requirement to us as a project
        • throttles we have today are kind of foreign… and not accurate in impl.
        • burst throttles could be moved to Nginx (or dropped entirely)
      • Markus: logging section needs some work still; could look at agents like Dave Grove has in Kube repo.
        • personally, logging retention on crit. path is NOT viable from a perf. point of view
      • Brendan: recommend looking at Akka model for logging… avoid locks, use pub-sub internally (asynch approach, use pub-sub bus no blocks)
      • Markus: how do you get the Context that you need on every log line? that is the major issue we have today… that is why all need threaded back to invoker today…
      • Markus: on “Container lifecycle”… Kube does not support pause/unpause
        • Dave employed an invoker agent… to help with this… able to reach underlying docker daemon to facilitate this…  
        • If container unused by X msec. the controller would still send the pause/unpause to the ctr. mgr.
        • today we have pause grace (50 sec, configurable)… under burst this path is not used, but should be quick
      • Tzu: what topics do you see in this proposal? Do we need to manage mult. topics for each action (like Dom. proposd?)
      • Markus: Like to avoid unbounded # of topics… as discussed on Dom proposal
        • single overflow topic… handles overflow sequentially; that is overlfow work is ongoing..
      • Chetan: option to use action name as partition key then have a way where consumers pulling from topic can “seek ahead”; container pools can pick up groups of actions…
        • goal: achieve consistent throughput was fund. goal; perhaps some middle ground helpful
    • Matt on project website redesign
      • Matt: shows screen…
      • goals:
        • Update to look/feel like similar open source projects
        • Reuse Jekyll for its plug-ins and reused templates, but decrease complexity of CSS (minimize)
        • works on mobile, tablet, laptop/desktop
        • 1-2 clicks away from any info
        • Full (curated) list of documentation (plans to review awsome openwhisk repo and better use markdown there)
          • rely on repo. markdown as canonical source of infor
      • James: ideas on getting people to language examples
      • Matt: yes,  plan to allow role drill-down to that level.
      • Justin: like the role of “Action developer”
      • Matt: we would like that granularity
      • will submit PR in a few days pending completion of Docs and Community page
      • Matt: still TODOs to link releases (widget?) and Medium blogs
      • Please engage us on dev list.
  • Release process: (Matt/Vincent)
    • Matt: Everyone should know our intrepid first Release Manager just had a new baby boy last week named Ryan… he is still trying to complete the formal “announce” last chats I had with him yesterday.
    • He also placed a requirement/opened an issue to publish source releases to our website
  • Mesos/Compose/Splunk update: (Dragos/Tyson)
    • No Update
  • Kubernetes:  (Dave Grove/Daisy)
    • No update
  • API Gateway (Matt Hamann/Dragos)
    • No update
  • Catalog/Packages/Samples (anyone)
    • No update
  • Tooling/Utilities (Carlos/Matt/Priti)


Confirm moderator for next call

  • Justin Halsall will volunteer, for Aug 1st meeting
  • adjourn 11:00 AM US Central
  • No labels