Sept 29th

Kube + OW meeting notes

Attendees: Dan Lavine, Ben Browning, David Grove, Joshua Smith

Start: 9:05

• DL: Past couple of weeks, we have made the build green again, updated CouchDB, added Kube pods for ApiGatewae and Redis.

• BB: Have some questions about upstream changes breaking us because.

• DL: This is something we have been discussing in IBM as Kube + OW is becoming more of a reality. We are getting some Vms so we can setup a CI. Also, since this is a new CI, I'm trying to drive this CI to be public to the community for anyone to see. For now though, the easiest thing to do is stand it up next to the other private CI jobs.

• BB: If it is just a matter of public CI, then I can create a public Jenkins image for this project. But I also know that Apache has its own CI infrastructure.

• Dan: That is true, I'm not sure who should officially have the CI images. Do you have any more info on this Dave?

• DG: I know we should be pushing to use the Apache infrastructure, but I know we are currently getting something setup on the internal CI. It should just be a matter of hosting it once it is set up.

• BB: I know there is one job on the public Apache infrastructure that just publishes Docker images, so someone in the community knows how to get resources.

• DG: We should also need a Kubernetes deployment, but I'm not sure if Apache is able to do that. That way we can use a real Kube environment.

• BB: Thats true, things can differ when using a real environment vs a local deployment.

9:11

• BB: I see some work for DeamonSets going on, I was wondering is there any advantage other than just running 1 Invoker per Kube node?

• DG: That is the main motivation, to have that guarantee.

• BB: There is already a way to do that with StateFul sets today with pod Affinity and AntiAffinity.

• DG: Using DeploymentSets it is just easier to see what is going on and you guarantee to get the correct count for the Invoker/Node count.

• BB: I'm asking is because we use the Kube API directly. For this, we just need the number of Invoker nodes needed to handle the load an Kube does all the distributing for us. So for us this doesn't really matter. We should just figure out the delta between the different styles of deployments on Kubernetes.

• DG: The main reason we are doing this is because we think that Kube is to slow for certain actions.

• BB: We think that Kube should eventually get there because there are some people at Red Hat in the Kube community working on these specific issues if they arise. We just have not have done any large scale testing. The small scale testing is great. I'm assuming you mean the cold start latency because pre-warmed pods don't have this issue. However I think we can extend the concept of pre-warming to other pods.

• DG: I think it would be great if we can push for upstream changes where we can just use the Kube APIs directly.

• BB: Its going to happen either way because Kubernetes wants to be able to run serverless workloads. There are some other projects and they all have the same limitations, so it will have to be solved in some way.

• DG: So I guess its which way we will be able to implement quicker.

• BB: Yeah so I guess its just a matter of getting out work into the core OpenWhisk repo. There are a couple of PRs out there to start getting some plug ability for workloads.

9:16

• DL: It would be cool if Kubernetes has the ability to download Docker containers for you but not actually run anything on them. It might be hard though if you would like to namespace these images, or only want to run Pods for the Kube nodes where Docker images have be pre-pulled.

• BB: I have toyed with this, and what I did was label certain nodes that only run Java images, or Python pods. But I can also do the same thing cluster wide if I want to. Then for on prem there are other solutions like setting up the docker registry locally.

• DL: Thats true, no-op containers could just be pulled and on-prem can handle these things differently because you have full controll.

9:19

• DL: Did you have anything you wanted to discuss there Joshua?

• JS: Not really, just interested to hear about OW on Kube and see whats going on. I'm a contractor and figure out if this would be useful for clients since Kubernetes is the next big thing.

• BB: Thinking in your shoes, the one thing we need to figure out is how does someone take our work and deploy it?

• JS: That is a good question

• BB: As OW gets closer to having actual releases, we need to start thinking about what those artifacts of a release are and how does someone consume those.

• JS: From my perspective, simplicity and stability will definitely be the keys. For adoption outside of the project, then these make things way easier for anyone looking at this as a solution.

• BB: I think that is a strength that OW + Kube can have. Just needing some yaml files, helm, or just a bash script to deploy things.

• JS: That would be great because when looking at the OpenWhisk repo that uses Ansible, if you don't use Ansible already as part of your deployments it makes it hard.

• DG: I just started learning about helm the last week and does it look like something that makes sense for you to use Ben?

• BB: For me I just wanted to make sure OW runs well on OpenShift and helm charts are not yet supported on OpenShift. My ideal thing is something that can be used to spit out a yaml file or helm chart. But its hard for doing just a local environment vs a production environment that has a lot more in it like persistent storage and clusters. So having something to generate common configs would be useful. Or we just chose one way and go with it, but that becomes harder for configuring the cluster.

• DG: I agree. Ideally there is just one set of things to maintain that anyone can use and I'm just wondering if helm is the standard most people use.

• BB: Helm isn't currently the standard as it still requires a server component to use. It is gaining popularity, but it still isn't the standard.

• JS: So I'm familiar with helm and I think one of the problems is that there actually isn't a standard. So it would be a risky time to start standardizing. I do think that creating some generation tools would be worth it in the long run, but I realize that that is a non trivial effort.

• BB: It even goes beyond just Kube though since it is currently maintained in Ansable, Kube, Docker Cmpose and its something that needs to be solved long term.

• DL: I wonder if there is a better way of doing full blow deployments. For a simpler piece, I think it would be good to generate one common config that any deployment can consume. In that config it would have info about release versions, images to use and common passwords shared across different components. Then everyone could just manually update a basic set of yaml files to insert their own features beyond that.

• BB: If we are only talking about Kube for a sec, then you could do some stuff with Pod preset files, or use configmaps for these things.

• DL: That Pod preset feature sounds neat, when did it come out?

• BB: I think 1.6 or maybe even 1.5. But that is another thing, we need to figure out what versions we are going to support.

• DL: I don't know either since that is an open ended question as things continually move forward, when do you stop maintaining backwards compatibility.

• BB: With releases we will have the ability to always deploy specific matching OW + Kube versions, we just need to figure out where the line in the sand is.

9:29

• DL: I don't know if any of you have actually run the test suite OW has agains Kubernets and I was talking with Kavitha and she was able to do it. So I think she will be submitting something back if need be to run tests. I had not even though about trying this.

• BB: I didn't until last week either. Someone was telling me that you just need to update the config file with properties to point to the Kube cluster. I think as part of build and CI plans we should run the full integration test suite.

• DG: That is definitely something we are going to do on the internal CI is run the test suite.

• BB: Resource wise, it might require a larger infrastructure than we currently use. So we might want to have a multi node cluster

• DG: I am able to run the test suite locally against OW, not OW on Kube locally. It takes about an hour and a half to fully run so its not to bad.

• BB: Yeah and hardware might not actually help since it probably is mostly witing and polling for status changes.

• BB: If you are focusing on CI internally right now, I might start conversations with the PMCs to get large Apache resources

9:33

• DL: Besides that for topics I do have an announcement. I will be ramping down my work for this OW on Kube as there are other teams that need some help with Kubernetes experience. I will still be around for a while to attend meetings and give some input. Hopefully the community will continue to drive this forward. I also saw that Dave made some PRs to maintain the Docker images I was maintaining. Hopefully as the CI comes along these can be made into more official images.

• BB: I figured once I saw that the Docker images were being moved to someone else. You will be missed Dan.

End 9:35