7/21/17 Kubernetes OpenWhisk meeting
Attendes: Dan Lavine, Matt Rutkowsi, Ben Browoning, Kavitha Devara
Agenda: Go over last 2 weeks of work. Whats next to move the proposal forward.
Start:
9:15
DL: I'll just go over what happened the last 2 weeks
Used the current PR for removing Consul https://github.com/apache/incubator-openwhisk/pull/2452 and rebased master OpenWhisk with this PR. Then built a number of docker images to test out OpenWhisk can be deployed all through yaml files.
Current controller and Invoker are Dan's Images, but should mimic master OpenWhisk once PR is merged
Dan also created Images for Kafka and CouchDB.
Fairly happy with the outcome of deploying OpenWhisk. Next couple of weeks is to try and work with people to build better Docker Images. That way we can stop using Dan's and use published OpenWhisk ones.
KD: (question) (via text, problem with speech) Why do we need these custom images?
DL: I don't care for them either. Built them to capture Ansible pre/post processing and built that logic directly into the docker images.
KD: (question) Can we just use kubctl commands directly?
DL: I would rather not do that. Ideally everything is configurable via environment variables. This way any of the hard setup happens in the background for you and don't need to worry about it.
9:20
DL: (question) Ben, have you taken a look at what Kafka is now doing during deployment?
Ben: No I have not, do you mean in general or for OpenWhisk?
Dan: For OpenWhisk, they have edited it so now there are additional properties like message byte lengths. We need to capture the ability to set those.
KD: (Question) (Back from having microphone issues) What I was getting at with the custom images, what I was trying to do was try to and execute kubectl commands to setup the components. What is the purpose for these custom images, I don't see how this is different?
Ben: There are Ansible scripts that run after an image deploys today.
Dan: Pulls up a couple of files and show how to move Ansible scripts to Docker images.
Ben: I think what the real question is, we should be able to do a Kubectl create with a directory that includes all the files we need to run, correct?
KD: Yeah exactly, this way we can get deployment configurations in.
Ben: These Docker images will essentially be that. Images with a number of scripts to configure OpenWhisk.
Dan: Shares screen
Takes a look at Kafka when it deploys Ansible
In the Ansible script, point out that it pulls the images and once Kafka is up and running, it then goes through and creates a bunch of topics with paramaters.
Next pull up Dan's Docker image that has Kakfa + an init script.
Dan: Thanks Ben for pointing me to the images you have also been creating
Dan: Looking at what I did for my Kafka Docker image, it starts Kafka and waits till it is up and running. Then it also goes and creates all the topics for Kafka once it is up and running. So I'm trying to move logic into one central location to deploy via environment variables.
KV: This is exactly what I was trying to do. I just used kubectl commands when targeting my instance. But I was having an issue with Kafka getting into a crash loop.
9:29
KV: (question) when I was trying to use Kubectl, I was looking at the current hostfile when trying to build these arguments, but the host file does not define zookeeprs route. How are things supposed to discover the Host?
DL: So in Kube, we split out zookeeper and Kafka:
Looking at the Zookeeper deployment, Zookeeper deploys a Service to create a KubeDNS entry.
Looking at the Kafka.yml, the DNS entry is set for the Zookeeper host.
KV: (question) So when does this DNS entry get created?
DL: When you deploy the Zookeeper yaml, it has a detention for both the Service and Pod inside of it. So when you do a kubectl create zookeeper.yml it will create both of them for you.
KV: I see, the Services are needed for Kube DNS entries.
DL: correct.
KV: (question) I thought if they are in the same namespace, they can talk to each other?
DL: Not exactly. What would need to be done is have both Pods defined in the same containers namespaces for the yaml file. Then they can talk to each other over local host. In a big cluster we are not saying where these have to run, so they could be running on different node.
KV: So this is the reason for all these services define?
DL: Yeah. There is a lot going on for deployments and concepts with Kube. Hard to wrap your head around it unless you start playing with it.
KV: (Question) So how do we see these services?
DL: You can use `kubectl services` to see the DNS entries. And Inspect them to see extra info like what Pods use them.
9:35
DL: Fairly happy with how things are deployed other than Kafka. Since topics are built into the image. Lose the ability to just run `kubectl scale ...`
KV: (Question) So this is the reason for keping things at the env level, rather tha Kube ConfigMap?
Ben: Well what we want to do is be able to set them via Environment variables, or a ConfigMap which is a list of environment variables. Not all versions of Kube can use the ConfigMap as environment variables though, think only newer version of Kube can do this.
Ben: I bring this up because there are many environment variables that are common and keep getting specified all over.
Dan: Down the line this will be needed and needed much more for making things HA and deploying everything smoothly.
Ben: There are other things to. Like other PRs to make OpenWhisk take in a list of Kafka addresses.
Ben: Back to Kafka, there are other things we can do. As long as the invoker have access to the topics script, it can be created with proper parameters.
Dan: So for Kube could have another Pod that can exec into the container and run the commands with proper topics.
Ben: Could just have the script be built into the Invoker image. Then you can create what is needed from the Invoker.
Dan: Oh you are right. I didn't even think about that.
Ben: There are other PRs out there too with other changes to Kafka we need to watch.
9:40
Dan: Great. I think it would be a win to start getting some of this logic into the OpenWhisk images. Then everyone deploying via Ansible, Kube, whatever will be happier with an easier process.
Only problem is I'm not sure where to put it. So I would like to work with someone to build these things.
Ben: I agree, think everyone will benefit.
9:41 General questions and discussion
Ben: We have a prototype with OpenWhisk talking to Kubernetes api and it is fully implemented. There are still some open questions about what we want to do for logging.
Today logs are not shipped to a 3rd party. Right now we just use `kubectl logs` which is quite expensive.
I think there is a more general issue for how to get these logs.
Any thoughts on bringing this discussion back up.
Dan: not sure about this, there was some thought about doing this, but the performance wount be as good. So the though was to side table this for now. However with new talks about different scheduling I think this is something that should be brought back up.
Matt: Absolutely, I think what we need to do is capture these issue and raise awareness so it does not get lost. Make issues, put on wiki and come up with a plan for moving these things forward.
Ben: To be fair, we have also been working hard with creating images with running on the Kube API.
Matt: I'm all for these changes on feature branches and getting this work out there.
Ben: I'm all for that and getting this out there on a branch for the main repo. The eventual goal is to get this all pushed back mainstream.
Matt: I think one of the main plusses with branches is you get more eyes on it. Then can get some feedback from others as well.
9:47
KV: So it looks like the Kube deployment task is done. What else needs to be addressed?
Dan: The main deployment for the first phase has been proven out, but there are other things that need to be addressed.
For example, when controller comes up, it should have retries when connecting to the database.
KV: is this for HA controller?
Dan: This isn't for HA. It is more for a clean deployment. Right now you could put everything into one giant yaml and have it be created at once. But process will keep failing until things are configured properly.
Would like to have things not keebp failing and being reselected by the system. Retry logic is a nicer way to get there.
Ben: That is currently what we do in our fork. Everything is in one giant yaml file. We have scripts in our docker images to have things curl for their dependencies and then trie to start.
Dan: I wasn't sure what approach I wanted. Right now I did it with health checks and things will just keep failing over until they eventually succeed.
Ben: That is definitely more of a common approach and I like that too.
When KV was asking about these deployment things and making things more robust/HA there is more work to be done there.
9:51
KV: I was wondering if there is more to be done fore logging as well. Since everything is done as JSON right now. Is there a way to get everything collected in one central place and show the status or health of the system? And are there any tools for that?
Ben: You mean a tool to get all these Kube logs and other output in one place?
KV: Correct. Is there one place to put all this stuff and have things be visual when tracing these logs.
Ben: Trying to think for OpenSource I'm not sure.
There are things for what RedHat is doing to get these things done for OpenShift.
Not sure of there is a more generic thing out there for anyone to use for OpenSource
KV: Right. I was just curious if anyone has put more thought if anyone has put more thought to get this in a visual field for health of the system.
Dan: I know I have never worked with anything like this at production scale. Deploying and monitoring processes for production.
KV: Thanks
9:54
Dan: I will submit all my current code as PRs to the Kube repo with the know fact that it relies on Consul being removed. Next week, my plans are to help with any changes blocking the Consul removal PR and to work with someone for smarter docker images.
End: 9:55