Lessons Learned from joining the Kubernetes community

Software can be complex and daunting, even more so in distributed systems. So when you or your company decide you’re going to give it a shot, it’s easy to get enamored with the technology and not think about the other things you and your team are going to need to learn to make participating rewarding for everyone.

When we first launched the Canonical Distribution of Kubernetes, our team was new, and while we knew how Kubernetes worked, and what ops expertise we were going to start to bring to market right away, we found the initial huge size of the Kubernetes community to be outright intimidating. Not the people of course, they’re great, it’s the learning curve that can really seem large. So we decided to just dive in head first, and then write about our experiences. While some of the things I mention here work great for individuals, if you have a team on individuals working on Kubernetes at your company then I hope some of these tips will be useful to you. This is by no means an exhaustive list, I’m still finding new things everyday.

Find your SIGs

Kubernetes is divided into a bunch of Special Interest Groups(SIGs). You can find a list here. Don’t be alarmed! Bookmark this page, I use this as my starting off point anytime we needed to find something out in more detail that we could find in the docs or public list. On this page, you’ll find contact information for the leads, and more importantly, when those SIGs meet. Meetings are open to public and (usually) recorded. Find someone on your team to attend these meetings regularly. This is important for a few reasons:

  • k8s moves fast, and if there’s an area you care about, you can miss important information about a feature you care about.
  • It’s high bandwidth since SIGs meet regularly, you won’t find long drawn out technical discussions on the mailing lists like you would on a project that only uses lists, these discussions move much faster when people talk face to face.
  • You get to meet people and put faces to names.
  • People get to see you and recognize your name (and optionally, your face). This will help you later on if you’re stuck and need help or if you want to start participating more.
  • Each team has a slack channel and google group (mailing list), so I prefer to sit in those channels as well as they usually have important information announced there, meeting reminders, and links to the important documents for that SIG.

There’s a SIG just for contributor experience

SIG-contribex - As it ends up there’s an entire SIG who work on improving contributor experience. I found this SIG relatively late when we started, you’ll find that asking quetions here will save you time in the long run. Even if you’re not asking questions yourself you can learn about how the mechanics of project works just by listening in on the conversation.

So many things in /community

https://github.com/kubernetes/community - This should be one of your first starting points, if not the starting point. I put the SIGs above this because I’ve found for most people they’re initially interested in one key area, and you can just go to that SIG directly first to get started then come back to this. That doesn’t mean this isn’t important, if I get lost in something this is usually the place I start to look for something. Try to get everyone on your team to have at least a working understanding of the concepts here, and of course don’t forget the CLA and Code of Conduct.

There’s a YouTube Channel

https://www.youtube.com/c/KubernetesCommunity - I found this channel to be very useful for “catching up”. Many SIGs publish their meetings relatively quickly, and tossing in the channel in the background can help you keep track of what’s going on.

If you don’t have the time do dig into all the SIG meetings, you can concentrate on the weekly community meeting, which is held weekly and a summary of many of things happening around the different SIGs. The community meetings also have demos, so it’s interesting to see how the ecosystem is building tools around k8s; if you can only make one meeting a week, this is probably the one to go to.

The Community Calendar and meetings

This sounds like advanced common sense but there’s a community calendar of events.

Additionally, I found that adding the SIG meetings to our team calendar helps. We like to rotate people around meetings so that they can get experience in what is happening around the project and to ensure that worst case if someone can’t make a meeting someone is there to take notes. If you’re getting started, do yourself a favor volunteer to take notes at a SIG meeting, you will find that you’ll need to pay closer attention to the material and for me it helps me understand concepts better when I have to write it down in a way that makes sense for others.

We also found it useful to not flood one meeting with multiple people. If it’s something important, sure, but if you just want to keep an eye on what’s going on there you can only send one person and then have that person give people a summary at your team standup or whatever. There are so many meetings that you don’t want to fall into the trap of having people sitting in meetings all day instead of getting things done.

OWNERS

Whatever area you’re working on, go up the tree and eventually you’ll find an OWNERS file that list who owns/reviews that section of the code or docs or whatever. I use this as a little checklist when I join the SIG meetings to keep track of who is who. When I eventually went to a SIG meeting at Kubecon, it was nice to meet people who will be reviewing your work or you’ll be having a working relationship with.

Find a buddy

At some point you’ll be sitting in slack and you’ll see some poor person who is asking the same sorts of questions you were. That’s one of the first places you can start to help, just find someone and start talking to them. For me it was “Hey I noticed you sit in SIG-onprem too, you doing bare metal? How’s it going for you?”

It’s too big!

This used to worry me because the project is so large I figured I would never understand the entire thing. That’s ok. It’s totally fine to not know every single thing that’s going on, that’s why people have these meetings and summaries in the first place, just concentrate on what’s important to you and the rest will start to fall into place.

But we only consume Kubernetes, why participate?

One of the biggest benefits of consuming an open source project is taking advantage of the open development process. At some point something you care about will be discussed in the community then you should take advantage of the economies of scale that having so many people working on something gives you. Even if you’re only using k8s on a beautifully set up public cloud from a vendor where you don’t have to worry about the gruesome details, your organization can still learn from all the work that is happening around the ecosystem. I learn about new tools and tips every single day, and even if your participation is “read-only”, you’ll find that there’s value in sharing expertise with peers.

Ongoing process

This post is already too long, so I’ll just have to keep posting more as I keep learning more. If you’ve got any tips to share please leave a comment, or post and send me a link to link to.

What a kickass ride this has been!

After 9 years, 20 Ubuntu releases, 14,455 edits and 359,186 miles in the air, it’s time for me to move on to something else. I’ve been privileged to work with some of the brightest people in the industry, and for that I’ll always be grateful.

But I won’t be going far, I’ll be starting next month at Heptio, where I will get to work with them in supporting and advancing Kubernetes. The community is rapidly expanding, and I’m looking forward to contributing to the machinery that helps keep it a great place for people to participate. If you’ve not yet given k8s a spin, you should, it’s great stuff and only getting better every release.

There are too many people who I’d like to thank, but you know who you are, and I’ll still see everyone around at conferences. I’ll be around for the next few weeks to tie things up, and then onward and upward!

Deploying Kubernetes on AWS, GCE, and Bare Metal

As part of of the weekly Kubernetes Community Meeting Marco Ceppi deploys a fully functional Kubernetes cluster on AWS, GCE, and bare metal:

If you’re interested in bare metal Kubernetes, we invite you to join us and other contributors in the sig-onprem.

Not sure where to get started? Check out our Getting Started documentation.

Canonical Distribution of Kubernetes - Release 1.5.3

We’re proud to announce support for Kubernetes 1.5.3 in the Canonical Distribution of Kubernetes. This is a pure upstream distribution of Kubernetes, designed to be easily deployable to public clouds, on-premise (ie vsphere, openstack), bare metal, and developer laptops. Kubernetes 1.5.2 is a patch release comprised of mostly bugfixes, and we encourage you to check out the release notes.

Getting Started:

Here’s the simplest way to get a Kubernetes 1.5.3 cluster up and running on an Ubuntu 16.04 system:

sudo snap install conjure-up --classic
conjure-up kubernetes

During the installation conjure-up will ask you what cloud you want to deploy on and prompt you for the proper credentials. If you’re deploying to local containers (LXD) see these instructions for localhost-specific considerations.

For production grade deployments and cluster lifecycle management it is recommended to read the full Canonical Distribution of Kubernetes documentation.

Home page: https://jujucharms.com/canonical-kubernetes/

Source code: https://github.com/juju-solutions/bundle-canonical-kubernetes

How to upgrade

With your kubernetes model selected, you can deploy the bundle to upgrade your cluster if on the 1.5.x series of kubernetes. At this time releases before 1.5.x have not been tested. Depending on which bundle you have previously deployed, run:

    juju deploy canonical-kubernetes

or

    juju deploy kubernetes-core

If you have made tweaks to your deployment bundle, such as deploying additional worker nodes as a different label, you will need to manually upgrade the components. The following command list assumes you have made no tweaks, but can be modified to work for your deployment.

juju upgrade-charm kubernetes-master
juju upgrade-charm kubernetes-worker
juju upgrade-charm etcd
juju upgrade-charm flannel
juju upgrade-charm easyrsa
juju upgrade-charm kubeapi-load-balancer

This will upgrade the charm code, and the resources to kubernetes 1.5.3 release of the Canonical Distribution of Kubernetes.

New features:

  • Full support for Kubernetes v1.5.3.
  • K8s master charm now properly keeps distributed master files in sync for an HA control plane.

General Fixes

  • #41251 - Fix UpdateAddonsTactic to use local repo
  • #42058 - Fix shebangs in charm actions to use python3
  • #41815 - enable DefaultTolerationSeconds admission controller by default
  • #41351 - Multi master patch
  • #41256 - Lint fixes for the master and worker Python code
  • #41919 - Juju: Disable anonymous auth on kubelet (Thanks to community member @raesene for pointing this out!)

etcd Specific Fixes

  • #74 - Add the ability to attach NRPE to etcd for monitoring with an external Nagios server.
  • #76 - Add the debug action to the etcd charm that makes it easier to collect debug information in the field.

Test Results

The Canonical Distribution of Kubernetes is a running daily tests to verify it works with the upstream code. As part of the Kubernetes test infrastructructure we upload daily test runs. The test results are available on the dashboard. Follow along with our progress here:

How to contact us:

We’re normally found in these Slack channels and attend these sig meetings regularly:

Operators are an important part of Kubernetes, we encourage you to participate with other members of the Kubernetes community!

We also monitor the Kubernetes mailing lists and other community channels, feel free to reach out to us. As always, PRs, recommendations, and bug reports are welcome: https://github.com/juju-solutions/bundle-canonical-kubernetes

Canonical Distribution of Kubernetes - Release 1.5.2

We’re proud to announce support for Kubernetes 1.5.2 in the Canonical Distribution of Kubernetes. This is a pure upstream distribution of Kubernetes, designed to be easily deployable to public clouds, on-premise (ie vsphere, openstack), bare metal, and developer laptops. Kubernetes 1.5.2 is a patch release comprised of mostly bugfixes, and we encourage you to check out the release notes.

Getting Started:

Here’s the simplest way to get a Kubernetes 1.5.2 cluster up and running on an Ubuntu 16.04 system:

sudo apt-add-repository ppa:juju/stable
sudo apt-add-repository ppa:conjure-up/next
sudo apt update
sudo apt install conjure-up
conjure-up kubernetes

During the installation conjure-up will ask you what cloud you want to deploy on and prompt you for the proper credentials. If you’re deploying to local containers (LXD) see these instructions for localhost-specific considerations.

For production grade deployments and cluster lifecycle management it is recommended to read the full Canonical Distribution of Kubernetes documentation.

Home page: https://jujucharms.com/canonical-kubernetes/

Source code: https://github.com/juju-solutions/bundle-canonical-kubernetes

How to upgrade

With your kubernetes model selected, you can deploy the bundle to upgrade your cluster if on the 1.5.x series of kubernetes. At this time releases before 1.5.x have not been tested. Depending on which bundle you have previously deployed, run:

    juju deploy canonical-kubernetes

or

    juju deploy kubernetes-core

If you have made tweaks to your deployment bundle, such as deploying additional worker nodes as a different label, you will need to manually upgrade the components. The following command list assumes you have made no tweaks, but can be modified to work for your deployment.

juju upgrade-charm kubernetes-master
juju upgrade-charm kubernetes-worker
juju upgrade-charm etcd
juju upgrade-charm flannel
juju upgrade-charm easyrsa
juju upgrade-charm kubeapi-load-balancer

This will upgrade the charm code, and the resources to kubernetes 1.5.2 release of the Canonical Distribution of Kubernetes.

New features:

  • Full support for Kubernetes v1.5.2.

General Fixes

  • #151 #187 It wasn’t very transparent to users that they should be using conjure-up when locally developing, conjure-up is now the defacto default mechanism for deploying CDK.

  • #173 Resolved permissions on ~/.kube on kubernetes-worker units

  • #169 Tuned the verbosity of the AddonTacticManager class during charm layer build process

  • #162 Added NO_PROXY configuration to prevent routing all requests through configured proxy [by @axinojolais]

  • #160 Resolved an error by flannel sometimes encountered during cni-relation-changed [by @spikebike]

  • #172 Resolved sporadic timeout issues between worker and apiserver due to nginx connection buffering [by @axinojolais]

  • #101 Work-around for offline installs attempting to contact pypi to install docker-compose

  • #95 Tuned verbosity of copy operations in the debug script for debugging the debug script.

Etcd layer-specific changes

  • #72 #70 Resolved a certificate-relation error where etcdctl would attempt to contact the cluster master before services were ready [by @javacruft]

Unfiled/un-scheduled fixes:

  • #190 Removal of assembled bundles from the repository. See bundle author/contributors notice below

Additional Feature(s):

  • We’ve open sourced our release management process scripts we’re using in a juju deployed jenkins model. These scripts contain the logic we’ve been running by hand, and give users a clear view into how we build, package, test, and release the CDK. You can see these scripts in the juju-solutions/kubernetes-jenkins repository. This is early work, and will continue to be iterated on / documented as we push towards the Kubernetes 1.6 release.

Notice to bundle authors and contributors:

The fix for #190 is a larger change that has landed in the bundle-canonical-kubernetes repository. Instead of maintaining several copies across several repositories of a single use-case bundle; we are now assembling the CDK based bundles as fragments (un-official nomenclature).

This affords us the freedom to rapidly iterate on a CDK based bundle and include partner technologies, such as different SDN vendors, Storage backend components, and other integration points. Keeping our CDK bundle succinct, and allowing the more complex solutions to be assembled easily, reliably, and repeatedly. This does change the contribution guidelines for end users.

Any changes to the core bundle should be placed in its respective fragment under the fragments directory. Once this has been placed/merged, the primary published bundles can be assembled by running ./bundle in the root of the repository. This process has been outlined in the repository README.md

We look forward to any feedback on how opaque/transparent this process is, and if it has any useful applications outside of our own release management process. The ./bundle python script is still very much geared towards our own release process, and how to assemble bundles targeted for the CDK. However we’re open to generalizing them and encourage feedback/contributions to make this more useful to more people.

How to contact us:

We’re normally found in these Slack channels and attend these sig meetings regularly:

Operators are an important part of Kubernetes, we encourage you to participate with other members of the Kubernetes community!

We also monitor the Kubernetes mailing lists and other community channels, feel free to reach out to us. As always, PRs, recommendations, and bug reports are welcome: https://github.com/juju-solutions/bundle-canonical-kubernetes

Fresh Kubernetes documentation available now

Over the past few months our team has been working real hard on the Canonical Distribution of Kubernetes. This is a pure-upstream distribution of k8s with our community’s operational expertise bundled in.

It means that we can use one set of operational code to get the same deployment on GCE, AWS, Azure, Joyent, OpenStack, and Bare Metal.

Like most young distributed systems, Kubernetes isn’t exactly famous for it’s ease of use, though there has been tremendous progress over the past 12 months. Our documentation on Kubernetes was nearly non-existent and it became obvious that we had to dive in there and bust it out. I’ve spent some time fixing it up and it’s been recently merged. 

You can find the Official Ubuntu Guides in the “Create a cluster” section. We’re taking what I call a “sig-cluster-lifecycle” approach to this documentation – the pages are organized into lifecycle topics based on what an operator would do. So “Backups”, or “Upgrades” instead one big page with sections. This will allow us to grow each section based on the expertise we learn on k8s for that given task. 

Over the past few months (and hopefully for Kubernetes 1.6) we will slowly be phasing out the documentation on our individual charm and layer pages to reduce duplication and move to a pure upstream workflow. 

On behalf of our team we hope you enjoy Kubernetes, and if you’re running into issues please let us know or you can find us in the Kubernetes slack channels.

Useful tool alert, explainshell.com

Ran into this while using the Stack Overflow; explainshell.com. Basically they take all of Ubuntu’s manpages and then parse them so you can paste in any Linux command and then see right away what each option does. Example:

rsync -chavzP –stats [email protected]:/path/to/copy /path/to/local/storage

It then takes the command you put in there and breaks down each of the flags. I like this for a few reasons. First of all, I hate reading manpages because I’m a human being. This shows me exactly what each option does without having to parse the entire manpage, it only tells me what I care about. Secondly, I prefer to learn by example instead of just reading manpages in their entirety. I can see this tool being very useful for people who are just starting to learn. Command on the internet confusing you? You can at least paste it in here and figure out what it’s doing before you end up being that guy.

Another example is bropages, which lets people submit example commands for each tool, and then users vote on the usefulness of the examples for a nice stackoverflow-like list of example commands. It seems as though they haven’t had a commit for almost a year, so not sure if people are still contributing to that, but it seems like a decent enough idea.