Using kubeadm to upgrade Kubernetes

I’ve started writing for the Heptio Blog, check out my new article on Upgrading to Kubernetes 1.8 with Kubeadm.

Also if you’re looking for more interactive help with Kubernetes, make sure you check out our brand new Kubernetes Office Hours, where we livestream developers answering user questions about Kubernetes. Starting tomorrow (18 October) at 1pm and 8pm UTC, hope to see you there!

 

Thoughts on the first Kubernetes Steering Election

The first steering committee election for Kubernetes is now over.  Congratulations to Aaron Crickenberger, Derek Carr, Michelle Noorali, Phillip Wittrock, Quinton Hoole and Timothy St. Clair, who will be joining the newly formed Kubernetes Steering Committe.

If you’re unfamiliar with what the SC does, you can check out their charter and backlog. I was fortunate to work alongside Paris Pittman on executing this election, hopefully the first of many “PB&J Productions”.

To give you some backstory on this, the Kubernetes community has been bootstrapping it’s governance over the past few years, and executing a proper election as stated in the charter was an important first step. Therefore it was critical for us to run an open election correctly.

Thankfully we can stand on the shoulders of giants. OpenStack and Debian are just two examples of projects with well formed processes that have stood the test of time. We then produced a voter’s guide to give people a place where they could find all the information they needed and the candidates a spot to fill in their platform statements.

This morning I submitted a pull request with our election notes and steps so that we can start building our institutional knowledge on the process, and of course, to share with whomever is interested.

Also, a big shout out to Cornell University for providing CIVS as a public service.

Can "cloud-native" concepts apply to home servers?

I’ve recently embarked on a journey of trying new things in my homelab. I’ve been experimenting with new methods of managing my services at home, and I have tried a few combinations of methods and operating systems.

Now that I am more familiar with the cloud native landscape, I was wondering if I could wean myself off the most stateful snowflake in my life, the trusty home server.

I’ve been collecting hardware recently and decided to go out of my comfort zone and run some home services via different combinations. Here’s what I found out.

What do I need to run?

I have a few things I run in house that I need to host:

The Unifi stuff depends on Java and MongoDB, and accesses hardware on the network. Pi-hole expects to basically be my DNS server, and Plex is the textbook definition of a stateful app; depending on the size of your video collection it can grow a substantial database that’s all on disk, so moving around means bringing gigs of stuff with it.

With this varied set of apps, we shall begin!

Old Reliable, traditional Ubuntu 16.04 with .debs

This has been working for me for years up to this point. As with anything involving third party packaging, maintenance can tend to get annoying.

For Unifi you need either an external Mongo repo(!) and/or a OpenJDK PPA(!) to get it to work.

Pi hole wants me to pipe a script to bash, and Plex just publishes one off debs with no repository, making that update a hassle.

There are some benefits here, once I configure unattended-upgrades things generally run fine. It’s a well understood system, and having the large repository of software is always good. Over time it tends to accumulate crap though.

Ubuntu 16.04 with Docker containers

The main advantage to this setup is I can crowdsource the maintenance of these apps to people who do a real good job, like the awesome guys at linuxserver.io. I can keep a lean and mostly stock host, toss away broken containers if need be, and keep everything nice and isolated.

What ppa do I need for unifi? Is my plex up to date? What java symlink do I need to fix? I don’t need to care anymore!

docker-compose was really good for this and relatively simple to grok, especially by stealing other people’s configs from github and modifying them to my needs.

Why not LXD containers?

I ran with this config for a while, but it had one major issue for me. LXD containers are system containers, that is, they run a copy of the OS. So now instead of maintaining one host I am now maintaining one host OS and many client OSes. Going into each one and installing/configuring the services felt like I was adding complexity. LXD is great, just not for my specific use case.

Ubuntu Core 16.04 with Docker containers

Finally, something totally different. I am pretty sure “home server” doesn’t rank high on the use case here, but I figured I would give it a shot. I took the same docker-compose files from before, except this time I deploy on top of ubuntu-core.

This gives me some nice features over the mutable-buntu. First off, atomic upgrades. Everytime it gets a new kernel it just reboots, and then on boot all the containers update and come back up.

This has a few teething issues. First off, if there’s a kernel update it’s just going to reboot, you can’t really control that. Another is, it really is small, so it’s missing tools. It can only install snaps, so no rsync, no git, no curl, no wget. I’m not going to run git out of a docker container. Also I couldn’t figure out how to run docker as the non-root user and I can’t seem to find the documentation on how to do that anywhere.

Container Linux with Docker containers

A slim OS designed to only run containers. This one definately has a more “work related” slant to it. There’s no normal installer, the installer basically takes a cloud-init-like yaml file and then dd’s the disk. Or just fire up your home PXE server. :) Most of the docs don’t even talk about how to configure the OS, the entire “state” is kept in this yaml file in git, it is expected that I can blow it away at any time and drop containers on it.

This comes with git, rsync, and curl/wget out of the box, so it’s nice to be able to have these core tools in there instead of totally missing. There’s also a toolbox command that will automagically fetch a traditional distro container (defaults to fedora) and then mounts the filesystem inside, so you can nano to your heart’s content.

This works really well. Container Linux lets me dictate the update policy as part of the config file, and if I have multiple servers I can cluster them together so that they will take turns rebooting without having a service go down. But as you can see, we quickly venture out of the “home server” use case with this one.

Container Linux with rkt/systemd

This is the setup I am enjoying the most. So instead of using the docker daemon, I create a systemd service file, like say /etc/systemd/system/unifi.service:

[Unit]
Description=Unifi
After=network.target
[Service]
Slice=machine.slice
Type=simple
ExecStart=/usr/bin/rkt --insecure-options=image run docker://linuxserver/unifi --volume config,kind=host,source=/home/jorge/config/unifi --mount volume=config,target=/config --net=host --dns=8.8.8.8
KillMode=mixed
Restart=always
[Install]
WantedBy=multi-user.target

Then I systemctl start unifi to start it, and systemctl enable unifi to enable it on boot. ContainerOS is set to reboot on Thursdays at 4am, containers update on boot. I can use journalctl and machinectl like I can with “normal” services.

Note that you can use this config on any OS that has systemd and rkt. Since ContainerOS has a section in it’s yaml file for writing systemd services, I can have one well maintained file that will just enable me to spit out and entire configured server in one shot. Yeah!

This one “feels” like the most future proof, as the OCI spec is now finalized it feels like over time every tool will just be able to consume these images. I don’t know what that means for rkt and/or docker, but you can similarly use docker in this manner as well.

What about state?

So far I’ve only really talked about the “installation problem”, I’ve continually left out the hard part, the state of the applications themselves. Reinstalling coreos will get me all the apps back but no data, that doesn’t sound very cloud native!

If you look at the systemd service file above, you see I keep the state in /home/jorge/config/plex. I do this for each of the services. I need to find a way to make sure that that is saved somewhere else than just local disk.

Saving this onto an NFS share earned overwhelming NOPE NOPE NOPE from the straw poll I took (and one even threatened to come over and fight me). And that’s kind of cheating by moving the problem.

So right now this is still up in the air, I fired up a quick duplicati instance to keep a copy on S3.

Really don’t want to add a ceph cluster to my home administration tasks. Suggestions here would be most welcome.

How have they been running?

I know what you’re thinking. Self rebooting servers and auto updating containers? You’re high.

Surprisingly in the last three months I have been running all five of these things side by side and they’ve all been rock solid.

I don’t know what to say here, some combination of great OS and container maintainers, or maybe not so much churn. I am going to try to keep these up and running as long as possible just to see what happens.

What’s left to try?

The obvious hole here is the Project Atomic stack, which will be next on the list.

And of course, it’s only a matter of time until one of these reboots breaks something or a container has a bad day.

If you think this blog post isn’t crazy enough, Chuck and I will be delving into Kubernetes for this later on as this is all just a warm up.

TLDRing your way to a Kubernetes Bare Metal cluster

Alex Ellis has an excellent tutorial on how to install Kubernetes in 10 minutes. It is a summarized version of what you can find in the official documentation. Read those first, this is a an even shorter version with my choices mixed in.

We’ll install 16.04 on some machines, I’m using three. I just chose to use Weave instead of sending you to a choose your-own-network page as you have other stuff to learn before you dive into an opinion on a networking overlay. We’re also in a lab environment so we assume some things like your machines are on the same network.

Prep the Operating System

First let’s take care of the OS. I set up automatic updates, ensure the latest kernel is installed, and then ensure we’re all up to date, whatever works for you:

sudo -s
dpkg-reconfigure unattended-upgrades
apt install linux-generic-hwe-16.04
apt update
apt dist-upgrade
reboot

Prep each node for Kubernetes:

This is just installing docker and adding the kubernetes repo, we’ll be root for these steps:

sudo -s
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main  
EOF

apt update
apt install -qy docker.io kubelet kubeadm kubernetes-cni

On the master:

Pick a machine to be a master, then on that one:

kubeadm init

And then follow the directions to copy your config file to your user account, we only have a few commands left needed with sudo so you can safely exit out and continue with your user account:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Let’s install the network, and then allow workloads to be scheduled on the master (for a lab we want to use all our hardware for workloads!):

kubectl apply -f https://git.io/weave-kube-1.6
kubectl taint nodes --all node-role.kubernetes.io/master-

On each worker node:

On each machine you want to be a worker (yours will be different, the output of kubeadm init will tell you what to do:

sudo kubeadm join --token 030b75.21ca2b9818ca75ef 192.168.1.202:6443 

You might need to tack on a --skip-preflight-checks, see #347, sorry for the inconvenience.

Ensuring your cluster works

It shouldn’t take long for the nodes to come online, just check em out:

$ kubectl get nodes
NAME       STATUS     AGE       VERSION
dahl       Ready      45m       v1.7.1
hyperion   NotReady   16s       v1.7.1
tediore    Ready      32m       v1.7.1

$ kubectl cluster-info
Kubernetes master is running at https://192.168.1.202:6443
KubeDNS is running at https://192.168.1.202:6443/api/v1/namespaces/kube-system/services/kube-dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'
$

Ok you’re cluster is rocking, now

Set up your laptop

I don’t like to be ssh’ed into my cluster unless I’m doing maintenance, so now that we know stuff is working let’s copy the kubernetes config from the master node to our local workstation. You should know how to copy files around systems already, but here’s mine for reference:

 sudo scp /etc/kubernetes/admin.conf [email protected]:/home/jorge/.kube/config

I don’t need the entire Kubernetes repo on my laptop, so we’ll just install the snap for kubectl and check that I can access the server:

  sudo snap install kubectl --classic
  kubectl get nodes
  kubectl cluster-info

Don’t forget to turn on autocompletion!

Deploy your first application

Let’s deploy the Kubernetes dashboard:

   kubectl create -f https://git.io/kube-dashboard
   kubectl proxy

Then hit up http://localhost:8001/ui.

That’s it, enjoy your new cluster!

Joining the Community

kubeadm is brought to you by SIG Cluster Lifecycle, they have regular meetings that anyone can attend, and you can give feedback on the mailing list. I’ll see you there!

Here's a pic from Ubuntu Down Under

While archiving a bunch of documents I found this pic of all of us from Ubuntu Down Under, thought I would share it!

Lessons Learned from joining the Kubernetes community

Software can be complex and daunting, even more so in distributed systems. So when you or your company decide you’re going to give it a shot, it’s easy to get enamored with the technology and not think about the other things you and your team are going to need to learn to make participating rewarding for everyone.

When we first launched the Canonical Distribution of Kubernetes, our team was new, and while we knew how Kubernetes worked, and what ops expertise we were going to start to bring to market right away, we found the initial huge size of the Kubernetes community to be outright intimidating. Not the people of course, they’re great, it’s the learning curve that can really seem large. So we decided to just dive in head first, and then write about our experiences. While some of the things I mention here work great for individuals, if you have a team on individuals working on Kubernetes at your company then I hope some of these tips will be useful to you. This is by no means an exhaustive list, I’m still finding new things everyday.

Find your SIGs

Kubernetes is divided into a bunch of Special Interest Groups(SIGs). You can find a list here. Don’t be alarmed! Bookmark this page, I use this as my starting off point anytime we needed to find something out in more detail that we could find in the docs or public list. On this page, you’ll find contact information for the leads, and more importantly, when those SIGs meet. Meetings are open to public and (usually) recorded. Find someone on your team to attend these meetings regularly. This is important for a few reasons:

  • k8s moves fast, and if there’s an area you care about, you can miss important information about a feature you care about.
  • It’s high bandwidth since SIGs meet regularly, you won’t find long drawn out technical discussions on the mailing lists like you would on a project that only uses lists, these discussions move much faster when people talk face to face.
  • You get to meet people and put faces to names.
  • People get to see you and recognize your name (and optionally, your face). This will help you later on if you’re stuck and need help or if you want to start participating more.
  • Each team has a slack channel and google group (mailing list), so I prefer to sit in those channels as well as they usually have important information announced there, meeting reminders, and links to the important documents for that SIG.

There’s a SIG just for contributor experience

SIG-contribex - As it ends up there’s an entire SIG who work on improving contributor experience. I found this SIG relatively late when we started, you’ll find that asking quetions here will save you time in the long run. Even if you’re not asking questions yourself you can learn about how the mechanics of project works just by listening in on the conversation.

So many things in /community

https://github.com/kubernetes/community - This should be one of your first starting points, if not the starting point. I put the SIGs above this because I’ve found for most people they’re initially interested in one key area, and you can just go to that SIG directly first to get started then come back to this. That doesn’t mean this isn’t important, if I get lost in something this is usually the place I start to look for something. Try to get everyone on your team to have at least a working understanding of the concepts here, and of course don’t forget the CLA and Code of Conduct.

There’s a YouTube Channel

https://www.youtube.com/c/KubernetesCommunity - I found this channel to be very useful for “catching up”. Many SIGs publish their meetings relatively quickly, and tossing in the channel in the background can help you keep track of what’s going on.

If you don’t have the time do dig into all the SIG meetings, you can concentrate on the weekly community meeting, which is held weekly and a summary of many of things happening around the different SIGs. The community meetings also have demos, so it’s interesting to see how the ecosystem is building tools around k8s; if you can only make one meeting a week, this is probably the one to go to.

The Community Calendar and meetings

This sounds like advanced common sense but there’s a community calendar of events.

Additionally, I found that adding the SIG meetings to our team calendar helps. We like to rotate people around meetings so that they can get experience in what is happening around the project and to ensure that worst case if someone can’t make a meeting someone is there to take notes. If you’re getting started, do yourself a favor volunteer to take notes at a SIG meeting, you will find that you’ll need to pay closer attention to the material and for me it helps me understand concepts better when I have to write it down in a way that makes sense for others.

We also found it useful to not flood one meeting with multiple people. If it’s something important, sure, but if you just want to keep an eye on what’s going on there you can only send one person and then have that person give people a summary at your team standup or whatever. There are so many meetings that you don’t want to fall into the trap of having people sitting in meetings all day instead of getting things done.

OWNERS

Whatever area you’re working on, go up the tree and eventually you’ll find an OWNERS file that list who owns/reviews that section of the code or docs or whatever. I use this as a little checklist when I join the SIG meetings to keep track of who is who. When I eventually went to a SIG meeting at Kubecon, it was nice to meet people who will be reviewing your work or you’ll be having a working relationship with.

Find a buddy

At some point you’ll be sitting in slack and you’ll see some poor person who is asking the same sorts of questions you were. That’s one of the first places you can start to help, just find someone and start talking to them. For me it was “Hey I noticed you sit in SIG-onprem too, you doing bare metal? How’s it going for you?”

It’s too big!

This used to worry me because the project is so large I figured I would never understand the entire thing. That’s ok. It’s totally fine to not know every single thing that’s going on, that’s why people have these meetings and summaries in the first place, just concentrate on what’s important to you and the rest will start to fall into place.

But we only consume Kubernetes, why participate?

One of the biggest benefits of consuming an open source project is taking advantage of the open development process. At some point something you care about will be discussed in the community then you should take advantage of the economies of scale that having so many people working on something gives you. Even if you’re only using k8s on a beautifully set up public cloud from a vendor where you don’t have to worry about the gruesome details, your organization can still learn from all the work that is happening around the ecosystem. I learn about new tools and tips every single day, and even if your participation is “read-only”, you’ll find that there’s value in sharing expertise with peers.

Ongoing process

This post is already too long, so I’ll just have to keep posting more as I keep learning more. If you’ve got any tips to share please leave a comment, or post and send me a link to link to.

What a kickass ride this has been!

After 9 years, 20 Ubuntu releases, 14,455 edits and 359,186 miles in the air, it’s time for me to move on to something else. I’ve been privileged to work with some of the brightest people in the industry, and for that I’ll always be grateful.

But I won’t be going far, I’ll be starting next month at Heptio, where I will get to work with them in supporting and advancing Kubernetes. The community is rapidly expanding, and I’m looking forward to contributing to the machinery that helps keep it a great place for people to participate. If you’ve not yet given k8s a spin, you should, it’s great stuff and only getting better every release.

There are too many people who I’d like to thank, but you know who you are, and I’ll still see everyone around at conferences. I’ll be around for the next few weeks to tie things up, and then onward and upward!