Table of contents
No headings in the article.
It has been a while since my last post, but as most of you know, every Year's end there are two events that keep you busy and occupied. So was I. And in my case I also had to fight my sicknesses. So the motivation to post something was very low on my side. But as the new year getting started I found my balance and routine again. So I thought it's time to pick up and post a new blog entry.
Since my last post I was able to enhance my little PI k8s cluster to two Raspi's and I want to give a little update on that.
In a previous post I mentioned I freed up a 4GB Raspi 4B+ and I wanted it to add it to my single node k8s cluster I was currently running. My old "mini" cluster was running k3os with k3s. So naturally I tried to setup the new node as agent with k3os and k3s
Anyway here is what I tried to accomplish:
So the Raspi Zero 2W with an attached 19'' Asus Monitor is my Monitoring solution. The Raspi Zero WH is my Weather gathering solution. In addition the two Zeros are building with "keepalived" a HA solution for my PiHole/DNS solution. My "mini" cluster based on the Raspi 4B+, runs currently Uptime Kuma to monitor all services and also Node-Red to generate the UI which is shown on the Asus Monitor from the Raspi Zero 2W. And my trusty old 2007 MacMini as my "ubuntu/docker" based host for services like, drone.io, gitea, portainer, registry and of course CouchDB. And last, but not least my good old Macbook Pro as main Work/Fun/Surf machine.
Ok, let's start. I did an install of Ubuntu on a SD card for the 4B+ 4GB, got it running and updated. Went through the takeover part on the k3s github.com page and failed. I tried that for two days and then I gave up. What I could do in December 2020, I was not able to reproduce in December 2021. Maybe it was because I used a newer version of Ubuntu 21.10 or I was not able to manipulate the "/boot/(firmware/)cmdline.txt" correctly. Anyhow after that frustration I gladly had the mandatory X-mas break and my mind was occupied with other and family stuff.
After the time out I decided to start over. And so I tore down my still running "mini" cluster with my Raspi 4B+ 8GB. First I tried to start the new k8s cluster with k3s on ubuntu. But I don't know why or what I did wrong or what I forgot to do, but I couldn't manage to boot ubuntu from an SSD on my Raspi 4B+ 8GB. It worked perfect with the 32GB SD card on the Raspi 4B+ 4GB, but not on the bigger with the SSD. There it worked fine with the SD card, but not with the SSD. And since I wanted the k8s master to boot from my 120GB SSD, that was another setback.
I invested again a couple of days to get it to work, but I failed, again. So I went to my Plan B. Plan B, I made up one evening before I fell asleep. My Plan B was using the experimental 64bit lite version of Raspberry OS Bullseye. And what can I say. It was as you would expect it. Flashed the SSD with the OS and it booted right away from the SSD, did the initial stuff and the master node was ready to become the k8s master.
I executed some commands on the Raspi "Master":
sudo rpi-eeprom-update -h # To check the info sudo rpi-eeprom-update -a -d # To actually update sudo raspi-config # To check the boot order and boot version sudo vi /boot/cmdline.txt # To add cgroup_memory=1 cgroup_enable=memory sudo apt update && sudo apt full-upgrade -y sudo apt autoclean sudo apt autoremove sync; sync; sudo reboot
I repeated the same with my Rasp 4B+ 4GB and in no time the agent node was also ready to become a k8s agent.
Next step was installing k8s. Naturally I went to k3s, cause it was designed for small devices like the Raspi. So I did the steps described on the k3s github page. And what can I say, I failed, yet again. It worked kinda, but not perfect, the "kubeconfig" was always in RO mode and hence I had problems. I tried it several times, but no success.
Of course I consulted my good "shiny" friend google or better his "ugly" pendant Duckduckgo to find help. But what I found was either to old and outdated or it didn't fit for my problem. But sometimes you have luck. So did I. At one point I came across an open source project "k3sup" - Shoutout to alexellis - and with this tool and the description it goes along with it, I was able to setup k3s successfully on both Raspi's 4B+. And the best part I could do it from my MacBook Pro. With two command lines my little mini cluster with two nodes was set up.
k3sup-darwin install --ip 192.168.x.y --k3s-extra-args '--write-kubeconfig-mode=644' --user pi k3sup-darwin join --ip 192.168.x.z --server-ip 192.168.x.y --user pi
One day later I deployed "Uptime Kuma" and "Node-Red" on my new k8s cluster and after 4 weeks of trial, error and success I was at the point I started out Mid-December, with the difference, I have now, two Raspi's in my cluster, both running Raspberry OS Bullseye 64bit lite with k3s. And that means I have finally space and resources to try more k8s stuff. For example I want to try out the Camunda Workflow Engine or finally migrate my docker based services from my old 2007 mac mini with 4GB to the k8s cluster, so it can go to retirement, eventually. And with the 120GB SSD on my "master" node I might also be able to try some NFS and file sharing things. I really want to try out if Samba Sharing is possible in k8s.
So till now, you wonder where is the Honeypot part. That's my fiends is a lucky coincident for you and for me. Ok, when I started to write this post, I searched on Youtube for some of the videos I watched in December to prepare for my k8s cluster setup, which I could include in my blog post, but as usual the Youtube algorithm gets you, and you stumble upon a video that distracts you from your original goal. Well, all I can say, the Youtube algorithm works and I got distracted.
Surprisingly it was a Honeypot for me, cause it is one thing to use a tool/technology like k8s, but it is something else if you know some of the history that is behind it. Since my love/hate relationship with k8s is still present, it was a nice break and for me also educational. I hope you enjoy the little k8s history from Honeypot, too.
As always apply this rule: "Questions, feel free to ask. If you have ideas or find errors, mistakes, problems or other things which bother or enjoy you, use your common sense and be a self-reliant human being."
Have a good one. Alex