title

Blog of René Jochum

Blogging about Programming, Security, Linux, Networking and Web Apps.

Howto Install Rancher k3s with MariaDB Galera

Installing Rancher k3s with MariaDB Galera


In this blog Post we gonna install a HA Rancher Kubernetes Cluster with a MariaDB Galera Cluster as Datastore.

Outline

  • [Prepare Ubuntu Bionic Server]()
  • [Install MariaDB Galera]()
  • [Deploy K3S]()
  • [Install kubectl and helm]()
  • [Install MetalLB]()
  • [Install cert-manager for Let’s Encrypt]()
  • [Install Rancher]()
  • [Forward HTTP/HTTPS to the Rancher Load Balancer IP]()

Prepare Ubuntu Bionic Server

You need 3 Nodes, 4 CPU, >8GiB RAM, 100GiB Disk, I have 3 Nodes 4 CPU, 24 GiB RAM, 250GiB Disk.

Install Ubuntu on one Server, remove snapd, ufw, cloud-init.
Then clone it and edit /etc/hosts /etc/hostname /etc/netplan/50-cloud-init.yaml and rm -f /etc/ssh/ssh_host_* - reboot.

Install MariaDB Galera

Install the MariaDB repo on each server.

sudo apt-get install software-properties-common
sudo apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc'
sudo add-apt-repository 'deb [arch=amd64,arm64,ppc64el] http://ftp.hosteurope.de/mirror/mariadb.org/repo/10.4/ubuntu bionic main'

sudo apt update
sudo apt install mariadb-server mariadb-client mariadb-backup
Secure MariaDB on each Node

Set Password with mysql_secure_installation:

pcdummy@rancher01:~$ sudo mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
haven't set the root password yet, you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password or using the unix_socket ensures that nobody
can log into the MariaDB root user without the proper authorisation.

You already have your root account protected, so you can safely answer 'n'.

Switch to unix_socket authentication [Y/n] n
 ... skipping.

You already have your root account protected, so you can safely answer 'n'.

Change the root password? [Y/n]  
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!
Configure Galera on each Node

Stop Mariadb

sudo systemctl stop mariadb

Liste on all Interfaces (if you want configure it to listen only on a specific address):

sudo sed -i 's/max_connections\t\t= 100/max_connections\t\t= 1000/g' /etc/mysql/my.cnf
sudo sed -i 's/bind-address\t\t= 127.0.0.1/#bind-address\t\t= 127.0.0.1/g' /etc/mysql/my.cnf

Enable Galera, Paste the following into /etc/mysql/mariadb.conf.d/99-cluster.cnf

[galera]

wsrep_on = on
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_cluster_address = gcomm://10.128.1.17,10.128.1.18,10.128.1.19 
wsrep_cluster_name = k3s_cluster_0
wsrep_provider_options="gcache.size=512M"
wsrep_sst_method = mariabackup
wsrep_sst_auth = root:SuperSecretRootPassword

default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
innodb_doublewrite = 1

binlog_format = ROW

And change the ip addresse for wsrep_cluster_address

Some tuning if you use this Galera cluster for other purposes

sudo nano /etc/mysql/mariadb.conf.d/98-tuning.cnf
[mysqld]
key_buffer_size=256M
thread_stack=192K
thread_cache_size=8
max_connections=1000
innodb_buffer_pool_size=2G
query_cache_limit=2M
query_cache_size=0
query_cache_type=0
table_open_cache=128
join_buffer_size=512k
table_definition_cache=-1
performance_schema=ON
innodb_log_file_size=256M
innodb_buffer_pool_instances=2
tmp_table_size=32M
max_heap_table_size=32M
Bootstrap the cluster

One one node run sudo galera_new_cluster

One the other 2 nodes run: sudo systemctl start mariadb.service

Check the MariaDB Galera Cluster
mysql -u root -p -e "SELECT * FROM information_schema.global_status WHERE variable_name IN ('WSREP_CLUSTER_STATUS','WSREP_LOCAL_STATE_COMMENT','WSREP_CLUSTER_SIZE','WSREP_EVS_REPL_LATENCY','WSREP_EVS_DELAYED','WSREP_READY');"
Create the k3s Database

One one node run:

mysql -u root -p
CREATE DATABASE `k3s`;
GRANT ALL PRIVILEGES ON `k3s`.* TO 'k3s'@'%' IDENTIFIED BY '<superSecret>';

Deploy k3s

Install k3s one each nodes, one after another:

curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://k3s:<superSecret>@tcp(localhost:3306)/k3s" --no-deploy servicelb --no-deploy traefik

Check the nodes after.

sudo k3s kubectl get nodes

One one node copy the config (I choose node1 for that):

mkdir ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown -R $(whoami): ~/.kube

Install kubectl and helm

Install kubectl

sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

Install helm to ~/bin

wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
tar xfz helm-v3.2.1-linux-amd64.tar.gz
mkdir ~/bin
mv linux-amd64/helm linux-amd64/tiller ~/bin
rm -rf linux-amd64

Init helm on the cluster

kubectl --namespace kube-system create serviceaccount tiller;
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller;
helm init --service-account tiller;

Install MetalLB

Install MetalLB (change the address range!)

helm install stable/metallb \
  --name metallb \
  --namespace kube-system \
  --set configInline.address-pools[0].name=default \
  --set configInline.address-pools[0].protocol=layer2 \
  --set configInline.address-pools[0].addresses[0]=10.128.3.1-10.128.3.254

Check the deployment

kubectl get pods -n kube-system -l app=metallb -o wide

Install cert-manager for Let’s Encrypt

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install jetstack/cert-manager \
  --name cert-manager \
  --namespace cert-manager \
  --version v0.12.0
$ kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-6bcdf8c5cc-5bcrg              1/1     Running   0          54s
cert-manager-cainjector-6659d6844d-zrr5h   1/1     Running   0          54s
cert-manager-webhook-547567b88f-ptrlg      1/1     Running   0          54s

Install Nginx

helm install nginx-ingress stable/nginx-ingress --namespace kube-system \
    --set controller.image.runAsUser=101 \
    --set defaultBackend.enabled=false

Install Rancher

helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
kubectl create namespace cattle-system
helm install rancher-stable/rancher \
  --name rancher \
  --namespace cattle-system \
  --set hostname=rancher.example.org \
  --set ingress.tls.source=letsEncrypt \
  --set letsEncrypt.email=support@example.org
$ kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...

Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available...
deployment "rancher" successfully rolled out

Forward HTTP/HTTPS to the Rancher Load Balancer IP

$ kubectl -n kube-system describe service/traefik
Name:                     traefik
Namespace:                kube-system
Labels:                   app=traefik
                          chart=traefik-1.81.0
                          heritage=Helm
                          release=traefik
Annotations:              field.cattle.io/publicEndpoints:
                            [{"addresses":["10.128.3.1"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["10.128.3.1"],...
Selector:                 app=traefik,release=traefik
Type:                     LoadBalancer
IP:                       10.43.47.63
LoadBalancer Ingress:     10.128.3.1
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32316/TCP
Endpoints:                10.42.0.6:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30752/TCP
Endpoints:                10.42.0.6:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age                  From                Message
  ----    ------        ----                 ----                -------
  Normal  IPAllocated   16m                  metallb-controller  Assigned IP "10.128.3.1"
  Normal  nodeAssigned  3m10s (x5 over 16m)  metallb-speaker     announcing from node "rancher01"

Here the IP is 10.128.3.1 i forward HTTP (80) and HTTPS (443) to it.

Wait for the Let’s Encrypt Cert

$ kubectl -n cattle-system describe certificate
Name:         tls-rancher-ingress
Namespace:    cattle-system
Labels:       app=rancher
              chart=rancher-2.4.3
              heritage=Tiller
              release=rancher
Annotations:  <none>
API Version:  cert-manager.io/v1alpha2
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-05-07T23:07:41Z
  Generation:          1
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  rancher
    UID:                   625bd78c-819a-4ba5-8ed0-4e8cf0497860
  Resource Version:        23050
  Self Link:               /apis/cert-manager.io/v1alpha2/namespaces/cattle-system/certificates/tls-rancher-ingress
  UID:                     6e0886ec-8b2d-4459-8c60-f994f269a146
Spec:
  Dns Names:
    rancher.example.org
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       Issuer
    Name:       rancher
  Secret Name:  tls-rancher-ingress
Status:
  Conditions:
    Last Transition Time:  2020-05-07T23:07:41Z
    Message:               Waiting for CertificateRequest "tls-rancher-ingress-2753661366" to complete
    Reason:                InProgress
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  GeneratedKey  18m   cert-manager  Generated a new private key
  Normal  Requested     18m   cert-manager  Created new CertificateRequest resource "tls-rancher-ingress-2753661366"

I had some troubles with cert-manager where it wasn’t able to access http:// without that it is not able to generate the certificate.

kauri.io - Some Informations from there
howtoforge Galera - Install Galera