License: KRIB is APLv2
This document provides information on how to use the Digital Rebar KRIB content add-on. Use of this content will enable the operator to install Kubernetes in either a Live Boot (immutable infrastructure pattern) mode, or via installed to local hard disk OS mode.
KRIB uses the kubeadm cluster deployment methodology coupled with Digital Rebar enhancements to help proctor the cluster Master election and secrets management. With this content pack, you can install Kubernetes in a zero-touch manner.
KRIB does also support production, highly available (HA) deployments, with multiple masters. To enable this configuration, we've chosen to manage the TLS certificates and etcd installation in the Workflow instead of using the kubeadm process.
This document assumes you have your Digital Rebar Provisioning endpoint fully configured, tested, and working. We assume that you are able to properly provision Machines in your environment as a base level requirement for use of the KRIB content add-on use. See Installing Krib for step by step instructions.
documentation source: https://gitlab.com/rackn/provision-content/blob/master/krib/._Documentation.meta
KRIB Video References¶
The following videos have been produced or presented by RackN related to the Digital Rebar KRIB solution.
- KRIB Zero Config Kubernetes Cluster channel on YouTube.
- KubeCon: Zero Configuration Pattern on Bare Metal on YouTube - RackN presentation at 2017 KubeCon/Cloud NativeCon in Austin TX
KRIB uses community
kubeadm for installation. That process relies on internet connectivity to download containers and other components.
Immutable -vs- Local Install Mode¶
The two primary deployment patterns that the Digital Rebar KRIB content pack supports are:
- Live Boot (immutable infrastructure pattern)
- Local Install (standard install-to-disk pattern)
The Live Boot mode uses an in-memory Linux image based on the Digital Rebar Sledgehammer (CentOS based) image. After each reboot of the Machine, the node is reloaded with the in-memory live boot image. This enforces the concept of immutable infrastructure - every time a node is booted, deployed, or needs updating, simply reload the latest Live Boot image with appropriate fixes, patches, enhancements, etc.
The Local Install mode mimics the traditional "install-to-my-disk" method that most people are familiar with.
KRIB is a Content Pack addition to Digital Rebar Provision. It uses the Clusters which provides atomic guarantees. This allows for Kubernetes master(s) to be dynamically elected, forcing all other nodes to wait until the kubeadm on the elected master to generate an installation token for the rest of the nodes. Once the Kubernetes master is bootstrapped, the Digital Rebar system facilitates the security token hand-off to rest of the cluster so they can join without any operator intervention.
Elected -vs- Specified Master¶
By default, the KRIB process will dynamically elect a Master for the Kubernetes cluster. This masters simply win the race-to-master election process and the rest of the cluster will coalesce around the elected master.
If you wish to specify a specific machines to be the designated masters, you can do so by setting a Param in the cluster Profile to the specific Machine that will be come the master. To do so, set the
krib/cluster-masters Param to a JSON structure with the Name, UUID and IP of the machines to become masters. You may add this Param to the Profile in the below specifications, as follows:
The Kubernetes Master will be built on this Machine specified by the
This MUST be in the cluster profile because all machines in the cluster must be able to see this parameter.
KRIB is a Content Pack and is installed in the standard method as any other Contents. We need the
krib.json content pack to fully support KRIB and install the helper utility contents for stage changes.
Please review Installing Krib for step by step instructions.
KRIB uses the Certs plugin to build TLS, you can install that from the RackN library
Using the Command Line (
drpcli) utility configured to your endpoint, use this process:
In the UX, follow this process:
- Open your DRP Endpoint: (eg. https://127.0.0.1:8092/ )
- Authenticate to your Endpoint
- Login with your
RackN Portal Loginaccount (upper right)
- Go to the left panel "Content Packages" menu
Kubernetes (KRIB: Kubernetes Rebar Immutable Bootstrapping)from the right side panel (you may need to select Browser for more Content or use the Catalog button)
- Select the Transfer button for both content packs to add the content to your local Digital Rebar endpoint
The basic outline for configuring KRIB follows the below steps:
- create a Profile to hold the Params for the KRIB configuration (you can also clone the
- add a Param of name
krib/cluster-profileto the Profile you created
- add a Param of name
etcd/cluster-profileto the Profile you created
- apply the Profile to the Machines you are going to add to the KRIB cluster
- change the Workflow on the Machines to
krib-live-clusterfor memory booting or
krib-install-clusterto install to Centos. You may clone these reference workflows to build custom actions.
- installation will start as soon as the Workflow has been set.
There are many configuration options available, review the
etcd/* parameters to learn more.
Configure with the CLI¶
The configuration of the Cluster includes several reference Workflow that can be used for installation. Depending on which Workflow you use, will determine if the cluster is built via install-to-local-disk or via an immutable pattern (live boot in-memory boot process). Outside of the Workflow differences, all remaining configuration elements are the same.
You must writeable create a Profile from YAML (or JSON if you prefer) with the Params stagemap and param required information. Modify the Name or other fields as appropriate - be sure you rename all subsequent fields appropriately.
echo ' --- Name: "my-k8s-cluster" Description: "Kubernetes install-to-local-disk" Params: krib/cluster-profile: "my-k8s-cluster" etcd/cluster-profile: "my-k8s-cluster" Meta: color: "purple" icon: "ship" title: "My Installed Kubernetes Cluster" render: "krib" reset-keeps": "krib/cluster-profile,etcd/cluster-profile" ' > /tmp/krib-config.yaml drpcli profiles create - < /tmp/krib-config.yaml
The following commands should be applied to all of the Machines you wish to enroll in your KRIB cluster. Each Machine needs to be referenced by the Digital Rebar Machine UUID. This example shows how to collect the UUIDs, then you will need to assign them to the
UUIDS variable. We re-use this variable throughout the below documentation within the shell function named my_machines. We also show the correct
drpcli command that should be run for you by the helper function, for your reference.
Create our helper shell function my_machines
List your Machines to determine which to apply the Profile to
IF YOU WANT to make ALL Machines in your endpoint use KRIB, do:
Otherwise - individually add them to the UUIDS variable, like:
Add the Profile to your machines that will be enrolled in the cluster
Change stage on the Machines to initiate the Workflow transition. YOU MUST select the correct stage, dependent on your install type (Immutable/Live Boot mode or install-to-local-disk mode). For Live Boot mode, select the stage
ssh-access and for the install-to-local-disk mode select the stage
Configure with the UX¶
The below example outlines the process for the UX.
RackN assumes the use of CentOS 7 BootEnv during this process. However, it should theoretically work on most of the BootEnvs. We have not tested it, and your mileage will absolutely vary...
- create a Profile for the Kubernetes Cluster (e.g.
my-k8s-cluster) or clone the
- add a Param to that Profile:
- add a Param to that Profile:
- Add the Profile (eg
my-k8s-cluster) to all the machines you want in the cluster.
- Change workflow on all the machines to
krib-install-clusterfor install-to-local-disk, or to
krib-live-clusterfor the Live Boot/Immutable Kubernetes mode
Then wait for them to complete. You can watch the Stage transitions via the Bulk Actions panel (which requires RackN Portal authentication to view).
The reason the Immutable Kubernetes/Live Boot mode does not need a reboot is because they are already running Sledgehammer and will start installing upon the stage change.
Who is my Master?¶
If you have not specified who the Kubernetes Master should be; and the master was chosen by election - you will need to determine which Machine is the cluster Master.
kubectl - on Master¶
You can log in to the Master node as identified above, and execute
kubectl commands as follows:
kubectl - from anywhere¶
Once the Kubernetes cluster build has been completed, you may use the
kubectl command to both verify and manage the cluster. You will need to download the conf file with the appropriate tokens and information to connect to and authenticate your
kubectl connections. Below is an example of doing this:
Advanced Stages - Helm and Sonobuoy¶
KRIB includes stages for advanced Kubernetes operating support.
The reference workflows already install Helm using the
krib-helm stage. To leverage this utility simply define the required JSON syntax for your charts as shown in the
krib-helm stage documentation.
Sonobuoy can be used to validate that the cluster conforms to community specification. Adding the
krib-sonobuoy stage will start a test run. It can be rerun to collect the results or configured to wait for them. Storing test results in the
files path requires setting the
unsafe/password parameter and is undesirable for production clusters.
Ingress/Egress Traffic, Dashboard Access, Istio¶
The Kubernetes dashboard is enabled within a default KRIB built cluster. However no Ingress traffic rules are set up. As such, you must access services from external connections by making changes to Kubernetes, or via the Kubernetes Dashboard via Proxy.
These are all issues relating to managing, operating, and running a Kubernetes cluster, and not restrictions that are imposed by Digital Rebar Provision. Please see the appropriate Kubernetes documentation on questions regarding operating, running, and administering Kubernetes (https://kubernetes.io/docs/home/).
For Istio via Helm, please consult the
krib-helm stage documentation for a reference install.
Kubernetes Dashboard via Proxy¶
You can get the admin-user security token with the following command:
Now copy the token from the token part printed on-screen so you can paste it into the
Enter token field of the dashboard log in screen.
Once you have obtained the
admin.conf configuration file and security tokens, you may use
kubectl in Proxy mode to the Master. Simply open a separate terminal/console session to dedicate to the Proxy connection, and do:
Now, in a local web browser (on the same machine you executed the Proxy command) open the following URL:
MetalLB Load Balancer¶
If your cluster is running on bare metal you will most likely need a LoadBalancer provider. You can easily add this to your cluster by adding the
krib-metallb stage after the
krib-config stage in your workflow. Currently only L2 mode is supported. You will need to set the
metallb/l2-ip-range param in your profile with the range of IP's you wish to use. This ip range must not be within the configured DHCP scope. See the MetalLB docs for more information (https://metallb.universe.tf/tutorial/layer2/).
You can add nginx-ingress to your cluster by adding the
krib-ingress-nginx stage to your workflow. This stage requires helm and tiller to be installed so should come after the
krib-helm stage in your workflow.
This stage also requires a cloud provider LoadBalancer service or on bare metal you can add the
krib-metallb stage before this stage in your workflow.
This stage includes support for cert-manager if your profile is properly configured. See
Kubernetes Dashboard via NGINX Ingress¶
If your workflow includes the NGINX Ingress stage the kubernetes dashboard will be accessable via
https://k8s-db.LOADBALANCER_IP.xip.io. The access url and cert-manager tls can also be configured by setting the appropriate params in your profile. See
Please consult Kubernetes Dashboard via Proxy for information on getting the login token
Rook Ceph Manager Dashboard¶
If you install the rook via the
krib-helm chart template and have
krib-ingress-nginx stage in your workflow an ingress will be created so you can access the Ceph Manager Dashboard at
https://rook-db.LOADBALANCER_IP.xip.io. The access url and cert-manager tls can also be configured by setting the appropriate params in your profile. See
The default username is
admin and you can get the generated password with the with the following command:
It is absolutely possible to build multiple Kubernetes KRIB clusters with this process. The only difference is each cluster should have a unique name and profile assigned to it. A given Machine may only participate in a single Kubernetes cluster type at any one time. You can install and operate both Live Boot/Immutable with install-to-disk cluster types in the same DRP Endpoint.