bonnyThe Elixir based Kubernetes Development Framework
Bonny: Kubernetes Development Framework
Extend the Kubernetes API with Elixir.
Bonny make it easy to create Kubernetes Operators, Controllers, and Custom Schedulers.
If Kubernetes CRDs and controllers are new to you, read up on the terminology.
Tutorials and Examples:
Important! These tutorials are for an older version of Bonny, but the add/1, modify/1, and delete/1 APIs are the same, as well as a new reconcile/1 function. Additionally a k8s has been added!
Feel free to message me on twitter if you need any help!
- HelloOperator Tutorial Part: 1 2 3
- HelloOperator source code
Talks
- Commandeering Kubernetes @ The Big Elixir 2019
Example Operators built with Bonny
- Eviction Operator - Bonny v 0.4
- Hello Operator - Bonny v 0.4
- Todo Operator - Bonny v 0.4
Installation
Bonny can be installed by adding bonny to your list of dependencies in mix.exs:
def deps do
[
{:bonny, "~> 0.4"}
]
end
Configuration
Bonny uses the k8s client under the hood.
The only configuration parameters required are :bonny controllers and a :k8s cluster:
config :k8s,
clusters: %{
default: %{ # `default` here must match `cluster_name` below
conn: "~/.kube/config"
}
}
config :bonny,
# Add each CRD Controller module for this operator to load here
# Defaults to none. This *must* be set.
controllers: [
MyApp.Controllers.V1.WebServer,
MyApp.Controllers.V1.Database,
MyApp.Controllers.V1.Memcached
],
# K8s.Cluster to use, defaults to :default
cluster_name: :default,
# The namespace to watch for Namespaced CRDs.
# Defaults to "default". `:all` for all namespaces
# Also configurable via environment variable `BONNY_POD_NAMESPACE`
namespace: "default",
# Set the Kubernetes API group for this operator.
# This can be overwritten using the @group attribute of a controller
group: "your-operator.example.com",
# Name must only consist of only lowercase letters and hyphens.
# Defaults to hyphenated mix app name
operator_name: "your-operator",
# Name must only consist of only lowercase letters and hyphens.
# Defaults to hyphenated mix app name
service_account_name: "your-operator",
# Labels to apply to the operator's resources.
labels: %{
"kewl": "true"
},
# Operator deployment resources. These are the defaults.
resources: %{
limits: %{cpu: "200m", memory: "200Mi"},
requests: %{cpu: "200m", memory: "200Mi"}
}
When configuring bonny to run in your cluster the mix bonny.gen.manifest command will generate a service account for you. To use that service account configure the k8s library like the following:
config :k8s,
clusters: %{
default: %{}
}
This will add a cluster named default. When no configuration information is provided, the k8s library will use the service account of the pod.
Running outside of a cluster
Running an operator outside of Kubernetes is not recommended for production use, but can be very useful when testing.
To start your operator and connect it to an existing cluster, one must first:
- Have configured your operator. The above example is a good place to start.
- Have some way of connecting to your cluster. The most common is to connect using your kubeconfig as in the example:
# config.exs
config :k8s,
clusters: %{
default: %{
conn: "~/.kube/config"
}
}
- If RBAC is enabled, you must have permissions for creating and modifying
CustomResourceDefinition,ClusterRole,ClusterRoleBindingandServiceAccount. - Generate a manifest
mix bonny.gen.manifestand install it using kubectlkubectl apply -f manifest.yaml
Now you are ready to run your operator
iex -S mix
Bonny Generators
There are a number of generators to help create a Kubernetes operator.
mix help | grep bonny
Generating an operator controller
An operator can have multiple controllers. Each controller handles the lifecycle of a custom resource.
By default controllers are generated in the V1 version scope.
mix bonny.gen.controller Widget widgets
You can specify the version flag to create a new version of a controller. Bonny will dispatch the controller for the given version. So old versions of resources can live alongside new versions.
mix bonny.gen.controller Widget widgets --version v2alpha1
Note: The one restriction with versions is that they will be camelized into a module name.
Open up your controller and add functionality for your resource's lifecycles:
- Add
- Modify
- Delete
- Reconcile; periodically called with each every instance of a CRD's resources
Each controller can create multiple resources.
For example, a todo app controller could deploy a Deployment and a Service.
Your operator can also have multiple controllers if you want to support multiple resources in your operator!
Check out the two test controllers:
Generating a dockerfile
The following command will generate a dockerfile for your operator. This will need to be pushed to a docker repository that your Kubernetes cluster can access.
Again, this Dockerfile is for your operator, not for the pods your operator may deploy.
You can skip this step when developing by running your operator external to the cluster.
mix bonny.gen.dockerfile
export BONNY_IMAGE=YOUR_IMAGE_NAME_HERE
docker build -t ${BONNY_IMAGE} .
docker push ${BONNY_IMAGE}:latest
Generating Kubernetes manifest for operator
This will generate the entire manifest for this operator including:
- CRD manifests
- RBAC
- Service Account
- Operator Deployment
The operator manifest generator requires the image flag to be passed if you plan to deploy the operator in your cluster. This is the docker image URL of your operators docker image created by mix bonny.gen.docker above.
mix bonny.gen.manifest --image ${BONNY_IMAGE}
You may omit the --image flag if you want to generate a manifest without the deployment so that you can develop locally running the operator outside of the cluster.
Note: YAML output is JSON formatted YAML. Sorry, elixirland isn't fond of YAML :D
By default the manifest will generate the service account and deployment in the "default" namespace.
To set the namespace explicitly:
mix bonny.gen.manifest --out - -n test
Alternatively you can apply it directly to kubectl:
mix bonny.gen.manifest --out - -n test | kubectl apply -f - -n test
Generating a resource
TODO: Need to support validation / OpenAPI.
Telemetry
Bonny uses the telemetry and notion library to emit event metrics.
Events: Bonny.Sys.Event.events()
[
[:bonny, :scheduler, :binding, :failed],
[:bonny, :scheduler, :binding, :succeeded],
[:bonny, :scheduler, :nodes, :fetch, :failed],
[:bonny, :scheduler, :nodes, :fetch, :succeeded],
[:bonny, :scheduler, :pods, :fetch, :failed],
[:bonny, :scheduler, :pods, :fetch, :succeeded],
[:bonny, :reconciler, :genserver, :down],
[:bonny, :reconciler, :reconcile, :failed],
[:bonny, :reconciler, :reconcile, :succeeded],
[:bonny, :reconciler, :run, :started],
[:bonny, :reconciler, :fetch, :failed],
[:bonny, :reconciler, :fetch, :succeeded],
[:bonny, :reconciler, :initialized],
[:bonny, :watcher, :genserver, :down],
[:bonny, :watcher, :chunk, :received],
[:bonny, :watcher, :watch, :timedout],
[:bonny, :watcher, :watch, :failed],
[:bonny, :watcher, :watch, :finished],
[:bonny, :watcher, :watch, :succeeded],
[:bonny, :watcher, :watch, :started],
[:bonny, :watcher, :initialized]
]
Terminology
A custom resource is an extension of the Kubernetes API that is not necessarily available on every Kubernetes cluster. In other words, it represents a customization of a particular Kubernetes installation.
CRD Custom Resource Definition:
The CustomResourceDefinition API resource allows you to define custom resources. Defining a CRD object creates a new custom resource with a name and schema that you specify. The Kubernetes API serves and handles the storage of your custom resource.
A custom controller is a controller that users can deploy and update on a running cluster, independently of the cluster’s own lifecycle. Custom controllers can work with any kind of resource, but they are especially effective when combined with custom resources. The Operator pattern is one example of such a combination. It allows developers to encode domain knowledge for specific applications into an extension of the Kubernetes API.
A set of application specific controllers deployed on Kubernetes and managed via kubectl and the Kubernetes API.
Testing
mix test