This is the final article about my Kube Log Exporter series. The first article introduced the basic design and how to execute the log exporter from your machine that runs the kubectl commands. The second article explained the necessary ServiceAccounts, ClusterRole and ClusterRoleBinding resources that we need to run the log exporter inside the cluster. And now in this article I explain how to define a cron job that runs the Kube Log Exporter automatically and regularly scheduled.
CronJob is in its basic specification similar to a
ReplicaSet: It defines a set of containers that run, and it defines its run conditions. These conditions are what makes a
CronJob a unique Kubernetes resource. Let's briefly discuss the most important ones:
scheduleTime- Uses the Linux crontab syntax to precisly define the exact, day, hour, minute and second when the job should run.
completions- The number of successful completions that need to be achieved before this cron job is considered successful. Potential use cases are jobs that optimize storage in a database, or clean up files and which may have a different set of success criteria.
parallelism- Controls if the the cron job can run in multiple parallel jobs or only sequentially
activeDeadlineSeconds- the maximum timespan in which the batch job needs to be finished. If it reaches this limit, the Kublet scheduler stops the cron job and considers it failed
There are many more options available, so take a look at the official Kubernetes documentation as well.
The KubeLogExporter cron job that I'm using needs to fulfil the following requirements:
- It needs to run every hour
- It needs to be successful, e.g. no error when reading or storing a log occurs
- It needs the use the service account that we discussed in the last article to have the proper access rights to namespaces, pods and logs
- It needs to use a persistent volume to store the log files, so that independent of the actual node where the job runs the same log files are amended
Let’s develop the
CronJob resource definition bit by bit. The first part fulfils the scheduling requirements. The schedule is defines at
spec.schedule to run each hour. The job will need to be completed exactly one time (
spec.jobTemplate.spec.completions) and will restart in case of an error (
apiVersion: batch/v1beta1 kind: CronJob metadata: name: kube-log-exporter-cron spec: schedule: "0 * * * *" jobTemplate: spec: completions: 1 template: spec: restartPolicy: OnError ...
The declaration of the
ServiceAccount is very simple: We add
apiVersion: batch/v1beta1 kind: CronJob metadata: name: kube-log-exporter-cron spec: schedule: "0 * * * *" jobTemplate: spec: completions: 1 template: spec: restartPolicy: OnError serviceAccountName: log-exporter-sa ...
Now we need to add the persistent volume declaration. The volume needs to be mounted only by one process, and I reserve 1Gi size for it.
kind: PersistentVolumeClaim metadata: name: log-exporter-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
In the Kubernetes distribution of my choice, K3S, persistent volumes are create automatically when the
PersistentVolumeClaim is defined. In another distribution you also need the setup the
PersistentVolume, but this is not in the focus of this article.
Now we use the
PersistentVolumeClaim to define a volume in
spec.jobTemplate.spec.volumes, and then reference this volume as a mounted volume in the container at
apiVersion: batch/v1beta1 kind: CronJob metadata: name: kube-log-exporter-cron spec: schedule: '0 * * * *' jobTemplate: spec: completions: 1 template: spec: serviceAccountName: log-exporter-sa containers: - name: kube-log-exporter image: docker.admantium.com/kube-log-exporter:0.1.9.12 volumeMounts: - name: log-exporter-volume mountPath: /etc/kube-log-exporter/logs restartPolicy: Never volumes: - name: log-exporter-volume persistentVolumeClaim: claimName: log-exporter-pvc
Executing the CronJob
Now we create the cron job with
kubectl create -f kube-log-exporter-cron-job.yaml. Once the job runs (for testing purposes you can also run the job every minute with
schedule: /1 ** **), we can see the job history.
kb describe cronjob kube-log-exporter 130 ↵ Name: kube-log-exporter-cron Namespace: default Labels: <none> Annotations: Schedule: 0 * * * * ... Last Schedule Time: Mon, 25 Aug 2020 19:00:00 +0200 Active Jobs: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 12m cronjob-controller Created job kube-log-exporter-cron-1590426000 Normal SawCompletedJob 12m cronjob-controller Saw completed job: kube-log-exporter-cron-1590426000, status: Complete Normal SuccessfulDelete 12m cronjob-controller Deleted job kube-log-exporter-cron-1590415200
And here is an example of the created logfiles.
ls -la /etc/kube-log-exporter/logs
-rw-r--r-- 1 root root 4515 Aug 25 19:00 lighthouse-78cc7475c7-74ctt_lighthouse.log -rw-r--r-- 1 root root 6012 Aug 25 19:00 lighthouse-78cc7475c7-gcl94_lighthouse.log -rw-r--r-- 1 root root 6873 Aug 25 19:00 lighthouse-78cc7475c7-k2cv7_lighthouse.log -rw-r--r-- 1 root root 7634 Aug 25 19:00 lighthouse-78cc7475c7-l7zpv_lighthouse.log -rw-r--r-- 1 root root 4636 Aug 25 19:00 lighthouse-78cc7475c7-wh2gk_lighthouse.log -rw-r--r-- 1 root root 25741 Aug 25 19:00 redis-6b746f4d9b-8tjds_redis.log ....
````sh > cat /etc/kube-log-exporter/logs/redis-6b746f4d9b-8tjds_redis.log 1:C 25 Aug 2020 16:21:04.675 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 1:C 25 Aug 2020 16:21:04.675 # Redis version=6.0.1, bits=64, commit=00000000, modified=0, pid=1, just started 1:C 25 Aug 2020 16:21:04.675 # Configuration loaded _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 6.0.1 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in standalone mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 7139 | `-._ `._ / _.-' | PID: 1 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' | `-._`-._ _.-'_.-' | `-._ `-.__.-' _.-' `-._ _.-' `-.__.-' 1:M 25 Aug 2020 16:21:04.678 # Server initialized
CronJob: Complete Resource Defintion
Here is the complete version again.
apiVersion: batch/v1beta1 kind: CronJob metadata: name: kube-log-exporter-cron spec: schedule: '0 * * * *' jobTemplate: spec: completions: 1 template: spec: serviceAccountName: log-exporter-sa containers: - name: kube-log-exporter image: docker.admantium.com/kube-log-exporter:0.1.9.12 args: ['node', 'cluster.js'] volumeMounts: - name: log-exporter-volume mountPath: /etc/kube-log-exporter/logs restartPolicy: Never volumes: - name: log-exporter-volume persistentVolumeClaim: claimName: log-exporter-pvc imagePullSecrets: - name: registry-secret
CronJob define periodic scheduled tasks in your cluster. Typical use cases are maintenance tasks such as leaning files, updating index or collect data. If you want to store log data in plain files, then using a cron job is a straightforward solution. This article showed how to define a cron job that uses the KubeLogExplorer to persists Pod log data in files as a persistent volume.