K
Q

What does Kubernetes cronjobs `startingDeadlineSeconds` exactly mean?

June 27, 2018

In Kubernetes cronjobs, It is stated in the limitations section that

Jobs may fail to run if the CronJob controller is not running or broken for a span of time from before the start time of the CronJob to start time plus startingDeadlineSeconds, or if the span covers multiple start times and concurrencyPolicy does not allow concurrency.

What I understand from this is that, If the 

startingDeadlineSeconds
is set to 
10
and the cronjob couldn't start for some reason at its scheduled time, then it can still be attempted to start again as long as those 
10
seconds haven't passed, however, after the 
10
seconds, it for sure won't be started, is this correct?

Also, If I have 

concurrencyPolicy
set to 
Forbid
, does K8s count it as a fail if a cronjob tries to be scheduled, when there is one already running?

-- Hesham Massoud
kubernetes
kubernetes-cronjob

Similar Questions

Is it possible to run containers on android devices?
kubectl wait --for=condition=complete --timeout=30s
Kubernetes CNI vs Kube-proxy
Script to continuously follow kubectl get pods
kubectl list / delete all completed jobs

1 Answer

June 28, 2018

After investigating the code base of the Kubernetes repo, so this is how the CronJob controller works:

  1. The CronJob controller will check the every 10 seconds the list of cronjobs in the given Kubernetes Client.

  2. For every CronJob, it checks how many schedules it missed in the duration from the 

    lastScheduleTime
    till now. If there are more than 100 missed schedules, then it doesn't start the job and records the event:

    "FailedNeedsStart", "Cannot determine if job needs to be started. Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew."

It is important to note, that if the field 

startingDeadlineSeconds
is set (not 
nil
), it will count how many missed jobs occurred from the value of 
startingDeadlineSeconds
till now. For example, if 
startingDeadlineSeconds
= 
200
, It will count how many missed jobs occurred in the last 
200
seconds. The exact implementation of counting how many missed schedules can be found here.

  1. In case there are not more than a 100 missed schedules from the previous step, the CronJob controller will check if the time 

    now
    is not after the time of its 
    scheduledTime + startingDeadlineSeconds
    , i.e. that it's not too late to start the job (passed the deadline). If it wasn't too late, the job will continue to be attempted to be started by the CronJob Controller. However, If it is already too late, then it doesn't start the job and records the event:

    "Missed starting window for {cronjob name}. Missed scheduled time to start a job {scheduledTime}"

It is also important to note, that if the field 

startingDeadlineSeconds
is not set, then it means there is no deadline at all. This means the job will be attempted to start by the CronJob controller without checking if it's later or not.

Therefore to answer the questions above:

1. If the startingDeadlineSeconds is set to 10 and the cronjob couldn't start for some reason at its scheduled time, then it can still be attempted to start again as long as those 10 seconds haven't passed, however, after the 10 seconds, it for sure won't be started, is this correct?

The CronJob controller will attempt to start the job and it will be successfully scheduled if the 10 seconds after it's schedule time haven't passed yet. However, if the deadline has passed, it won't be started this run, and it will be counted as a missed schedule in later executions.

2. If I have concurrencyPolicy set to Forbid, does K8s count it as a fail if a cronjob tries to be scheduled, when there is one already running?

Yes, it will be counted as a missed schedule. Since missed schedules are calculated as I stated above in point 2.

-- Hesham Massoud
Source: StackOverflow