effective-creation-timeout & new metrics for use by dependency-watchdog

[elankath]

What this PR does / why we need it:

Which issue(s) this PR fixes:
Fixes #1103
Fixes #1098

Special notes for your reviewer:

Release note:

Support node.machine.sapcloud.io/effective-creation-timeout on MachineDeployment to override shoot configured machineControllerManager.machineCreationTimeout for use by dependency-watchdog

Support new metrics: machine_create_duration_seconds, machine_initialize_duration_seconds, machine_join_duration_seconds,machine_drain_duration_seconds,machine_delete_duration_seconds

[gardener-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from elankath. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[elankath]

Test Log-1

Testing done with virtual provider. Real provider test logs will be attached to PR's raised for the specific providers.

Effective Machine Creation Timeout

1. Check MCD

k get mcd -n shoot--i034796--aw                                              
NAME                           READY   DESIRED   UP-TO-DATE   AVAILABLE   AGE
shoot--i034796--aw-fruit-z1            2         2                        4m5s
shoot--i034796--aw-weapon-z1           2         2                        4m5s

2. Add effective-creation-timeout annotation to MCD

k annotate mcd --overwrite shoot--i034796--aw-weapon-z1 -n shoot--i034796--aw node.machine.sapcloud.io/effective-creation-timeout="2m"
machinedeployment.machine.sapcloud.io/shoot--i034796--aw-weapon-z1 annotated

2. Check annotation propagated to MachineSet

k get mcs -n shoot--i034796--aw                                              git:main*
NAME                                 DESIRED   CURRENT   READY   AGE
shoot--i034796--aw-fruit-z1-79c97    2         2                 7m20s
shoot--i034796--aw-weapon-z1-c998d   2         2                 7m20s


k get mcs -n shoot--i034796--aw shoot--i034796--aw-weapon-z1-c998d -o custom-columns=NAME:.metadata.name,ANNOTATIONS:.metadata.annotations
NAME                                 ANNOTATIONS
shoot--i034796--aw-weapon-z1-c998d   map[bingo:lingo deployment.kubernetes.io/desired-replicas:2 deployment.kubernetes.io/max-replicas:3 deployment.kubernetes.io/revision:1 deployment.kubernetes.io/revision-history:1,1 greeting:howdy machine.sapcloud.io/last-deployment-replica-change-by-scaler-time:2026-05-28T14:13:17Z node.machine.sapcloud.io/effective-creation-timeout:2m] <-- Changed to 2m

2. Increase replicas & check annotation propagated to new Machine object

k scale mcd shoot--i034796--aw-weapon-z1 -n shoot--i034796--aw --replicas=3  
machinedeployment.machine.sapcloud.io/shoot--i034796--aw-weapon-z1 scaled

k get mc -n shoot--i034796--aw                                               
NAME                                       STATUS   AGE   NODE
shoot--i034796--aw-fruit-z1-79c97-4s9ln             30m
shoot--i034796--aw-fruit-z1-79c97-kgztg             30m
shoot--i034796--aw-weapon-z1-c998d-ksnqm            30m
shoot--i034796--aw-weapon-z1-c998d-mntb7            30m
shoot--i034796--aw-weapon-z1-c998d-p8png            9s


k get mc -n shoot--i034796--aw shoot--i034796--aw-weapon-z1-c998d-p8png -o custom-columns=NAME:.metadata.name,ANNOTATIONS:.metadata.annotations
NAME                                       ANNOTATIONS
shoot--i034796--aw-weapon-z1-c998d-p8png   map[bingo:lingo deployment.kubernetes.io/desired-replicas:3 deployment.kubernetes.io/max-replicas:4 deployment.kubernetes.io/revision:1 deployment.kubernetes.io/revision-history:1,1 greeting:howdy machine.sapcloud.io/last-deployment-replica-change-by-scaler-time:2026-05-28T14:13:17Z node.machine.sapcloud.io/effective-creation-timeout:2m] <-- New Machine object has 2m

3. Case: Creation Success Metrics

Machines should be created and new Node joins the cluster. init, create metrics available.

k get no shoot--i034796--aw-weapon-z1-c998d-p8png                            
NAME                                       STATUS   ROLES    AGE   VERSION
shoot--i034796--aw-weapon-z1-c998d-p8png   Ready    <none>   13m

From local prometheus http://127.0.0.1:9090/prometheus
NOTE: Virtual provider randomizes create and join durations.

I0610 11:26:15.480417   13619 machine.go:193] reconcileClusterMachine: Start for "shoot--i034796--aw-weapon-z1-c998d-p8png" with phase:"Running", description:"Machine shoot--i034796--aw-weapon-z1-c998d-p8png successfully joined the cluster in 5m13.183556s"


|mcm_machine_machine_create_duration_seconds{instance="localhost:10259", job="am_1", machine_deployment="shoot--i034796--aw-fruit-z1", namespace="shoot--i034796--aw"}|214|

|mcm_machine_machine_create_duration_seconds{instance="localhost:10259", job="am_1", machine_deployment="shoot--i034796--aw-weapon-z1", namespace="shoot--i034796--aw"}|307|

mcm_machine_machine_join_duration_seconds{instance="localhost:10259", job="am_1", machine_deployment="shoot--i034796--aw-fruit-z1", namespace="shoot--i034796--aw"} |214|

mcm_machine_machine_join_duration_seconds{instance="localhost:10259", job="am_1", machine_deployment="shoot--i034796--aw-weapon-z1", namespace="shoot--i034796--aw"}|313|

4. Case: Machine Join Failure

• Before state: weapon MCD replica count = 1
• Increased virtual provider InstanceDelays.JoinMin to 130s above the effective-creation-timeout of 2m

• Increased weapon MCD replica count to 2.

k get mcd -nshoot--i034796--aw                                               
NAME                           READY   DESIRED   UP-TO-DATE   AVAILABLE   AGE
shoot--i034796--aw-fruit-z1    2       2         2            2           3h25m
shoot--i034796--aw-weapon-z1   1       1         1            1           3h25m


k scale mcd shoot--i034796--aw-weapon-z1 -n shoot--i034796--aw --replicas=2 
machinedeployment.machine.sapcloud.io/shoot--i034796--aw-weapon-z1 scaled

k get mcd -nshoot--i034796--aw; k get no -n shoot--i034796--aw              
NAME                           READY   DESIRED   UP-TO-DATE   AVAILABLE   AGE
shoot--i034796--aw-fruit-z1    2       2         2            2           3h27m
shoot--i034796--aw-weapon-z1   1       2         2            1           3h27m <-- replice inc to 2
NAME                                       STATUS     ROLES    AGE    VERSION
shoot--i034796--aw-fruit-z1-79c97-4s9ln    Ready      <none>   172m
shoot--i034796--aw-fruit-z1-79c97-kgztg    Ready      <none>   172m
shoot--i034796--aw-weapon-z1-c998d-4xk78   NotReady   <none>   12s <-- will NOT join within 2m
shoot--i034796--aw-weapon-z1-c998d-m2wz9   Ready      <none>   46m

Machine object 4xk78 is Terminated after effective-creation-timeout of 2m and new object xs276 is created. This behaviour will keep repeating. num_failed_join is incremented.

k get mc -nshoot--i034796--aw; k get no -n shoot--i034796--aw                git:main*
NAME                                       STATUS        AGE     NODE
shoot--i034796--aw-fruit-z1-79c97-4s9ln    Running       3h30m   shoot--i034796--aw-fruit-z1-79c97-4s9ln
shoot--i034796--aw-fruit-z1-79c97-kgztg    Running       3h30m   shoot--i034796--aw-fruit-z1-79c97-kgztg
shoot--i034796--aw-weapon-z1-c998d-4xk78   Terminating   2m31s   shoot--i034796--aw-weapon-z1-c998d-4xk78
shoot--i034796--aw-weapon-z1-c998d-m2wz9   Running       48m     shoot--i034796--aw-weapon-z1-c998d-m2wz9
shoot--i034796--aw-weapon-z1-c998d-xs276   Pending       15s     shoot--i034796--aw-weapon-z1-c998d-xs276
NAME                                       STATUS                        ROLES    AGE     VERSION
shoot--i034796--aw-fruit-z1-79c97-4s9ln    Ready                         <none>   174m
shoot--i034796--aw-fruit-z1-79c97-kgztg    Ready                         <none>   174m
shoot--i034796--aw-weapon-z1-c998d-4xk78   NotReady,SchedulingDisabled   <none>   2m30s
shoot--i034796--aw-weapon-z1-c998d-m2wz9   Ready                         <none>   48m
shoot--i034796--aw-weapon-z1-c998d-xs276   NotReady                      <none>   12s

E0610 14:10:02.395176   26065 machine_util.go:1142] Machine shoot--i034796--aw-weapon-z1-c998d-4xk78 failed to join the cluster in 2m0s minutes.
I0610 14:10:02.395230   26065 metrics.go:240] incremented num_failed_join metric due to "shoot--i034796--aw-weapon-z1-c998d-4xk78" with labels instance_type=m5.xlarge,machine_deployment=shoot--i034796--aw-weapon-z1,namespace=shoot--i034796--aw,zone=eu-west-1c

From Prometheus

|mcm_machine_num_failed_join{instance="localhost:10259", instance_type="m5.xlarge", job="am_1", machine_deployment="shoot--i034796--aw-weapon-z1", namespace="shoot--i034796--aw", zone="eu-west-1c"}|1|

[aaronfern]

/assign

[aaronfern]

Just a few minor commits from me. In general lgtm once the comments are resolved.

[aaronfern]

Consider writing an faq for this value.
You could title it such as "How to override a MachineDeployment.Spec.Template.Spec.MachineCreationTimeout?" and over there you can explain any limitations too.

Could be helpful for ops.

[aaronfern]

Seems like copied by mistake

[aaronfern]

Would it be possible to add some unit tests for this function?

[takoverflow]

Can this be renamed to GetEffectiveMachineCreationTimeout instead i.e. removing the FromRuntimeObject suffix; since that's evident from the function signature.