Github Actions Runner pods gets deleted as soon as it comes up

[meetatmoe]

I've configured Github Action controller using official guide.

When the job is triggered, runner pod gets created but as soon as pod gets initialized, it gets deleted.
Here are the error logs before pod gets deleted.

runner √ Connected to GitHub
runner [RUNNER 2024-05-02 08:56:30Z INFO Terminal] WRITE LINE: 
runner 
runner [RUNNER 2024-05-02 08:56:31Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
runner [RUNNER 2024-05-02 08:56:31Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
runner [RUNNER 2024-05-02 08:56:31Z ERR  GitHubActionsService] POST request to https://pipelinesghubeus9.actions.githubusercontent.com/REDACTED/_apis/oauth2/token failed. HTTP Status: BadRequest
runner [RUNNER 2024-05-02 08:56:31Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
runner [RUNNER 2024-05-02 08:56:31Z ERR  MessageListener] Catch exception during create session.
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener] GitHub.Services.OAuth.VssOAuthTokenRequestException: Registration a87a388f-7975-4edd-ae39-51647b21f3e0 was not found.
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.OAuth.VssOAuthTokenProvider.OnGetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.Common.IssuedTokenProvider.GetTokenOperation.GetTokenAsync(VssTraceActivity traceActivity)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.Common.IssuedTokenProvider.GetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
runner [RUNNER 2024-05-02 08:56:32Z ERR  MessageListener]    at GitHub.Runner.Listener.MessageListener.CreateSessionAsync(CancellationToken token)
runner [RUNNER 2024-05-02 08:56:32Z ERR  Terminal] WRITE ERROR: Failed to create a session. The runner registration has been deleted from the server, please re-configure. Runner registrations are automatically deleted for runners that have not connected to the service recently.
runner Failed to create a session. The runner registration has been deleted from the server, please re-configure. Runner registrations are automatically deleted for runners that have not connected to the service recently.
runner [RUNNER 2024-05-02 08:56:32Z INFO Listener] Runner execution has finished with return code 1
runner Runner listener exit with terminated error, stop the service, no retry needed.
runner Exiting runner...
dind time="2024-05-02T08:56:32.030184965Z" level=info msg="Processing signal 'terminated'"
dind time="2024-05-02T08:56:32.031672419Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
dind time="2024-05-02T08:56:32.037650221Z" level=info msg="Daemon shutdown complete"
dind time="2024-05-02T08:56:32.037712487Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
dind time="2024-05-02T08:56:32.037779240Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby

[snimesh17]

facing exact same issue.

any solution you found?

[pulkitanz]

I am still seeing this happening across a lot of our pods very intermittently. Another issue is that once the pod is deleted then the Ephemeral Runner Resource is also not cleaned up.

[rp-martinbarmaymon]

Has anyone found a fix or workaround for this? We started experiencing this a few weeks ago and the issue is intermittent. Some days we have no errors, and on other days, we see errors in 20-30% of our pods. These errors are concentrated in short bursts, all occurring within a few minutes.

[andrey-petrenko-develeap]

How do you deal with it anyway when you encounter it?
For us last time the only thing that seemed to help is force reinstalling the scale set helm chart