OceanCron shut down despite min_idle_instances = 1

[Trek Hopton]

It has been observed that OceanCron shutdown unexpectedly on 08/09/2023.

The configuration is this:

automatic_scaling:
  min_idle_instances: 1
  max_idle_instances: 1
  min_instances: 1
  max_instances: 1

I have compiled a timeline to help debugging. See sheet here: https://docs.google.com/spreadsheets/d/1V633n_pJuXlxsnUKmQ-7p-ot0SfzsNIWBNpypDC83B8/edit#gid=0

8/9/2023
00:30	OceanCron shutdown	[pid1] Received terminated signal [session:HVLTMLG]
		[pid1] shutdown [session:HVLTMLG]
08:50	No crons ran in OceanCron
09:00	No broadcasts were setup in VidGrind
09:10	Streams were observed by Cath to have not started via repeat broadcast
12:55	Cron set triggered OceanCron session to start	/cron/set/492735809745/Willunga_Tcam
		[pid1] started [session:GHS6MJ0]

It is unknown what caused this shutdown.

[Alan Noble]

Can you include the App Engine logs too?

[Trek Hopton]

These are the Logs from the shutdown and restart:

INFO 2023-09-07T15:00:21.725049Z [pid1] Received terminated signal [session:HVLTMLG]
INFO 2023-09-07T15:00:21.725393Z [pid1] Sending terminated signal to processes [session:HVLTMLG]
INFO 2023-09-07T15:00:21.725669Z [pid1] Waiting 2s to allow shutdown processing [session:HVLTMLG]
INFO 2023-09-07T15:00:21.725677Z [pid1-nginx] Propagating terminated signal to nginx (pid 21) [session:HVLTMLG]
INFO 2023-09-07T15:00:21.725943Z [pid1-app] Propagating terminated signal to app (pid 11) [session:HVLTMLG]
WARNING 2023-09-07T15:00:21.732880Z [pid1-app] Exit -1 app (pid 11): /layers/google.go.build/bin/main [session:HVLTMLG]
INFO 2023-09-07T15:00:21.753675Z [pid1-nginx] Exit 0 nginx (pid 21): /usr/sbin/nginx -c /tmp/nginxconf-912421153/nginx.conf [session:HVLTMLG]
INFO 2023-09-07T15:00:23.726849Z [pid1] Stopping all processes [session:HVLTMLG]
INFO 2023-09-07T15:00:23.727153Z [pid1] Done [session:HVLTMLG]
INFO 2023-09-07T15:00:23.727165Z [pid1] shutdown [session:HVLTMLG]
INFO 2023-09-08T03:25:30.419985Z [protoPayload.method: GET] [protoPayload.status: 200] [protoPayload.responseSize: 240 B] [protoPayload.latency: 31.481 s] [protoPayload.userAgent: Go-http-client 2.0] /cron/set/492735809745/Willunga_Tcam
INFO 2023-09-08T03:25:30.572091Z [pid1] started [session:GHS6MJ0]
DEBUG 2023-09-08T03:25:30.573007Z [pid1-app] Using app start info from /srv/.googleconfig/app_start.json: &main.appStart{Entrypoint:struct { Type string "json:\"type\""; UnparsedValue string "json:\"unparsed_value\""; Command string "json:\"command\""; WorkDir string "json:\"workdir\"" }{Type:"Generated", UnparsedValue:"", Command:"main", WorkDir:""}, EntrypointFromAppYAML:"", EntrypointContents:"", Runtime:"go118"} [session:GHS6MJ0]
INFO 2023-09-08T03:25:30.573306Z [pid1] Starting processes [app nginx] [session:GHS6MJ0]
INFO 2023-09-08T03:25:30.573742Z [pid1-app] app has no prerequisites, starting immediately [session:GHS6MJ0]
INFO 2023-09-08T03:25:30.573955Z [pid1-nginx] nginx waiting for any of 4 prerequisite(s): [portbind:tcp:127.0.0.1:8081 portbind:tcp:localhost:8080 portbind:tcp:localhost:8081 portbind:unix:/tmp/google-config/app.sock] [session:GHS6MJ0]
INFO 2023-09-08T03:25:30.658873Z [pid1-app] Starting app (pid 11): /layers/google.go.build/bin/main [session:GHS6MJ0]
DEFAULT 2023-09-08T03:25:31.049355Z 2023/09/08 03:25:31 Running in App Engine mode
DEFAULT 2023-09-08T03:25:31.789548Z 2023/09/08 03:25:31 cron mode

‌

[Trek Hopton]

I wonder if we need to enable warmup requests.

The documentation says that they are required for min_instances and min_idle_instances to work properly.

https://cloud.google.com/appengine/docs/standard/reference/app-yaml?tab=python#scaling_elements

This is how we enable them.

https://cloud.google.com/appengine/docs/standard/configuring-warmup-requests?tab=go

I think we should add the following to our cron-app.yaml:

inbound_services:
- warmup

‌

[Alan Noble]

Yes, makes sense.

We also need to implement the actual handler though, i.e., handle the /_ah/warmup request.

[Trek Hopton]

I’m not sure we do need to implement the handler unless we want to do extra warmup tasks:

With warmup requests enabled, App Engine issues GET requests to /_ah/warmup. You can implement handlers for this request to perform application-specific tasks, such as pre-caching application data.

I think the request itself would warm up the instance even if we don’t have a handler. Might be worth making an empty one for now though just in case we do ever want to perform warmup tasks.

[Alan Noble]

Yes, I was thinking of just making it empty for now. Taking a look right now.

[Trek Hopton]

Okay I’ll leave it to you

[Alan Noble]

Implemented in https://bitbucket.org/ausocean/vidgrind/commits/fa42284d076ee6be353b1f79543af05c3312232e