error calling "repeatBroadcast" at site=492735809745

[Alan Noble]

Is there some kind of config issue with the Willunga site? The other sites seem to be calling repeatBroadcast just fine.

Log message below:

2023-08-22T23:30:00.695819Z 2023/08/22 23:30:00 cron: error calling "repeatBroadcast" at site=492735809745 with "{\"SKey\":492735809745,\"Name\":\"Willunga_Tcam\",\"ID\":\"2csj6KricO0\",\"SID\":\"\",\"CID\":\"\",\"StreamName\":\"Willunga_Tcam\",\"Description\":\"Testing permanent broadcast function\",\"Privacy\":\"Unlisted\",\"Resolution\":\"1080p\",\"StartTime\":\"2023-08-18T09:00\",\"StartTimeUnix\":\"1692315000\",\"Start\":\"2023-08-17T23:30:00Z\",\"EndTime\":\"2023-08-18T17:30\",\"EndTimeUnix\":\"1692345600\",\"End\":\"2023-08-18T08:00:00Z\",\"VidforwardHost\":\"35.201.16.14:8080\",\"DeviceName\":\"Willunga_Tcam\",\"OnActions\":\"Willunga_Tcam.mode=Normal,ESP_4869E7.Power1=true\",\"OffActions\":\"Willunga_Tcam.mode=Paused,ESP_4869E7.Power1=false\",\"RTMPVar\":\"Willunga_Tcam.RTMPURL\",\"ScheduleTime\":\"9:00\",\"Active\":false,\"Slate\":false,\"LastStatusCheck\":\"0001-01-01T00:00:00Z\",\"LastChatMsg\":\"0001-01-01T00:00:00Z\",\"LastHealthCheck\":\"0001-01-01T00:00:00Z\",\"Issues\":0,\"SendMsg\":false,\"SensorList\":[{\"SendMsg\":false,\"Sensor\":{\"SKey\":492735809745,\"SID\":\"AirTemperature\",\"Pin\":\"ESP_0D2895.X50\",\"Quantity\":\"MWH\",\"Func\":\"linear\",\"Args\":\"0.1,-273.15\",\"Scale\":1,\"Offset\":0,\"Units\":\"C\",\"Format\":\"round2\"},\"Name\":\"airtemperature\",\"DeviceMac\":154664809867413},{\"SendMsg\":false,\"Sensor\":{\"SKey\":492735809745,\"SID\":\"AnalogValue\",\"Pin\":\"ESP_0D2895.X10\",\"Quantity\":\"OTH\",\"Func\":\"none\",\"Args\":\"\",\"Scale\":1,\"Offset\":0,\"Units\":\"\",\"Format\":\"round2\"},\"Name\":\"analogvalue\",\"DeviceMac\":154664809867413},{\"SendMsg\":false,\"Sensor\":{\"SKey\":492735809745,\"SID\":\"BatteryVoltage\",\"Pin\":\"ESP_0D2895.A0\",\"Quantity\":\"DCV\",\"Func\":\"scale\",\"Args\":\"0.0281\",\"Scale\":1,\"Offset\":0,\"Units\":\"V\",\"Format\":\"round1\"},\"Name\":\"batteryvoltage\",\"DeviceMac\":154664809867413},{\"SendMsg\":false,\"Sensor\":{\"SKey\":492735809745,\"SID\":\"Humidity\",\"Pin\":\"ESP_0D2895.X51\",\"Quantity\":\"MMB\",\"Func\":\"scale\",\"Args\":\"0.1\",\"Scale\":1,\"Offset\":0,\"Units\":\"%\",\"Format\":\"round2\"},\"Name\":\"humidity\",\"DeviceMac\":154664809867413},{\"SendMsg\":false,\"Sensor\":{\"SKey\":492735809745,\"SID\":\"AValue\",\"Pin\":\"ESP_198942.X10\",\"Quantity\":\"OTH\",\"Func\":\"none\",\"Args\":\"\",\"Scale\":1,\"Offset\":0,\"Units\":\"\",\"Format\":\"round2\"},\"Name\":\"avalue\",\"DeviceMac\":102046260365634},{\"SendMsg\":false,\"Sensor\":{\"SKey\":492735809745,\"SID\":\"AirTemperature2\",\"Pin\":…
  {
    "textPayload": "2023/08/22 23:30:00 cron: error calling \"repeatBroadcast\" at site=492735809745 with \"{\\\"SKey\\\":492735809745,\\\"Name\\\":\\\"Willunga_Tcam\\\",\\\"ID\\\":\\\"2csj6KricO0\\\",\\\"SID\\\":\\\"\\\",\\\"CID\\\":\\\"\\\",\\\"StreamName\\\":\\\"Willunga_Tcam\\\",\\\"Description\\\":\\\"Testing permanent broadcast function\\\",\\\"Privacy\\\":\\\"Unlisted\\\",\\\"Resolution\\\":\\\"1080p\\\",\\\"StartTime\\\":\\\"2023-08-18T09:00\\\",\\\"StartTimeUnix\\\":\\\"1692315000\\\",\\\"Start\\\":\\\"2023-08-17T23:30:00Z\\\",\\\"EndTime\\\":\\\"2023-08-18T17:30\\\",\\\"EndTimeUnix\\\":\\\"1692345600\\\",\\\"End\\\":\\\"2023-08-18T08:00:00Z\\\",\\\"VidforwardHost\\\":\\\"35.201.16.14:8080\\\",\\\"DeviceName\\\":\\\"Willunga_Tcam\\\",\\\"OnActions\\\":\\\"Willunga_Tcam.mode=Normal,ESP_4869E7.Power1=true\\\",\\\"OffActions\\\":\\\"Willunga_Tcam.mode=Paused,ESP_4869E7.Power1=false\\\",\\\"RTMPVar\\\":\\\"Willunga_Tcam.RTMPURL\\\",\\\"ScheduleTime\\\":\\\"9:00\\\",\\\"Active\\\":false,\\\"Slate\\\":false,\\\"LastStatusCheck\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"LastChatMsg\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"LastHealthCheck\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"Issues\\\":0,\\\"SendMsg\\\":false,\\\"SensorList\\\":[{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"AirTemperature\\\",\\\"Pin\\\":\\\"ESP_0D2895.X50\\\",\\\"Quantity\\\":\\\"MWH\\\",\\\"Func\\\":\\\"linear\\\",\\\"Args\\\":\\\"0.1,-273.15\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"C\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"airtemperature\\\",\\\"DeviceMac\\\":154664809867413},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"AnalogValue\\\",\\\"Pin\\\":\\\"ESP_0D2895.X10\\\",\\\"Quantity\\\":\\\"OTH\\\",\\\"Func\\\":\\\"none\\\",\\\"Args\\\":\\\"\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"analogvalue\\\",\\\"DeviceMac\\\":154664809867413},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"BatteryVoltage\\\",\\\"Pin\\\":\\\"ESP_0D2895.A0\\\",\\\"Quantity\\\":\\\"DCV\\\",\\\"Func\\\":\\\"scale\\\",\\\"Args\\\":\\\"0.0281\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"V\\\",\\\"Format\\\":\\\"round1\\\"},\\\"Name\\\":\\\"batteryvoltage\\\",\\\"DeviceMac\\\":154664809867413},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"Humidity\\\",\\\"Pin\\\":\\\"ESP_0D2895.X51\\\",\\\"Quantity\\\":\\\"MMB\\\",\\\"Func\\\":\\\"scale\\\",\\\"Args\\\":\\\"0.1\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"%\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"humidity\\\",\\\"DeviceMac\\\":154664809867413},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"AValue\\\",\\\"Pin\\\":\\\"ESP_198942.X10\\\",\\\"Quantity\\\":\\\"OTH\\\",\\\"Func\\\":\\\"none\\\",\\\"Args\\\":\\\"\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"avalue\\\",\\\"DeviceMac\\\":102046260365634},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"AirTemperature2\\\",\\\"Pin\\\":\\\"ESP_198942.X50\\\",\\\"Quantity\\\":\\\"MWH\\\",\\\"Func\\\":\\\"linear\\\",\\\"Args\\\":\\\"0.1,-273.15\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"C\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"airtemperature2\\\",\\\"DeviceMac\\\":102046260365634},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"Humidity2\\\",\\\"Pin\\\":\\\"ESP_198942.X51\\\",\\\"Quantity\\\":\\\"MMB\\\",\\\"Func\\\":\\\"scale\\\",\\\"Args\\\":\\\"0.1\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"%\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"humidity2\\\",\\\"DeviceMac\\\":102046260365634},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"Voltage\\\",\\\"Pin\\\":\\\"ESP_198942.A0\\\",\\\"Quantity\\\":\\\"DCV\\\",\\\"Func\\\":\\\"scale\\\",\\\"Args\\\":\\\"0.0294\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"V\\\",\\\"Format\\\":\\\"round1\\\"},\\\"Name\\\":\\\"voltage\\\",\\\"DeviceMac\\\":102046260365634},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"AnalogValue09\\\",\\\"Pin\\\":\\\"ESP_466B09.X10\\\",\\\"Quantity\\\":\\\"OTH\\\",\\\"Func\\\":\\\"none\\\",\\\"Args\\\":\\\"\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"analogvalue09\\\",\\\"DeviceMac\\\":172194353539849},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"BatteryVoltage09\\\",\\\"Pin\\\":\\\"ESP_466B09.A0\\\",\\\"Quantity\\\":\\\"DCV\\\",\\\"Func\\\":\\\"scale\\\",\\\"Args\\\":\\\"0.0293224\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"V\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"batteryvoltage09\\\",\\\"DeviceMac\\\":172194353539849},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"Humidity09\\\",\\\"Pin\\\":\\\"ESP_466B09.X51\\\",\\\"Quantity\\\":\\\"MMB\\\",\\\"Func\\\":\\\"scale\\\",\\\"Args\\\":\\\"0.1\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"%\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"humidity09\\\",\\\"DeviceMac\\\":172194353539849},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"Temperature09\\\",\\\"Pin\\\":\\\"ESP_466B09.X50\\\",\\\"Quantity\\\":\\\"MWH\\\",\\\"Func\\\":\\\"linear\\\",\\\"Args\\\":\\\"0.1,-273.15\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"C\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"temperature09\\\",\\\"DeviceMac\\\":172194353539849},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"Water_Temperature09\\\",\\\"Pin\\\":\\\"ESP_466B09.X60\\\",\\\"Quantity\\\":\\\"MTW\\\",\\\"Func\\\":\\\"linear\\\",\\\"Args\\\":\\\"0.1,-273.15\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"C\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"water_temperature09\\\",\\\"DeviceMac\\\":172194353539849},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"AirTemperatureE7\\\",\\\"Pin\\\":\\\"ESP_4869E7.X50\\\",\\\"Quantity\\\":\\\"MWH\\\",\\\"Func\\\":\\\"linear\\\",\\\"Args\\\":\\\"0.1,-273.15\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"C\\\",\\\"Format\\\":\\\"round1\\\"},\\\"Name\\\":\\\"airtemperaturee7\\\",\\\"DeviceMac\\\":207804699273703},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"AnalogValueE7\\\",\\\"Pin\\\":\\\"ESP_4869E7.X10\\\",\\\"Quantity\\\":\\\"OTH\\\",\\\"Func\\\":\\\"none\\\",\\\"Args\\\":\\\"\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"\\\",\\\"Format\\\":\\\"\\\"},\\\"Name\\\":\\\"analogvaluee7\\\",\\\"DeviceMac\\\":207804699273703},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"BatteryVoltageE7\\\",\\\"Pin\\\":\\\"ESP_4869E7.A0\\\",\\\"Quantity\\\":\\\"DCV\\\",\\\"Func\\\":\\\"scale\\\",\\\"Args\\\":\\\"0.0289\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"V\\\",\\\"Format\\\":\\\"round1\\\"},\\\"Name\\\":\\\"batteryvoltagee7\\\",\\\"DeviceMac\\\":207804699273703},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"HumidityE7\\\",\\\"Pin\\\":\\\"ESP_4869E7.X51\\\",\\\"Quantity\\\":\\\"MMB\\\",\\\"Func\\\":\\\"scale\\\",\\\"Args\\\":\\\"0.1\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"%\\\",\\\"Format\\\":\\\"round2\\\"},\\\"Name\\\":\\\"humiditye7\\\",\\\"DeviceMac\\\":207804699273703},{\\\"SendMsg\\\":false,\\\"Sensor\\\":{\\\"SKey\\\":492735809745,\\\"SID\\\":\\\"WaterTemperatureE7\\\",\\\"Pin\\\":\\\"ESP_4869E7.X60\\\",\\\"Quantity\\\":\\\"MTW\\\",\\\"Func\\\":\\\"linear\\\",\\\"Args\\\":\\\"0.1,-273.15\\\",\\\"Scale\\\":1,\\\"Offset\\\":0,\\\"Units\\\":\\\"C\\\",\\\"Format\\\":\\\"round1\\\"},\\\"Name\\\":\\\"watertemperaturee7\\\",\\\"DeviceMac\\\":207804699273703}],\\\"RTMPKey\\\":\\\"9eg5-828q-daw6-bzhw-9w9c\\\",\\\"UsingVidforward\\\":false,\\\"CamOn\\\":\\\"35 10 * * *\\\",\\\"CamOff\\\":\\\"01 17 * * *\\\",\\\"CheckingHealth\\\":true,\\\"AttemptingToStart\\\":false,\\\"ListingSecondaries\\\":false}\": could not set up broadcast: could not broadcast stream: could not get service: could not get youtube credentials token: could not load token: could not get reader for object: could not get object: error getting bucket named: ausocean: googleapi: Error 403: [e-mail] does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden resp: {0 map[]}",
    "insertId": "64e544f8000a9e0bc9d7b923",
    "resource": {
      "type": "gae_app",
      "labels": {
        "zone": "australia-southeast1-2",
        "version_id": "1",
        "project_id": "oceancron",
        "module_id": "default"
      }
    },
    "timestamp": "2023-08-22T23:30:00.695819Z",
    "labels": {
      "clone_id": "00c61b117cc18a8f74d8c7e126b3be240dc81d13dec661f6a01d990e5202d4b6f59aee06081875f940a1558a931e8ea7bd484464d66368e3a94fdc"
    },
    "logName": "projects/oceancron/logs/stderr",
    "receiveTimestamp": "2023-08-22T23:30:01.005174386Z"
  }

[Trek Hopton]

Yeah, I’m seeing this for stony point too. I think the important part is:

could not set up broadcast: could not broadcast stream: could not get service: could not get youtube credentials token: could not load token: could not get reader for object: could not get object: error getting bucket named: ausocean: googleapi: Error 403: [e-mail] does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist

Not sure why this is happening yet.

[Alan Noble]

Okay, so the OceanCron service does not have access to YouTube. I thought adding YOUTUBE_SECRETS to app.yaml would suffice, but evidently not.

I'm guessing YT also needs to grant access to the OceanCron service account?

Saxon?

Alan

[Trek Hopton]

Looking in the bucket, [e-mail] has access but I noticed that the credentials were updated last night, I’m wondering if that has anything to do with it.

[Trek Hopton]

Apparently the credentials auto refresh, no one has touched them recently. I started the broadcasts fine on VidGrind so for some reason OceanCron is having access issues despite the permissions in the bucket being set to ‘Storage Object Viewer’ like VidGrind.

[Trek Hopton]

Looks like the issue is that broadcast/storage.go line 62 is trying to get the bucket “ausocean” but OceanCron doesn’t have storage.buckets.get permission.

See the roles here: https://cloud.google.com/storage/docs/access-control/iam-roles

Storage Object Viewer only has storage.objects.get permissions.

We could give OceanCron Storage Insight Collector Service role which has storage.buckets.get.

[Alan Noble]

The Storage Object Viewer role is sufficient to view bucket objects, which should be sufficient. In fact, OceanCron utilizes this role in order to access datastore credentials, so that much is clearly working.

[Trek Hopton]

Yeah, I think so too. Perhaps the issue is how the code for YouTube is checking if we have access to the bucket metadata with a call to bkt.Attrs() .

bkt := c.Bucket(bktName)
_, err = bkt.Attrs(ctx)
if err != nil {
    return nil, fmt.Errorf("error getting bucket named: %s: %w", bktName, err)
}

obj := bkt.Object(objName)
_, err = obj.Attrs(ctx)
if err != nil {
    return obj, fmt.Errorf("error getting object named: %s: %w", objName, err)
}

The code for other secrets is the following:

bkt := clt.Bucket(url[:sep])
obj := bkt.Object(url[sep+1:])
r, err := obj.NewReader(ctx)        
if err != nil {
    return m, fmt.Errorf("cannot create GSB reader: %w", err)
}

‌

[Alan Noble]

I don’t think that’s the problem. Storage Object Viewer sufficient to call the Bucket.Attrs function (which is just reading info).

[Trek Hopton]

Resolved in PR #289