Cron job scheduler

[Alan Noble]

As part of the process of porting functionality from NetReceiver to VidGrind, it is desirable to migrate cron job functionality from the former.

Rather than using GAE’s cron.yaml, is proposed to integrate cron job scheduling into VidGrind to enable a tighter integration.

Cron jobs are represented by the ‘Cron’ datastore type (already implemented).

[Saxon Milton]

As discussed from meeting, it would be useful if this could also offer scheduling of more custom actions. Maybe something that would allow scheduling of a callback function? The scheduling that will be required for use with the youtube api will be more complex than adjusting variable values for example.

[Alan Noble]

This could be achieved with the addition of a call action to iotds.Cron. Cron.Var could then store the name of the function to be called (using reflect.Call) and Cron.Data its string parameters.

[kortschak]

It would be helpful here if there were a (perhaps approximate) mapping between the ds Cron records and the standard format cron spec (https://en.wikipedia.org/wiki/Cron). This doesn't need to have an implementation — I'm happy to write that — but at least a behaviour spec.

[Alan Noble]

By all means proceed.

Also, it is not imperative that we remain with the current record spec. We can migrate if necessary.

[kortschak]

I just need to confirm the meanings of the fields. This is what I glean from reading the python.

In the Go iotds.Cron type:

• Time/TOD refer to single event per day crons, where Time is for numerical time and TOD for symbolic time (these are both simple, but here for completeness). In this interpretation the time.Time's *time.Location is time.Local and the date part of the time.Time is not relevant.
• Repeat/Minutes refer to repeating jobs where Repeat is true for these and Minutes is the time between repetitions. It's not entirely clear to me how these should interact with Time/TOD.
• Action is what to do: the set of actions are "set", "del", "email" and "sms".
• Var/Data are for variable setting (or delete).

The remaining fields are obvious.

Is this correct? The reference to "since start of UTC" in the Minutes field docs is unclear to me; in the code it just fires on essentially time%minutes == 0. Is this comment because the clock is in UTC?

[kortschak]

This is my interpretation of the current information:

// Spec returns the cron rendered as a standard cron spec line for the given
// geographic location.
func (c *Cron) Spec(lat, lon float64) string {
    if !c.Enabled {
        return ""
    }
    if c.Repeat {
        // This is not exactly the meaning of the repeat, but it is
        // close enough for our purposes.
        return fmt.Sprintf("@every %v", time.Duration(c.Minutes)*time.Minute)
    }
    if c.TOD != "" {
        switch tod := strings.ToLower(c.TOD); tod {
        case "sunrise", "noon", "sunset":
            return fmt.Sprintf("@%s %v %v", tod, lat, lon)
        default:
            // Midnight is not solar time, but rather just 0:00.
            return "@" + tod
        }
    }
    return fmt.Sprintf("%d %d * * *", c.Time.Minute(), c.Time.Hour())
}

[Alan Noble]

That looks correct.

[kortschak]

It's not quite right (the minutes field does not mean what I thought), looking at the help.html doc in netreceiver. Also, there is something there that's not possible to express exactly the same way; "Crons may also use a repeating time, represented by asterisk () followed by 1 to 4 digits. For example, 30 means repeat every 30 minutes, and *800 means repeat every 8 hours (not 800 minutes)."

The issue arises because a cron spec cannot say, for example every 5 hours and 30 minutes. There is a way to do this with robfigs cron, but the start time is independent of timezone, rather just being from the initialisation of the cron job. Do we care about that? In the first instance I'm going to say no and send a PR on that basis.

[kortschak]

Is there currently a way to obtain all the sites that a running instance of vidgrind could be aware of? (I don't see one). This is needed so that on start-up the cron scheduler can ensure that all the cron jobs are scheduled. On edit, it's easy to hook in schedule updates, but obviously it should not be needed to edit a cron job on the /set/crons/edit end point in order to make it run.

[Saxon Milton]

I don’t think there is, but we could add a function to the iotsvc/iotds/site.go file:

func GetAllSites(ctx context.Context, store Store) ([]Site, error) {
  q := store.NewQuery(typeSite, false)
  var sites []Site
  _, err := store.GetAll(ctx, q, &sites)
  return sites, err 
}

(I think this is what it would look like)

‌

[kortschak]

Cool. Out of curiosity, why are the GetAllX functions storing into []T rather than []*T?

[Saxon Milton]

I don’t think there would be a particular reason, but Alan would know if there was.

[Alan Noble]

GAE’s datastore.Client.GetAll() takes []T , so we kept that interface.

[Alan Noble]

Re the earlier comment, we’ve decided not to implement repeating crons, right?

[kortschak]

We did, but I figured out how to get them for free. So they still exist.

[Alan Noble]

In that case, by all means change the implementation of repeat crons to comply with the spec.

[kortschak]

The current master already does. The pending PR is just to wire that in.

[Saxon Milton]

Resolved by PR #138, PR #136, PR #133, PR #131