Extract Items Timeout Action

[Former user]

If a web-resource groovy fails during "extract Items", Serviio immediately repeats the execution (refreshes) rather than process the items extracted to that point if any, and refreshing at the default or set expiry time.

As a result, if a webresource site changes its format, or goes out of service to be replaced by a dummy page, as has happened with hahasport, anyone with the groovy resourceUrl enabled will experience a continuing execution of the groovy.

Similarly, if the groovy times out, after extracting a number of items, the groovy will immediately and repetitively execute again, tying up the system, rather than process the items so extracted.

Serviio should process any items found and only refresh the groovy at the default or item expiry time.

[Former user]

That logic is ok if the resource is actually down or the link format has changed. But if the page simply times-out or the internet goes down for a moment, one would have to wait till the next item expiry update for the items to show up, or would have to refresh the feed manually. Why not retrying for 2-3 times over a period of 2-3 minutes and only then wait till the next item expiry update?

[Petr Nejedly]

When the error happens, an exception is thrown and I don't have any items available up to that time to me. So the only thing I can do is to fail the whole resource.

I could to what mqojdn suggests and try up to 3 times and if it still fails don't try again until the next refresh time.

Is that going to improve the experience?

[Former user]

If items were found before the exception they should extract and create a menu, but it should schedule 2 more immediate refreshes and then schedule a refresh as provided by the extracted items if any or the console and quit.

If no items were found then it should just schedule 2 more immediate refreshes and then schedule a refresh as provided by the console and quit.

I think I asked before if a default refresh time for the plug in can be set in extract items to override the console value if its earlier, and remove the dependence on "expires on" in Extract URL. Currently my plugins allow a refresh time to be set but I have to pass this thru to add to one or more items in the ContentURLContainer because there is no plugin wide value. Something like: return new WebResourceContainer(title: pageTitle, thumbnailUrl: pageThumb, items: items, expiresOn: expirydate) That would allow control over the later refresh when no items are found.

A good example recently is that Billboard20 went black for 6 hours twice over the past 2 days. and kept refreshing. This fix would have tried 3 times and then refreshed per the console, but with an expireson value for the plugin I could have set it to a more frequent value.

[Petr Nejedly]

As I said, it's not possible to process the feed partially. Once the exception is throws the caller doesn't get any of the data processes up to the point.

So I can only retry 2 times after that, but there will not be any items available from the previous partially successful run.

[Former user]

OK. Try twice and then fail until next refresh time, assuming that we can give the feed a refresh time to be used when no item urls with expiry date have been extracted.

[Petr Nejedly]

It would just re-try again once the feed expires (ie in this case based on the expiry time in the console)

[Former user]

Since the refresh is a an attribute of the feed and not the items, I don't understand why expires on cannot be be specified for the "Web Resource Container", to override the console value if sooner, and be overridden by any sooner item. Do I need to request this as an enhancement? We should be able to retry every half hour or so for these intermittent failures not have to wait a day for a retry.

[Petr Nejedly]

Looking at the logs you provided in beta forum, it seems to e the error is happening in the extractItems plugin method. Can you confirm? If that's the case there is no way to get the expiry time for the whole container, because no container is returned (only an exception is thrown).

If it's happening in extractUrl it should already skip the item as it is now, right?

Also, if the default 24 hours is too long for retry, the should be able to refresh the feed by right-clicking on it in the console and forcing refresh.

[Former user]

So it would appear that the only solution that meets my spec is to prevent exceptions by doing my own checks within the groovy, like checking for page availability, detecting too long an execution before it times out etc, and then processing the items to that point or generate a dummy item with a reasonable retry expireson.

Will the try twice and then wait til console timeout be in the next beta?

Thanks

[Petr Nejedly]

Yes it will.