1. Petr Nejedly Avatar Petr Nejedly
  2. Serviio
  3. serviio
  4. Issues

ffmpeg not retried for cached items

Issue #727 resolved
Former user created an issue

See http://forum.serviio.org/viewtopic.php?f=22&t=14421

Comments (13)

  2. Petr Nejedly repo owner

    added new attribute to WebResourceItem that enables to mark certain items as volatile.

    boolean volatile – items marked as volatile are expected to be offline for periods of time, so that if Serviio fails to retrieve technical metadata from the URL it will try again later, rather than assuming the URL is invalid and ignoring the item.

  3. Former user Account Deleted

    I don't understand this solution. Any feed may be offline for periods of time for various unforeseen reasons, so all feeds are by definition volatile. I had expected the fix to be as simple as removing the item from the item cache if the ffmpeg to get metadata failed, so that on the next expiry it would be reextracted and re ffmpeged.

  5. Petr Nejedly repo owner

    You might get a bug in the plugin, or some additional security on the back end, of FFmpeg not supporting the file format and you'd end up running FFmpeg for the (invalid) URLs all the time. People will be complaining about CPU usage, etc. This has been there since the beginning, and the volatile flag will enable for some streams to be re-tried, when there is high possibility the feed comes up and down frequently and it makes sense to try to keep checking.

  6. Former user Account Deleted

    OK, I sense from your response that the failure to retry the ffmpeg has nothing to do with it having been item cached, and that standard behavior is and always has been that no stream (extracturl cachekey) will have ffmpeg retried once it has failed once.

    I do not recall ever having seen this happen, and I was under the impression that ffmpeg would always be rerun if its data was not present in the metadata cache for that extracturl cachekey. In fact, here is an example from the last 2 days, so my impression remains "volatile" is implicit and the reported ffmpeg failure is a bug tied to the item cache.

    2013-12-07 16:11:10,116 DEBUG [FeedUpdaterThread] Retrieving information about the video stream 'jsc1'
2013-12-07 16:11:10,116 DEBUG [FFMPEGWrapper] Invoking FFMPEG to retrieve media information for file: rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=4cb9829f81edb91d4568709224985a4f pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1
2013-12-07 16:11:10,117 DEBUG [ProcessExecutor] Starting C:\Program Files\Serviio\lib\ffdump.bat -analyzeduration 10000000 -user-agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.83 Safari/537.1 -i rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=4cb9829f81edb91d4568709224985a4f pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1
2013-12-07 16:11:11,279 DEBUG [AbstractCDSLibraryIndexingListener] Library updated, notifying CDS
2013-12-07 16:11:13,577 DEBUG [FeedUpdaterThread] Cannot get information about the URL, it might have expired already. Trying again.
2013-12-07 16:11:13,577 DEBUG [FeedItemUrlExtractor] IGoal: Starting extraction of url for item: jsc1
2013-12-07 16:11:14,027 DEBUG [FeedItemUrlExtractor] IGoal: call kbps
2013-12-07 16:11:14,789 DEBUG [FeedItemUrlExtractor] IGoal: Extracted Url for jsc1
2013-12-07 16:11:14,789 DEBUG [FeedItemUrlExtractor] IGoal: Finished extraction of url: ContentURLContainer [fileType=VIDEO, contentUrl=rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=f7ab11eafa4fe151986acd623cf0c8ea pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1, thumbnailUrl=http://i.imgur.com/yPh1I8Q.jpg, expiresImmediately=true, cacheKey=IGoal_jsc1, live=true, userAgent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.83 Safari/537.1]
2013-12-07 16:11:14,790 DEBUG [FFMPEGWrapper] Invoking FFMPEG to retrieve media information for file: rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=f7ab11eafa4fe151986acd623cf0c8ea pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1
2013-12-07 16:11:14,790 DEBUG [ProcessExecutor] Starting C:\Program Files\Serviio\lib\ffdump.bat -analyzeduration 10000000 -user-agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.83 Safari/537.1 -i rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=f7ab11eafa4fe151986acd623cf0c8ea pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1
2013-12-07 16:11:27,337 WARN  [FeedUpdaterThread] Failed to retrieve online item information for rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=f7ab11eafa4fe151986acd623cf0c8ea pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1. It might not play.
java.io.IOException: org.serviio.library.local.metadata.extractor.InvalidMediaFormatException: Unknown video file type.

2013-12-08 01:15:39,238 DEBUG [FeedUpdaterThread] Retrieving information about the video stream 'jsc1'
2013-12-08 01:15:39,238 DEBUG [FFMPEGWrapper] Invoking FFMPEG to retrieve media information for file: rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=7c9fae5becea0084aee5a543b59f766c pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1
2013-12-08 01:15:39,238 DEBUG [ProcessExecutor] Starting C:\Program Files\Serviio\lib\ffdump.bat -analyzeduration 10000000 -user-agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.83 Safari/537.1 -i rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=7c9fae5becea0084aee5a543b59f766c pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1
2013-12-08 01:15:40,088 DEBUG [AbstractCDSLibraryIndexingListener] Library updated, notifying CDS
2013-12-08 01:15:57,007 DEBUG [FeedUpdaterThread] Cannot get information about the URL, it might have expired already. Trying again.
2013-12-08 01:15:57,007 DEBUG [FeedItemUrlExtractor] IGoal: Starting extraction of url for item: jsc1
2013-12-08 01:15:57,460 DEBUG [FeedItemUrlExtractor] IGoal: call kbps
2013-12-08 01:15:58,190 DEBUG [FeedItemUrlExtractor] IGoal: Extracted Url for jsc1
2013-12-08 01:15:58,190 DEBUG [FeedItemUrlExtractor] IGoal: Finished extraction of url: ContentURLContainer [fileType=VIDEO, contentUrl=rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=8c335635b649657ca33afd49072837dd pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1, thumbnailUrl=http://i.imgur.com/yPh1I8Q.jpg, expiresImmediately=true, cacheKey=IGoal_jsc1, live=true, userAgent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.83 Safari/537.1]
2013-12-08 01:15:58,190 DEBUG [FFMPEGWrapper] Invoking FFMPEG to retrieve media information for file: rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=8c335635b649657ca33afd49072837dd pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1
2013-12-08 01:15:58,191 DEBUG [ProcessExecutor] Starting C:\Program Files\Serviio\lib\ffdump.bat -analyzeduration 10000000 -user-agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.83 Safari/537.1 -i rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=8c335635b649657ca33afd49072837dd pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1
2013-12-08 01:16:03,741 DEBUG [SearchManager] Committing search index
2013-12-08 01:16:06,727 WARN  [FeedUpdaterThread] Failed to retrieve online item information for rtmp://198.204.255.66:1935/load/ playpath=jsc001?Key=8c335635b649657ca33afd49072837dd pageUrl=http://www.kbps.tv/ swfUrl=http://www.kbps.tv//jl/player.swf live=1. It might not play.
java.io.IOException: org.serviio.library.local.metadata.extractor.InvalidMediaFormatException: Unknown video file type.
  8. Petr Nejedly repo owner

    Are you sure channel1's technical metadata in your example had not been extracted before and stored in the tech metadata cache with the cache key SkysportsPlus_ch1_ch1x ? If that is the case, FFmpeg would not run again and that cached technoical metadata would be used.

  9. Former user Account Deleted

    Yes, As I recollect I created the 1x cachekey to illustrate the problem for you so it was new, but the real proof is the fact that ffmpeg ran on the forced refresh which I think proves it was not previously cached.

  10. Petr Nejedly repo owner

    Refresh doesn't remove anything from the technical metadata cache, so the FFmpeg is not tried again for the same cacheKey of ContentURLContainer

  11. Former user Account Deleted

    Exactly, so the fact that it did run again on the forced refresh using 1x proves that 1x was not metadata cached when ffmpeg failed the first time due to an offline stream, so ffmpeg should have run again for 1x on the expiry refresh, which it did not, which is a bug, apparently due to the fact that the "item" was found to have been previously "item cached" ie: Item with key 'SkysportsPlus_ch11+19:00Knicks at Bobcats 21:00Nuggets at Suns ' already found in the cache, skipping URL extraction

  14. Log in to comment
Assignee
Type
bug
Priority
minor
Status
resolved
Component
server
Version
1.4
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!