Low speed NAS optimisation

[Richard Hauswald]

Hi there, I like subtler a lot since I discovered the options to change default audio processing :)

I have a huge library of MKVs I want to convert to mp4/m4v. When I am converting directly on my network drive (mounted) the process takes longer than local of course. But manually copying the files from my NAS to my MacBook (16GB RAM, SSD), convert them using subler and copy the converted result back to my NAS is faster than using subler directly on the mounted network drive. I assume this is due to parallel READ/WRITE requests to the NAS HDDs. MY HDDs are awfully slow with random READ/WRITE/parallel access - but they provide acceptable performance when it comes to continuous READ/WRITE.

Processing files like READ 100MB, convert and then WRITE the result would speed up the conversion on slow drives IMHO. The conversion could be done in parallel to the READ to speed up things further. But There should be no parallel READ and WRITE.

Any thoughts on this?

Cheers, Richard

PS: I chose major since it was the default priority and IMHO my feature request shouldn't be treated as minor :)

[Richard Hauswald]

I did some performance testing to provide evidence for my believes:

• Local to Local: 15 seconds
• Local to Local second time: 3 seconds (file system cache hits)
• Local to NAS(since the device read from is a different one than the device written to there is no parallel READ/Write happening): 32 second
• Local to NAS(since the device read from is a different one than the device written to there is no parallel READ/Write happening) second time: 30 seconds (file system cache hits - even though 2 seconds don't really make a difference when saving to the NAS - but still....)
• NAS to NAS: 302 seconds
• NAS to NAS second time: 32 seconds (file system cache hits and prevents the disk from parallel READ/WRITE/random access )

Preventing parallel READ/WRITE comes with a performance gain of factor 10 on non SSD drives and is still noticeable on SSD drives even though they don't spin their heads around :). Most decent RDBMS (postgres does for sure) build up a small write buffer in memory before writing to the HDD to reduce parallel READ/WRITE as much as possible to take advantage of this performance gain. Maybe subler could try as well ?

[Emory Lundberg]

Another approach would be to tune, change, or alter your method of mounting your remote filesystem. CIFS/AFP/NFS each have different options for improving performance and altering behavior for sync/async read/write operations that would be very relevant to your use case.

e.g. my media filesystems are presented via NFSv3, so I enable mount options for rsize/wsize in addition to allow_async et al on the clients (and other options on the server) precisely to address this sort of thing.

OS X has a filesystem cache in kernel memory that also gets involved, so doing anything that puts you at odds with that is probably a bad idea. I'd look at tuning your network filesystem protocols before asking other people to work around your network filesystem configuration.

[Richard Hauswald]

Emory,

Since you suggested that this is only related to my networking options I did another performance test. I used a different mkv than in my last performance test. Thats why the Local to Local time is a bit different this time. I was also clearing the file system cache in between these runs using 'sudo purge'. Here are my results:

• Local to Local: 20 seconds
• USB drive(Samsung Extreme USB 3.0) to Local: 7.5 seconds

IMHO these numbers indicate that the issue also exists when there is no networking involved.

Thanks for the pointer to rsize/wsize. I'll give that one a try as a workaround,

Richard