Max file size supported is 4MB - should support bigger files

[Alistair Spark]

During testing I noticed this error when running the queue:

Scheduled task failed: Prepare submissions for annotation (assignfeedback_editpdf\task\convert_submissions),BadRequest: Maximum request length exceeded.

It looks like the standard upload endpoint will only accept files up to 4MB

To upload bigger files, chunking is required via a different method:

https://docs.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0

As far as I can see that isn't the method this plugin is using.

[Alistair Spark]

It looks like that would recommended chunking would be 10MiB per chunk but could go up to 60 MiB.

Maximum file size which can be uploaded would then be limited by Onedrvie's maximum file size - this was increased to 10GB in 2014 (from 2GB). It may be something the plugin should check before starting an upload - is the filesize < onedrivemaxfilesize, if not then skip and let the next available converter deal with it.

[Neill Magill]

I have a change that will allow it to handle files of the same sizes as the OneDrive repository bundled with Moodle.

You are a reviewer on the ticket in case you want to give the change a once over.

[Alistair Spark]

Sorry missed this, I'll take a look now.

N.B the Onedrive max file size was increased to 15GB in December - https://support.office.com/en-gb/article/invalid-file-names-and-file-types-in-onedrive-onedrive-for-business-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa#individualfilesize

[Neill Magill]

Hopefully the change to 60 MiB is now enough for almost all assignment submissions. It is larger than Turnitin will allow for a submission (40 MB)

I with this expect this change would work for the vast majority of submission files now based on the following stats for compatible assignment submissions from our database:

ext|max|avg|stddev|count ---|----|----|------|-----| csv|9.6760|0.14601180|0.8200534182296352|887 doc|220.6487|0.39701390|2.1274845961762407|72171 docx|237.0699|0.50417144|2.5943786984651087|278863 odp|154.8056|5.59631605|17.26660408830398|91 ods|3.0612|0.10442915|0.4339255588148428|49 odt|44.6467|0.31895776|1.8645290097819835|2027 pot|6.6284|6.62841797|0|1 potx|1.7786|0.78065753|0.6499081117781884|4 pps|15.2065|4.45910645|6.23645246282029|4 ppsx|202.2536|18.00394304|41.77425717247629|40 ppt|135.8564|3.11974670|8.355542531082532|1799 pptm|211.2054|7.14998634|24.101326194961064|166 pptx|248.4034|5.09679027|14.321054345083693|20607 rtf|244.7602|3.04516263|19.152617784893028|465 xls|36.5981|0.40865216|1.633680580801678|2749 xlsx|91.2512|0.16590794|1.0580432261950814|24551

Using:

SELECT 
    SUBSTR(filename, 1-LOCATE('.', REVERSE(filename))) AS ext, 
    MAX(filesize) / (1024*1024) AS max, 
    avg(filesize) / (1024*1024) as avg, 
    stddev(filesize) / (1024*1024) as stddev, 
    count(*) 
FROM mdl_files
WHERE 
    component = 'assignsubmission_file' 
    AND filearea = 'submission_files' 
    AND filename <> '.' 
    AND LOCATE('.', REVERSE(filename)) > 0 
    AND SUBSTR(filename, 1-LOCATE('.', REVERSE(filename))) IN ('csv', 'doc', 'docx', 'odp', 'ods', 'odt', 'pot', 'potm', 'potx', 'pps', 'ppsx', 'ppsxm', 'ppt', 'pptm', 'pptx','rtf', 'xls', 'xlsx')
GROUP BY SUBSTR(filename, 1-LOCATE('.', REVERSE(filename)));

In total 332 of 404477 files are over 60 MiB.

[Neill Magill]

Is the change as it currently is enough for you?

I expect I will be able to look at implementing chunking at some point in the future if it is not, or we would be happy to look at a pull request for it should you need it faster?

[Alistair Spark]

Hi Neill, Yes that will be fine for now, thanks a lot for sorting that out so quickly. Much appreciated.

Chunking would be great, will see if we can get a PR done for it.

[Neill Magill]

I have put the new version into the Moodle plugins directory

[Segun Babalola]

Hi,

I’ve implemented chunked transfers for this converter plugin to allow it support larger file conversions. Please review https://bitbucket.org/isprynt/moodle-onedrive-converter and merge if acceptable.

All feedback welcome.

Best,

Segun

[Neill Magill]

Thanks @segunbabalola ,

I plan to take a look at it either later today or early next week.

Could you submit a pull request with me as the reviewer?

[Segun Babalola]

PR submitted

[Neill Magill]

@segunbabalola thanks. I have taken a look and made a couple of comments on the request.

It looks good, there are just a couple of minor issues with it. Let us know if you will resolve them.

[Neill Magill]

The change has been merged and passed our automatic tests overnight.

Thanks for your contribution @segunbabalola