Add option to automatically fail/reject groups that did not submit anything for first deadline

[Andreas Lööw]

Every time I start grading after a deadline, I see partial or empty submissions with comments like the following:

did not finish in time

and

Hi! We were not able to finish the compiler. This due to time spent on (just) understanding Haskell...

and

This is not yet a complete solution.

What I have to do then is to say, "Ok, rejecting, please resubmit when you are done" and then press reject.

This is because there are often two guaranteed markings: One if you submit before the first deadline, and one after you have addressed the things said in a potential first rejection.

The reason students submit broken/partial/empty things is because they cannot submit anything after the first deadline if the haven't submitted anything before -- without asking for an extension, that is.

This could be automated, by simply rejecting all groups that didn't submit anything for the first deadline with a template message like "You did not submit anything for the first deadline, so now there's just one grading left." or something like that. Or one could simply allow students to submit after the first deadline... Just something that automates away this pointless clicking of rejecting partial/empty submissions.

Possibly related to #72.

[Andreas Lööw]

Ping @dschoepe and @bafain: This is relevant for e.g. the plt course -- any comments?

[Daniel Schoepe]

I think this should be integrated with the automated checks feature, so one can also auto-reject submissions that fail a given test suite. Since empty/partial solutions will also fail the tests, this should cover both this case and solutions that fail tests without a corresponding comment that the solution is incomplete.

I guess a general approach would be to extend the automated checks infrastructure so that the test script can also cause the submission to be automatically rejected in addition to just leaving a comment.

[Andreas Lööw]

@dschoepe: Yes, for courses with a working auto checker that would be optimal. But many courses, such as e.g. plt, do not have such a thing. I.e., integrating this with the auto checker would only help courses with an auto checker (but in cases where such a thing exists, having it integrated would be better as one would then also check that e.g. submitted makefiles work). So even if the auto checker would be extended in this way, some courses would still benefit from what is suggested in this issue (i.e., something simpler, not integrated with any auto checker).

[Evgeny Kotelnikov]

This could be automated, by simply rejecting all groups that didn't submit anything for the first deadline with a template message like "You did not submit anything for the first deadline, so now there's just one grading left."

What do you mean by "one grading left"? Fire does not enforce a limit on the number of resubmissions.

Or one could simply allow students to submit after the first deadline...

For that you just need to not have the first deadline set up at all.

I think this should be integrated with the automated checks feature, so one can also auto-reject submissions that fail a given test suite.

Yes, I am planning to add the "reject on failure" and "accept on success" options to the automatic checks.

In my opinion the culprit of this problem is Chalmers's culture of lenient grading. Students never face repercussions for missing deadlines, and most courses do not restrict the number of resubmissions. Really, submission deadlines are a mean of encouragement and not a mean of control. As a result, strict enforcement of deadlines becomes an obstacle for when once again a grader has to soften the rules because this is the norm.

The reason students submit broken/partial/empty things is because they cannot submit anything after the first deadline if the haven't submitted anything before -- without asking for an extension, that is.

I think that students should ask for a deadline extension if they missed the deadline. I am not keen to add more loopholes to the submission system which allow students to bypass the course restrictions.

[Andreas Lööw]

What do you mean by "one grading left"? Fire does not enforce a limit on the number of resubmissions.

Yes, I was referring to the Fire-external course concept of having a fix number of guaranteed gradings, where missing the first deadline uses up one of these gradings.

For that you just need to not have the first deadline set up at all.

But there is a first deadline, it's just that the Fire deadline semantics do not coincide with the course deadline semantics (i.e., you should still be able to submit something after the deadline). (Having the deadline, and it's countdown, clearly visible in the UI is helpful/useful information for the students I think.)

I think that students should ask for a deadline extension if they missed the deadline. I am not keen to add more loopholes to the submission system which allow students to bypass the course restrictions.

It would be more flexible to let each course responsable define what a deadline means, rather than having a fixed concept in Fire that all courses must adhere to. But I agree with what you say to some extent, and agree that it's part of the problem -- because now I'm not entirely sure what the exact deadline semantics is for the course I had in mind when writing this issue... Maybe one can see it as, currently, Fire only supports hard deadlines, whereas adding an option to allow students to submit after a first deadline would extend fire with support for soft deadlines (i.e., how most deadlines are handled in practice).

[Evgeny Kotelnikov]

It would be more flexible to let each course responsable define what a deadline means, rather than having a fixed concept in Fire that all courses must adhere to.

I am open to suggestions, bug this sounds very vague.

Maybe one can see it as, currently, Fire only supports hard deadlines, whereas adding an option to allow students to submit after a first deadline would extend fire with support for soft deadlines (i.e., how most deadlines are handled in practice).

Yes, this is the way I pretty much interpret it myself. Students usually get deadline extensions unconditionally, but the hassle of requesting one is enough of an incentive for most students to follow the deadlines anyway. I personally haven't been annoyed by empty submissions enough to seriously think about changing the submission-grading workflow.

[Andreas Lööw]

I am open to suggestions, bu[t] this sounds very vague.

Yes, as a suggestion this is too vague to mean anything. What I meant was that there are two ways one could go about this: 1) Fire offers one particular doctrine of deadlines, that all courses must subject themselves to, or 2) Fire offers settings where one can configure deadlines so they coincide with the notion of deadlines used in one's particular course. In this specific case we are only considering hard vs. soft deadlines, where soft deadlines means that students are allowed to submit after the first deadline, without asking for special permissions -- but, in the particular case I have in mind this means that students that do that lose one guaranteed grading; which is a Fire-external notion, so having some kind of marker in the system that a submission was late is still valuable.

But after that you have said what you have said, I'm not entirely sure if the deadlines are meant to be hard or soft in the course I had I mind, I just know that in practice they are soft, as students "unconditionally" get deadline extensions just by asking, or in most cases, by just sending in partial solutions with a Fire comment saying that they will send in a full solution later (after a manual reject from a TA).

What's good about alternative 1 is that it keeps the Fire system simple (and consequently easier to understand for everyone), and takes less implementation effort (because such a system exists already). What's not good about it is how it handles soft deadlines, because it involves too many manual steps (and confuses students).

So what is (still) clear is that this practice of sending in partial solutions must stop, but I'm not longer sure what to do about it...

I personally haven't been annoyed by empty submissions enough to seriously think about changing the submission-grading workflow.

I guess I'm more easily annoyed then.

[Andreas Lööw]

Actually, the web page for the course I had in mind says this:

We guarantee two gradings per lab: one for the version submitted before the ordinary deadline for that lab, the other for a resubmission before the final deadline. If your first submission is after the lab deadline, only one grading is guaranteed.

So I guess soft deadlines is what's indented after all.

[Evgeny Kotelnikov]

It should be possible in the future to set up a limit on the number of resubmissions, this was suggested in #129. I have not thought about refining it to account differently for solutions sent before or after the first deadline.