Assign grader on demand/unrealized submissions

[JonasDuregard]

This concerns a different (and IMO superior) algorithm for assigning submissions to graders.

The algorithm can be described as follows: * In addition to the submission lists for each grader there is a global list of pending submissions for every task. * Every grader has a grading quota for every task, a number indicating how many submissions they need to grade. These numbers should always add up to the total number of submitting groups, and they are calculated from the relative weights assigned to graders for the tasks.

When a student submits a (first) solution it is added to the common submission list, and the grading quota is increased for one of the graders (based on their assigned weights and current numbers).

When a grader accesses their list of submissions, if their quota exceed their current total of graded and pending submissions there should be a button for fetching the oldest pending submission from the global list to their individual list.

Advantages/disadvantages

There are many advantages to this algorithm, here are some good ones: 1. There is a higher degree of fairness, when a (first) submission is graded it will tend to be the oldest pending one, not just locally oldest for a particular grader. 2. Its easier for graders to plan the time they spend grading. As long as there are submissions on the global list they can for instance do most of their grading before the deadline if they want to. 3. If grader weights are changed after deadline, the required numbers for graders change without requiring previously assigned submissions to be shuffled around (See Issue #90). Of course this does not apply if graders have already filled their grading quota.

I can not think of any significant disadvantages for graders. If a grader wants to have all their current submissions at once, it would be easy to add a "fetch all" button and if they just click it whenever they access a lab it would be very similar to how it works today.

I suppose it would no longer be possible to tell students who will grade their submission when they submit, which is a bit unfortunate.

Alternative definition

Another way to specify this algorithm is that it adds a fourth state a submission can be in. Where today submissions would be either pending, passed or rejected, with this algorithm first submissions could also be "unrealized" meaning that the grader has been assigned a submission but it has not been decided which one. When the grader wants to access one of these submissions it is realized into a pending submission by fetching the oldest submission from the global list.

This can be implemented with various levels of transparency to the grader (up to automatically realizing all their labs when they log in and access their list for a task), but IMO a good way would be to list these as pending submission but instead of a group number it just says "unrealized submission" (better word?) and when the grader accesses it (or downloads all submissions) it is automatically realized. Described like this, it seems like a fairly small change that would probably not be hard to implement and would not cause any extra complexity for graders.

[Evgeny Kotelnikov]

Thanks for sharing!

When a grader accesses their list of submissions

That is, visits the lab page with the list of submissions?

1. There is a higher degree of fairness, when a (first) submission is graded it will tend to be the oldest pending one

Agree.

2. Its easier for graders to plan the time they spend grading.

Can you elaborate on that? Currently, submissions are assigned round-robin with the weight taken into account. So, a grader is never assigned less or more (modulo rounding error) of what they are supposed to grade. They are free to choose when to grade and in which order. What exactly can be improved here with respect of planning the time?

3. If grader weights are changed after deadline, the required numbers for graders change without requiring previously assigned submissions to be shuffled around

Agree.

I can not think of any significant disadvantages for graders.

Some things will get a bit convoluted, namely the indicators of how many submissions are left to grade and notifications of new submissions. I guess, the indicators can calculate how many submissions are supposed to be assigned based on the quota and show that number. The notification system can probably do a similar estimation with the new submissions. Although, in such case we wouldn't be able include direct links to newly assigned submission, because we wouldn't know what they are (I dislike this lack of transparency). And, that is except for those submissions, that were reassigned to you by another grader (that should still be allowed, right?), we would need to notify about them as well.

Not that it can't be done, but why all the hustle really? I actually don't see any significant advantages for graders in this model.

Another thing is that I think it's a bad practice to have some non-trivial action be performed simply by visiting a page. We have that in one place (a new submission is created when a student simply visits the submission page), and has been causing all kinds of trouble. Non-trivial actions should be performed explicitly by pressing a button.

when the grader accesses it (or downloads all submissions) it is automatically realized

Does this model assumes that a grader goes to the list of all unrealized submissions and chooses one? I don't see any good reasons for a grader to have this choice. Also, that would break advantage №1.

[JonasDuregard]

That is, visits the lab page with the list of submissions?

Yes.

They are free to choose when to grade and in which order

The "when" is certainly not free in the Databases course, we require graders to give feedback within three calendar days from submission.

Can you elaborate on that?

Suppose a course has a 72 hour limit on grading (like the databases course has now). Currently that means every grader needs to grade labs fairly often (every other day or so) to keep up. With the global queue a few days of absence by one grader (something that happens several times every year) would often be compensated automatically by other graders.

The algorithm essentially emulates graders manually switching one of their submissions for the globally oldest one before grading anything, so it will have all the advantages of that (including giving the previously assigned grader a respite).

In fact the whole reason I wrote this suggestion was that it felt silly for me to grade a 1 hour old submission when other submissions where approaching the 72 hour limit, simply because I did not know if the assigned grader was already working on it so doing the switch described above might cause a race condition.

In my opinion the current system mixes two concerns that should be separate: How many assignments each grader gets and in which order they are graded. For the first question, the weighted round-robin system is perfectly reasonable, but letting the technicalities of that algorithm dictate the order in which assignments are graded does not make sense. If we disregard the question of weights, surely everyone would agree that the only sensible solution would be for graders to fetch submissions from a global queue (think of it as a simple worker pool, graders are workers/threads and submissions are jobs). So why should the introduction of weights change this simple principle?

Some things will get a bit convoluted, namely the indicators of how many submissions are left to grade and notifications of new submissions.

I don's see that this would be more complex. The "unrealized" submissions can be counted just as realized ones are today. Regarding the links in the email, as a grader I have no use at all for them because when I review a submission I always start with the oldest one I'm assigned (and I assume the email only links to new ones?). A link to my oldest submission would work better, or even just a link to the assignment page.

Does this model assumes that a grader goes to the list of all unrealized submissions and chooses one? I don't see any good reasons for a grader to have this choice.

If you mean choose different unrealized submission, such a choice would affect nothing, because they are all identical (they are all just placeholders to be replaced by a submission from the global queue, which one is replaced does not matter - it is always replaced by the oldest submission). So listing the unrealized submissions would just be a way to keep the current interface, counters etc. as intact as possible, it could just as well just show the number of unrealized submission and have a single button for realizing any one of them.

If you are asking if it should be possible to move other groups than the first one from the global queue I suppose that would be good in rare cases, as long as the default behavior is to always realize to the oldest submission.

Another thing is that I think it's a bad practice to have some non-trivial action be performed simply by visiting a page.

I tend to agree and that is a completely optional mode for this algorithm, that would require no changes to the front end.