auto reject based on something

[Luciano Bello]

We would like to autoreject if the submission is not in a minimal shape. Something like the pre-commit.sh in git.

I can picture that like, when a new submission arrives, ask to an external computer to do something with it and accepted or not.

[Gregoire Detrez]

Yes, I'd like that too :-D The problem is that we need to be able to let this other computer get the assignment files in a secure manner. Which means managing auth tokens of some sort, or ssh keys.

I have some ideas on doing that with git+ssh but it'll still require some work

[Evgeny Kotelnikov]

It would certainly be useful to have some sort of hooks for that, not an ad hoc solutoin.

I can imagine a very simple solution: graders are allowed to attach a Python script to a lab. There will be Fire's API exposed (as Python function/classes/etc), so you have access to attached files and everything. You can run external processes (compilers, lints, etc), available on the server. Every time there's a new submission, script is executed; after that there's an icon next to the submittion (red or green) with the transcript of the check.

[Luciano Bello]

I think is better to externalize the check (for flexibility and maintenance). Grader can set a server where Fire send the things (can be over SSL) and get an answer. If you define the API, the grader just need to follow it.

[Evgeny Kotelnikov]

It does sound like like it's more "correct" way to go, but it also requires setting up complex infrastructure with SSL access, authorization tokens, remote API as so forth. Does the benefit of doing things correctly outweight the extra effort needed for the implementation?

What are the drawbacks of local Python scripts that I described?

[Gregoire Detrez]

I agree with Luciano, this has to be done on a different machine. Courses can have all sort of crazy requirements for weird tools and specific versions and it will be hell to maintain. And more generally, executing user code on the server is not a good idea: it would need to be sandboxed, with limited resources so it doesn't crash the server, etc etc

[Evgeny Kotelnikov]

Well, that depends on the kind of things you would like to check. If the majority of courses require complicated environments then separate servers will surely be more convinient. I'm not sure if that it's the case. In my datastructures course with Java code submission simple "Try to compile" button would already helped a lot. And it would be much easier to implement it locally.

Remember that any new infrastrucre is a lot of code and a lot of bugs.

[Gregoire Detrez]

That mean you at least need a java compiler, the database course would need psql, some courses will need a haskell compiler, not necessarily the same version...

And this is only one part of the problem: we would need to sandbox all of those which probably means firing and controlling virtual machines or linux containers to run the hook. I believe this will be a lot more infrastructure than some http hooks with an API.

[Luciano Bello]

In compilers we need Java with some obscure libs. Haskell with another bunch of libs. Even run some in-house tests.

[JonasDuregard]

How about this, if you can require a code from the user upon submission (and also provide instructions on how they obtain the code). Then the code could be provided when they upload their submission to a separate server (a bit unfortunate, requiring them to upload the same files twice) or by running a testing application provided on the course webpage or on the Chalmers computer system. It could also be as simple as requiring them to write "yes" when asked if they have done the required testing, which could be enough in many cases.

How fancy the codes are can vary from a simple code word to some sort of cryptographic signing. Using the former should not be a problem as long as you are aware that it might happen that someone tries to bypass the check (I doubt anyone will actually do it).

[Evgeny Kotelnikov]

I think it's about as complex as implementing a hooks system.

[JonasDuregard]

Certainly requesting a passphrase is simpler?

Something like this:

answer = prompt("Confirm that you have used the testing tool by writing I HAVE TESTED");
if (answer != "I HAVE TESTED") {
  abortSubmission("Incorrect passphrase");
}

Of course a more general solution would be better, but simpler?

[Evgeny Kotelnikov]

I thought you meant that the testing server would generate a code when all the tests have passed and that code would be checked by Fire.

Making a student confirm that they run the tests is of course simple.

[H E]

I know whatever solution that involves JavaScript is not considered serious but I suggest the following solution:

Allowing course responsible to define onclick javascript Event for submit button. This way the course responsible can perform some basics checks or even send the files to another server for further assessment. Sample JavaScript can also be provided.

[Gregoire Detrez]

I think this solution has two major problems:

• The javascript will be executed on the student browser, so they could easily change it to do whatever they want (like bypassing the checks)
• if the file is send to a separate server this way, there need to be a mechanism to get the response from this server. And in that case, the response need to reach the student browser, not just the server, which makes is really complicated.

[Gregoire Detrez]

Maybe we should divide this issue in two:

Simple checks on the file list could be implemented in fire itself and used to refuse incorrect submissions (e.g. allowing only some extensions, disallowing empty submission, etc..).

More complex tests (linter, compilation, test-suite, etc) should be run asynchronously on a separate server. This probably means that the submission should be allowed (we don't wait for the external server). Also, in that case, the pass/fail feedback should be shown as an indication to the grader but not necessarily used to automatically reject submissions (a bit like the way github shows build status on pull requests).

[Evgeny Kotelnikov]

Maybe we should divide this issue in two

Agree. Actually, I thought that part 1 in one of the issues already.

[H E]

I was aware of the weaknesses. Javascript can be disabled but if a student cheats they will eventually be caught when the submission is checked by a grader. Sending file to another server and looking at the response and showing an error msg in the student's browser [the complicated mechanism] can all be done in the javascript. I think in order to keep the Fire system simple, the Fire should not deal with all these special cases (file type, file content, result of execution), sandboxing and ...

[Gregoire Detrez]

I have implemented a simple rule that allows only a predefined set of extension. I also have been talking with the TAs for the AI course to implement the hook system. I'll try to post the progress here.

[Gregoire Detrez]

To be tested in LP4 2016