AssertException: Cannot open /dev/null -- Bad file descriptor raised in init_env at src/env.c:92

[Gilbert Wilson]

When trying to execute a monit process with Saltstack (start, status, etc) I get the following error:

AssertException: Cannot open /dev/null -- Bad file descriptor
raised in init_env at src/env.c:92

It appears that this happens because Saltstack does not always provide an environment that passes Monit's initialization test. The problem is the test on line 91 of env.c:

if (fstat(i, &st) == -1 && open("/dev/null", O_RDWR) != i)
                    THROW(AssertException, "Cannot open /dev/null -- %s", STRERROR);
    }

To test/fix this issue, we made the following c-wrapper that corrects the test and passes it's arguments on to monit:

// Ensure that std descriptors (0, 1 and 2) are open and fstat'able. If not, remap to /dev/null.
    for (int i = 0; i < 3; i++) {
            struct stat st;
            if (fstat(i, &st) == -1 && close(i) == 0 && open("/dev/null", O_RDWR) != i) {
                    perror("descriptor remapping failed");
        return 1;
    }
    }
// Hand off to monit (found via the current path), passing along the same arguments we received.
execvp("monit", (void *)argv);

But, it would be nice to have it patched so I don't need to do anything like this!

I also filed a bug with the Saltstack folks that has related information (if needed): https://github.com/saltstack/salt/issues/32190

[Tildeslash]

It seems that errno=EBADF was set by fstat(i) on line 91, not by the open() call, which cannot produce EBADF.

It then means, that the "i" filedescriptor was closed already (EBADF on close = the argument is not a valid open file descriptor), the open() call most probably succeeded, but it returned different filedescriptor then expected.

Since we start at fd=0 for stdin (and as you noted at https://github.com/saltstack/salt/issues/32190, when you set stdin then monit works), it seems that although stdin with fd=0 was not set, either stdout with fd=1 and/or stderr with fd=2 were opened and the open() call probably returned something like fd=3, where monit expected fd=0.

The call of close(i) in your wrapper should fail with EBADF same way as ftstat(i), as the filedescriptor should be invalid (closed already). If it succeeded and the "i" filedescriptor was in fact open, it is very strange ... seems more like kernel bug.

Can you get output of "lsof -p $$" when executed via Saltstack? (so we can see which filedescritors are open)

We have also improved the logging little bit, so it'll be clear what failed: https://bitbucket.org/tildeslash/monit/commits/1931eff04fcd/

[Tildeslash]

Tried to create simple test program to simulate a closed stdin with stdout+stderr opened and trying the same sanity check and on-demand reopen as monit (use "gcc -std=c99 -o open open.c" to compile):

#include <assert.h>
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#include <sys/stat.h>

int main(int argc, char **argv) {
        // Close stdin to simulate issue #337
        close(0);

        for (int i = 0; i < 3; i++) {
                struct stat st;
                int rv = fstat(i, &st);
                if (rv != -1) {
                        printf("fd %d: fstat OK\n", i);
                } else {
                        printf("fd %d: fstat returned -1\n", i);
                        // Note: uncomment if you want to test close() on fd as done in the wraper (we also expect close will fail with EBADF here)
                        /*
                          assert(close(i) == -1);
                          printf("fd %d: close returned -1 (OK)\n", i);
                         */
                        int rv = open("/dev/null", O_RDWR);
                        if (rv != i) {
                                printf("fd %d: open returned %d whereas %d was expected\n", i, rv, i);
                                return 1;
                        } else {
                                printf("fd %d: open returned %d (OK)\n", i, i);
                        }
                }
        }
        return 0;
}

Tested on FreeBSD 10.2, MacOSX and linux, works fine on all platforms.

[Tildeslash]

Removing version: 5.16 (automated comment)

[Tildeslash]

unable to reproduce, works fine on MacOSX, linux, FreeBSD ... kernel bug?