name collision with two experiments with the same name but different users

[dd1]

two experiments using the same name but running under different user accounts will collide over the posix shared memory name under /dev/shm. This is because shared memory name does not contain the user name. Fix requested by Stefan. K.O.

[Stefan Ritt]

Wouldn’t a

shm_name = msprintf("%s_%s_SHM", cm_get_path(), name);
while(shm_name.find('/') != std::string::npos)
  shm_name.replace(shm_name.find('/'), 1 ,"_");

just do it?

Since each user runs their experiment in a separate directory (usually /home/<user>/online or so) we have a nice namespace separation. That’s also how it was implemented before we had SYSV shared memory.

[dd1]

Changing name of shared memory will break binary compatibility of MIDAS. So this will be POSIXv3_SHM in .SHM_TYPE.TXT. Existing experiments will continue to use POSIXv2_SHM (unless they edit .SHM_TYPE.TXT). Newly created experiments will use POSIXv3_SHM. K.O.

[dd1]

I think using cm_get_path() as part of the shared memory name will work. But I will need to check for limitations on the name length. K.O.

[dd1]

I was thinking of just adding the username, but this still has collision if some user creates 10 experiments all with the same name in 10 different directories.

So I like Stefan’s idea better. I hope we do not run into a limit on shared memory name length.

K.O.

‌

[dd1]

macos shm_open() name limit is 31 bytes (+NUL), see PSHMNAMELENGTH is posix_shm.h. so on macos using pathname of experiment directory will not work. K.O.

[Stefan Ritt]

Then back to user name + experiment name?

[dd1]

Shared memory name must contain: experiment name (32 bytes), buffer name (32 bytes) and if we add user name (unlimited? bytes), we are even more in trouble with the MacOS shared memory name limit.

On Linux, we are okey max name length is around 256 bytes (NAME_MAX), so we are okey with adding the user name, but if we add the experiment directory path, we could be in trouble.

On BSD, I am not sure what the limits are.

K.O.

‌

[dd1]

On MacOS, do we have a use case where there are multiple users doing something at the same time? Most Mac laptops are single user and going forward, I do not see Apple building any serious multi-user machines. So maybe we stay with POSIXv2 shared memory on MacOS (no user name in the shared memory name)? K.O.

[dd1]

On MacOS, there is no “ls -l /dev/shm”, so we do not need nice looking names for share memory objects. We can use “token_shmname”, i.e. “12345_ODB”, “12345_SYSTEM”, token can be a random number (i.e. time ODB is first created) and “encode” the user name, the experiment name and experiment directory by putting it in a file “.SHM_TOKEN.TXT” in the experiment directory. This would be POSIXv4 shared memory.

This scheme would also work on Linux, but for doing things like “ls -l /dev/shm” and “rm /dev/shm/expt_SYSTEM” it is nice to have the experiment name as part of the shared memory file name.

K.O.

‌

[dd1]

So I am now looking at two new things:

• POSIXv3 - add experiment directory to the shared memory name (so we do not have to give different names to experiments that live in different directories)
• POSIXv4 - use names “token_shmname”, this is for MacOS with it’s short names.

K.O.

‌

[Stefan Ritt]

Actually user names can be 31 chars long, but in reality they never are. So we could truncate the experiment name to let’s say 12 chars, the user name to 12 chars, add the shared memory name (which does not exceed 6 chars if I’m not mistaken) and we end up with 12+12+6+1=31 chars at most. Most names would never be truncated, so the whole string looks nice on all systems.

[dd1]

Hmm… I have seen to many “user1”, “user2” and “expt1”, “expt2” to be comfortable with truncating such things.

Event buffer names seem to be 32 byte long (NAME_LENGTH), at least they are 32 byte long in ODB /eq/xxx/common, this presumes same length for shared memory objects. K.O.

[dd1]

We could add them together and if we do not fit, complain “please use a shorter experiment name and/or shorter event buffer name”. K.O.

[dd1]

googled for “max user name length”, there seems to be some confusion. traditional length is 8 bytes, “maximum” length is 32 bytes (31+NUL?), but not clear who defines this limit. We definitely have usernames longer than 8 bytes at TRIUMF. K.O.

[Stefan Ritt]

Another way would be to concatenate user, experiment and shared memory name, then limit it to 31 chars cutting from the LEFT. In 99% if all cases no truncation would be necessary. Like

olchanski_acdaqmini_SYSTEM10 = 28 chars

and

olchanski_acdaqmini_VERYLONGSYSTEM = 35 chars

would be truncated to

anski_acdaqmini_VERYLONGSYSTEM = 31 chars

sorry for the name truncation, but still kind of readable.

‌

‌

‌

‌

[dd1]

ok, done. I added:

• POSIXv3_SHM with names “uid_expt_shmname”, i.e. “1900_myexpt_ODB” (no SHM prefix as in POSIXv2_SHM)
• POSIXv4_SHM added the experiment directory to this.

New defaults: MacOS: POSIXv3_SHM, Linux: POSIXv4_SHM.

If shm_open() returns “name too long”, I tell the user to use shorter names and to fall back shared memory types that use shorter names. (last one to fall back to is POSIX_SHM which does not generate names longer than 32 bytes, good enough even for MacOS).

If shm_open() fails due to file permission, I tell the user to switch to POSIXv4_SHM.

good enough?

K.O.

‌

[Stefan Ritt]

Perfect! Clever idea with the uid! Thank you.

[dd1]

fixed dc86fe9 to 203db44. K.O.