Example of using shared memory databases

[Bert Bogaerts]

Dear,

I am trying to use the shared memory databases but it does not seem to work.

I used the following command to put the database into memory:

kma shm -t_db /db/kma/bacteria.ATG/bacteria.ATG -shmLvl 29

So if I understand correctly this should put all of the database files in memory:

bacteria.ATG.comp.b 
bacteria.ATG.length.b
bacteria.ATG.name
bacteria.ATG.seq.b
bacteria.name

But when I use the following command:

kma -i ../input/reads.fastq.gz -t_db /db/kma/bacteria.ATG/bacteria.ATG -shm 29 -o kma_out

The database is loaded into memory again.

Could you let me know if I am using wrong commands?

‌

Best,

Bert

[ptlcc]

Dear Bert

If the “kma shm” completed successfully the database should be in shared memory.
You might notice that the loading of the database on the kma call is now close to zero seconds. If you use gnu-time or top to survey the program, it will look like it has loaded the database into memory. But in fact these programs shows you how much memory the program have access to, in effect you can easily get above 100% memory usage when using shm and surveying with e.g. top.

Btw. there is not a large difference between -shmLvl 29 and 1 for bacteria.ATG, assuming it is the one from the cge ftp-site. Additionally I would recommend the flags “-1t1 -mem_mode“ on that database, as it contain complete genomes.

Best,
Philip

[Bert Bogaerts]

Dear Philip,

The database loading is very fast indeed, but even when I omit the ‘-shm’ option the total time used for DB loading is 0.00s.

I still do not think the memory is shared. Because if I run a single job I have no issues, but as soon as I launch a couple in parallel I run out memory, and the program crashes with the ‘failed to allocate memory’ message.

If all the runs share the same database in memory then the additional RAM usage for multiple runs should be limited I think?

Best,

Bert

‌

[ptlcc]

Dear Bert

That is because kma checks if the database is in shared memory before loading from the disk, the extra “-shm“ flags causes kma to fail with an error message if the database is not in shared memory.

If you add the “-mem_mode -1t1“ flags and just run the shm command with default level sharing (“-shmLvl 1“) then you should not have problems with the extra jobs.

Best,

[Bert Bogaerts]

Dear Philip,

Ok, thank you!

I tested it and no memory issues anymore.

Best regards,

Bert

[ptlcc]

Great.