need validation of event size

[Stefan Ritt]

Similar to event contents validation, we need a validation of event sizes. If events get bigger than MAX_EVENT_SIZE of half of SYSTEM buffer size, strange things happen. I would prefer a clear error message and rejection of too large events.

[dd1]

I will look into this. K.O.

[dd1]

Looking into this now.

a) MAX_EVENT_SIZE is not used when sending events into an event buffer. The only requirement is that it fits into the buffer. Not clear if event size must be half the size of the event buffer or not (min 2 events per buffer or permit 1 event per buffer).

b) need to check for correct error messages when sending events that do not (or should not) fit into the event buffer.

c) MAX_EVENT_SIZE is still used by the event receiving code and the event dispatcher - and this is the mlogger main data path. I should replace it with automatic memory allocation (bm_receive_event() -> bm_receive_event_alloc()). But need to measure the memory allocation overhead.

d) once MAX_EVENT_SIZE is no longer used by the event receiver, there will be no limit on event size anywhere inside midas. Except for the buffer size, of course.

Getting there, K.O.

[Stefan Ritt]

I would say that at least two events should fit into the buffer so one event can be retrieved by the consumer while one event can be placed into the buffer by the producer at the same time. Otherwise buffer management does not make sense.

[dd1]

MAX_EVENT_SIZE removed from mserver RPC call receiver (via _net_recv_buffer), memory is now allocated automatically as needed.

MAX_EVENT_SIZE remains used by bm_poll_event() (via _event_buffer and _bm_max_event_size). The bm_poll_event() is normally only used by a remote-connected analyzer. Internally, it uses bm_receive_event(), uses RPC_BM_RECEIVE_EVENT, and there is no corresponding bm_receive_event_realloc(), and we need to know the maximum event size ahead of time.

K.O.

[dd1]

hmm... in the mserver RPC call receiver (rpc_server_receive()), we never actually use the RPC code to receive events, there is a special event receiver code for that (rpc_send_event() & co).

So actually MAX_EVENT_SIZE was used to set the maximum RPC call size. So if we make an RPC call bigger than MAX_EVENT_SIZE, i.e. write something big to ODB via db_set_data(), we will bomb. Luckily, MAX_EVENT_SIZE is usually 4 Mbytes or bigger, so this does not happen in practice.

Now the max RPC size is disconnected from the max event size. (and the max rpc size is now unlimited).

K.O.

[dd1]

remove the check the MAX_EVENT_SIZE from mfe.c as there is no size limit anymore. Attempt by the frontend to send an event bigger than the size of the SYSTEM buffer produces these error messages, which I think are clear enough, I think I do not need to add words saying "please increase the buffer size in /experiment/buffer sizes").

[midas.c:8532:bm_send_event,ERROR] total event size (1024040) larger than size (838860) of buffer 'SYSTEM'

K.O.

[dd1]

Checked event buffer read size.

Locally connected mlogger and mdump can receive events of any size regardless of MAX_EVENT_SIZE setting in ODB.

Remote connected mdump requires ODB MAX_EVENT_SIZE set to bigger than biggest event otherwise events are truncated with these error messages:

[mdump,ERROR] [midas.c:8922:bm_read_buffer,ERROR] buffer size 26 is smaller than event size 102436, event truncated. buffer "SYSTEM"

and

[mdump,ERROR] [midas.c:9527:bm_poll_event,ERROR] received event was truncated, buffer size 26 is too small, see messages and increase /Experiment/MAX_EVENT_SIZE in ODB

K.O.

[dd1]

I think both on the reader side and the writer side the error messages for sending "too big" events are good now.

Closing this ticket.

K.O.

[dd1]

Error messages for "too big" events have been improved. As of commit 6fc0d39. K.O.