Insert + find

kisyad avatarkisyad created an issue

I'm encountering problems when I insert a record and try to find it it immediately afterwards with pool size > 1.

The find doesn't return the just inserted record.

It seems that the insert and find order get switched somewhere.

I wrote a small test case:

-module(emongo_write_read_test).

-export([empty_test/1, test_mongo/3]).

empty_test(Database) ->
	application:start(emongo),
	emongo:add_pool(pool1, "localhost", 27017, Database, 1),

	emongo:delete(pool1, "bugtest").

test_mongo(Database, Num, Pools) ->

	application:start(emongo),
	emongo:add_pool(pool1, "localhost", 27017, Database, Pools),

	Name = "Name" ++ integer_to_list(Num),
	
	ok = emongo:insert(pool1, "bugtest", [{name, Name}]),
	case emongo:find(pool1, "bugtest", [{name, Name}]) of
		[_Rec] ->
			test_mongo(Database, Num + 1, Pools);
		[] ->
			io:format("Couldn't find [~p] after insert\n", [Name]),
			{error, Name}
	end.
erlc -o ebin/ t/emongo_write_read_test.erl
erl -pa ./ebin -eval "emongo_write_read_test:empty_test(\"bugtest\")" -s init stop -noshell
erl -pa ./ebin -eval "emongo_write_read_test:test_mongo(\"bugtest\", 1, 5)" -s init stop -noshell

Output: 
Couldn't find ["Name52"] after insert

From mongod (verbose) log:
Fri Jun 11 23:12:44 insert bugtest.bugtest 0ms
Fri Jun 11 23:12:44 runQuery: bugtest.bugtest{ name: "Name51" }
Fri Jun 11 23:12:44    used cursor: 0x101010ab0
Fri Jun 11 23:12:44 query bugtest.bugtest ntoreturn:0 reslen:75 nreturned:1 0ms
Fri Jun 11 23:12:44 runQuery: bugtest.bugtest{ name: "Name52" }
Fri Jun 11 23:12:44    used cursor: 0x101010ab0
Fri Jun 11 23:12:44 query bugtest.bugtest ntoreturn:0 reslen:36 nreturned:0 0ms
Fri Jun 11 23:12:44 insert bugtest.bugtest 0ms

Notice how the query and insert order has switched for "Name52". Query becomes before insert.

Am I misunderstanding how pools should be used, or is there a bug here?

Comments (9)

  1. Jacob Perkins

    This issue totally make sense when pool size is greater than 1. What happens is emongo starts a bunch of connections and uses them in a round robin fashion. So insert hits the first connection, and find hits the second connection. Since find is faster than insert, it will fail unless there's a long enough delay so the insert can complete. So you probably need to add a delay between insert & find, or just assume you won't be able to find an object immediately after an insert call.

  2. kisyad

    Yeah, that's what I was assuming, just wanted to make sure that I wasn't missing something.

    We are porting our app from mnesia/postgres to mongo and encountered couple of these situations. We will just have to restructure a bit.

    Thanks for your response.

  3. Dmitry Belyaev

    Another solution is to use synchronized version of insert. I haven't created it yet but it must be like emongo:update_sync/5 or emongo:delete_sync/3. It must ask server for getLastError and wait for server to reply.

    But even this method won't help you if you have more than one mongodb server. If you configure one of them to be slave, insert_sync something on master and ask slave for newly inserted record you probably won't find it yet. That's why I'm using only one server for writing and reading.

  4. Dmitry Belyaev
    ok = emongo:insert(pool1, "bugtest", [{name, Name}]),
    case emongo:find(pool1, "bugtest", [{name, Name}]) of
    

    with recent "sequences" it is possible like that:

    case emongo:sequence(pool1, emongo:insert_seq("bugtest", [{name, Name}],
           emongo:find_all_seq("bugtest", [{name, Name}], []))) of
    

    all operations will be executed one by one hence find will be run after element is inserted.

  5. kisyad

    Thanks Dmitry,

    Eventual insert_sync would be great because many of the little issues I face come from event handling where I cannot use the explicit sequence.

    For example:

    1. Process handles event A in an event handler by inserting a record
    2. Process handles event B in a different event handler by looking up the record

    In these cases it would be great to be able to make sure that event handling doesn't proceed before the insert has been performed.

  6. Dmitry Belyaev

    This new interface allows to combine operations as you like:

    emongo:sequence(pool1, emongo:insert_seq("bugtest", [{name, Name}], emongo:synchronous())).

    This is exactly what insert_sync would do.

    emongo:synchronous/0 (of course, inside sequence/2) just calls getLastError thus waits for all previous operations to commit.

    Yes, it's requires more text but allows more control. Any ideas on how to improve this would be appreciated.

  7. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.