Multiple custodians running the bot on the same pair

[Desrever Nu]

If they are not synchronised , one's wall shift can happen before some other bot and they could eat each other walls.

[Desrever Nu]

First temporary action is preventing two custodians to act on the same non-usd pair.

Secondly, we have a proposal by JL to use the liquidityinfo as a mean of syncronizing bots.

[Michael Witrant]

If you synchronize bots you'll have to handle a lot of new problems. You start a system where one bot can easily break all the others.

I think we can instead rely on market data here. For example we could have a delay before eating walls on a price change to give enough time for the other bots and the arbitrageurs to handle the price change.

When a price change is detected, we would remove our orders immediately but we would place new orders only when there's nothing to eat or when a certain delay has passed (a few seconds for example).

If the other bots haven't removed their walls before the delay then we would eat them. But we would have to do that anyway even if the bots were synchronized.

[Michael Witrant]

Actually the bot would not remove the orders immediately. It should just adjust the wall so that it doesn't eat any order until a certain delay has passed.

For example if the price increases, the bot would adjust the sell wall immediately and only adjust the buy wall after a certain delay has passed or when there's no sell order in the way.

I made a drawing to illustrate:

We can start more simple and just add a delay between the sell wall update and the buy wall update, without considering active orders. That would mean that bots have X seconds to move their sell wall when the price increases (and X seconds to move their buy wall after the price decreases).

What do you think?

[Desrever Nu]

Brilliant. A few notes from a first analysis of mine:

• note: from the internal functioning of the bot, we always have the wall splitted in two (more or less equals) orders : two buy walls, and two sell walls. This allows for the custodian to make smooth movements, and adjust the price of an order without being left without walls. Even in the code there is already an implementation of the same mechanism that moves the orders one after the other, but is currently not being used since it need more testing. Your model can also work with two orders the same way, I just wanted to highlight this.

• This mechanism would rely on market data. I guess you had orderbooks in mind. So assuming we have access to the orderbook (now the API wrapper do not include orderbooks, it must be upgraded), how can we differentiate a custodian's order from other orders? I can see that sometime the size of the order is not enough.

• Currently there are two different threads : processA that runs every yy seconds and execute orders and cleans things up, processB that runs every xx seconds to detect a price change. Both xx and yy can be controlled by custodians. When a price change is detected, processB notifies processA about the change in price by updating the values of some variables. This means that the "wall shift" right now is not instantaneous and there is already a delay. Moreover, different custodians could have different values for the time spacing. Different custodians can also choose to track the PEG price from a different source. All of the above is to say that, from the moment that the PEG price actually change to the moment that we actual procedure to shift walls start, there is an interval of time that passes that cannot be controlled and changes from one custodian to another.

• I like the solution you proposed. One case where walls get eaten up is when one of the two bot, for whatever reason, gets stuck and does not adjust its price. Another case is where bot A is tracking the price of PEG from one feed, while bot B from another one that does not update its price according to price A.

• How does your design work with 3,4,5,...10 bots running?

• We should always make two cases : when price increases ( walls shift rightwards ) and when price decreases ( walls shift leftwards)

[Michael Witrant]

This mechanism would rely on market data. I guess you had orderbooks in mind.

Yes.

But in the simplified design I described after the drawings we don't even need that. We just wait a few seconds before we move the other wall.

how can we differentiate a custodian's order from other orders?

We don't. We adjust and we give a few seconds to the whole market to adjust (bots, arbitrageurs and traders). I think we just don't need to rush eating orders here. There's one wall that we can and should move immediately, but the other one can be delayed.

If I'm wrong and we need to consume these orders as soon as possible, but still not consume other bots' wall, then we will need some kind of synchronization between the bots. But it would be weird if we wanted to consume other users' orders while not consuming other bots' orders. We can't differentiate them when we eat them anyway.

Different custodians can also choose to track the PEG price from a different source.

They need to agree to some extent here. I don't see how the bots can work on the same pair with different prices. If we force them to avoid collision, then the result will be larger spread. We must choose what happens when they disagree on price: either some bots get their wall eaten, or we increase the spread. I think the former is better for us because increasing the spread means a bad peg. And custodians should try to agree on price sources to avoid this situation as much as possible.

from the moment that the PEG price actually change to the moment that we actual procedure to shift walls start, there is an interval of time that passes that cannot be controlled

That's the reason of the delay. Custodians should agree on a maximum interval that is allowed and should configure their bot accordingly. If their bot takes more time to adjust, they risk having their wall eaten by another bot.

For example if they agree (or we tell them) that bots have a maximum of 10 seconds to adjust their wall, then xx + yy + zz should be < 10 seconds. zz being the maximum time it takes for the bot to process the price change (cancel orders and make new ones).

I say 10 seconds as an example, I have no idea what are the timings we expect here.

One case where walls get eaten up is when one of the two bot, for whatever reason, gets stuck and does not adjust its price

Yes that will always happen whatever the solution. The problem is the same: what do we want when that happens? Either a larger spread because the other bots hold their move, or its wall eaten. Or something intermediate like holding for X seconds and then the wall eaten. The latter is probably the best for us.

How does your design work with 3,4,5,...10 bots running?

It works the same. The bot considers all the others as traders, so whether they are bots or not shouldn't change anything. And as long as all the bots comply with the delays, they should never get their wall eaten.

[Desrever Nu]

Ok, so let's make sure we are on the same page and we both talk about the very simple strategy :

When the secondary peg price moves :
    Compute the new buy and sell prices.
    If prices are greater than current orders' prices
        All bots should immediately try to : cancel their active BUY orders, re-place them at new price
        All bots should wait N seconds, cancel their active SELL orders then re-place their SELL orders at new price
   If prices are smaller than previous prices, do the same as above in the opposite order.

Is the above reasoning correct?

They need to agree to some extent here. I don't see how the bots can work on the same pair with different prices.

In that case, we can work out a solution to limit custodian's freedom in choosing price feeds. The only reason I left "price feeds" as option parameters, its because I wanted custodians to choose the best feed for each situation. Let me provide you an example : Bot needs to act on the currency NBT/BTC on bter. Where would you take the BTC/USD price from? At first I suggested that is "always from the same exchange BTC/USD market" if there is one. After one week we observed that despite bter has a btc/usd pair, it was completely unreliable and a 20% off with other markets. So we had to choose an alternative price feed. in some other case (hypotetical btc-e), it will be more than suggested to use btc-e itself as price feed. So this cannot really be done algorithmically, but must be somehow hardcoded : for each exchange and pair, we should pre-determine the best price feed(s) available. What do you think? I believe that once we have this system in place, it will be much easier to apply your suggested strategy as we already know all bots reads from the same source (at least within the same market)

For example if they agree (or we tell them) that bots have a maximum of 10 seconds to adjust their wall, then xx + yy + zz should be < 10 seconds

Think that in the example above we are taling about a minimum of 60 seconds to refresh the price (xx) . It could potentially be less than that, but many API just does not update their one price more than once a minute. (Only exchange's direct price does that almost real time ( btce, bter ...) Others (blockchain, coinbase, bitcoinaverage... ) can take more than a minute to display a new price.

So we are looking at xx+yy+zz >> 60 . This means that on an average shift case (that can happen multiple time during a 10 minutes span) , the market will experience a "lack of walls" for more than 30% of the time. Which is quite in strong contrast with the whole mechanism I was working on to avoid leaving the market without walls, not even for a second.

I am not able to assess if being left with intermittent walls is bad or good, and if any evil player can work out a solution against our bot. Need @pennybreaker opinion

[Michael Witrant]

All bots should immediately try to : cancel their active BUY orders, re-place them at new price

When the price increases they should immediately update their SELL orders, not the BUY orders. And delay the update of the BUY order.

Is the above reasoning correct?

Apart from the above inversion, yes.

So this cannot really be done algorithmically, but must be somehow hardcoded : for each exchange and pair, we should pre-determine the best price feed(s) available. What do you think?

I think leaving that up to the custodians is still a good idea. I guess if a feed becomes unreliable they are the first one who will see it and they should be able to react immediately. I think we should just encourage them to find a consensus for each exchange. They should be able to change that whenever they need and not have to wait for an emergency release. A compromise may be to have default values. But I think they should learn they have to manage that from the beginning.

Maybe we can also have our own feeds. That would probably make consensus easier.

This means that on an average shift case [...] the market will experience a "lack of walls" for more than 30% of the time

In my proposal the walls are always there (see the drawings). You only remove a wall when you know where you will put the new one.

You correctly described the process here:

All bots should wait N seconds, cancel their active SELL orders then re-place their SELL orders at new price

The bots wait and then they move the wall. So during the waiting period, we have a larger spread but we still have walls.

[Desrever Nu]

When the price increases they should immediately update their SELL orders, not the BUY orders.

Sorry, I had Bitcoin price in mind. You are correct.

I think leaving that up to the custodians is still a good idea.

In that case, the requirement will be that, as you said, they need to reach consensus on which feeds to use in which order. ( If the first feed fails, the bot will read from the second, and so on).

The bots wait and then they move the wall.

That makes sense and makes me less worried about lack of walls. I can start working on it later this afternoon.

[Ben Rossi]

@sigmike and @desrever : I'd like to add to the proposal and get feedback on the idea.

Do you two see a benefit for synchronizing bot actions across the entire network by using a public time server? I know that getting sub-second precision across a globally disperate network is VERY hard to do, so what I'm envisioning is:

1. When a bot is started it first checks into a common set of public timestamp servers to check that the system's clock and the remote servers clock's unix epoch timestamp match (or are within some small variance).
2. Using this synchronized time stamp, operations are then carried out on a system wide interval. Instead of allowing the bot to be customized on the period that operations run, we normalize it so that, for example, every minute (at the top of the minute) the bot will check to see if the prices are correct.
3. Operations then then be carried out by all bots at a predicatable time, which should decrease the time that each bot needs to wait between cancelling their walls and putting back up their walls.

Any value?

[Michael Witrant]

@bshares that would help reduce the delay, but there would still be no guarantee. For example if the price feed also updates at the top of the minute, then some bots may get the old price and some others the new one. These situations would probably be rare though.

Using feeds that update frequently and reducing the delay between price scans may be easier and more efficient.

[Sam Griffiths]

I'd like to add some thoughts to this issue.

For the pricefeeds, I think we should definitely have a service that aggregates the available feeds and weights them as the shareholders see fit. This should be the standard pricefeed for all bots and should act as the Nu view of the current prices of all peg currencies. The Nu feed could be updated as often as is required for bots to function properly.
The ability of custodians to alter which feeds are used should remain as this may form part of a strategy (also a shame to remove it if it already exists).

With regards the walls, I think that it would be helpful for bots to be aware of which orders belong to itself, which orders belong to other instances of NuBot and which are unknown. Even if we decide that, as Mike said, we treat all walls the same and eat them whether they belong to a different instance of NuBot or not, knowing that we are eating a NuBot wall means that alerts can be sent and custodians made aware.
How extensible is the liquidityinfo function in Nu? could we add some more information such as the exchange name and the order numbers the bot has in place? I'm assuming that liquidityinfo isn;tiincluded in blocks but just exists in the memory pool. What are the limits on the amount of data we can send?

e.g.
{ "B7mmVdVQ1SNNcT9zuQRK1B3Cbvo8vHeoB1" : { "buy" : 55966.68510000, "sell" : 530368.27330000, "exchange" : "bter", "orders" : { "buy_1" : 12345, "buy_2" : 12355, "sell_1" : 23312, "sell_2" : 22123 } }, "B9fv9Di2Y4RxUHzrwjtfq5fetAhqULzWKL" : { "buy" : 54838.31630000, "sell" : 133053.87470000, "exchange" : "poloniex", "orders" : { "buy_1" : 33345, "buy_2" : 34355, "sell_1" : 21233, "sell_2" : 22231 } }, "total" : { "buy" : 133523.37390000, "sell" : 680040.47520000 } }

[Desrever Nu]

I think we should definitely have a service that aggregates the available feeds and weights them as the shareholders see fit

@bshares ^

[Desrever Nu]

How extensible is the liquidityinfo function in Nu?

Thats a question for @sigmike

[Michael Witrant]

The liquidityinfo function is fully flexible. We can put whatever we want in it. But it implies a mandatory upgrade of all the nodes. Without an upgrade of the nodes we may still be able to add more data, but it will propagate very badly.

[Desrever Nu]

@all, after some experiment with the implemented solutions I realised that the current implementation of the above proposed solution will fail in many cases, causing bot-to-bot order filling : a spike or a change that goes undetected for one bot but not the other. A price feed failing just for a few seconds : enough for making one bot using the backup price, not enough for the others to detect the price change.

Imagine the following situation : two bots, identical config, same price feeds, same price checking interval .

The bots will likely eat each other orders out. I suggest we move this item to 0.1.4 so we can start merging the PR and design the required modification.

I would declare 0.1.3 not safe for multiple custodian.

[Ben Rossi]

After discussing this with @desrever , I concur that we should move this functionality to (at least) 0.1.4. We have quite a few major changes in 0.1.3 outside of this enhancement that make it safer for a custodian to run a single market by themselves. Let's get those released and then work on resolving the "bad path" scenarios for this in 0.1.4.

[Sam Griffiths]

I maintain that it would help for the NuBots to be aware of which orders on an exchange belong to another NuBot or even itself. As @sigmike says, this would require another mandatory upgrade of the client. I propose that to try this method out, we create a simple web service that can accept order numbers from a bot, store them and display them upon request to a bot. In this way we can simulate what would be achieved by expanding the liquidityinfo call without having to, potentially unecessarily, upgrade everyone's client.

[Michael Witrant]

If this communication only matters to bots (and I guess it does) it may be better to use another channel than the Nu network. You'll get better performances and flexibility. Maybe just through a message queuing service.

I agree knowing whether you've eaten other bot's wall to issue warnings would be useful. But since it's just a warning and the bot should work even when this information is broken, I think it's not the priority.

@desrever I understand this is an implementation problem and not a design flaw. Am I right? The examples you gave in the charts should work with the design (although it depends what the wall eating delay is).

[Sam Griffiths]

Good point @sigmike . I hadn't considered that the bots would need to continue working even if the communication channel was unavailable.

I have implemented a fix for issue #28 which may alleviate some of the issues pointed out in @desrever s images above. When updating to a new price, a sanity check is carried out against the last known good price. If there is a difference of more than 10% between them, the new price is ignored and the last good price is used instead. That should reduce the amount of occasions when a sudden spike in price could cause bots walls to be eaten as in figure 2.

[Desrever Nu]

I understand this is an implementation problem and not a design flaw. Am I right?

Unless I misinterpreted the design, It looks like a design flaw to me. @sigmike please correct me if I am wrong.

Take the second image I posted : both b1 and b2 read the price from the same feed, every minute (but shifted[***] one from another). b1 detects a price change, b2 doesn't see the spike. So b1 will try to shift his orders (waiting also) while b2 will simply do nothing = orders getting eaten up. I am not sure how the "wait" design decision can prevent the orders to be eaten.

Now, for this very specific case the re is one possible fix that will not require radical changes. Aka, make sure that all the bots reads from the same price feed at the same time, avoiding [***]. This could be accomplished using a central time service (?) .

Thoughts?

[Michael Witrant]

Indeed, but it depends what the delay is. If the delay is longer than the time between 2 price updates, the design would work. b2 would see the lower price and would never move the wall at the height of the spike.

In this example the last triggered shift would only update the target value of the already delayed wall shift. Since there was already a bigger increase planned I think it's ok to not delay again the shift and just update the target price.

The wall would still be eaten if there was repeated spikes that only one bot sees. But that's an extreme situation similar to the bots seeing different prices.

Having the delay longer than the update interval may make the implementation more complex, or maybe even impractical. It depends on the flexibility of the current implementation.

I think it'd be a good thing that custodians agree on a time to fetch the prices. Given the custodians already agree on the price update interval, an easy way to do that without involving them would be to synchronize the updates from a reference time. For example instead of waiting interval seconds between updates you would wait (floor(now / interval) + 1) * interval - now to wait up to the next interval slot (now being the number of seconds since a reference time).

The custodians should make their system use NTP or something so that their system clock is pretty accurate. If we can't rely on that then you could use an NTP client inside the bot.

Another solution that would make everything easier but more centralized would be to use a push price feed service. Bots would connect to it and the service would push the new price everytime it changes. So all the bots would receive the same price at the same time. A message queuing services or even XMPP would work.

[Desrever Nu]

I must admit I am a bit confused by your drawing.

If the delay is longer than the time between 2 price updates

it is. The wait interval is indeed computed and is longer than the price checking interval.

//Compute the waiting time as the strategyInterval + refreshPrice interval + 10 seconda to take down orders
long wait_time = (1000 * (Global.options.getSecondaryPegOptions().getRefreshTime() + Global.options.getExecuteStrategyInterval() + 10));

Still, I don't see how this saves the wall to be eaten unless there is a misunderstanding : are you assuming that, after waiting , the bot reads the price again before placing the order? because I remember discussing it with you on hipchat and we said that it doesn't have to : the price used would be the one at the top of the spike.

And even if we change the implementation to make the bot read the price again after the waiting interval, there are situations where, reading the price again after the wait time, b2 will encounter a price different from the one read by b1. And you probably already spotted this :

The wall would still be eaten if there was repeated spikes that only one bot sees

So, lets look at the alternatives :

1. Sync using NTP client inside the bot
2. Make custodian using the same NTP locally
3. Build a push service

I would probably discard option number #2 because it adds another assumption that is hard to control. And when choosing between 1 and 3, I think that 1) is simpler to implement and can obtain very similar results to 3, in respect to this issue. However long term we can evaluate the push service which will make the bot easier to operate with respect to other concerns.

use a push price feed service.

that is what we are looking at, also proposed by @bshares. Would you still it consider it centralized if we make the service redundant on multiple servers? It will simply pull the data from several external services and act as a funnel, I don't think there is nothing wrong with this approach.

We can make the aforementioned service open-source and accessible by third parties, to prevent people from worring on how the price is computed.

Do you agree that the way to fix it now is 1. Sync using NTP client inside the bot ?

[Desrever Nu]

Java NTP client ready, waiting for confirmation before integrating it with the bot.

[Michael Witrant]

are you assuming that, after waiting , the bot reads the price again before placing the order?

Yes. But the action after this new price is know would be different if there's already a wall shift pending, and the exact action would also depend on whether the previously planed shift is bigger or smaller than the new shift.

because I remember discussing it with you on hipchat and we said that it doesn't have to : the price used would be the one at the top of the spike.

Maybe I don't remember. Anyway the last example you provided made me think it was needed in these some circumstances.

So, lets look at the alternatives

1. Sync using NTP client inside the bot
2. Make custodian using the same NTP locally
3. Build a push service

Using NTP reduces a lot the probability of the problem happening, but it can still happen (in your last example just move the b1 and b2 price reading closer and make the price spike sharper). Using a push service reduces the probability even more, but it can still happen too (it probably requires some kind of network or system failure though).

Would you still it consider it centralized if we make the service redundant on multiple servers?

Somewhat, but it may not be a problem. The question is not so much about the level of decentralization but the consequences of the failure of the remote service and the probability it happens. For example what happens if one of the servers is down, if some packets are lost, if one of the servers sends invalid data, etc.

The answer may be that these events are so unlikely that we can tolerate some walls being eaten under these circumstances. It depends how much a wall eating costs.

If the cost is too high then we should probably go for a more predictable solution where the bots communicate directly and do not take any action until a consensus is clear. Reaching a consensus may be difficult to implement but the situations where a wall is eaten could be clearly defined and would probably only happen when a bot does not behave correctly.

Do you agree that the way to fix it now is 1. Sync using NTP client inside the bot ?

Yes that looks like a good solution if we can tolerate some wall being eaten when a big spike starts just between the request of 2 bots. I guess this event is rare enough, but again it depends on the cost of its consequences.

[Desrever Nu]

After last commit the bots read prices in sync, using an NTP client (with multiple servers as backups) .

[Desrever Nu]

temporarily marked this a resolve. Will open a new issue for mid-long term design changes desired for increased stability.