NuBot needs a system to prevent orders to be placed at very low prices

[Desrever Nu]

a "0" price is automatically rejected by all exchanges. For USD, the price is always the result of a sum operation so I don't see how it could fail.

For non-USD prices, that's more tricky. The price is computed every time it needs to, introducing possible (yet unlikely faults). The bot reads the price from multiple feeds at the time and execute a sanity check, such that if the price is more than 10% off from the average of the others, it will discard the feed.

However, if all the feeds fails coherently at the same time (very very unlikley but still... ) the bot has no way to determine that the price he has been given is "wrong".

I could and should add an additional control based on recent history of orders and prevent place orders that are more than some % points off.

[Sam Griffiths]

I have implemented a potential fix in branch feature/28 in PriceMonitorTriggerTask.java there is a method called updateLastPrice(). This is called when a new price is found and sets it as the new price. I have added a sanity check here to compare the new value against the last known good value. if it differs by more than 10%, the last known good price is treated as the new price and the update uses that value instead. i'm not totally sure that this is the best approach but it is a start. Comments would be welcome

[Desrever Nu]

Very good start. So the only dangerous price now is the first one. Any idea on how we could make sure that the first price read is not ridiculously below market?

I have some ideas but I don't want to bias

[Desrever Nu]

after reading your comment on #11,

I think we should review this approach. There is no way to distinguish a spike (price going quickly back to average) from a sudden price drop/rise. So I am afraid that the intervention on issue #28 can make harm when price have a sudden change : if the price doesn't go back within the 10%, the bot will never update the price again, lagging behind and causing arbitrage fun.

[Sam Griffiths]

Good point. we need to track the existing price as well as the updated prices. I will have a re-think on this.

[Sam Griffiths]

Second Attempt

A fixed length FIFO queue of prices now exists, the average of which gives us a moving average.
Each time a new potential price is detected, it is compared to the moving average to date.
If the price is >10% different it is added to the moving average queue which alters the moving average in it's favour.
the price information is then re-requested from the feed providers without updating the price or moving the wall.
Only when the potential price is within the 10% boundary is the price updated and the wall shifted.

Very large and sudden changes in price could mean that the walls are slow to move as the moving average would need time to catch up and be within 10% of the new price.
Conversely, when the walls do move, the new price has been fairly stable for a few minutes which should give other NuBots time to have adjusted too.

Thoughts on this approach?

I considered using latest trade feeds from the exchange to follow the market trend but this is reliant on the trade ticker methods being implemented in the exchange wrappers.
Mostly these aren't as yet. I think this would be a valuable addition to this logic but the necessary work needs to be carried out on the wrappers.

[Desrever Nu]

After our discussion on HipChat, it emerged that this approach could work.

Optimally it still needs two main addition:

1. A timeout to be sure that the total execution time of the PriceTracking task never gets > 60 seconds
2. On startup, fill the MA before placing new orders. And expose an option.json test parameter "bypass-control"(boolean) to speed up testing and avoid waiting for it =)

[Sam Griffiths]

Both of these suggestions have been implemented.
The method which gets the prices from the exchanges will automatically fill the moving average queue if it ever has less than the required number of data points. It uses every data point from every exchange it contacts to speed the process up and give a fairer average.
A timer is started whenever the price feeds are updated. If this timer gets to within ~1 second of the time when an automated price feed update would have taken place, We assume that there is some error with the price feeds and take action.
The action is currently to send notifications of the issue via hipchat and email, cancel all the orders to avoid arbitrage against the bot and to shutdown the bot.
All numeric parameters for this approach are held in variables (required percentage, timeout, closeness to timeout to trigger action.) It is hoped that these vales can be tweaked after experiencing some real world scenarios so that the action becomes a rarity (if it ever occurs at all - As there is no pause between moving the average figure and re-fetching the currency data, I would hope that the Moving average would close to within the required percentage of the incoming price well within the time out period. It does need real world experience though).

[Sam Griffiths]

Tested the new methodology by spoofing the btce feed with a file stored on localhost. the log output of the test is here: http://pastebin.com/3VZ78BWv. During the test I hiked the price from ~$300 to ~$400, then to ~$4000 then to ~$4000000000 (through a few iterations). Each time the moving average moved to within 10% of the incoming price within a number of seconds. During the last price hike, the bot started off here:

Nov 12, 2014 10:55:04 AM com.nubits.nubot.tasks.PriceMonitorTriggerTask updateLastPrice
WARNING: Latest price 4.444444482975E9 is -199.96000053289023% outside of the moving average of 444482.97499999974.
Shifting moving average and re-fetching exchange rate data.
Nov 12, 2014 10:55:04 AM com.nubits.nubot.tasks.PriceMonitorTriggerTask executeUpdatePrice
INFO:
GETTING PRICES
MA = 1.4857781630833334E8

and ended up here:

Nov 12, 2014 10:55:04 AM com.nubits.nubot.tasks.PriceMonitorTriggerTask updateLastPrice
WARNING: Latest price 4.444444482975E9 is -10.525207666015433% outside of the moving average of 4.000044482975002E9.
Shifting moving average and re-fetching exchange rate data.
Nov 12, 2014 10:55:04 AM com.nubits.nubot.tasks.PriceMonitorTriggerTask executeUpdatePrice
INFO:
GETTING PRICES
MA = 4.1481778163083353E9

I know that when dealing with real exchanges there will be network latency to deal with but I feel confident that the size of price hike seen here will never happen in the real world so the edge case where the bot has to clear orders and shutdown will never happen.

[Sam Griffiths]

Tested at length on the internal exchange. Bot functions as normal and reacts to price changes. Feel that this could be merged into develop but don;t want to merge my own issue

[Desrever Nu]

The solution proposed and implemented comply with the requirements.

External QA needed before release on 0.1.4

Merged into develop

https://bitbucket.org/JordanLeePeershares/nubottrading/commits/16aa49bb0bb3be228ad324d34fdfcad7cd04531a