* stream/buffer: Adds key map optimisation (cherry-pick)
* stream/buffer: Add buffer to DataHandler intermediary. Add field InitialUpdate bool to toggle when first update is seen for initial sync.
* whoops
* buffer: Add difference benchmarks for reference
* glorious: nits (reverting out of context changes)
* RM unused error that will be used later
* purge: benchmark
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* fixes tests
* pair formats and extra fixes
* quick change before shazbert sees
* sneaky lint
* adds bybit mock testing and fixes test
* whoops
* error response instead
* classic forgetting to lint
* bybit live test no longer auto-records results
* ty thrasher- Update exchanges/bybit/bybit_wrapper.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
---------
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Bitstamp: Fix orderbook snapshot blanking LastUpdate
Drive-By: Remove:
`Bitstamp BTC/USDT spot orderbook has zero bid price, filtering.`
It's unactionable and happens every snapshot right now
* Bitstamp: Use ob timestamps for LastUpdate
Obviously.
* SubmitOrder- order.Buy replaced with IsLong
Progresses #1270
* Btcmarkets: order.Buy and order.Sell changed to order.Bid and order.Ask
* Bithumb: SubmitOrder- order.Buy replaced with IsLong
Progresses #1270
* Binanceus: SubmitOrder- order.Buy replaced with IsLong
Progresses #1720
* Bistamp: SubmitOrder- order.Buy replaced with IsLong
Progresses #1270
* Bitfinex: Changed order Side to IsLong
* Binance:switch added to SubmitOrder
* Binance: Replaced Buy and Sell in SubmitOrder with IsLong and IsShort
* Binance:SubmitOrder if statement for order side simplified
* Bitfinex: Fix ws new/modify Order Fields
Fixes Type, Amount and Symbol for new order, and IsShort for modify
order
* Bitfinex: Fix ws order type for margin
Also unifies the pair symbol method to standardised fixCasing
* Fix lint warning on shadow err
* Bitfinex: Remove CustomID duplication for ws
* Kraken: Fix GetActiveOrders AssetType and CancelOrder wrappers
AssetType wasn't set when calling GetActiveOrders and no default case handling was in CancelOrder
* Kraken: Add basic CancelOrder wrapper asset coverage
* Kraken: Add AssetType to GetOrderInfo
* Kraken: Add order status for open orders
* Bitfinex: Order Execution Limits and Testing
* Bitfinex: added proposed changes
* Bitfinex: lint fixes- removed the unnecessary whitespace and replaced the value with indexing
* Bitfinex: Moved data len check before the string conversion, moved the error for the array conversion before the info len check
* Bitfinex: Change GetSiteInfoConfigData to return slice, add tests
* Bitfinex: Fixed lint issue by preallocating pairs
* kraken: fix websocket orderbook processing
* kraken: clean
* glorious: nits also add todo
* more: add required checksum to TODO for future, move variable to top
* glorious: get mad
* orderbook: fix retrieve issue
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Kraken: Load asset limits for pairs
* Kraken: Add test for spot order limit loading
* Kraken: move Limits to UpdateExecutionLimits
* Kraken: Whitespace fixed
* Kraken: added ErrNotYetImplemented for the futures
* Kraken: Fix FetchTradablePairs Loading Limits
* Kraken: changed the error to match the Go 1.13 error package
---------
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
* Kraken: Fix Balances reporting 0 Free
This switches to BalancesEx(tended) to get Held assets so we can
construct a Total, Held and Free.
Without this fix we get 0 balance from GetFree()
Haven't added tests because no construct for mocks and it's private
data. Considering adding mocks a bridge to far for this fix.
* Kraken: Simplify GetBalance
* Okx: Fix panic during shutdown due to race
Error:
```
panic: runtime error: slice bounds out of range [:13340] with capacity 8192
bufio.(*Reader).Read(0x14000038540, {0x1400059cffc?, 0x4?, 0x0?})
/usr/local/go/src/bufio/bufio.go:250 +0x33c
github.com/gorilla/websocket.(*messageReader).Read(0x1400098bb78, {0x1400059cffc, 0x4, 0x4})
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1050 +0x208
io.ReadAll({0x12f36c618, 0x1400098bb78})
/usr/local/go/src/io/io.go:701 +0xe4
io/ioutil.ReadAll(...)
/usr/local/go/src/io/ioutil/ioutil.go:27
github.com/gorilla/websocket.(*Conn).ReadMessage(0x3?)
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1097 +0x54
github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*WebsocketConnection).ReadMessage(0x140006622d0)
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/stream/websocket_connection.go:217 +0x30
github.com/thrasher-corp/gocryptotrader/exchanges/okx.(*Okx).wsFunnelConnectionData(0x140019321e0?, {0x106909450, 0x140006622d0})
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/okx/okx_websocket.go:346 +0x94
created by github.com/thrasher-corp/gocryptotrader/exchanges/okx.(*Okx).WsConnect
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/okx/okx_websocket.go:234 +0x134
exit status 2
```
This happens when there's a race in calls to bufio because it over-reads. See [this comment](https://github.com/golang/go/issues/42289#issuecomment-723393783)
Detected using go -race:
```
WARNING: DATA RACE
Read at 0x00c000818bc0 by goroutine 2156:
github.com/gorilla/websocket.(*Conn).NextReader()
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1000 +0x38
github.com/gorilla/websocket.(*Conn).ReadMessage()
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1093 +0x28
github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*WebsocketConnection).ReadMessage()
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/stream/websocket_connection.go:217 +0x44
github.com/thrasher-corp/gocryptotrader/exchanges/okx.(*Okx).wsFunnelConnectionData()
```
Because we started a new wsFunnelConnectionData for each re-connect.
This bug might apply to other exchanges.
* Okx: Fix websocket waitgroup going negative
Move the waitgroup additions to the actual places that use them
* Okx: Add nolint for revive
* Okx: Move wg Adds to outside goros
There is a risk of a race condition if we let the goros Add themselves.
* Okx: Simplify websocket reading
This fixes the issue that the WsRead and Multiplexer were intrinsically
linked to the websocket, even though they need to survive both
disconnects and Disable/Enables.
Messages are now handled in a goro, which means they might not be
sequential, but there's a very high chance that messages of the same
codepath will be handled sequentially. So orderbook, ticked and order
messages should be sequential
* Okx: Switch to blocking processing of ws msgs
* Okx: Remove nolint from Setup
Actioning a review comment: @gloriousCode prefers to avoid having to nolint this in favour of
a func call return.
* Okx: Remove redundant Wg use inside WsReadData
* Okx: Fix WsMultiplexer Re-Run() shutdown
* SyncManager: Optimise and fixes
This is a fairly invasive change which addresses the amount of work the
sync manager does each cycle and the cycle intervals.
We switch to using discrete locks for each type of work on each pair,
so each worker can take a discrete chunk of work safely.
For performance and simplicity we now use a map for the currencyPairs.
* fix reporting when a websocket is reconnected
* fix not switching REST off after websocket available again
* fix race condition in isProcessing flag
This PR still could go further by avoiding cycling through everything
each time, and by pushing some one-time work for adding enabled pairs
down to a later stage off the hot path.
This was the smallest chunk of refactoring I felt could address
everything without changing too much.
Significant manual testing done with a variety of Timeouts to test for
edgecases and handling.
* SyncManager: Fix ticker/orderbook tracker linked
* SyncManager: Fix sync complete logging in update
* SyncManager: Fix pair format breaking sync key
Kraken seems to always switch to XBT_USDT format, but websockets still
pass around XBTUSDT format. Just to be safe this just removes the
delimiter to avoid any such issues
* SyncManager: Remove unused error
* SyncManager: Remove unused IsProcessing flag
* SyncManager: Fix Update test add() pair format
We had to unify pair format inside sync manager, so test needs to do the
same
* engine: Fix data race on err var
Found with go run -race:
```
WARNING: DATA RACE
Write at 0x00c0005c5740 by goroutine 125: github.com/thrasher-corp/gocryptotrader/engine.(*Engine).Start.func2()
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/engine/engine.go:524 +0x58
Previous write at 0x00c0005c5740 by main goroutine: github.com/thrasher-corp/gocryptotrader/engine.(*Engine).Start()
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/engine/engine.go:576 +0x47e4
```
err should be localised, and that avoids this race and doesn't introduce
any other risk
* engine: Fix linter shadow gripe
* engine: Add nolint for indent-error-flow
These are false positives in revive, fixed in PR#821.
This commit can be reverted once a revive release is cut from master,
which is likely to be the next one
* switch over to package defined const for time layout
* bump appveyor playa
* bumperino to latest while setting patherino
* whoooops
* bump VS version set GOROOT
* puge build cache
* Revert "puge build cache"
This reverts commit 315bb578afc19529457f435e52af2172f5143bc5.
* bumperino to test
* purge setting of golang directory for version and allow default
* purge cache state when file change
* whoops
* thrasher: nits
* don't need to flusherino the cacherino
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
