* Kraken: Load asset limits for pairs
* Kraken: Add test for spot order limit loading
* Kraken: move Limits to UpdateExecutionLimits
* Kraken: Whitespace fixed
* Kraken: added ErrNotYetImplemented for the futures
* Kraken: Fix FetchTradablePairs Loading Limits
* Kraken: changed the error to match the Go 1.13 error package
---------
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
* Kraken: Fix Balances reporting 0 Free
This switches to BalancesEx(tended) to get Held assets so we can
construct a Total, Held and Free.
Without this fix we get 0 balance from GetFree()
Haven't added tests because no construct for mocks and it's private
data. Considering adding mocks a bridge to far for this fix.
* Kraken: Simplify GetBalance
* Okx: Fix panic during shutdown due to race
Error:
```
panic: runtime error: slice bounds out of range [:13340] with capacity 8192
bufio.(*Reader).Read(0x14000038540, {0x1400059cffc?, 0x4?, 0x0?})
/usr/local/go/src/bufio/bufio.go:250 +0x33c
github.com/gorilla/websocket.(*messageReader).Read(0x1400098bb78, {0x1400059cffc, 0x4, 0x4})
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1050 +0x208
io.ReadAll({0x12f36c618, 0x1400098bb78})
/usr/local/go/src/io/io.go:701 +0xe4
io/ioutil.ReadAll(...)
/usr/local/go/src/io/ioutil/ioutil.go:27
github.com/gorilla/websocket.(*Conn).ReadMessage(0x3?)
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1097 +0x54
github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*WebsocketConnection).ReadMessage(0x140006622d0)
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/stream/websocket_connection.go:217 +0x30
github.com/thrasher-corp/gocryptotrader/exchanges/okx.(*Okx).wsFunnelConnectionData(0x140019321e0?, {0x106909450, 0x140006622d0})
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/okx/okx_websocket.go:346 +0x94
created by github.com/thrasher-corp/gocryptotrader/exchanges/okx.(*Okx).WsConnect
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/okx/okx_websocket.go:234 +0x134
exit status 2
```
This happens when there's a race in calls to bufio because it over-reads. See [this comment](https://github.com/golang/go/issues/42289#issuecomment-723393783)
Detected using go -race:
```
WARNING: DATA RACE
Read at 0x00c000818bc0 by goroutine 2156:
github.com/gorilla/websocket.(*Conn).NextReader()
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1000 +0x38
github.com/gorilla/websocket.(*Conn).ReadMessage()
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1093 +0x28
github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*WebsocketConnection).ReadMessage()
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/stream/websocket_connection.go:217 +0x44
github.com/thrasher-corp/gocryptotrader/exchanges/okx.(*Okx).wsFunnelConnectionData()
```
Because we started a new wsFunnelConnectionData for each re-connect.
This bug might apply to other exchanges.
* Okx: Fix websocket waitgroup going negative
Move the waitgroup additions to the actual places that use them
* Okx: Add nolint for revive
* Okx: Move wg Adds to outside goros
There is a risk of a race condition if we let the goros Add themselves.
* Okx: Simplify websocket reading
This fixes the issue that the WsRead and Multiplexer were intrinsically
linked to the websocket, even though they need to survive both
disconnects and Disable/Enables.
Messages are now handled in a goro, which means they might not be
sequential, but there's a very high chance that messages of the same
codepath will be handled sequentially. So orderbook, ticked and order
messages should be sequential
* Okx: Switch to blocking processing of ws msgs
* Okx: Remove nolint from Setup
Actioning a review comment: @gloriousCode prefers to avoid having to nolint this in favour of
a func call return.
* Okx: Remove redundant Wg use inside WsReadData
* Okx: Fix WsMultiplexer Re-Run() shutdown
* SyncManager: Optimise and fixes
This is a fairly invasive change which addresses the amount of work the
sync manager does each cycle and the cycle intervals.
We switch to using discrete locks for each type of work on each pair,
so each worker can take a discrete chunk of work safely.
For performance and simplicity we now use a map for the currencyPairs.
* fix reporting when a websocket is reconnected
* fix not switching REST off after websocket available again
* fix race condition in isProcessing flag
This PR still could go further by avoiding cycling through everything
each time, and by pushing some one-time work for adding enabled pairs
down to a later stage off the hot path.
This was the smallest chunk of refactoring I felt could address
everything without changing too much.
Significant manual testing done with a variety of Timeouts to test for
edgecases and handling.
* SyncManager: Fix ticker/orderbook tracker linked
* SyncManager: Fix sync complete logging in update
* SyncManager: Fix pair format breaking sync key
Kraken seems to always switch to XBT_USDT format, but websockets still
pass around XBTUSDT format. Just to be safe this just removes the
delimiter to avoid any such issues
* SyncManager: Remove unused error
* SyncManager: Remove unused IsProcessing flag
* SyncManager: Fix Update test add() pair format
We had to unify pair format inside sync manager, so test needs to do the
same
* engine: Fix data race on err var
Found with go run -race:
```
WARNING: DATA RACE
Write at 0x00c0005c5740 by goroutine 125: github.com/thrasher-corp/gocryptotrader/engine.(*Engine).Start.func2()
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/engine/engine.go:524 +0x58
Previous write at 0x00c0005c5740 by main goroutine: github.com/thrasher-corp/gocryptotrader/engine.(*Engine).Start()
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/engine/engine.go:576 +0x47e4
```
err should be localised, and that avoids this race and doesn't introduce
any other risk
* engine: Fix linter shadow gripe
* engine: Add nolint for indent-error-flow
These are false positives in revive, fixed in PR#821.
This commit can be reverted once a revive release is cut from master,
which is likely to be the next one
* switch over to package defined const for time layout
* bump appveyor playa
* bumperino to latest while setting patherino
* whoooops
* bump VS version set GOROOT
* puge build cache
* Revert "puge build cache"
This reverts commit 315bb578afc19529457f435e52af2172f5143bc5.
* bumperino to test
* purge setting of golang directory for version and allow default
* purge cache state when file change
* whoops
* thrasher: nits
* don't need to flusherino the cacherino
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* initial concept of a nice validation tester for exchanges
* adds some datahandler design
* expand testing
* more tests and fixes
* minor end of day fix for bithumb
* fixes implementation issues
* more test coverage and improvements, but not sure if i should continue
* fix more wrapper implementations
* adds error type, more fixes
* changes signature, fixes implementations
* fixes more wrapper implementations
* one more bit
* more cleanup
* WOW things work?
* lintle 1/1337
* mini bump
* fixes all linting
* neaten
* GetOrderInfo+ asset pair fixes+improvements
* adds new websocket test
* expand ws testing
* fix bug, expand tests, improve implementation
* code coverage of a lot of new codes
* fixes everything
* reverts accidental changes
* minor fixes from reviewing code
* removes Bitfinex cancelBatchOrder implementation
* fixes dumb baby typo for babies
* mini nit fixes
* so many nits to address
* addresses all the nits
* Titlecase
* switcheroo
* removes websocket testing for now
* fix appveyor, minor test fix
* fixes typo, re-kindles killed kode
* skip binance wrapper tests when running CI
* expired context, huobi okx fixes
* kodespull
* fix ordering
* time fix because why not
* fix exmo, others
* hopefully this fixes all of my life's problems
* last thing today
* huobi, more like hypotrophy
* golangci-lint, more like mypooroldknee-splint
* fix huobi times by removing them
* should fix okx currency issues
* blocks the application
* adds last little contingency for pairs
* addresses most nits and new problems
* lovely fixed before seeing why okx sucks
* fixes issues with okx websocket
* the classic receieieivaier
* lintle
* adds test and fixes existing tests
* expands error handling messages during setup
* fixes dumb okx bugs introduced
* quick fix for lint and exmo
* fixes nixes
* fix exmo deposit issue
* lint
* fixes issue with extra asset runs missing
* fix surprise race
* all the lint and merge fixes
* fixes surprise bugs in OKx
* fixes issues with times and chains
* fixing all the merge stuff
* merge fix
* rm logs and a panic potential
* lovely lint lament
* an easy demonstration of scenario, but not of initial purpose
* put it in the bin
* Revert "put it in the bin"
This reverts commit 15c6490f713233d43f10957367fcbf18e3818bdd.
* re-add after immediate error popup
* fix mini poor test design
* okx okay
* merge fixes
* fixes issues discovered in lovely test
* I FORGOT TO COMMIT THIS
* nit fixaroonaboo
* forgoetten test fix
* revert old okx asset intrument work
* fixes
* revert problems I didnt understand. update bybit
* fix merge bugs
* test cleanup
* further improvements
* reshuffle and lint
* rm redundant CI_TEST by rm the CI_TEST field that is redundant
* path fix
* move to its own section, dont run on 32 bit + appveyor
* lint
* fix lbank
* address nits
* let it rip
* fix failing test time range
* niteroo boogaloo
* mod tidy, use common.SimpleTimeFormat
* Add test for dispatch channel reuse
* Dispatcher - make chans bidirectional
* No need to to keep the type assertion separate from the Get()
* Unexport Pipe's channel and add getter
* Okx: Fix Instrument unmarshal
This fixes: `json: invalid use of ,string struct tag, trying to unmarshal "" into float64`
when TwapSz or other fields are empty, which happened briefly today.
As a DriveBy it also simplifies the exposed Instrument type to native
types only.
* Okx: Add missing InstrumentFamily to Instrument
* Okx: Add Instrument Unmarshal tests
DriveBy: Removes stray newline in type conversions
* Okx: Fix empty line upsetting linter
* orders: Fix manager not processing immediately
This ensures that orders are processed as soon as orderManager claims to have started.
Specifically this will fix an eager call to CancelAllOrders not finding
anything until after the first Interval run.
* orders: Fix reset timer call
This fix is small niggle.
The existing code was more complicated than it needed to be, whilst also
ignoring the return of timer.Reset and the conditional need to drain the
channel. It's unlikely but possible during transient resource
starvation.
* OrderManager: Simplify startup process
This simplifies the sequence and Adds to the wg first.
We won't bail from the run() during any type of processOrders, and
that's something we can fix later
* bitfinex: filter security pairs from spot trading
* asset: add securities type
* bitfinex: remove print
* linter: fix
* gomod tidy
* glorious: nits
* bitfinex: just some orderbook fun and added a pass through for bitfinex funding rate when rate == 0
* bitfinex: move update function from testmain to localised test
* test: fix up
* glorious:nits
* bitfinex; fix casing test
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
