* Websocket: Remove IsInit and simplify SetProxyAddress
IsInit was basically the same as IsConnected.
Any time Connect was called both would be set to true.
Any time we had a disconnect they'd both be set to false
Shutdown() incorrectly didn't setInit(false)
SetProxyAddress simplified to only reconnect a connected Websocket.
Any other state means it hasn't been Connected, or it's about to
reconnect anyway.
There's no handling for IsConnecting previously, either, so I've wrapped
that behind the main mutex.
* Websocket: Expand and Assertify tests
* Websocket: Simplify state transistions
* Websocket: Simplify Connecting/Connected state
* Websocket: Tests and errors for websocket
* Websocket: Make WebsocketNotEnabled a real error
This allows for testing and avoids the repetition.
If each returned error is a error.New() you can never use errors.Is()
* Websocket: Add more testable errors
* Websocket: Improve GenerateMessageID test
Testing just the last id doesn't feel very robust
* Websocket: Protect Setup() from races
* Websocket: Use atomics instead of mutex
This was spurred by looking at the setState call in trafficMonitor and
the effect on blocking and efficiency.
With the new atomic types in Go 1.19, and the small types in use here,
atomics should be safe for our usage. bools should be truly atomic,
and uint32 is atomic when the accepted value range is less than one byte/uint8 since
that can be written atomicly by concurrent processors.
Maybe that's not even a factor any more, however we don't even have to worry enough to check.
* Websocket: Fix and simplify traffic monitor
trafficMonitor had a check throttle at the end of the for loop to stop it just gobbling the (blocking) trafficAlert channel non-stop.
That makes sense, except that nothing is sent to the trafficAlert channel if there's no listener.
So that means that it's out by one second on the trafficAlert, because any traffic received during the pause is doesn't try to send a traffic alert.
The unstopped timer is deliberately leaked for later GC when shutdown.
It won't delay/block anything, and it's a trivial memory leak during an infrequent event.
Deliberately Choosing to recreate the timer each time instead of using Stop, drain and reset
* Websocket: Split traficMonitor test on behaviours
* Websocket: Remove trafficMonitor connected status
trafficMonitor does not need to set the connection to be connected.
Connect() does that. Anything after that should result in a full
shutdown and restart. It can't and shouldn't become connected
unexpectedly, and this is most likely a race anyway.
Also dropped trafficCheckInterval to 100ms to mitigate races of traffic
alerts being buffered for too long.
* Websocket: Set disconnected earlier in Shutdown
This caused a possible race where state is still connected, but we start
to trigger interested actors via ShutdownC and Wait.
They may check state and then call Shutdown again, such as
trafficMonitor
* Websocket: Wait 5s for slow tests to pass traffic draining
Keep getting failures upstream on test rigs.
Think they can be very contended, so this pushes the boundary right out
to 5s
* CI/build: Update Go version, linters and fix minor issues
* Bump golangci-lint to v1.56.1
* BinanceUS: Make uint usage consistent
* Throw blank identifiers into the trash
* Websockets: Move Subscription to its own package
This allows the small type to be imported from both `config` and from
`stream` without an import cycle, so we don't have to repeat ourselves
* Subs: Renamed Currency to Pair
This was being mis-used through much of the code, and since we're
already touching everything, we might as well fix it
* Websockets: Add Subscription configuration
* Binance: Add subscription configuration
* Kucoin: Subscription configuration
* Simplify GenerateDefaultSubs
* Improve TestGenSubs coverage
* Test Candle Sub generation
* Support Candle intervals
* Full responsibility for formatting Channel name on GenerateDefaultSubs
OR consumer of Subscribe
* Simplify generatePayloads as a result
* Fix test coverage of asset types in processMarketSnapshot
* Exchanges: Abstract ParallelChanOp
* Tests: Generic ws mock instances
* Kucoin: Fix intermittent conflict in test currs
Use isolated test instance for `TestGetOpenInterest`.
`TestGetOpenInterest` would occassionally change pairs before
GenerateDefault Subs.
* Websockets: Add keys to websocket subscriptions
* This switches all RO uses of the mutex to use a RLock method.
* The mutex used for discrete field access has had scope drift from
name 'connectionMutex' so rename to more appropriate fieldsMutex
* The mutex used for Set/CanUseAuthEndpoints moves from the
subscriptions endpoint to the fieldsMutex
* Add GetSubscription by key
* Expose stream.Matcher type
* Bitfinex: Subscribe and Unsubscribe atomicly
* Fix Auth failures ignored
* This change makes it so that Subscribe and Unsubscribe wait for success
** Tells the DataHandler about errors
** Errors are returned to consumers
* Subscribes concurrently to the channels
* It also simplifies the chanId to stream mapping
* Removes unable to locate chanID: %d errors which are just noise
* Paves the way for unified channelSubscription id handling
* Adds support for subId for Book subscriptions, which is more robust
* Vastly simplifies what we need to test TestWsSubscribedResponse
This test was working to ensure that the various fancy key parsing
mechanisms all worked. Now that we use subId, we just need a thorough
test of that
* Expose Match.Set in order to capture websocket incoming data
Can't see another way of doing this. Doesn't seem too bad
* Allow tests to run with auth or WS
These flags made it difficult to run the tests whilst working on
websockets
* Enable API auth and WS in testconfig
This change minimises the changes requires for a full test run against
live endpoints, so that new contributors have a clearer testing path.
I cannot see any reason to turn WS off and Auth endpoints off when we're
not going to run API tests without Creds being set, and we're not going
to do live fire tests without canManipulateRealOrders
* TestWsSubscribe and various fixes
** Enables the websocket for live non-authed integration tests by default
** Adds an integration test for subscriptions
** Changes the Ws tests to respect canManipulateRealOrders
** Uses WsConnect instead of setupWs; fixes seqNo config not sent for WS tests
** Allows api creds to live in config/testdata.json which might be
less likely to accidentally commit, and less obtrusive
* Bitfinex: Support period and timeframe for Candles
* Fixes manual Subscribe() symbol or key formatting
* Unifies handling of params for DefaultSubscriptions and manual
subsrciptions
* Bitfinex: Handle conf and info WS channel events
* Bitfinex: Better tests for subscriptions
* fixup! Websockets: Add keys to websocket subscriptions
* fixup! Bitfinex: Subscribe and Unsubscribe atomicly
* fixup! Websockets: Add keys to websocket subscriptions
* Websockets: Add Pending subscription status
Add a status tracker so that Sub/Unsub can prevent duplicates,
and also fixes when first message comes before we have added the sub
to the tracker
* Websockets: Add State instead of pending
This change allows more clarity about the current state and
checks for specifically already Unsubing
* Bitfinex: Fix first sub message maybe lost
The only link we have between a sub req and the sub resp is the subID.
And the only link we have between a sub message and the sub is the chanID.
We can't derive a link using Pair or anything else.
This meant that by sending the resp and its chanID down the IncomingData
channel, we allowed the channel reader to maybe process the next
message, the first message on the channel, before the runtime executed
the switch back to subscribeToChan waiting on the chan.
To fix this, we key initially on subId.(string), and then replace it
with chanId.(int64) when we have it *inside* the wsHandleData so we
know we've procedurally handled it before the next message.
subscribeToChan is then free to remove the subId keyed Sub regardless of
error or not
If there's an error, we don't need to inline handling because there
won't be any second update.
Expands test coverage to make sure those subId keyed subscriptions are
removed.
* Websocket: Validate state in SetChanState
* fixup! Bitfinex: Fix first sub message maybe lost
* Websockets: Rename RemoveUnsuccessfulSubs
Implementation doesn't imply Unsuccessful or need to.
This change supports the registering of Pending subs
* Bitfinex: Fix race in Tests
* okx: books 5 (cherry-pick)
* okx: shift types to types file, remove commented code and updated field name to better reflect pushed type
* linter: fix
* remove slowness
* * Introduce function checksubscriptions and shift check of subscriptions to internal websocket package
* Shift Max websocket connection int to Websocket setup (temp) for this use case only.
* glorious: nits
* linter: fix
* websocket: don't try and subscribed with nothing to subscribe to.
* Update exchanges/stream/websocket_test.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* glorious: nits
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Okx: Fix WS order fields
* Fixes float64 with string annotation erroring on empty strings:
Okx Order Push Data error json: invalid use of ,string struct tag, trying to unmarshal "" into float64
Specifically this came from px field from a market order
* Switch to convert.StringToFloat64 instead of okxNumericalValue
* Fix typo in Notional* field names; Ironically prevented them from erroring
* Okx: Add tests for first order fields
* Okx: CID and maybe set WS order Filled time
* Tests: Set TestFixtureToDataHandler to t.Helper
* Orders: Add UnmarshalJSON to order.Side
* Okx: Fix FillTime not parsed for PendingOrder
* Okx: Switch to order.Side Unmarshal throughout
* Okx: Add Fee and FeeAsset to order processing
* Okx: Fix WS order.Detail amounts and Test
This fixes Amount vs QuoteAmount for market sells where tgtCcy is
quote_ccy
* Add comment to order.Side.UnmarshalJSON
* Okx: Replace PendingOrderItem Unmarshal with local types
* Okx: string type for WS order reduceOnly
Note: Not yet in unit tests, since it's not part of the spot tests I was
originally fixing. I'll circle back to adding full test support for
Reduce only and deleveraging positions.
* Okx: Fix TestOrderPushData Amount
We were expecting 0 when we're given a quoteAmount
In reality, we'll calculate the size from the price
* Okx: Fix order and remAmount in wsOrders
Improved handling for Float64 issues and boundaries when the order is
fully executed but not yet marked as Filled
* Fix ErrSideIsInvalid in tests
* Okx: Remove redundant asset types from conversion
* Okx: Fix list of private channels for subscription
* Okx: Fix ws order update Acc and Rem amounts
The amounts aren't treated as deltas by anything receiving them
upstream, e.g. orderManager or wsRoutineManager.
So we should treat them the same as the REST implementation
* Okx: Add context to stream unmarshall errors
I encountered a series of errors with unmarshalling from okx and have no
idea what the messages which weren't valid json actually were.
This fix adds the raw message for context to the error
* Okx: Reduce ping interval
Seeing occassional disconnects and suspect dropping the ping interval
down might help
* Fix ClientSupplier and ClientSupplied inconsist
This simply removes the Supplie. part in favour of the simpler name used
in binance. Seems safe since its already inconsistent and not populated
in at least ws order handlin
* Okx: Fix ClientOrderID ignored in ws orders
* Okx: Standardise Rfq abbrev; Fix RfqIDs
* Okx: Fix RfqIds outside of ClientRfqIds
* Okx: Simplify TestUpdateOrderExecutionLimits
* Okx: Fix panic on no avgPrice and QuoteAmount
* Okx: Fix panic during shutdown due to race
Error:
```
panic: runtime error: slice bounds out of range [:13340] with capacity 8192
bufio.(*Reader).Read(0x14000038540, {0x1400059cffc?, 0x4?, 0x0?})
/usr/local/go/src/bufio/bufio.go:250 +0x33c
github.com/gorilla/websocket.(*messageReader).Read(0x1400098bb78, {0x1400059cffc, 0x4, 0x4})
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1050 +0x208
io.ReadAll({0x12f36c618, 0x1400098bb78})
/usr/local/go/src/io/io.go:701 +0xe4
io/ioutil.ReadAll(...)
/usr/local/go/src/io/ioutil/ioutil.go:27
github.com/gorilla/websocket.(*Conn).ReadMessage(0x3?)
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1097 +0x54
github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*WebsocketConnection).ReadMessage(0x140006622d0)
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/stream/websocket_connection.go:217 +0x30
github.com/thrasher-corp/gocryptotrader/exchanges/okx.(*Okx).wsFunnelConnectionData(0x140019321e0?, {0x106909450, 0x140006622d0})
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/okx/okx_websocket.go:346 +0x94
created by github.com/thrasher-corp/gocryptotrader/exchanges/okx.(*Okx).WsConnect
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/okx/okx_websocket.go:234 +0x134
exit status 2
```
This happens when there's a race in calls to bufio because it over-reads. See [this comment](https://github.com/golang/go/issues/42289#issuecomment-723393783)
Detected using go -race:
```
WARNING: DATA RACE
Read at 0x00c000818bc0 by goroutine 2156:
github.com/gorilla/websocket.(*Conn).NextReader()
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1000 +0x38
github.com/gorilla/websocket.(*Conn).ReadMessage()
/Users/gbjk/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1093 +0x28
github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*WebsocketConnection).ReadMessage()
/Users/gbjk/go/pkg/mod/github.com/gbjk/gocryptotrader@v0.0.0-20230619070715-ae6f283f6be6/exchanges/stream/websocket_connection.go:217 +0x44
github.com/thrasher-corp/gocryptotrader/exchanges/okx.(*Okx).wsFunnelConnectionData()
```
Because we started a new wsFunnelConnectionData for each re-connect.
This bug might apply to other exchanges.
* Okx: Fix websocket waitgroup going negative
Move the waitgroup additions to the actual places that use them
* Okx: Add nolint for revive
* Okx: Move wg Adds to outside goros
There is a risk of a race condition if we let the goros Add themselves.
* Okx: Simplify websocket reading
This fixes the issue that the WsRead and Multiplexer were intrinsically
linked to the websocket, even though they need to survive both
disconnects and Disable/Enables.
Messages are now handled in a goro, which means they might not be
sequential, but there's a very high chance that messages of the same
codepath will be handled sequentially. So orderbook, ticked and order
messages should be sequential
* Okx: Switch to blocking processing of ws msgs
* Okx: Remove nolint from Setup
Actioning a review comment: @gloriousCode prefers to avoid having to nolint this in favour of
a func call return.
* Okx: Remove redundant Wg use inside WsReadData
* Okx: Fix WsMultiplexer Re-Run() shutdown
* initial concept of a nice validation tester for exchanges
* adds some datahandler design
* expand testing
* more tests and fixes
* minor end of day fix for bithumb
* fixes implementation issues
* more test coverage and improvements, but not sure if i should continue
* fix more wrapper implementations
* adds error type, more fixes
* changes signature, fixes implementations
* fixes more wrapper implementations
* one more bit
* more cleanup
* WOW things work?
* lintle 1/1337
* mini bump
* fixes all linting
* neaten
* GetOrderInfo+ asset pair fixes+improvements
* adds new websocket test
* expand ws testing
* fix bug, expand tests, improve implementation
* code coverage of a lot of new codes
* fixes everything
* reverts accidental changes
* minor fixes from reviewing code
* removes Bitfinex cancelBatchOrder implementation
* fixes dumb baby typo for babies
* mini nit fixes
* so many nits to address
* addresses all the nits
* Titlecase
* switcheroo
* removes websocket testing for now
* fix appveyor, minor test fix
* fixes typo, re-kindles killed kode
* skip binance wrapper tests when running CI
* expired context, huobi okx fixes
* kodespull
* fix ordering
* time fix because why not
* fix exmo, others
* hopefully this fixes all of my life's problems
* last thing today
* huobi, more like hypotrophy
* golangci-lint, more like mypooroldknee-splint
* fix huobi times by removing them
* should fix okx currency issues
* blocks the application
* adds last little contingency for pairs
* addresses most nits and new problems
* lovely fixed before seeing why okx sucks
* fixes issues with okx websocket
* the classic receieieivaier
* lintle
* adds test and fixes existing tests
* expands error handling messages during setup
* fixes dumb okx bugs introduced
* quick fix for lint and exmo
* fixes nixes
* fix exmo deposit issue
* lint
* fixes issue with extra asset runs missing
* fix surprise race
* all the lint and merge fixes
* fixes surprise bugs in OKx
* fixes issues with times and chains
* fixing all the merge stuff
* merge fix
* rm logs and a panic potential
* lovely lint lament
* an easy demonstration of scenario, but not of initial purpose
* put it in the bin
* Revert "put it in the bin"
This reverts commit 15c6490f713233d43f10957367fcbf18e3818bdd.
* re-add after immediate error popup
* fix mini poor test design
* okx okay
* merge fixes
* fixes issues discovered in lovely test
* I FORGOT TO COMMIT THIS
* nit fixaroonaboo
* forgoetten test fix
* revert old okx asset intrument work
* fixes
* revert problems I didnt understand. update bybit
* fix merge bugs
* test cleanup
* further improvements
* reshuffle and lint
* rm redundant CI_TEST by rm the CI_TEST field that is redundant
* path fix
* move to its own section, dont run on 32 bit + appveyor
* lint
* fix lbank
* address nits
* let it rip
* fix failing test time range
* niteroo boogaloo
* mod tidy, use common.SimpleTimeFormat
Ping handler for okx is not the RFC compat version gorilla would
support. Even if it was, I think the UseGorillaHandler is
broken/misconceived because it does not send keepalive pings, it responds to them.
This also increases the delay to 27, in keeping with the 30 second
timeout documented.
I think this issue was hidden by the noisy default subs.
When just using funding-rates, it showed straight away.
* common: adjust common error slice to allow multi errors.Is matching and conform to interface better
* zb: forgot to save?
* linties: fixies
* linties: word change as well.
* nitters: glorious
* buts
* nitters: fix glorious bug
* Update common/common.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* nitters: shifty
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* public endpoints methods added
* Completing mapping REST endpoints
* Binanceus Wrapper methods -Partially
* Binanceus Wrapper methods -Partially
* BinaWra functions with test funs; Not Completed
* Finalizing wrapper methods & test
* Fix & Complete wrapper functions
* Finalizing wrapper methods & test
* Adding Stream Datas
* WS Test functions
* CI: Fix golangci-lint linter issues
* CI: Fix reverting unnessesary changes and type conversion issues
* CI: Fix reverting unnessesary changes and type conversion issues
* Adding Public endpoints and tests
* Adding Market and Public Endpoints
* Adding Public endoints
* Public Trading Endpoints & Authenticated Trade order methods
* Adding Authenticated Methods and Tests
* Adding algo and Funding Authenticated endpoints
* Adding funding trading endpoints and correspondint tests
* adding authenticated endpoints
* Completing Block Trading endpoints and added subaccount endpoints
* Completing sub account and grid Trading endpoints
* Adding Rate Limit and missing endpoints
* Wrapper and Websocket handlers
* Fixing Websocket Test and Push Data Handler Issues
* Fixing Websocket Test and Push Data Handler Issues
* Fixing linter issues, package dependency, and other slight tempos
* Fixing linter and slight tempos
* Update on test functions, and Rest and Websocket Endpoint handlers
* Remove okex, adding comments, and slight fixes on endpoints.
* Fixing linter issues and adding comments
* Slight code changes, updating documentation, and n and linter issues
* Fix context and configuration endpoint issues
* slight fixes on config and test files
* adding some missing test and fix linter issues
* fix linter issue
* Slight fixes on code structure, shorthand exp,and ot and other
* Fix slight linter issue
* Slight code fixes and fixing linter issues
* fixing linter iissues
* fixing linter iissues
* slight linter issue fix
* slight linter issue fix
* Fix on models, type convert funcs and endpoints
* Adding Error messages map and update of models
* Fix on error message string and linter issues
* Fix slight linter issue
* Fix slight linter issue
* Fixing type converts, models, and linter issues
* Adding Ws fixes
* Slight fix on websocket and other issues
* Adding slight websocket fixes
* Remove 'books5' channel default subscription
* Small changes on default subscription and checksum
* Fix slight websocket tempos
* Fix Wrapper function tempost and linter issues
* Resolving slight naming and other issues
* Resolve slight pointer issues
* resolve slight linter issues
* Resolve config files issue
* Update websocket and wrapper funcs with test and docs
* fixs on websocket multiplexer, types, and other slight issues
* fix slight linter issues
* slight update on web-socket orderbook and tickers
* fix slight issues and websocket runtime errors
* Slight unit test fix and assing simple semaphore
* FIx race issue
* Update on authenticated endpoints
* Fix wsSetupRun check in websocket 'setupWsAuth' func
* Update wsSetupRun check in websocket 'setupWsAuth' func
* Slight update on websocket handling
* Fix some race conditions
* fix slight tempos
* fix authenticated test issues
* Update on conditional statements
* slight update on unit test
* fix unit test tempos
* Fix slight tempos
* Change check map from struct valued to bool valued
* slight fix trial
* Slight unit test update
* Fix websocket timeout error
* Updating websocket subscription endpoints, and unit tests
* update unit tests
* Slight issue on wrapper method 'GetActiveOrders'
* Overall code update
* Addressing missing review comments
* Fix unit test tempo and linter issue
* Monor fix
* Slight update
* Slight unit test fix
* Slight fixes
* Slight fixes
* Fixing on missing review comments
* Adding WS Fixes
* slight fix
* Monor fix on unit test
* Minor convert issue
* Minor change on WS
* Monor logic fix
* Fix code structure and logic issues
* Fixing small typos
* fix slight data format issue
* Update on trade and order wrapper methods
* Adding slight update
* fix on order detail
* Slight update on FetchTradablePairs wrapper method
* Slight update on wrapper
* Update on deserialization and other slight issues
* Final update
* Resolve missing review comments
* Slight update on config and unit test
* minor fix on GetDepositAddress param
* Minor fix
