* gateio: Add multi asset websocket support WIP.
* meow
* Add tests and shenanigans
* integrate flushing and for enabling/disabling pairs from rpc shenanigans
* some changes
* linter: fixes strikes again.
* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.
* Add subscription tests (state functional)
* glorious:nits + proxy handling
* Spelling
* linter: fixerino
* instead of nil, dont do nil.
* clean up nils
* cya nils
* don't need to set URL or check if its running
* stop ping handler routine leak
* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response
* Allow rollback on connect on any error across all connections
* fix shadow jutsu
* glorious/gk: nitters - adds in ws mock server
* linter: fix
* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.
* gk: nits
* Leak issue and edge case
* gk: nits
* gk: drain brain
* glorious: nits
* Update exchanges/stream/websocket.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* glorious: nits
* add tests
* linter: fix
* After merge
* Add error connection info
* Fix edge case where it does not reconnect made by an already closed connection
* stream coverage
* glorious: nits
* glorious: nits removed asset error handling in stream package
* linter: fix
* rm block
* Add basic readme
* fix asset enabled flush cycle for multi connection
* spella: fix
* linter: fix
* Add glorious suggestions, fix some race thing
* reinstate name before any routine gets spawned
* stop on error in mock tests
* glorious: nits
* glorious: nits found in CI build
* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.
* mutex across shutdown and connect for protection
* lint: fix
* test time withoffset, reinstate stop
* fix whoops
* const trafficCheckInterval; rm testmain
* y
* fix lint
* bump time check window
* stream: fix intermittant test failures while testing routines and remove code that is not needed.
* spells
* cant do what I did
* protect race due to routine.
* update testURL
* use mock websocket connection instead of test URL's
* linter: fix
* remove url because its throwing errors on CI builds
* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.
* remove another superfluous url thats not really set up for this
* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly
* linter: fixerino uperino
* glorious: panix
* linter: things
* whoops
* defer lock and use functions that don't require locking in SetProxyAddress
* lint: fix
* thrasher: nits
---------
Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Websocket: Use ErrSubscribedAlready
instead of errChannelAlreadySubscribed
* Subscriptions: Replace Pair with Pairs
Given that some subscriptions have multiple pairs, support that as the
standard.
* Docs: Update subscriptions in add new exch
* RPC: Update Subscription Pairs
* Linter: Disable testifylint.Len
We deliberately use Equal over Len to avoid spamming the contents of large Slices
* Websocket: Add suffix to state consts
* Binance: Subscription Pairs support
* Bitfinex: Subscription Pairs support
* Bithumb: Subscription Pairs support
* Bitmex: Subscription Pairs support
* Bitstamp: Subscription Pairs support
* BTCMarkets: Subscription Pairs support
* BTSE: Subscription Pairs support
* Coinbase: Subscription Pairs support
* Coinut: Subscription Pairs support
* GateIO: Subscription Pairs support
* Gemini: Subscription Pairs support and improvement
* Hitbtc: Subscription Pairs support
* Huboi: Subscription Pairs support
* Kucoin: Subscription Pairs support
* Okcoin: Subscription Pairs support
* Poloniex: Subscription Pairs support
* Kraken: Add subscription Pairs support
Note: This is a naieve implementation because we want to rebase the
kraken websocket rewrite on top of this
* Bybit: Subscription Pairs support
* Okx: Subscription Pairs support
* Bitmex: Subsription configuration
* Fixes unauthenticated websocket left as CanUseAuth
* Fixes auth subs happening privately
* CoinbasePro: Subscription Configuration
* Consolidate ProductIDs when all subscriptions are for the same list
* Websocket: Log actual sent message when Verbose
* Subscriptions: Improve clarity of which key is which in Match
* Subscriptions: Lint fix for HugeParam
* Subscriptions: Add AddPairs and move keys from test
* Subscriptions: Simplify subscription keys and add key types
* Subscriptions: Add List.GroupPairs Rename sub.AddPairs
* Subscription: Fix ExactKey not matching 0 pairs
* Subscriptions: Remove unused IdentityKey and HasPairKey
* Subscriptions: Fix GetKey test
* Subscriptions: Test coverage improvements
* Websocket: Change State on Add/Remove
* Subscriptions: Improve error context
* Subscriptions: Fix Enable: false subs not ignored
* Bitfinex: Fix WsAuth test failing on DataHandler
DataHandler is eaten by dataMonitor now, so we need to use ToRoutine
* Deribit: Subscription Pairs support
* Websocket: Accept nil lists for checkSubscriptions
If the user passes in a nil (implicitly empty) list, we would not panic.
Therefore the burden of correctness about that data lies with them.
The list of subscriptions is empty, and that's okay, and possibly
convenient
* Websocket: Add context to NilPointer errors
* Subscriptions: Add context to nil errors
* Exchange: Fix error expectations in UnsubToWSChans
* Websocket: Remove IsInit and simplify SetProxyAddress
IsInit was basically the same as IsConnected.
Any time Connect was called both would be set to true.
Any time we had a disconnect they'd both be set to false
Shutdown() incorrectly didn't setInit(false)
SetProxyAddress simplified to only reconnect a connected Websocket.
Any other state means it hasn't been Connected, or it's about to
reconnect anyway.
There's no handling for IsConnecting previously, either, so I've wrapped
that behind the main mutex.
* Websocket: Expand and Assertify tests
* Websocket: Simplify state transistions
* Websocket: Simplify Connecting/Connected state
* Websocket: Tests and errors for websocket
* Websocket: Make WebsocketNotEnabled a real error
This allows for testing and avoids the repetition.
If each returned error is a error.New() you can never use errors.Is()
* Websocket: Add more testable errors
* Websocket: Improve GenerateMessageID test
Testing just the last id doesn't feel very robust
* Websocket: Protect Setup() from races
* Websocket: Use atomics instead of mutex
This was spurred by looking at the setState call in trafficMonitor and
the effect on blocking and efficiency.
With the new atomic types in Go 1.19, and the small types in use here,
atomics should be safe for our usage. bools should be truly atomic,
and uint32 is atomic when the accepted value range is less than one byte/uint8 since
that can be written atomicly by concurrent processors.
Maybe that's not even a factor any more, however we don't even have to worry enough to check.
* Websocket: Fix and simplify traffic monitor
trafficMonitor had a check throttle at the end of the for loop to stop it just gobbling the (blocking) trafficAlert channel non-stop.
That makes sense, except that nothing is sent to the trafficAlert channel if there's no listener.
So that means that it's out by one second on the trafficAlert, because any traffic received during the pause is doesn't try to send a traffic alert.
The unstopped timer is deliberately leaked for later GC when shutdown.
It won't delay/block anything, and it's a trivial memory leak during an infrequent event.
Deliberately Choosing to recreate the timer each time instead of using Stop, drain and reset
* Websocket: Split traficMonitor test on behaviours
* Websocket: Remove trafficMonitor connected status
trafficMonitor does not need to set the connection to be connected.
Connect() does that. Anything after that should result in a full
shutdown and restart. It can't and shouldn't become connected
unexpectedly, and this is most likely a race anyway.
Also dropped trafficCheckInterval to 100ms to mitigate races of traffic
alerts being buffered for too long.
* Websocket: Set disconnected earlier in Shutdown
This caused a possible race where state is still connected, but we start
to trigger interested actors via ShutdownC and Wait.
They may check state and then call Shutdown again, such as
trafficMonitor
* Websocket: Wait 5s for slow tests to pass traffic draining
Keep getting failures upstream on test rigs.
Think they can be very contended, so this pushes the boundary right out
to 5s
* CI/build: Update Go version, linters and fix minor issues
* Bump golangci-lint to v1.56.1
* BinanceUS: Make uint usage consistent
* Throw blank identifiers into the trash
* Websockets: Move Subscription to its own package
This allows the small type to be imported from both `config` and from
`stream` without an import cycle, so we don't have to repeat ourselves
* Subs: Renamed Currency to Pair
This was being mis-used through much of the code, and since we're
already touching everything, we might as well fix it
* Websockets: Add Subscription configuration
* Binance: Add subscription configuration
* Kucoin: Subscription configuration
* Simplify GenerateDefaultSubs
* Improve TestGenSubs coverage
* Test Candle Sub generation
* Support Candle intervals
* Full responsibility for formatting Channel name on GenerateDefaultSubs
OR consumer of Subscribe
* Simplify generatePayloads as a result
* Fix test coverage of asset types in processMarketSnapshot
* Exchanges: Abstract ParallelChanOp
* Tests: Generic ws mock instances
* Kucoin: Fix intermittent conflict in test currs
Use isolated test instance for `TestGetOpenInterest`.
`TestGetOpenInterest` would occassionally change pairs before
GenerateDefault Subs.
* switch over to package defined const for time layout
* bump appveyor playa
* bumperino to latest while setting patherino
* whoooops
* bump VS version set GOROOT
* puge build cache
* Revert "puge build cache"
This reverts commit 315bb578afc19529457f435e52af2172f5143bc5.
* bumperino to test
* purge setting of golang directory for version and allow default
* purge cache state when file change
* whoops
* thrasher: nits
* don't need to flusherino the cacherino
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* websocket: select case error if no receiver, add in functionality to reset to initial sync for books on a new websocket connection
* websocket: fix tests
* websocket: log error instead of losing it
* websocket: fix whoopsie
* exchanges: fix test
* websocket: force requirement of specific functionality
* exchanges: fix tests
* exchanges/websocket: move waitgroup add before scheduling across exchanges
* gateio: add feature subscribe
* bithumb/bittrex: include connection state reset, fix reconnection bug for Bithumb
* huobi: Add listen to shutdown to routine so it actually returns and stops being a naughty boy.
* huobi: add missing waitgroup add.
* exchanges: bleed comms channels
* binance: fix reconnection bug with buffer
* bithumb: fix reconnection bug with ws orderbook when websocket is diabled/enabled
* bithumb/bittrex: add bleeders for ws orderbook jobs
* linter: fix
* kraken: reduce code block from double assertion
* This bug ruined my day.
* glorious: error checking
* zb: add correct path for websocket connection
* exchange: Add verbosity when config conflicts and overwrites default values
* zb: add https to path
* exchanges: glorious nits
* stream: Add checkAndSetMonitoring to reduce potential routine bundling, increase timeout and check state in tests
* stream: remove check that is not needed.
* glorious: nits addr.
* lint: test
