* don't need to verify if checksum available; don't retroactively verify snapshots
* glorious: nits + allow checksum/validation to be done on updateByIDAndAction
* possible slice manipulation issue fix for test
* thrasher: nit
* this should fix it now
* for subsystem usage disallow deploying a book twice
* reverts strict policy using deploy, it's used in a bunch of tests I don't want to touch just yet, left a note
* rm unused variable
* glorious: nits
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* bybit: Add time field for orderbook and handle correct timings
* use UpdateID as this is the localised ID for this pair, rm check result.UpdateID == 1 just use snapshot
* general quick fixes across from #1672 which can be used straight away
* fix test
* glorious: nits
* thrasher: nits
* thrasher: nits thanks @thrasher-
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Bybit: Subscription configuration for spot
* Bybit: Enable candles ws sub by default
* Orderbook: Use a RW mutex for depth
* Orderbook: Fix race on depth.VerifyOrderbook
Despite being protected by an ob level mutex, this needed to privatise
and protect the option var.
The subscription pairs do not need to be validated as enabled or
available. The check was just belt-and-braces and didn't have a specific
use-case in mind.
Coinbase has a use-case for wanting to subscribe to BTC-USD when it's
not enabled.
Moreover, it shouldn't be our job to check this. You want a sub expanded
with these pairs? Fine. Done. If it doesn't work, you can work out why
* Exchanges: Allow empty batches in ParallelChanOp
In keeping with both common.Batch and "It's not my responsibility",
ParallelChanOp should just do nothing when given an empty list (and
implicitly an empty batch size.
Whatever it's going to do, it'll delegate to common.Batch,
and this allows us to just inline calls:
```
return common.AppendError(
b.ParallelChanOp(subs.Public(), func(l subscription.List) error { return b.manageSubs(wsSubscribeMethod, l, wsPublicStream) }, len(subs)),
b.ParallelChanOp(subs.Private(), func(l subscription.List) error { return b.manageSubs(wsSubscribeMethod, l, wsPrivateStream) }, len(subs)),
)
```
* Bitmex: Test config updates
* Bitmex: Sub Templating
* Bitmex: Enable websocket for tests
* Bitmex: Handle subscription errors
This switches to multiplexing so that we know which errors belong to
which stream, particularly for the auth attempt
* Bitmex: Fix ws order side err going to data stream
Shouldn't fall into classification error if it's actually a parsing
error
* Cherry_pickable
* gateio/websocket: use millisecond time for more accurate push time
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* gateio: Add multi asset websocket support WIP.
* meow
* Add tests and shenanigans
* integrate flushing and for enabling/disabling pairs from rpc shenanigans
* some changes
* linter: fixes strikes again.
* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.
* Add subscription tests (state functional)
* glorious:nits + proxy handling
* Spelling
* linter: fixerino
* instead of nil, dont do nil.
* clean up nils
* cya nils
* don't need to set URL or check if its running
* stop ping handler routine leak
* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response
* Allow rollback on connect on any error across all connections
* fix shadow jutsu
* glorious/gk: nitters - adds in ws mock server
* linter: fix
* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.
* gk: nits
* Leak issue and edge case
* gk: nits
* gk: drain brain
* glorious: nits
* Update exchanges/stream/websocket.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* glorious: nits
* add tests
* linter: fix
* After merge
* Add error connection info
* Fix edge case where it does not reconnect made by an already closed connection
* stream coverage
* glorious: nits
* glorious: nits removed asset error handling in stream package
* linter: fix
* rm block
* Add basic readme
* fix asset enabled flush cycle for multi connection
* spella: fix
* linter: fix
* Add glorious suggestions, fix some race thing
* reinstate name before any routine gets spawned
* stop on error in mock tests
* glorious: nits
* glorious: nits found in CI build
* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.
* mutex across shutdown and connect for protection
* lint: fix
* test time withoffset, reinstate stop
* fix whoops
* const trafficCheckInterval; rm testmain
* y
* fix lint
* bump time check window
* stream: fix intermittant test failures while testing routines and remove code that is not needed.
* spells
* cant do what I did
* protect race due to routine.
* update testURL
* use mock websocket connection instead of test URL's
* linter: fix
* remove url because its throwing errors on CI builds
* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.
* remove another superfluous url thats not really set up for this
* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly
* linter: fixerino uperino
* glorious: panix
* linter: things
* whoops
* defer lock and use functions that don't require locking in SetProxyAddress
* lint: fix
* thrasher: nits
---------
Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Convert: Fix TimeFromUnixTimestampDecimal using local
All parsed times should be in UTC
* Subscriptions: Add IgnoringAssetsKey
* Tests: Pass tb to curried WS handlers
* Websocket: Make ErrNoMessageListener a public error
* Kraken: Fix URLMap ignored for websocket URLs
* Kraken: Move SeedAssets from Setup to Bootstrap
Having SeedAssets in Setup is cruel and unusual because it calls the
API. Most other interactive data seeding happens in Bootstrap.
This made it so that fixing and creating unit tests for Kraken was
painfully slow, particularly on flaky internet.
* Kraken: Remove convert test
Duplicate of convert_test.go TestTimeFromUnixTimestampDecimal
* Kraken: Test config upgrades
* Kraken: Sub Channel improvements
* Use Websocket subscriptionChannels instead of local slice
* Remove ChannelID - Deprecated in docs
* Simplify ping handlers and hardcodes message
* Add Depth as configurable orderbook channel param
* Simplify auth/non-auth channel updates
* Add configurable Book depth
* Add configurable Candle timeframes
Kraken: Simplify all WS handlers with reqId
* Kraken: Subscription templating
* Generate N+ subs for pairs
If we generate one sub for all pairs, but then fan it out in the
responses, we end up with a mis-match between the sub store and
GenerateSubs, and when we do FlushChannels it will try to resub
everything again.
* Kraken: Rename channelName var throughout
Avoid shadowing func of same name
* Kraken: Add TestEnforceStandardChannelNames
* Websocket: Fix Resubscribe erroring Duplicate
* consolidate type to types package and share across code base
* rm convert type and convert codebase
* rm irrelavant test cases
* Fix tests
* glorious nits
* Update exchanges/gateio/gateio_types.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* thrasher: nits
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* types and orderbook handling fix
* Minor types and endpoints update
* Minor fixes on Kucoin
* Add benchmarking test
* Unit tests update and minor endpoints update
* Adding updates and newly added endpoints
* Add and correct funding, withdrawal, and deposit endpoints
* linter and codespell fix
* Adding and correcting spot trading endpoints
* Completed Spot HF endpoints
* Minor fix
* Added OCO and Margin HF trade and other endpoints
* Adding missing endpoints
* Updating Isolated margin, margin v3, and v3market lending endpoints
* minor codespell fix
* Completed adding and fixing futures endpoints
* wrapper update and fix unit tests
* Updating endpoint ratelimits
* Complete ratelimiter setup and endpoint functions update
* Unit test configuration and update
* fix linter issue
* Added a ratelimiter test and heavy update on unit tests
* Adding websocket update based on ChangeLog
* Added newly added Earn General, Kucoin Earn and Staking endpoints
* Added VIP lending endpoints
* Minor linter and endpoints fix
* Added unit tests, publicised functions, and minor updates
* Update on wrapper funcs, unit tests, and other methods
* Enexport exchange specific websocket methods
* remove deprecated topic
* Update wrapper based on Type, add and fix unit tests
* Added a margin configuration endpoint and unit test
* Update methods, types, and unit tests
* Update error declaration and handling unit tests
* Update method parameters and error handling
* Updating unit tests and error handling
* Update methods arguments, added and update unit tests
* Fix unit tests and wrapper methods
* Resolving unit test issues and fix faulty endpoints
* Fix on unit tests and working on passphrase errors
* Minor fixed on websocket and endpoint url
* comment and wrapper filters issue fix
* Unit tests and other minor updates
* Update wrapper functions, endpoint methods, and unit tests
* change require to change on two unit tests
* Update unit tests, types, and endpoints
* Refine and update wrapper tempo for minor adjustments
* Remove code that enabled logging
* Update wrapper functions, missing endpoints, response and parameter values, and unit tests
* removed High-frequency orders from wrapper functions, and updated unit tests and types
* Added missing fields and minor update on wrapper
* Update types
* Update tests and websocket channels
* Update unit tests and methods error returns
* CI/build: Update Go version, linters and fix minor issues
* linters: Add intrange, copyloopvar, additional go vet linters to match gopls and fix issues
* Kraken: Fix TestGetOpenInterest
We see daily failures on OpenInterest for Kraken.
This fix assumes that the issue might be related to volume of ETHUSD
open interest, and switches the single pair test to XBTUSD instead
Also isolates the test from others, since we're changing stored pairs
and we might be colliding on the global k
* Kraken: Handle Errors field in futures response
* Add in extra field for colocation monitoring
* rm tags
* populate through gateio orderbooks
* populate through incremental updates
* Add insert at field for orderbook depth
* I AM A BAD SPELLLLLLA
* add it in
* c change
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Currency: Variadic Pairs.Add
This version of Pairs.Add is simpler and [more
performant](https://gist.github.com/gbjk/06a1fc1832d04ee41213ca518938cf74)
Behavioural difference: If there's nothing to add, the same slice is
returned unaltered. This seems like good sauce
* Currency: Variadic Remove
* Common: Add Batch function
* Common: Add common.SortStrings for stringers
* Subscriptions: Add batching to templates
* Subscriptions: Sort list of pairs
* Kucoin: Switch to sub templating
* Kucoin: Simplify channel prefix usage
* Kucoin: Fix race on fetchedFuturesOrderbook
* Subscriptions: Filter AssetPairs
Now only the assetPairs relevant to the subscription are in the context
* Subscriptions: Respect subscription Pairs
* Subscriptions: Trim AssetSeparator early
We want to trim before checking for "AssetSeparator vs All" because a
template should be allowed to reuse a range template and generate just one trailing AssetSeparator
whilst using a specific Asset
* Kucoin: Fix empty margin asset added
* Kucoin: Add Subscription batching
Turns out that contary to the documentation, kucoin supports batching of
all symbols and currencies
* Kucoin: Fix checkSubscriptions and coverage
* Subscriptions: Simplify error checking
This reduces the complexity of error checking to just be "do we get the
correct numbers".
Fixes Asset.All with only one asset erroring on xpandPairs, because we
trimmed the only asset separator, and then errored that we're not
xpanding Assets and the asset on the sub is asset.All
This use-case conflicted with commit 6bbd546d74, which required:
```
Subscriptions: Trim AssetSeparator early
We want to trim before checking for "AssetSeparator vs All" because a
template should be allowed to reuse a range template and generate just one trailing AssetSeparator
whilst using a specific Asset
```
Now we set up the assets earlier, and we remove the check for xpandAssets, since the number of asset lines matching is all that matters.
I've removed the asset tests for this, but they were correctly erroring
on the number of asset lines instead.
Everything hits coverage, as well.
* Kucoin: Remove deprecated fundingBook endpoint
* BTCMarkets: Use common.Batch
* Kraken: Simplify REST error and result handling
* Kraken: Rid excessive map lookups or large copies when iterating
* Kraken: Prealloc orderbook types.Numer size
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
---------
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Making Pairs(Add) Variatic
* Slight improvements
* Implementation overhaul
* Improvements
* Changing code which can use the variadic functionality
* Fixing silliness
* thing left over
* Subscriptions: Add List.AssetPairs
* Subscriptions: Add Template and QualifiedChannel
These fields separate the concept of what the channel is from the
qualified resource name
* Subscriptions: Add List.SetStates()
* Subscriptions: Add List.QualifiedChannels
* Subscriptions: Rename testsubs.EqualLists
* Binance: Switch to ExpandTemplates
* Binance: Update ConfigTest format
* Subscriptions: Test Coverage improvements
* Subscriptions: Reenterant List.ExpandTemplates
* Subscriptions: Move templates from subscriptions to exchanges
* Binance: Inline subscription template and improvements
