* Exchanges: Allow empty batches in ParallelChanOp
In keeping with both common.Batch and "It's not my responsibility",
ParallelChanOp should just do nothing when given an empty list (and
implicitly an empty batch size.
Whatever it's going to do, it'll delegate to common.Batch,
and this allows us to just inline calls:
```
return common.AppendError(
b.ParallelChanOp(subs.Public(), func(l subscription.List) error { return b.manageSubs(wsSubscribeMethod, l, wsPublicStream) }, len(subs)),
b.ParallelChanOp(subs.Private(), func(l subscription.List) error { return b.manageSubs(wsSubscribeMethod, l, wsPrivateStream) }, len(subs)),
)
```
* Bitmex: Test config updates
* Bitmex: Sub Templating
* Bitmex: Enable websocket for tests
* Bitmex: Handle subscription errors
This switches to multiplexing so that we know which errors belong to
which stream, particularly for the auth attempt
* Bitmex: Fix ws order side err going to data stream
Shouldn't fall into classification error if it's actually a parsing
error
* Cherry_pickable
* gateio/websocket: use millisecond time for more accurate push time
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* gateio: Add multi asset websocket support WIP.
* meow
* Add tests and shenanigans
* integrate flushing and for enabling/disabling pairs from rpc shenanigans
* some changes
* linter: fixes strikes again.
* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.
* Add subscription tests (state functional)
* glorious:nits + proxy handling
* Spelling
* linter: fixerino
* instead of nil, dont do nil.
* clean up nils
* cya nils
* don't need to set URL or check if its running
* stop ping handler routine leak
* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response
* Allow rollback on connect on any error across all connections
* fix shadow jutsu
* glorious/gk: nitters - adds in ws mock server
* linter: fix
* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.
* gk: nits
* Leak issue and edge case
* gk: nits
* gk: drain brain
* glorious: nits
* Update exchanges/stream/websocket.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* glorious: nits
* add tests
* linter: fix
* After merge
* Add error connection info
* Fix edge case where it does not reconnect made by an already closed connection
* stream coverage
* glorious: nits
* glorious: nits removed asset error handling in stream package
* linter: fix
* rm block
* Add basic readme
* fix asset enabled flush cycle for multi connection
* spella: fix
* linter: fix
* Add glorious suggestions, fix some race thing
* reinstate name before any routine gets spawned
* stop on error in mock tests
* glorious: nits
* glorious: nits found in CI build
* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.
* mutex across shutdown and connect for protection
* lint: fix
* test time withoffset, reinstate stop
* fix whoops
* const trafficCheckInterval; rm testmain
* y
* fix lint
* bump time check window
* stream: fix intermittant test failures while testing routines and remove code that is not needed.
* spells
* cant do what I did
* protect race due to routine.
* update testURL
* use mock websocket connection instead of test URL's
* linter: fix
* remove url because its throwing errors on CI builds
* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.
* remove another superfluous url thats not really set up for this
* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly
* linter: fixerino uperino
* glorious: panix
* linter: things
* whoops
* defer lock and use functions that don't require locking in SetProxyAddress
* lint: fix
* thrasher: nits
---------
Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Convert: Fix TimeFromUnixTimestampDecimal using local
All parsed times should be in UTC
* Subscriptions: Add IgnoringAssetsKey
* Tests: Pass tb to curried WS handlers
* Websocket: Make ErrNoMessageListener a public error
* Kraken: Fix URLMap ignored for websocket URLs
* Kraken: Move SeedAssets from Setup to Bootstrap
Having SeedAssets in Setup is cruel and unusual because it calls the
API. Most other interactive data seeding happens in Bootstrap.
This made it so that fixing and creating unit tests for Kraken was
painfully slow, particularly on flaky internet.
* Kraken: Remove convert test
Duplicate of convert_test.go TestTimeFromUnixTimestampDecimal
* Kraken: Test config upgrades
* Kraken: Sub Channel improvements
* Use Websocket subscriptionChannels instead of local slice
* Remove ChannelID - Deprecated in docs
* Simplify ping handlers and hardcodes message
* Add Depth as configurable orderbook channel param
* Simplify auth/non-auth channel updates
* Add configurable Book depth
* Add configurable Candle timeframes
Kraken: Simplify all WS handlers with reqId
* Kraken: Subscription templating
* Generate N+ subs for pairs
If we generate one sub for all pairs, but then fan it out in the
responses, we end up with a mis-match between the sub store and
GenerateSubs, and when we do FlushChannels it will try to resub
everything again.
* Kraken: Rename channelName var throughout
Avoid shadowing func of same name
* Kraken: Add TestEnforceStandardChannelNames
* Websocket: Fix Resubscribe erroring Duplicate
* consolidate type to types package and share across code base
* rm convert type and convert codebase
* rm irrelavant test cases
* Fix tests
* glorious nits
* Update exchanges/gateio/gateio_types.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* thrasher: nits
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* types and orderbook handling fix
* Minor types and endpoints update
* Minor fixes on Kucoin
* Add benchmarking test
* Unit tests update and minor endpoints update
* Adding updates and newly added endpoints
* Add and correct funding, withdrawal, and deposit endpoints
* linter and codespell fix
* Adding and correcting spot trading endpoints
* Completed Spot HF endpoints
* Minor fix
* Added OCO and Margin HF trade and other endpoints
* Adding missing endpoints
* Updating Isolated margin, margin v3, and v3market lending endpoints
* minor codespell fix
* Completed adding and fixing futures endpoints
* wrapper update and fix unit tests
* Updating endpoint ratelimits
* Complete ratelimiter setup and endpoint functions update
* Unit test configuration and update
* fix linter issue
* Added a ratelimiter test and heavy update on unit tests
* Adding websocket update based on ChangeLog
* Added newly added Earn General, Kucoin Earn and Staking endpoints
* Added VIP lending endpoints
* Minor linter and endpoints fix
* Added unit tests, publicised functions, and minor updates
* Update on wrapper funcs, unit tests, and other methods
* Enexport exchange specific websocket methods
* remove deprecated topic
* Update wrapper based on Type, add and fix unit tests
* Added a margin configuration endpoint and unit test
* Update methods, types, and unit tests
* Update error declaration and handling unit tests
* Update method parameters and error handling
* Updating unit tests and error handling
* Update methods arguments, added and update unit tests
* Fix unit tests and wrapper methods
* Resolving unit test issues and fix faulty endpoints
* Fix on unit tests and working on passphrase errors
* Minor fixed on websocket and endpoint url
* comment and wrapper filters issue fix
* Unit tests and other minor updates
* Update wrapper functions, endpoint methods, and unit tests
* change require to change on two unit tests
* Update unit tests, types, and endpoints
* Refine and update wrapper tempo for minor adjustments
* Remove code that enabled logging
* Update wrapper functions, missing endpoints, response and parameter values, and unit tests
* removed High-frequency orders from wrapper functions, and updated unit tests and types
* Added missing fields and minor update on wrapper
* Update types
* Update tests and websocket channels
* Update unit tests and methods error returns
* CI/build: Update Go version, linters and fix minor issues
* linters: Add intrange, copyloopvar, additional go vet linters to match gopls and fix issues
* Kraken: Fix TestGetOpenInterest
We see daily failures on OpenInterest for Kraken.
This fix assumes that the issue might be related to volume of ETHUSD
open interest, and switches the single pair test to XBTUSD instead
Also isolates the test from others, since we're changing stored pairs
and we might be colliding on the global k
* Kraken: Handle Errors field in futures response
* Add in extra field for colocation monitoring
* rm tags
* populate through gateio orderbooks
* populate through incremental updates
* Add insert at field for orderbook depth
* I AM A BAD SPELLLLLLA
* add it in
* c change
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Currency: Variadic Pairs.Add
This version of Pairs.Add is simpler and [more
performant](https://gist.github.com/gbjk/06a1fc1832d04ee41213ca518938cf74)
Behavioural difference: If there's nothing to add, the same slice is
returned unaltered. This seems like good sauce
* Currency: Variadic Remove
* Common: Add Batch function
* Common: Add common.SortStrings for stringers
* Subscriptions: Add batching to templates
* Subscriptions: Sort list of pairs
* Kucoin: Switch to sub templating
* Kucoin: Simplify channel prefix usage
* Kucoin: Fix race on fetchedFuturesOrderbook
* Subscriptions: Filter AssetPairs
Now only the assetPairs relevant to the subscription are in the context
* Subscriptions: Respect subscription Pairs
* Subscriptions: Trim AssetSeparator early
We want to trim before checking for "AssetSeparator vs All" because a
template should be allowed to reuse a range template and generate just one trailing AssetSeparator
whilst using a specific Asset
* Kucoin: Fix empty margin asset added
* Kucoin: Add Subscription batching
Turns out that contary to the documentation, kucoin supports batching of
all symbols and currencies
* Kucoin: Fix checkSubscriptions and coverage
* Subscriptions: Simplify error checking
This reduces the complexity of error checking to just be "do we get the
correct numbers".
Fixes Asset.All with only one asset erroring on xpandPairs, because we
trimmed the only asset separator, and then errored that we're not
xpanding Assets and the asset on the sub is asset.All
This use-case conflicted with commit 6bbd546d74, which required:
```
Subscriptions: Trim AssetSeparator early
We want to trim before checking for "AssetSeparator vs All" because a
template should be allowed to reuse a range template and generate just one trailing AssetSeparator
whilst using a specific Asset
```
Now we set up the assets earlier, and we remove the check for xpandAssets, since the number of asset lines matching is all that matters.
I've removed the asset tests for this, but they were correctly erroring
on the number of asset lines instead.
Everything hits coverage, as well.
* Kucoin: Remove deprecated fundingBook endpoint
* BTCMarkets: Use common.Batch
* Kraken: Simplify REST error and result handling
* Kraken: Rid excessive map lookups or large copies when iterating
* Kraken: Prealloc orderbook types.Numer size
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
---------
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Making Pairs(Add) Variatic
* Slight improvements
* Implementation overhaul
* Improvements
* Changing code which can use the variadic functionality
* Fixing silliness
* thing left over
* Subscriptions: Add List.AssetPairs
* Subscriptions: Add Template and QualifiedChannel
These fields separate the concept of what the channel is from the
qualified resource name
* Subscriptions: Add List.SetStates()
* Subscriptions: Add List.QualifiedChannels
* Subscriptions: Rename testsubs.EqualLists
* Binance: Switch to ExpandTemplates
* Binance: Update ConfigTest format
* Subscriptions: Test Coverage improvements
* Subscriptions: Reenterant List.ExpandTemplates
* Subscriptions: Move templates from subscriptions to exchanges
* Binance: Inline subscription template and improvements
* Add in initial handling for quote/base currency deployment requirements
* include client order ID checking
* glorious: suggestions
* spell and fix
* linter/context/test
* rework tests and order side specific requirements
* linter
* mend panic at the disco
* mending more panics at the disco
* anudda fix brudda
* glorious: NITTTTTT BOOOOOMB
* leftover things and stuff
* whoops
* tie in gateio
* glorious: nit fixes life and everything.
* thrasher: nits
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Common: DriveBy sanitisation of vars
* BTSE: Unify to exchange OrderLimits
* BTSE: Remove SeedAssets and test GetOrderExcutionLimit
* BTSE: Fix handling for K_* pairs
In addition to the M_ pairs we previously handled, BTSE has now
introduced K_* pairs (K_SATS-USD*).
This change moves the handling over to look for an exponent, and moves
the filtering to happen on all market data.
The original MarketSummary is still availiable, but I can't see any of
our current use-cases wanting to get the unfiltered list
* BTSE: Fix marketSummary futures field
BTSE returns no futures field for futures api marketInfo, and the documentation for the futures api shows returning false.
When we know we asked for futures, it makes the data flow much saner if
we can trust this field and not have to track what asset we asked for
* BTSE: Abstract marketPair.Pair()
* BTSE: Fix UpdateTicker symbol format
* gateio: update FuturesAccount struct
* gateio: update Position struct
* gateio: rm redundant checks and add in actual required checks
* export GateioTime
* linter: fix
* linter: again fix
* issue: fix
* gateio: update fee struct and such
* Update exchanges/gateio/gateio_types.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Update exchanges/gateio/gateio_convert.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* use Time type acrost file
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Websocket: Use ErrSubscribedAlready
instead of errChannelAlreadySubscribed
* Subscriptions: Replace Pair with Pairs
Given that some subscriptions have multiple pairs, support that as the
standard.
* Docs: Update subscriptions in add new exch
* RPC: Update Subscription Pairs
* Linter: Disable testifylint.Len
We deliberately use Equal over Len to avoid spamming the contents of large Slices
* Websocket: Add suffix to state consts
* Binance: Subscription Pairs support
* Bitfinex: Subscription Pairs support
* Bithumb: Subscription Pairs support
* Bitmex: Subscription Pairs support
* Bitstamp: Subscription Pairs support
* BTCMarkets: Subscription Pairs support
* BTSE: Subscription Pairs support
* Coinbase: Subscription Pairs support
* Coinut: Subscription Pairs support
* GateIO: Subscription Pairs support
* Gemini: Subscription Pairs support and improvement
* Hitbtc: Subscription Pairs support
* Huboi: Subscription Pairs support
* Kucoin: Subscription Pairs support
* Okcoin: Subscription Pairs support
* Poloniex: Subscription Pairs support
* Kraken: Add subscription Pairs support
Note: This is a naieve implementation because we want to rebase the
kraken websocket rewrite on top of this
* Bybit: Subscription Pairs support
* Okx: Subscription Pairs support
* Bitmex: Subsription configuration
* Fixes unauthenticated websocket left as CanUseAuth
* Fixes auth subs happening privately
* CoinbasePro: Subscription Configuration
* Consolidate ProductIDs when all subscriptions are for the same list
* Websocket: Log actual sent message when Verbose
* Subscriptions: Improve clarity of which key is which in Match
* Subscriptions: Lint fix for HugeParam
* Subscriptions: Add AddPairs and move keys from test
* Subscriptions: Simplify subscription keys and add key types
* Subscriptions: Add List.GroupPairs Rename sub.AddPairs
* Subscription: Fix ExactKey not matching 0 pairs
* Subscriptions: Remove unused IdentityKey and HasPairKey
* Subscriptions: Fix GetKey test
* Subscriptions: Test coverage improvements
* Websocket: Change State on Add/Remove
* Subscriptions: Improve error context
* Subscriptions: Fix Enable: false subs not ignored
* Bitfinex: Fix WsAuth test failing on DataHandler
DataHandler is eaten by dataMonitor now, so we need to use ToRoutine
* Deribit: Subscription Pairs support
* Websocket: Accept nil lists for checkSubscriptions
If the user passes in a nil (implicitly empty) list, we would not panic.
Therefore the burden of correctness about that data lies with them.
The list of subscriptions is empty, and that's okay, and possibly
convenient
* Websocket: Add context to NilPointer errors
* Subscriptions: Add context to nil errors
* Exchange: Fix error expectations in UnsubToWSChans
* initial consolidation of rate limiting code to request package to reduce bespoke code implementation
* continued
* finish abstraction
* lint
* exchanges: fix tests
* linter: fix
* poloniex: fix auth rate limit not being set
* ratelimiter: convert from token to weight
* glorious: nits addressed with fire
* linter: rip
* change func name set -> get
* fix test
* derbit: impl
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* deribit implementation
* add ws impll
* cleanup
* Update deribit_wrapper.go
* Add missing endpoints
* Fix config file
* asset type update
* Update code structure
* Update authenticated private endpoints unit tests
* Updating websocket
* Updating websocket connection and subscription handling
* Finishing up adding subscription push data
* Adding websocket public endpoint
* Adding WS endpoints
* Adding websocket unit tests
* Minor clean-up
* Integrating websocket endpoints into the wrapper funcs
* Updating exchange documentations
* Fixing test issues
* Code cleaning-up
* fix test issues
* Updating validations and logic errors
* Updating wrapper issues
* fix test issues
* Slight test update
* Unit test and code structure update
* Update websocket tempos
* Slight update on code structure
* Minor update on unit tests
* Update depending on review comments
* Minor code fix and doc re-generating
* Update on Candlestick wrapper functions
* Minor updates
* minor unit test updates
* Minor updates on weboscket and unit tests
* minor linter fix
* codespell and rate limiter issues
* single linter issue fix
* adding rate limiter
* Add ratelimiter to websocket conn and overall code update
* fix websocket push data issue
* Implementing missing wrapper function
* Websocket fix
* Minor update on missing endpoint and other
* fixing websocket issues and cleaningup
* Minor tempo fix
* Minor linter issues
* unit test update
* Indexing error fix
* Websocket connection fix
* string formatting fix
* Small fix on unit tests
* fix minor json conversion issue
* websocket and documentation update
* websocket, wrapper and unit test updates
* Documentation and unit tests update
* Fix unit tests
* wrapper fix for new change
* Unit test fix
* timestamp conversion and unit tests update
* Minor instrument ID conversion fix
* instrument formats and unit test update
* formatting and unit test fix
* config update
* Updating websocket and adding the Spot support
* Add small unit test fix
* unit test and websocket handlers update
* Linter issues fix
* minor documentation and code update
* Minor fix
* added a wrapper func GetLatestFundingRates
* Types, wrapper update, and unit tests
* Minor config update
* fix wrapper unit tests
* Resolve all panic and wrapper test issues
* minor unit test fix
* fix issues and adding newly added endpoints
* updates and added remaining endpoints with unit tests
* Update unit tests using assert
* Added missing endpoints and unit tests
* Minor updates and clean-ups
* Resolve tradable pair fetching panic
* Mutex fix
* Added Options assets test and minor fixes
* subscription mothod updated
* Remove misadded code lines
* resolve websocket
* Updating tests, types, endpoint methods and others
* Added GetFuturesContractDetails and minor fix
* fix linter issue
* revert change on candlestic time
* Added filters to candles
* minor unit test and wrapper fix
* Minor unit tests update
* cahnge param key for GetOrderMarginByID
* updating unit tests and resolve issues
* Update websocket unit tests
* Minor fix based on review
* Revert unit test change
* fix pair config issue
* Added missing wrapper functions
* Fix missing review changes
* Fix options request pair formatting
* fix AllExchangeWrappers test issue
* Changes with unit test and wrapper based on the review
* Fix to options reg-exp
* wrapper functions fix
* Update MaximumFundingRateHistory filter and minor fixes
* Fix besed on review comment
* Fix issues on review comment
* linter fix
* fix minor unit test issue
* Fix unit test issues
* Update trade order cancellation responses
* fix config files issue
* lint update config files
* Update unit tests
* Update return values and response handling
* added missing endpoint and fixes based on review comment
* toggle useTestNet back
* Update cancel by label and other fix
* fix forgotten cancel all response type
* update CancelResp type
* Fix unmarshaling error
* updated websocket orderbook load issue
* fix websocket lock and groups
* Change Items to Tranche and fix linter issues
* Fix orderbook issue
* Update unit tests offline error handling, and endpoints argument and error handling
* Contributors documentation update and change error return type
* Updated unit tests based on review comment
* Update unit tests and removed password change endpoint
* Fix race condition
* Update on tests, test pairs, and wrapper config
* Update test tradable pairs loading
* Update unit tests, fix linter issues, and update wrapper functions
* remove credentials
* Update test and fix authentication method and few authenticated endpoints
* fix codespell issue
* group the repeated currency code check to a func
* added unit test for repeated pair check func
* Added a base coin and related updates
---------
Co-authored-by: E Sequeira <earncef@earncef.com>
