* Kline: Fix Raw Short, Marshal and Unmarshal
* Deribit: Rename GenerateDefaultSubs
* Deribit: Remove custom GetDefaultConfig
Moved to exchange base by #1472
* Deribit: Straight Rename of eps to endpoints
Since I had to ask what this abbreviation meant, I think we should
abandon it
* Deribit: Add Subscription configuration
* Deribit: Fix race on Setup with optionsRegex
Calling Setup twice would race on the assignment to this package var.
There was an option to just move the assignment to the package var declaration, but this
change improves the performance and allocations:
```
BenchmarkOptionPairToString-8 1000000 1239 ns/op 485 B/op 10 allocs/op
BenchmarkOptionPairToString2-8 3473804 656.2 ns/op 348 B/op 7 allocs/op
```
I've also removed the t.Run because even success the -v output from
tests would be very noisy, and I don't think we were getting any benefit
from it at all:
```
=== RUN TestOptionPairToString
=== PAUSE TestOptionPairToString
=== CONT TestOptionPairToString
=== RUN TestOptionPairToString/BTC-30MAY24-61000-C
=== PAUSE TestOptionPairToString/BTC-30MAY24-61000-C
=== RUN TestOptionPairToString/ETH-1JUN24-3200-P
=== PAUSE TestOptionPairToString/ETH-1JUN24-3200-P
=== RUN TestOptionPairToString/SOL_USDC-31MAY24-162-P
=== PAUSE TestOptionPairToString/SOL_USDC-31MAY24-162-P
=== RUN TestOptionPairToString/MATIC_USDC-6APR24-0d98-P
=== PAUSE TestOptionPairToString/MATIC_USDC-6APR24-0d98-P
=== CONT TestOptionPairToString/BTC-30MAY24-61000-C
=== CONT TestOptionPairToString/SOL_USDC-31MAY24-162-P
=== CONT TestOptionPairToString/ETH-1JUN24-3200-P
=== CONT TestOptionPairToString/MATIC_USDC-6APR24-0d98-P
--- PASS: TestOptionPairToString (0.00s)
--- PASS: TestOptionPairToString/BTC-30MAY24-61000-C (0.00s)
--- PASS: TestOptionPairToString/ETH-1JUN24-3200-P (0.00s)
--- PASS: TestOptionPairToString/SOL_USDC-31MAY24-162-P (0.00s)
--- PASS: TestOptionPairToString/MATIC_USDC-6APR24-0d98-P (0.00s)
```
( And that got worse with me adding more tests )
* gateio: Add multi asset websocket support WIP.
* meow
* Add tests and shenanigans
* integrate flushing and for enabling/disabling pairs from rpc shenanigans
* some changes
* linter: fixes strikes again.
* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.
* Add subscription tests (state functional)
* glorious:nits + proxy handling
* Spelling
* linter: fixerino
* instead of nil, dont do nil.
* clean up nils
* cya nils
* don't need to set URL or check if its running
* stream match update
* update tests
* linter: fix
* glorious: nits + handle context cancellations
* stop ping handler routine leak
* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response
* Allow rollback on connect on any error across all connections
* fix shadow jutsu
* glorious/gk: nitters - adds in ws mock server
* linter: fix
* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.
* glorious: whooops
* gk: nits
* Leak issue and edge case
* Websocket: Add SendMessageReturnResponses
* whooooooopsie
* gk: nitssssss
* Update exchanges/stream/stream_match.go
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
* Update exchanges/stream/stream_match_test.go
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
* linter: appease the linter gods
* gk: nits
* gk: drain brain
* started
* more changes before merge match pr
* gateio: still building out
* gateio: finish spot
* fix up tests in gateio
* Add tests for stream package
* rm unused field
* glorious: nits
* rn files, specifically set function names to asset and offload routing to websocket type.
* linter: fix
* glorious: nits
* add counter and update gateio
* fix collision issue
* Update exchanges/stream/websocket.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* glorious: nits
* add tests
* linter: fix
* After merge
* Add error connection info
* upgrade to upstream merge
* Fix edge case where it does not reconnect made by an already closed connection
* stream coverage
* glorious: nits
* glorious: nits removed asset error handling in stream package
* linter: fix
* rm block
* Add basic readme
* fix asset enabled flush cycle for multi connection
* spella: fix
* linter: fix
* Add glorious suggestions, fix some race thing
* reinstate name before any routine gets spawned
* stop on error in mock tests
* glorious: nits
* glorious: nits found in CI build
* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.
* mutex across shutdown and connect for protection
* lint: fix
* test time withoffset, reinstate stop
* fix whoops
* const trafficCheckInterval; rm testmain
* y
* fix lint
* bump time check window
* stream: fix intermittant test failures while testing routines and remove code that is not needed.
* spells
* cant do what I did
* protect race due to routine.
* update testURL
* use mock websocket connection instead of test URL's
* linter: fix
* remove url because its throwing errors on CI builds
* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.
* remove another superfluous url thats not really set up for this
* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly
* linter: fixerino uperino
* glorious: panix
* linter: things
* whoops
* dont need to make consecutive Unix() calls
* websocket: fix potential panic on error and no responses and adding waitForResponses
* rm json parser and handle in json package instead
* linter: fix
* linter: fix again
* * change field name OutboundRequestSignature to WrapperDefinedConnectionSignature for agnostic inbound and outbound connections.
* change method name GetOutboundConnection to GetConnection for agnostic inbound and outbound connections.
* drop outbound field map for improved performance just using a range and field check (less complex as well)
* change field name connections to connectionToWrapper for better clarity
* spells and magic and wands
* glorious: nits
* comparable check for signature
* mv err var
* glorious: nits and stuff
* attempt to fix race
* glorious: nits
* gk: nits; engine log cleanup
* gk: nits; OCD
* gk: nits; move function change file names
* gk: nits; 🚀
* gk: nits; convert variadic function and message inspection to interface and include a specific function for that handling so as to not need nil on every call
* gk: nits; continued
* gk: engine nits; rm loaded exchange
* gk: nits; drop WebsocketLoginResponse
* stream: Add match method EnsureMatchWithData
* gk: nits; rn Inspect to IsFinal
* gk: nits; rn to MessageFilter
* linter: fix
* gateio: update rate limit definitions (cherry-pick)
* Add test and missing
* Shared REST rate limit definitions with Websocket service, set lookup item to nil for systems that do not require rate limiting; add glorious nit
* integrate rate limits for websocket trading spot
* bitstamp: fix issue
* glorious: nits
* ch name and commentary
* fix bug add test
* rm a thing
* fix test
* Update engine/engine.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* thrasher: nits
* Update exchanges/stream/stream_match_test.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Update exchanges/stream/stream_match_test.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* GK: nits rn websocket functions
* explicit function names for single to multi outbound orders
* linter: fix
---------
Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* exchanges: Fix gateio/coinbase test failures
* OKX: Fix TestGetAssetsFromInstrumentTypeOrID race
* GateIO: Add/improve comments
* GateIO: Rid additional API call for FetchTradablePairs and provide additional context for test
* GateIO: Prompt test reviewers to take action if BTC settlement is supported again
* gateio: update rate limit definitions (cherry-pick)
* Add test and missing
* Shared REST rate limit definitions with Websocket service, set lookup item to nil for systems that do not require rate limiting; add glorious nit
* gateio: fix race
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Bitmex: Add index asset to test config
* Subscriptions: Fix unsupported WS assets included
fixed 1692
* Bitmex: Assertify and fix tests
* Subscriptions: Fix all asset subs erroring on no enabled assets
Only subs with Empty asset should do anything when assets are empty.
If asset is set to all and no assets are enabled, we should return
nothing
* Assets: Fix LinearContracts support
* Assets: Fix False positives with IsValid
supportedFlag&a is true for every even number, making the test pretty
close to meaningless.
[This
fix](5b33bc5324)
was a viable fix maintaining bit shifted iota, but there's no benefit to
it at all.
Simplifying.
* add fixes and test
* glorious: nits, privatise upper case field, add item field for case sensitivity for format checks, rm dead code.
* fix potential panic
* gk/glorious: nits
* gk: nits and other things
* improve commentary lol
* glorious+gk: nits and improvements (with no sillyness this time)
* Update currency/pairs.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* thrasher: nitssssss
* linter: fix
* bye bye silly boy
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* updates some rate limits
* WOAH FRIDAY FAILURES
* remove secret code, DONT LOOK
* types.Number, rv ob num, fr changes
* ever so slight neaten
* Remove FundingRateInfo for CMF only
* revert to using the funding rate intervals
* don't need to verify if checksum available; don't retroactively verify snapshots
* glorious: nits + allow checksum/validation to be done on updateByIDAndAction
* possible slice manipulation issue fix for test
* thrasher: nit
* this should fix it now
* for subsystem usage disallow deploying a book twice
* reverts strict policy using deploy, it's used in a bunch of tests I don't want to touch just yet, left a note
* rm unused variable
* glorious: nits
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* bybit: Add time field for orderbook and handle correct timings
* use UpdateID as this is the localised ID for this pair, rm check result.UpdateID == 1 just use snapshot
* general quick fixes across from #1672 which can be used straight away
* fix test
* glorious: nits
* thrasher: nits
* thrasher: nits thanks @thrasher-
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Bybit: Subscription configuration for spot
* Bybit: Enable candles ws sub by default
* Orderbook: Use a RW mutex for depth
* Orderbook: Fix race on depth.VerifyOrderbook
Despite being protected by an ob level mutex, this needed to privatise
and protect the option var.
The subscription pairs do not need to be validated as enabled or
available. The check was just belt-and-braces and didn't have a specific
use-case in mind.
Coinbase has a use-case for wanting to subscribe to BTC-USD when it's
not enabled.
Moreover, it shouldn't be our job to check this. You want a sub expanded
with these pairs? Fine. Done. If it doesn't work, you can work out why
* Exchanges: Allow empty batches in ParallelChanOp
In keeping with both common.Batch and "It's not my responsibility",
ParallelChanOp should just do nothing when given an empty list (and
implicitly an empty batch size.
Whatever it's going to do, it'll delegate to common.Batch,
and this allows us to just inline calls:
```
return common.AppendError(
b.ParallelChanOp(subs.Public(), func(l subscription.List) error { return b.manageSubs(wsSubscribeMethod, l, wsPublicStream) }, len(subs)),
b.ParallelChanOp(subs.Private(), func(l subscription.List) error { return b.manageSubs(wsSubscribeMethod, l, wsPrivateStream) }, len(subs)),
)
```
* Bitmex: Test config updates
* Bitmex: Sub Templating
* Bitmex: Enable websocket for tests
* Bitmex: Handle subscription errors
This switches to multiplexing so that we know which errors belong to
which stream, particularly for the auth attempt
* Bitmex: Fix ws order side err going to data stream
Shouldn't fall into classification error if it's actually a parsing
error
* Cherry_pickable
* gateio/websocket: use millisecond time for more accurate push time
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* gateio: Add multi asset websocket support WIP.
* meow
* Add tests and shenanigans
* integrate flushing and for enabling/disabling pairs from rpc shenanigans
* some changes
* linter: fixes strikes again.
* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.
* Add subscription tests (state functional)
* glorious:nits + proxy handling
* Spelling
* linter: fixerino
* instead of nil, dont do nil.
* clean up nils
* cya nils
* don't need to set URL or check if its running
* stop ping handler routine leak
* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response
* Allow rollback on connect on any error across all connections
* fix shadow jutsu
* glorious/gk: nitters - adds in ws mock server
* linter: fix
* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.
* gk: nits
* Leak issue and edge case
* gk: nits
* gk: drain brain
* glorious: nits
* Update exchanges/stream/websocket.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* glorious: nits
* add tests
* linter: fix
* After merge
* Add error connection info
* Fix edge case where it does not reconnect made by an already closed connection
* stream coverage
* glorious: nits
* glorious: nits removed asset error handling in stream package
* linter: fix
* rm block
* Add basic readme
* fix asset enabled flush cycle for multi connection
* spella: fix
* linter: fix
* Add glorious suggestions, fix some race thing
* reinstate name before any routine gets spawned
* stop on error in mock tests
* glorious: nits
* glorious: nits found in CI build
* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.
* mutex across shutdown and connect for protection
* lint: fix
* test time withoffset, reinstate stop
* fix whoops
* const trafficCheckInterval; rm testmain
* y
* fix lint
* bump time check window
* stream: fix intermittant test failures while testing routines and remove code that is not needed.
* spells
* cant do what I did
* protect race due to routine.
* update testURL
* use mock websocket connection instead of test URL's
* linter: fix
* remove url because its throwing errors on CI builds
* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.
* remove another superfluous url thats not really set up for this
* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly
* linter: fixerino uperino
* glorious: panix
* linter: things
* whoops
* defer lock and use functions that don't require locking in SetProxyAddress
* lint: fix
* thrasher: nits
---------
Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Convert: Fix TimeFromUnixTimestampDecimal using local
All parsed times should be in UTC
* Subscriptions: Add IgnoringAssetsKey
* Tests: Pass tb to curried WS handlers
* Websocket: Make ErrNoMessageListener a public error
* Kraken: Fix URLMap ignored for websocket URLs
* Kraken: Move SeedAssets from Setup to Bootstrap
Having SeedAssets in Setup is cruel and unusual because it calls the
API. Most other interactive data seeding happens in Bootstrap.
This made it so that fixing and creating unit tests for Kraken was
painfully slow, particularly on flaky internet.
* Kraken: Remove convert test
Duplicate of convert_test.go TestTimeFromUnixTimestampDecimal
* Kraken: Test config upgrades
* Kraken: Sub Channel improvements
* Use Websocket subscriptionChannels instead of local slice
* Remove ChannelID - Deprecated in docs
* Simplify ping handlers and hardcodes message
* Add Depth as configurable orderbook channel param
* Simplify auth/non-auth channel updates
* Add configurable Book depth
* Add configurable Candle timeframes
Kraken: Simplify all WS handlers with reqId
* Kraken: Subscription templating
* Generate N+ subs for pairs
If we generate one sub for all pairs, but then fan it out in the
responses, we end up with a mis-match between the sub store and
GenerateSubs, and when we do FlushChannels it will try to resub
everything again.
* Kraken: Rename channelName var throughout
Avoid shadowing func of same name
* Kraken: Add TestEnforceStandardChannelNames
* Websocket: Fix Resubscribe erroring Duplicate
* consolidate type to types package and share across code base
* rm convert type and convert codebase
* rm irrelavant test cases
* Fix tests
* glorious nits
* Update exchanges/gateio/gateio_types.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* thrasher: nits
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* types and orderbook handling fix
* Minor types and endpoints update
* Minor fixes on Kucoin
* Add benchmarking test
* Unit tests update and minor endpoints update
* Adding updates and newly added endpoints
* Add and correct funding, withdrawal, and deposit endpoints
* linter and codespell fix
* Adding and correcting spot trading endpoints
* Completed Spot HF endpoints
* Minor fix
* Added OCO and Margin HF trade and other endpoints
* Adding missing endpoints
* Updating Isolated margin, margin v3, and v3market lending endpoints
* minor codespell fix
* Completed adding and fixing futures endpoints
* wrapper update and fix unit tests
* Updating endpoint ratelimits
* Complete ratelimiter setup and endpoint functions update
* Unit test configuration and update
* fix linter issue
* Added a ratelimiter test and heavy update on unit tests
* Adding websocket update based on ChangeLog
* Added newly added Earn General, Kucoin Earn and Staking endpoints
* Added VIP lending endpoints
* Minor linter and endpoints fix
* Added unit tests, publicised functions, and minor updates
* Update on wrapper funcs, unit tests, and other methods
* Enexport exchange specific websocket methods
* remove deprecated topic
* Update wrapper based on Type, add and fix unit tests
* Added a margin configuration endpoint and unit test
* Update methods, types, and unit tests
* Update error declaration and handling unit tests
* Update method parameters and error handling
* Updating unit tests and error handling
* Update methods arguments, added and update unit tests
* Fix unit tests and wrapper methods
* Resolving unit test issues and fix faulty endpoints
* Fix on unit tests and working on passphrase errors
* Minor fixed on websocket and endpoint url
* comment and wrapper filters issue fix
* Unit tests and other minor updates
* Update wrapper functions, endpoint methods, and unit tests
* change require to change on two unit tests
* Update unit tests, types, and endpoints
* Refine and update wrapper tempo for minor adjustments
* Remove code that enabled logging
* Update wrapper functions, missing endpoints, response and parameter values, and unit tests
* removed High-frequency orders from wrapper functions, and updated unit tests and types
* Added missing fields and minor update on wrapper
* Update types
* Update tests and websocket channels
* Update unit tests and methods error returns
