* CI/build: Update Go version, linters and fix minor issues
* linters: Add intrange, copyloopvar, additional go vet linters to match gopls and fix issues
* Kraken: Fix TestGetOpenInterest
We see daily failures on OpenInterest for Kraken.
This fix assumes that the issue might be related to volume of ETHUSD
open interest, and switches the single pair test to XBTUSD instead
Also isolates the test from others, since we're changing stored pairs
and we might be colliding on the global k
* Kraken: Handle Errors field in futures response
* Add in extra field for colocation monitoring
* rm tags
* populate through gateio orderbooks
* populate through incremental updates
* Add insert at field for orderbook depth
* I AM A BAD SPELLLLLLA
* add it in
* c change
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Currency: Variadic Pairs.Add
This version of Pairs.Add is simpler and [more
performant](https://gist.github.com/gbjk/06a1fc1832d04ee41213ca518938cf74)
Behavioural difference: If there's nothing to add, the same slice is
returned unaltered. This seems like good sauce
* Currency: Variadic Remove
* Common: Add Batch function
* Common: Add common.SortStrings for stringers
* Subscriptions: Add batching to templates
* Subscriptions: Sort list of pairs
* Kucoin: Switch to sub templating
* Kucoin: Simplify channel prefix usage
* Kucoin: Fix race on fetchedFuturesOrderbook
* Subscriptions: Filter AssetPairs
Now only the assetPairs relevant to the subscription are in the context
* Subscriptions: Respect subscription Pairs
* Subscriptions: Trim AssetSeparator early
We want to trim before checking for "AssetSeparator vs All" because a
template should be allowed to reuse a range template and generate just one trailing AssetSeparator
whilst using a specific Asset
* Kucoin: Fix empty margin asset added
* Kucoin: Add Subscription batching
Turns out that contary to the documentation, kucoin supports batching of
all symbols and currencies
* Kucoin: Fix checkSubscriptions and coverage
* Subscriptions: Simplify error checking
This reduces the complexity of error checking to just be "do we get the
correct numbers".
Fixes Asset.All with only one asset erroring on xpandPairs, because we
trimmed the only asset separator, and then errored that we're not
xpanding Assets and the asset on the sub is asset.All
This use-case conflicted with commit 6bbd546d74, which required:
```
Subscriptions: Trim AssetSeparator early
We want to trim before checking for "AssetSeparator vs All" because a
template should be allowed to reuse a range template and generate just one trailing AssetSeparator
whilst using a specific Asset
```
Now we set up the assets earlier, and we remove the check for xpandAssets, since the number of asset lines matching is all that matters.
I've removed the asset tests for this, but they were correctly erroring
on the number of asset lines instead.
Everything hits coverage, as well.
* Kucoin: Remove deprecated fundingBook endpoint
* BTCMarkets: Use common.Batch
* Kraken: Simplify REST error and result handling
* Kraken: Rid excessive map lookups or large copies when iterating
* Kraken: Prealloc orderbook types.Numer size
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
---------
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Making Pairs(Add) Variatic
* Slight improvements
* Implementation overhaul
* Improvements
* Changing code which can use the variadic functionality
* Fixing silliness
* thing left over
* Subscriptions: Add List.AssetPairs
* Subscriptions: Add Template and QualifiedChannel
These fields separate the concept of what the channel is from the
qualified resource name
* Subscriptions: Add List.SetStates()
* Subscriptions: Add List.QualifiedChannels
* Subscriptions: Rename testsubs.EqualLists
* Binance: Switch to ExpandTemplates
* Binance: Update ConfigTest format
* Subscriptions: Test Coverage improvements
* Subscriptions: Reenterant List.ExpandTemplates
* Subscriptions: Move templates from subscriptions to exchanges
* Binance: Inline subscription template and improvements
* Add in initial handling for quote/base currency deployment requirements
* include client order ID checking
* glorious: suggestions
* spell and fix
* linter/context/test
* rework tests and order side specific requirements
* linter
* mend panic at the disco
* mending more panics at the disco
* anudda fix brudda
* glorious: NITTTTTT BOOOOOMB
* leftover things and stuff
* whoops
* tie in gateio
* glorious: nit fixes life and everything.
* thrasher: nits
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Common: DriveBy sanitisation of vars
* BTSE: Unify to exchange OrderLimits
* BTSE: Remove SeedAssets and test GetOrderExcutionLimit
* BTSE: Fix handling for K_* pairs
In addition to the M_ pairs we previously handled, BTSE has now
introduced K_* pairs (K_SATS-USD*).
This change moves the handling over to look for an exponent, and moves
the filtering to happen on all market data.
The original MarketSummary is still availiable, but I can't see any of
our current use-cases wanting to get the unfiltered list
* BTSE: Fix marketSummary futures field
BTSE returns no futures field for futures api marketInfo, and the documentation for the futures api shows returning false.
When we know we asked for futures, it makes the data flow much saner if
we can trust this field and not have to track what asset we asked for
* BTSE: Abstract marketPair.Pair()
* BTSE: Fix UpdateTicker symbol format
* gateio: update FuturesAccount struct
* gateio: update Position struct
* gateio: rm redundant checks and add in actual required checks
* export GateioTime
* linter: fix
* linter: again fix
* issue: fix
* gateio: update fee struct and such
* Update exchanges/gateio/gateio_types.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Update exchanges/gateio/gateio_convert.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* use Time type acrost file
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Websocket: Use ErrSubscribedAlready
instead of errChannelAlreadySubscribed
* Subscriptions: Replace Pair with Pairs
Given that some subscriptions have multiple pairs, support that as the
standard.
* Docs: Update subscriptions in add new exch
* RPC: Update Subscription Pairs
* Linter: Disable testifylint.Len
We deliberately use Equal over Len to avoid spamming the contents of large Slices
* Websocket: Add suffix to state consts
* Binance: Subscription Pairs support
* Bitfinex: Subscription Pairs support
* Bithumb: Subscription Pairs support
* Bitmex: Subscription Pairs support
* Bitstamp: Subscription Pairs support
* BTCMarkets: Subscription Pairs support
* BTSE: Subscription Pairs support
* Coinbase: Subscription Pairs support
* Coinut: Subscription Pairs support
* GateIO: Subscription Pairs support
* Gemini: Subscription Pairs support and improvement
* Hitbtc: Subscription Pairs support
* Huboi: Subscription Pairs support
* Kucoin: Subscription Pairs support
* Okcoin: Subscription Pairs support
* Poloniex: Subscription Pairs support
* Kraken: Add subscription Pairs support
Note: This is a naieve implementation because we want to rebase the
kraken websocket rewrite on top of this
* Bybit: Subscription Pairs support
* Okx: Subscription Pairs support
* Bitmex: Subsription configuration
* Fixes unauthenticated websocket left as CanUseAuth
* Fixes auth subs happening privately
* CoinbasePro: Subscription Configuration
* Consolidate ProductIDs when all subscriptions are for the same list
* Websocket: Log actual sent message when Verbose
* Subscriptions: Improve clarity of which key is which in Match
* Subscriptions: Lint fix for HugeParam
* Subscriptions: Add AddPairs and move keys from test
* Subscriptions: Simplify subscription keys and add key types
* Subscriptions: Add List.GroupPairs Rename sub.AddPairs
* Subscription: Fix ExactKey not matching 0 pairs
* Subscriptions: Remove unused IdentityKey and HasPairKey
* Subscriptions: Fix GetKey test
* Subscriptions: Test coverage improvements
* Websocket: Change State on Add/Remove
* Subscriptions: Improve error context
* Subscriptions: Fix Enable: false subs not ignored
* Bitfinex: Fix WsAuth test failing on DataHandler
DataHandler is eaten by dataMonitor now, so we need to use ToRoutine
* Deribit: Subscription Pairs support
* Websocket: Accept nil lists for checkSubscriptions
If the user passes in a nil (implicitly empty) list, we would not panic.
Therefore the burden of correctness about that data lies with them.
The list of subscriptions is empty, and that's okay, and possibly
convenient
* Websocket: Add context to NilPointer errors
* Subscriptions: Add context to nil errors
* Exchange: Fix error expectations in UnsubToWSChans
* initial consolidation of rate limiting code to request package to reduce bespoke code implementation
* continued
* finish abstraction
* lint
* exchanges: fix tests
* linter: fix
* poloniex: fix auth rate limit not being set
* ratelimiter: convert from token to weight
* glorious: nits addressed with fire
* linter: rip
* change func name set -> get
* fix test
* derbit: impl
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* deribit implementation
* add ws impll
* cleanup
* Update deribit_wrapper.go
* Add missing endpoints
* Fix config file
* asset type update
* Update code structure
* Update authenticated private endpoints unit tests
* Updating websocket
* Updating websocket connection and subscription handling
* Finishing up adding subscription push data
* Adding websocket public endpoint
* Adding WS endpoints
* Adding websocket unit tests
* Minor clean-up
* Integrating websocket endpoints into the wrapper funcs
* Updating exchange documentations
* Fixing test issues
* Code cleaning-up
* fix test issues
* Updating validations and logic errors
* Updating wrapper issues
* fix test issues
* Slight test update
* Unit test and code structure update
* Update websocket tempos
* Slight update on code structure
* Minor update on unit tests
* Update depending on review comments
* Minor code fix and doc re-generating
* Update on Candlestick wrapper functions
* Minor updates
* minor unit test updates
* Minor updates on weboscket and unit tests
* minor linter fix
* codespell and rate limiter issues
* single linter issue fix
* adding rate limiter
* Add ratelimiter to websocket conn and overall code update
* fix websocket push data issue
* Implementing missing wrapper function
* Websocket fix
* Minor update on missing endpoint and other
* fixing websocket issues and cleaningup
* Minor tempo fix
* Minor linter issues
* unit test update
* Indexing error fix
* Websocket connection fix
* string formatting fix
* Small fix on unit tests
* fix minor json conversion issue
* websocket and documentation update
* websocket, wrapper and unit test updates
* Documentation and unit tests update
* Fix unit tests
* wrapper fix for new change
* Unit test fix
* timestamp conversion and unit tests update
* Minor instrument ID conversion fix
* instrument formats and unit test update
* formatting and unit test fix
* config update
* Updating websocket and adding the Spot support
* Add small unit test fix
* unit test and websocket handlers update
* Linter issues fix
* minor documentation and code update
* Minor fix
* added a wrapper func GetLatestFundingRates
* Types, wrapper update, and unit tests
* Minor config update
* fix wrapper unit tests
* Resolve all panic and wrapper test issues
* minor unit test fix
* fix issues and adding newly added endpoints
* updates and added remaining endpoints with unit tests
* Update unit tests using assert
* Added missing endpoints and unit tests
* Minor updates and clean-ups
* Resolve tradable pair fetching panic
* Mutex fix
* Added Options assets test and minor fixes
* subscription mothod updated
* Remove misadded code lines
* resolve websocket
* Updating tests, types, endpoint methods and others
* Added GetFuturesContractDetails and minor fix
* fix linter issue
* revert change on candlestic time
* Added filters to candles
* minor unit test and wrapper fix
* Minor unit tests update
* cahnge param key for GetOrderMarginByID
* updating unit tests and resolve issues
* Update websocket unit tests
* Minor fix based on review
* Revert unit test change
* fix pair config issue
* Added missing wrapper functions
* Fix missing review changes
* Fix options request pair formatting
* fix AllExchangeWrappers test issue
* Changes with unit test and wrapper based on the review
* Fix to options reg-exp
* wrapper functions fix
* Update MaximumFundingRateHistory filter and minor fixes
* Fix besed on review comment
* Fix issues on review comment
* linter fix
* fix minor unit test issue
* Fix unit test issues
* Update trade order cancellation responses
* fix config files issue
* lint update config files
* Update unit tests
* Update return values and response handling
* added missing endpoint and fixes based on review comment
* toggle useTestNet back
* Update cancel by label and other fix
* fix forgotten cancel all response type
* update CancelResp type
* Fix unmarshaling error
* updated websocket orderbook load issue
* fix websocket lock and groups
* Change Items to Tranche and fix linter issues
* Fix orderbook issue
* Update unit tests offline error handling, and endpoints argument and error handling
* Contributors documentation update and change error return type
* Updated unit tests based on review comment
* Update unit tests and removed password change endpoint
* Fix race condition
* Update on tests, test pairs, and wrapper config
* Update test tradable pairs loading
* Update unit tests, fix linter issues, and update wrapper functions
* remove credentials
* Update test and fix authentication method and few authenticated endpoints
* fix codespell issue
* group the repeated currency code check to a func
* added unit test for repeated pair check func
* Added a base coin and related updates
---------
Co-authored-by: E Sequeira <earncef@earncef.com>
* currency/manager: remove deadlock potential in concurrent operations
* fix kucoin test, lets see what this does shalllllll weeeeee.
* another fix in the deep dark depths
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Tests: Move and simplify TestFixtureToDataHandler
* Currency: Fix PairsManager.Load breaking matcher
* Tests: Add multi-instance cache to UpdatePairsOnce
* Kraken: Fix TestUpdateTickers race error
Calling StorePairs on global instance can lead to race
* Bitfinex: Fix TestUpdateTickers racing intermittently
* Currency: Fix concurrent access to PM formats
* Currency: Fix SupportsAsset implementation
This should delegate entirely to PairManager's IsAssetSupported
* Okx: Fix PM intrusion, rm GetPairFromInstrumentID
* Exchange: Fix SetGlobalPairsManager to set asset enabled
* Bitflyer: Fix race on set TestGetCurrURL
TestGetCurrencyTradeURL would fail sometimes due to sequencing of
enabling futures but not having pairs for it.
* Tests: Simplify usage pattern for FixtureToDH
* initial purge and benchmarks proof before rn overhaul
* rn LinkedList -> Tranche(s) and purge references
* roll out acrost exchanges
* linterino
* rn silly billy label
* linter strikes AAAAAGAIN!
* fix some things
* rm comment
* Add actual comparison from master to branch benchmark for sorting algorithms
* lower case via git mv YAAY!
* drop code
* convert type name
* glorious: nits
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Test: Internal exchange coverage
* Tests: Rename exchange.TestInstance to Setup
This package function is either going to be imported and used as just exchange.Setup,
or more likely testexch.Setup.
This removes the Stutter of having
internal/testing/exchange.TestInstance which is implicit given the
package path
* gateio: fix spot deployment issue
* fix status bug add test
* to actual return type
* fix linter
* ch type
* glorious: nits
* rm space
* glorious: nits
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Currency: Remove Pair Index formatting/parsing
This feature was originally for exchanges with only one pair (e.g. KRW) which made parsing easier.
However there's no examples of this left, and we can reduce complexity
overall by removing it.
* Exchange: Partial assertify tests and fixes
* Currency: Fix panic on a delimiterless small currency
* Kraken: Fix wsCancelOrders not erroring order id
We were using the "cancel many" facility of the Kraken api.
However since that doesn't actually report errors individually, it seems
saner to just multiplex over it.
We were going to get N+ responses anyway. Might as well send N+ requests
* Common: Add ErrorCollector and methods
* improv. timed mutex
* Add all protection back in and jankyness because races. :'(
* Add intial benchmarkeroos
* Add master benchmarks
* goodness me
* what?
* what again?
* glorious: nits
* just a swaperino instead
* clean up package nonce so that we only need to aquire mutex once
* unlock before checking master
* commentary
* wha
* more comment
* ch comment
* nonce: Allow for broad customisation externally with a ~2ns overhead
* glorious: nits maybe works?
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Shift wrapper function GetDefaultConfig to exchange.Base method definition, to ensure set defaults doesn't get called twice and to reduce code
* rm alphapoint bootstrap method as is defined as exchange.Base method
* add tests
* glorious: make it a function and make it IBOTEXCHANGE
---------
Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
WsConnect is calling GetInstruments, and when that fails, erroring out
and not subscribing to anything.
The response to get instruments is an object {}, which skips the
IncomingWithData check in wsReadData for arrays [].
The check in wsHandleData depended on client_ord_id, but I figure so
long as there's a nonce which matches, we can palm it off.
This results simutaneously in having to move the login handler back to
it's waiting nonce-parser, and also simplifying/deduping it.
Failed again due to kraken re-setting limits due to price increases.
Switch from specific PriceStepIncrementSize and MinimumBaseAmount to just ensuring they are positive so we don't have to do this again
* Websocket: Prevent panic on Disconnect/Connect
Previously we'd set the websocket to disconnected when *either* of the
Conn/AuthConn got a disconnect message.
This meant that:
* The other connection was still functioning
* A user would be free to call Connect()
* wsReadData() may not have exited
Any call to Connect would be acceptable, and we'd probably get a panic
from the underlying shared/re-used gorilla websocket:
`panic: runtime error: slice bounds out of range [:13501] with capacity 8192`
when a new wsReadData goro is started and the old tries to ReadMessage
as well, overallocating the buffer.
This wouldn't normally occur because trafficMonitor would see traffic
(on either connection) and then set the websocket to being connected
again.
The solution is to treat a Disconnect on either websocket as a call to
Shutdown the whole websocket, and then trafficMonitor can reconnect it
properly.
Unit Testing for this has been difficult. So far I've had to rely on a
disruption inside websocket's connectionMonitor itself to reproduce the
panic, but from there it's been very consistent.
* Websocket: Fix race on DataHandler during shutdown
Previously we would encounter situations where shutdown would race and
fail TestConnectionMessageErrors, demonstrating that consumers might not
be told why their connection is closing.
* Do not drain DataHandler on dataMonitor shutdown
Consumers should be allowed to process whatever messages were in flight
prior to a socket shutdown
* Ensure all DataHandler messages are sent to ToRoutine before shutdown
This avoids a race where we'd read a message from DataHandler, but then
process a shutdown before trying to relay it. The relay is non-blocking
anyway, so we can trust that we'd pick up the Shutdown during the next
loop.
* Send errors to DataHandler before starting a shutdown
This just reduces the chance of a race on shutdown processing
* Websocket: Log counter of dropped messages
* gateio: fix unmarshal bug and update fields
* gateio: fix wrapper function function, add helper methods
* update order types and add kucoin wrapper fix
* currency pairs
* Add tests
* gateio; inspect error and continue for no funds in account, kucoin: fetch all settlement amounts
* futures: order fixit
* finish off gateio updates for market orders
* cute line
* Update exchanges/kucoin/kucoin_wrapper.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Update exchanges/kucoin/kucoin_wrapper.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Update exchanges/gateio/gateio.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Update exchanges/gateio/gateio_wrapper.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* Update exchanges/gateio/gateio_wrapper.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* glorious: nits
* glorious: nits - filter by pair match and fix bug where the endpoint returns details instead of message
* Add fix for leverage check (non-merge) my ip has been blocked from gateio still... scammmmmmmm
* glorious: nitters
* Update exchanges/gateio/gateio_test.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Update exchanges/gateio/gateio_test.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Update exchanges/gateio/gateio_test.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* kucoin: quick batching support for ticker/trades and orderbooks
* fix test
* kucoin: move pieces add commentry
* kucoin: optimise listOfAssetsCurrencyPairEnabledFor and refactor implementations, address specific orderbook channel subscription handling
* glorious: nits
* thx @thrasher-: nits addressed
* rm types and tests that are not needed
* rm subs checking code, and convert to types.Number
* not needed anymore
* fix tests
* set up reader routine to process updates before init a potential slow websocket subscriber
* implement glorious suggestion
* glorious: nitters
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Websocket: Remove IsInit and simplify SetProxyAddress
IsInit was basically the same as IsConnected.
Any time Connect was called both would be set to true.
Any time we had a disconnect they'd both be set to false
Shutdown() incorrectly didn't setInit(false)
SetProxyAddress simplified to only reconnect a connected Websocket.
Any other state means it hasn't been Connected, or it's about to
reconnect anyway.
There's no handling for IsConnecting previously, either, so I've wrapped
that behind the main mutex.
* Websocket: Expand and Assertify tests
* Websocket: Simplify state transistions
* Websocket: Simplify Connecting/Connected state
* Websocket: Tests and errors for websocket
* Websocket: Make WebsocketNotEnabled a real error
This allows for testing and avoids the repetition.
If each returned error is a error.New() you can never use errors.Is()
* Websocket: Add more testable errors
* Websocket: Improve GenerateMessageID test
Testing just the last id doesn't feel very robust
* Websocket: Protect Setup() from races
* Websocket: Use atomics instead of mutex
This was spurred by looking at the setState call in trafficMonitor and
the effect on blocking and efficiency.
With the new atomic types in Go 1.19, and the small types in use here,
atomics should be safe for our usage. bools should be truly atomic,
and uint32 is atomic when the accepted value range is less than one byte/uint8 since
that can be written atomicly by concurrent processors.
Maybe that's not even a factor any more, however we don't even have to worry enough to check.
* Websocket: Fix and simplify traffic monitor
trafficMonitor had a check throttle at the end of the for loop to stop it just gobbling the (blocking) trafficAlert channel non-stop.
That makes sense, except that nothing is sent to the trafficAlert channel if there's no listener.
So that means that it's out by one second on the trafficAlert, because any traffic received during the pause is doesn't try to send a traffic alert.
The unstopped timer is deliberately leaked for later GC when shutdown.
It won't delay/block anything, and it's a trivial memory leak during an infrequent event.
Deliberately Choosing to recreate the timer each time instead of using Stop, drain and reset
* Websocket: Split traficMonitor test on behaviours
* Websocket: Remove trafficMonitor connected status
trafficMonitor does not need to set the connection to be connected.
Connect() does that. Anything after that should result in a full
shutdown and restart. It can't and shouldn't become connected
unexpectedly, and this is most likely a race anyway.
Also dropped trafficCheckInterval to 100ms to mitigate races of traffic
alerts being buffered for too long.
* Websocket: Set disconnected earlier in Shutdown
This caused a possible race where state is still connected, but we start
to trigger interested actors via ShutdownC and Wait.
They may check state and then call Shutdown again, such as
trafficMonitor
* Websocket: Wait 5s for slow tests to pass traffic draining
Keep getting failures upstream on test rigs.
Think they can be very contended, so this pushes the boundary right out
to 5s
Kucoin delisted XMR temporarily for margin trading.
That seems quite likely long term too, due to XMR's privacy features.
Replace XMR with TRX to hopefully avoid in future
