* kucoin: quick batching support for ticker/trades and orderbooks
* fix test
* kucoin: move pieces add commentry
* kucoin: optimise listOfAssetsCurrencyPairEnabledFor and refactor implementations, address specific orderbook channel subscription handling
* glorious: nits
* thx @thrasher-: nits addressed
* rm types and tests that are not needed
* rm subs checking code, and convert to types.Number
* not needed anymore
* fix tests
* set up reader routine to process updates before init a potential slow websocket subscriber
* implement glorious suggestion
* glorious: nitters
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Websocket: Remove IsInit and simplify SetProxyAddress
IsInit was basically the same as IsConnected.
Any time Connect was called both would be set to true.
Any time we had a disconnect they'd both be set to false
Shutdown() incorrectly didn't setInit(false)
SetProxyAddress simplified to only reconnect a connected Websocket.
Any other state means it hasn't been Connected, or it's about to
reconnect anyway.
There's no handling for IsConnecting previously, either, so I've wrapped
that behind the main mutex.
* Websocket: Expand and Assertify tests
* Websocket: Simplify state transistions
* Websocket: Simplify Connecting/Connected state
* Websocket: Tests and errors for websocket
* Websocket: Make WebsocketNotEnabled a real error
This allows for testing and avoids the repetition.
If each returned error is a error.New() you can never use errors.Is()
* Websocket: Add more testable errors
* Websocket: Improve GenerateMessageID test
Testing just the last id doesn't feel very robust
* Websocket: Protect Setup() from races
* Websocket: Use atomics instead of mutex
This was spurred by looking at the setState call in trafficMonitor and
the effect on blocking and efficiency.
With the new atomic types in Go 1.19, and the small types in use here,
atomics should be safe for our usage. bools should be truly atomic,
and uint32 is atomic when the accepted value range is less than one byte/uint8 since
that can be written atomicly by concurrent processors.
Maybe that's not even a factor any more, however we don't even have to worry enough to check.
* Websocket: Fix and simplify traffic monitor
trafficMonitor had a check throttle at the end of the for loop to stop it just gobbling the (blocking) trafficAlert channel non-stop.
That makes sense, except that nothing is sent to the trafficAlert channel if there's no listener.
So that means that it's out by one second on the trafficAlert, because any traffic received during the pause is doesn't try to send a traffic alert.
The unstopped timer is deliberately leaked for later GC when shutdown.
It won't delay/block anything, and it's a trivial memory leak during an infrequent event.
Deliberately Choosing to recreate the timer each time instead of using Stop, drain and reset
* Websocket: Split traficMonitor test on behaviours
* Websocket: Remove trafficMonitor connected status
trafficMonitor does not need to set the connection to be connected.
Connect() does that. Anything after that should result in a full
shutdown and restart. It can't and shouldn't become connected
unexpectedly, and this is most likely a race anyway.
Also dropped trafficCheckInterval to 100ms to mitigate races of traffic
alerts being buffered for too long.
* Websocket: Set disconnected earlier in Shutdown
This caused a possible race where state is still connected, but we start
to trigger interested actors via ShutdownC and Wait.
They may check state and then call Shutdown again, such as
trafficMonitor
* Websocket: Wait 5s for slow tests to pass traffic draining
Keep getting failures upstream on test rigs.
Think they can be very contended, so this pushes the boundary right out
to 5s
Kucoin delisted XMR temporarily for margin trading.
That seems quite likely long term too, due to XMR's privacy features.
Replace XMR with TRX to hopefully avoid in future
Retain the .msg field of a go fmt.Errorf .msg field returned by .Error()
when wrapping multiple errors.
This fixes a situation where a nested stack of errors would lose
formatting information, which is often used to supply identifying
context.
e.g.
```
err = fmt.Errorf("%w `%s`: %w", errParsingField, fieldName,
parsingError)
errs = common.AppendError(errs, err)
```
This isn't really an issue with our implementation; Calling Unwrap() on a
fmt.Errorf() which returns a wrapErrors will lose that formatting.
Our issue was that we were using just Unwrap() to bind together our
chain-of-custody.
* CI/build: Update Go version, linters and fix minor issues
* Bump golangci-lint to v1.56.1
* BinanceUS: Make uint usage consistent
* Throw blank identifiers into the trash
Calls for limits for margin asset were not doing anything if spot is
enabled.
This caused intermittent failures when the sequence of tests meant
margin was tested before spot, since the limits wouldn't be loaded.
But it also highlighted that API users calling Update Limits for margin
outside the context of a loop on all assets would not get an update.
Also intermittently when the loop of UpdateLimits on assets puts margin first
Our build toolchain is using 1.21 since #1313
This hides issues on 1.20 from go.mod
Notably uses of implied Generics and slices package seem okay, but would
fail for anyone running 1.20 locally
* engine/gRPC proxy: Fix mux regression and enhance test coverage
* Use a temp dir for TLS creds and add credentials test tables
* Update GetRPCEndpoints grpcProxyName ListenAddr field
* Log unauthorised access attempts
* Binance: Fix subscription failures ignored
* Testing: Fix race on shared config singleton
* Config: Privatise Global config var
We should *either* use a private var *or* use an accessor, but it
doesn't make sense to mix paradigms.
Since GetConfig() is well established this instead removes the limited uses of direct public access and adds a Setter
* Zip: Fix test failure on http mocks
* Exchanges: Remove Pair upgrade handling
Now redundant behind #1401. These paths should never be met.
Several legacy coin upgrade paths being deprecated as well: ZUSD and CNY
Expecting any users with bad config from 3+ years ago would have to
reset anyway.
Also: At the time the intention of this was to upgrade the config
format.
However now, instead, it'd mostly serve to reset enabled pairs if
there's a config mistake, which doesn't feel right.
* Kraken: Fix typo in Kraken type struct
* Exchanges: Abstract exchange Start() and Run()
* Exchanges: Add test for abstracted Start
* Exchanges: Move Start to Bootstrap
* Simplify waitgroup usage
* Add call to exchange.Bootstrap to allow overide or supplementation
* Exchanges: Concurrent common bootstap actions
* Gateio: Remove incorrect Run in test
* GateIO: Fix pair dependencies in tests
This ensures that the pairs are initialised no more than needed and
kind-of just-in-time.
Better pattern might be to use a function to get these pairs when we
need them.
* Exchanges: Complete UpdatePairs before ExecLims
If we're going to update pairs, it needs to complete before we check for
limits to avoid errors on old pairs
* Exchanges: Remove Start and Run from tmpl
Since they're replaced by bootstrap now and shouldn't need customisation
normally
* Alphapoint: Move Start to Bootstrap
* GateIO: Fix linter shadow var
* Websockets: Fix subscription failure on reconnect
If a websocket is disconnected then the subscription map was left with
old subscriptions, causing:
`Okx websocket: subscription failure, channel already subscribed for ...`
* fixup! Websockets: Fix subscription failure on reconnect
Fixes review comment https://github.com/thrasher-corp/gocryptotrader/pull/1457#discussion_r1468947509
* Bitfinex: Handle Errors in tickers without panic
* Bitfinex: Use TypeAssertError for tickers
* Bitfinex: Refactor and improve ticker handling
* Unify Ticker response handling
* Simplify/Humanify errors in parsing ticker responses
* Remove polymorphic response handling for < 10 fields, seems [antiquated according to docs](https://docs.bitfinex.com/reference/rest-public-ticker)
* Add test coverage for tickers
* Bitfinex: Ignore resp format errs in GetTickers
We're still getting:
`received 'invalid ticker response format for tALT2612:USD field BidSize
from [100 <nil> 100 <nil> <nil> <nil> <nil> <nil> <nil> <nil>]`
too frequently right now.
Considered:
* Special casing tALT2612:*; However if it's happening with this curr
it'll probably happen again
* Warning about the error; However it'd just be persistent,
unactionable and annoying noise in the logs of a running server
So the conclusion is to just silently ignore it
* Bitfinex: Accept locked market in TestUpdateTicker
* Websockets: Move Subscription to its own package
This allows the small type to be imported from both `config` and from
`stream` without an import cycle, so we don't have to repeat ourselves
* Subs: Renamed Currency to Pair
This was being mis-used through much of the code, and since we're
already touching everything, we might as well fix it
* Websockets: Add Subscription configuration
* Binance: Add subscription configuration
* Kucoin: Subscription configuration
* Simplify GenerateDefaultSubs
* Improve TestGenSubs coverage
* Test Candle Sub generation
* Support Candle intervals
* Full responsibility for formatting Channel name on GenerateDefaultSubs
OR consumer of Subscribe
* Simplify generatePayloads as a result
* Fix test coverage of asset types in processMarketSnapshot
* Exchanges: Abstract ParallelChanOp
* Tests: Generic ws mock instances
* Kucoin: Fix intermittent conflict in test currs
Use isolated test instance for `TestGetOpenInterest`.
`TestGetOpenInterest` would occassionally change pairs before
GenerateDefault Subs.
* Dispatch: Assertify tests
* Dispatch: Fix race during stop
If we have blocking writers, then we need to synchronise them exiting
before closing off their channels.
* Dispatch: Rename Routes mutex for clarity
* adds open interest to exchanges
* ADDS TESTING YEAH
* New endpoints, BTSE, RPCS, cached
* slight design change, begin gateio
You will need to get cached for
each exchange that supports it
* gateio, huobi, rpc
* fix up kraken, cache retrieval
* okx, gateio
* finalising all implementations and tests
* definitely my final ever commit on this
* Well, well, well
* final v2
* quick fix of bug
* test coverage, assert notempty, test helper
Added a new testhelper for currency
management because its very annoying
in a parallel test setting which wastes
so much space otherwise
* minimises REST requests for Open Interest
* types.Number merge misses
* Minimises Kraken REST calls
* len change, value -> pointer receiver
* further fixup
* fixes gateio, batch calculates open interest
* single gateio, lint const fixes
* rejig and more thorough oi for huobi
* formatting expansion
* minor fix for handling expiring contracts
* rm unused Binance strings
* add bybit support, fix bybit issues
* oopsie doopsie, dont look at my whoopsie
* Fix issue, remove feature
* move an irrelevant function for the pr
* mini bybit upgrades
* fixes cli request bug
This test was failing with `-tags mock_test_off` because the API imposes
a default limit of 500 now, so we'd just get 500 trades and no batching.
If the user doesn't provide a limit, that means we need to assume
batching
* Adding Bybit public endpoints
* Completed adding market endpoints
* Added trade endpoints
* Adding position endpoints
* completing position endpoints
* Adding Pre-upgrade endpoints
* Completed adding Pre-upgrade and Account endpoints
* Added asset endpoints
* Added user endpoints and unit tests
* Adding spot leverage and margin trade endpoints
* spot margin trade added
* added spot-margin-trade, institutional lending, c2c lending, and broker
* Adding wrapper funnctions
* Working on wrapper public methods
* Added wrapper functions and unit tests
* Added websocket support with unit tests
* Update websocket handlers and added rate-limiter
* wrapper function, websocket handlers, and linter issues fixe
* unit tests fixes and codespell correction
* Update documentation
* Minor websocket handling fix and URL consts merging
* types, unit test other updates
* Updated websocket and methods based on review
* Added GetFeeByType method with unit test and fixes
* add filter for Unified and Normal endpoints
* Mock recording and unit tests update
* minor linter issue fix
* websocket and rest tests and fix
* change asset types and wrapper methods update
* rm: forgotten panic message
* endpoints, websocket and unit tests update
* Added and updated endpoints and unit test
* linter and spell fix
* unit test and orders update
* Update on endpoints, fields, config pairs and formating, and unit tests
* minor update on responses
* Fix unit test and types
* Unit tests, models, and wrapper issues fix and mock test recording
* rm print statement
* Fix issue, add FundingRate wrapper func, mock record
* minor type and unit test update
* Update on order handling and unit test
* Minor test
* Minor fix in wrapper
* unit tests update, recording, and documentation update
* Unit tests and minor wrapper function update
* minor unit test fix
* Added newly added endpoints, unit tests, and mock recording
* Rename GetInstruments -> GetInstrumentInfo
* doc update
* Minor unit tests update
* Minor unit test and wrapper update
* Fix linter issue
* Add unit test and minor updates
* Revert websocket error declaration
* Revert websocket error declaration
* Balace --> Balance
* Fix config issues
* Added next funding time minor fix
* Update GetLatestFundingRates and record mock test data
* Added LatestFundingRate time
* Fix test issue of TestAllExchangeWrappers
* config pairs update
* configtest spot pairs update
* Minor update on options UpdateOrderExecutionLimits wrapper func
* Linter issue fix and added new currency codes
* Added new currency codes
* Update bybit pairs in config_example
* config assets pair format update
* Bitfinex: Fix TestUpdateTickers on ICE
Accept 5% failure rate of available pairs not working as tickers.
When Bitfinex [delisted ICE](https://www.bitfinex.com/posts/990) it's still coming back in all pub:conf and pub:info listings:
```
❯ curl -s https://api.bitfinex.com/v2/conf/pub:info:pair | jq -c '.[0][] | select(.[0] | test("(BTC|ICE|ZIL)USD")) ' ["BTCUSD",[null,null,null,"0.00006","2000.0",null,null,null,0.1,0.05,null,null]]
["ICEUSD",[null,null,null,"4.0","25000.0",null,null,null,null,null,null,null]]
["ZILUSD",[null,null,null,"40.0","1500000.0",null,null,null,null,null,null,null]]
```
_( I included ZIL to show a tradable pair without Margin fields )_
There's absolutely no sign it's not a tradable pair _until_ you ask for a ticker for it:
```
❯ curl -s https://api.bitfinex.com/v2/ticker/tICEUSD ["error",10020,"symbol: invalid"]⏎
❯ curl -s https://api.bitfinex.com/v2/ticker/tBTCUSD [42854,11.8920918,42855,12.71095442,-290,-0.00672292,42846,725.08132142,43288,41850]⏎
```
* fixup! Bitfinex: Fix TestUpdateTickers on ICE
* fixup! Bitfinex: Fix TestUpdateTickers on ICE
* Bitfinex: Fix UpdateTickers stopping on first error
* CI: Add macOS, Windows and Linux arm64 support
* Modify arm64 job name to be inline with the others
* linter: Prevent cache from causing "Cannot open: File exists" issues
* Use setup-go and setup-node's inbuilt caching
