* Config: Tighten config version handling as uint16
This constrains the versions to uint16 and improves error handling.
LatestVersion becomes literally that.
Fixes handling for negative or overflowing versions in config
* Config: Rename LatestVersion to UseLatestVersion
If one asset is enabled but has no enabled pairs, we should ignore that
asset, even for non-pair related subscriptions.
That matches the existing code, but wasn't happening in the context of
asset.All subscriptions with just one asset in this state.
If a user wanted to have non-pair subscriptions, they should use
asset.Empty. Would expect other breakages with no pairs enabled, too.
Note: No enabled pairs for an enabled asset is a transient issue which
can occur due to enableOnePair racing against no available pairs. The
second run, available pairs would be populated and enableOnePair would
work. This situation could persist due to running with --dry or
containerised, though.
Fixes:
`
Okx websocket: subscription failure, myOrders all : subscription template did not generate the expected number of pair records for spread: Got 1;
Expected 0
`
Relates to #1420
* Send trades down the DataHandler
When the tradeFeed is enabled, send the trades down the DataHandler
Add TestWSProcessTrades
* Update assertions
* Add check against null references in slices
* Add an error for ws parsing field to common
* Update kraken websocket ProcessTrades
Send individual trades down the DataHandler
Send multiple trades in test
Test error if the trade length is too short
Nits
* Fix nits
* Add check for subscription store insertion and validation with tests
* bybit: fix subscriptions
* deribit: fix subscriptions
* linter: fix
* glorious/nits: add test for updateChannelSubscriptions RM GetChannelDifference method as its only used locally and Diff method can be accessed directly
* glorious/nits: add to store in the loop; add correct formatting to template for edge case perps with settlement
* spelling: fix
* glorious/nit: silly billy
* gk: nits
* gk/nits: split PartitionByPresence into Contained and Missing
* gk/nits: formatPerpetualPairWithSettlement -> formatChannelPair
* stream/websocket: stop full websocket disconnect on Connect when encountering subscription specific error paths
* stream/websocket: rm nil assignment
* glorious: nits
* gk: niterinos
* Update exchanges/stream/websocket_test.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Update exchanges/stream/websocket_test.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Update exchanges/stream/websocket_test.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* thrasher: nits
---------
Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* CI: Replace docker amd64 with native runner, switch Docker test arch to amd64
* tests: Remove spaces and new lines from test JSON timestamp fields
* builds: Disable sonic for arm64 due to OKX unmarshal issue
* nits: Improve Dockerfile comment and fix build tag
* tag optional sonic and allow full library conversion
* Add workflow and disallow arm and darwin usage
* Add basic hotswap benchmark
* linter: fix
* use bash
* linter: fix?
* Fix whoopsie, add to make file, also add mention in features list.
* test enforcement
* actually read documentation see if this works
* linter: fix
* linter: fix
* sonic: bump tagged version
* encoding/json: drop build tag arch and os filters
* encoding/json: consolidate tests
* encoding/json: log build tag usage
* rm superfluous builds
* glorious/nits: add template change and regen docs
* glorious/nits: update commentary on nolint directive
* glorious/nits: rm init func and log results in main.go
* Test to actually pull flag in
* linter: fix
* thrasher: nits
* gk: nits 4 goflags goooooooooo!
* gk: nits rn
* make sonic default json implementation
* screen 386
* linter: fix
* Add commentary
* glorious: nits Makefile not working
* gk: nits
* gk: nits whoops
* whoopsirino
* mention 32bit systems won't be sonic
* gk: super-duper nit of extremes
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* Config: Rename DecryptConfigFile to DecryptConfigData
Because this isn't really a file, it's a byte slice
* Config: Rename EncryptConfigFile to EncryptConfigData
Because it's not actually a file
* Config: Fix config version downgrade
Fixes#1769
* Bitfinex: Fix WS trade processing
* Add handling for funding trades
Fixes#1746
* Linter: Disable shadow linting for err
It's been a year, and I'm still getting caught out by govet demanding I
don't shadow a var I was deliberately shadowing.
Made worse by an increase in clashes with stylecheck when they both want
opposite things on the same line.
* Bitfinex: Move websocket constants to websocket file
* Bitfinex: Rename channel consts
* Bitfinex: Send individual WS trades down the DataHandler
* update acccount ratelimits and added missing endpoints
* completed mapping missing trade accoung REST endpoints and Rate Limit
* added orderbook trading missing REST endpoints
* Added few missing endpoints and unit tests
* Completed grid trading and signal bot trading with unit tests
* Added Recurring Buy endpoints and corresponding unit tests
* Added copy trading endpoints and unit tests
* added newly added block trading and spread endpoints
* completed mapping spread endpoints
* Added new endpoints and unit tests
* Added round 1: Okx types and converts update.
* Update endpoints handling and types update
* Removed constants, updated unit tests, and updated endpoint methods
* Slight endpoint and unit test update
* Added spread and other websocket endpoints and update
* completed Spread WS Orderbook handler
* Added missing spread channels and handlers
* Adding Bussinss websocket and missing subscriptions, update unit tests, and endpoints
* Added spread endpoints to wrapper and unit tests update
* Added missing websocket subscriptions and copy trading endpoints
* Added missing endpoints and re-organize business websocket handlers
* Docs update
* Endpoints code updates
* types, unit test and endpoints update
* Minor unit tests update
* spelling fix
* fix unit test issues
* Updating unit tests error handling
* Updating unit tests error handling
* Unit tests assertion handling update
* Unit tests update
* Resolve linter issues
* linter issues fix
* Orderbook unit test error fix
* Minor fixes
* Change on test handling and types
* Updating unit tests and cleanups
* Fix unit test issues
* Add ratelimit test and update unit tests and methods
* Update method parameters and error declarations
* Added lending endpoints, unit tests, and update endpoint methods and error declarations
* Update ratelimiters, add missing lending and trading endpoints and unit tests
* Update websocket authentication and subscription handling
* Minor update to unit test and types
* Types, error handling, and other minor updates
* Update unit tests and endpoint functions
* error declarations update and unit tests
* Overall update on unit endpoint, unit tests, and types
* Adding review fixes
* Update on endpoints, order types, and unit tests
* Update unit tests and endpoint functions
* Update on endpoint and update missing parameters
* Implemented and tested newly added trading endpoints
* endpoints update and unit tests
* Update missing endpoints and update overall code
* added and testing funding and fiat related endpoints
* Update on convert and fiat related endpoints
* linter fix, types, endpoints, and unit tests update
* linter issues fix
* revert changes on tempos
* Fix Panic and update websocket auth calls handling
* config linter issue fix
* Fix panic issues and update unit tests
* Fix race condition, TestAllExchangeWrappers unit test issues
* Fix withdrawal manager test issue
* Rename ClosePositionForContractrID --> ClosePositionForContractID
* Rename ClosePositionForContractrID --> ClosePositionForContractID
* Fix error
* endpoints update and fix unit test issues
* added unit tests, updated comments, and code sections
* revert change in defaultSubscriptions
* few types comments update
* Minor changes
* remove lending endpoints
* rm mistakenly added code
* fix unit test
* minor unit test fix
* Adding offline error tests, update endpoints function, config update
* Update unit test coverage for offline error handliing
* Updating wrapper algo order coverage, endpoint calls, and unit tests
* Updating wrapper trade functions to accomodate algo orders
* update wrapper unit tests
* Fix wrapper order functions offline errors handling
* Tested and updated wrapper order functions
* Address review comments
* update order unit tests, and okx endpoint functions
* finalize affected order, endpoint, and margin endpoints
* Slight change on margin unit test
* fix margin unit test issues
* Minor change on unit test
* updates on contract settlement and future contract wrapper function
* add test coverage for contract functions and minor fix on wrapper
* Overall update and unit testing
* codespell, unit tests, type declaration and naming, and code-structure updates
* margin types value and validation function fix
* Update tests and helper funcs
* Improve test coverage
* helper functions and unit tests update
* Fix margin unit test
* Minor review updates
* minor fix on if statement
* Update helper functions
* error handling and functions naming update
* update comment
* minor error return fixes
* minor unit test fix
* Minor fix on spread websocket orders handling
* codespell fix
* skip orderbook depth with incomplete price
* skip orderbook depth with incomplete price
* Codespell: Fix new violations in 2.4.0
* Codespell: Move currency ignoring to regex
* Codespell: Add vendor to ignored files
* Codespell: Remove github action params duplicated from .codespellrc
.codespellrc already exists in the project
* Codespell: Remove unused freez from ignore words
* GateIO: Fix GetFuturesContractDetails for Deliveries
Was returning the product of all the contracts, so 1444 instead of 38
contracts.
* GateIO: Fix GetOpenInterest returning asset.ErrNotEnabled
Using wrong error for pair not enabled
* GateIO: Rename GetSingleContract and GetSingleDeliveryContracts
Especially fixes GetSingleContract, which seems misleading to not say
Futures.
There's a load of `GetSingle*` here that should probably also be fixed,
but these two justified a dyno
* GateIO: Rename GateIOGetPersonalTradingHistory to GetMySpotTradingHistory
* GateIO: Rename GetMyPersonalTradingHistory to GetMyFuturesTradingHistory
* GateIO: Remove duplicate DeliveryTradingHistory
* GateIO: Rename Get*PersonalTradingHistory to GetMy*TradingHistory
* Linter: Disable shadow linting for err
It's been a year, and I'm still getting caught out by govet demanding I
don't shadow a var I was deliberately shadowing.
Made worse by an increase in clashes with stylecheck when they both want
opposite things on the same line.
* GateIO: Add missing Futures and tradinghistory fields
* GateIO: Improve WS Header parsing
This unifies handling for time_ms and time in response headers, since
options and delivery have only time, but spot has time_ms as well.
We use the better of the two results.
Also [improves performance 2x](https://gist.github.com/gbjk/7cacb63b9a256e745534bb05ca853c48)
* GateIO: Use time_ms WS fields where available
Removes the deprecated _time json fields and populates our Time fields
with the time_ms values
As a GCT user with spot and margin assets enabled, but only margin asset enabled websocket subscriptions,
I should still get subscriptions for all the pairs in margin which are also in spot
Currently it only works when spot subscriptions are enabled. Otherwise the spot pairs are ignored.
Fixes#1755
Thusfar ExpandTemplates had only been called internally.
All methods consumers might want to get, in this case GetPairFormat,
need to be in the interface we're passing out, otherwise users can't
call them (or things that use them, like ExpandTemplates)
* Kline: Fix Raw Short, Marshal and Unmarshal
* Deribit: Rename GenerateDefaultSubs
* Deribit: Remove custom GetDefaultConfig
Moved to exchange base by #1472
* Deribit: Straight Rename of eps to endpoints
Since I had to ask what this abbreviation meant, I think we should
abandon it
* Deribit: Add Subscription configuration
* Deribit: Fix race on Setup with optionsRegex
Calling Setup twice would race on the assignment to this package var.
There was an option to just move the assignment to the package var declaration, but this
change improves the performance and allocations:
```
BenchmarkOptionPairToString-8 1000000 1239 ns/op 485 B/op 10 allocs/op
BenchmarkOptionPairToString2-8 3473804 656.2 ns/op 348 B/op 7 allocs/op
```
I've also removed the t.Run because even success the -v output from
tests would be very noisy, and I don't think we were getting any benefit
from it at all:
```
=== RUN TestOptionPairToString
=== PAUSE TestOptionPairToString
=== CONT TestOptionPairToString
=== RUN TestOptionPairToString/BTC-30MAY24-61000-C
=== PAUSE TestOptionPairToString/BTC-30MAY24-61000-C
=== RUN TestOptionPairToString/ETH-1JUN24-3200-P
=== PAUSE TestOptionPairToString/ETH-1JUN24-3200-P
=== RUN TestOptionPairToString/SOL_USDC-31MAY24-162-P
=== PAUSE TestOptionPairToString/SOL_USDC-31MAY24-162-P
=== RUN TestOptionPairToString/MATIC_USDC-6APR24-0d98-P
=== PAUSE TestOptionPairToString/MATIC_USDC-6APR24-0d98-P
=== CONT TestOptionPairToString/BTC-30MAY24-61000-C
=== CONT TestOptionPairToString/SOL_USDC-31MAY24-162-P
=== CONT TestOptionPairToString/ETH-1JUN24-3200-P
=== CONT TestOptionPairToString/MATIC_USDC-6APR24-0d98-P
--- PASS: TestOptionPairToString (0.00s)
--- PASS: TestOptionPairToString/BTC-30MAY24-61000-C (0.00s)
--- PASS: TestOptionPairToString/ETH-1JUN24-3200-P (0.00s)
--- PASS: TestOptionPairToString/SOL_USDC-31MAY24-162-P (0.00s)
--- PASS: TestOptionPairToString/MATIC_USDC-6APR24-0d98-P (0.00s)
```
( And that got worse with me adding more tests )
* gateio: Add multi asset websocket support WIP.
* meow
* Add tests and shenanigans
* integrate flushing and for enabling/disabling pairs from rpc shenanigans
* some changes
* linter: fixes strikes again.
* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.
* Add subscription tests (state functional)
* glorious:nits + proxy handling
* Spelling
* linter: fixerino
* instead of nil, dont do nil.
* clean up nils
* cya nils
* don't need to set URL or check if its running
* stream match update
* update tests
* linter: fix
* glorious: nits + handle context cancellations
* stop ping handler routine leak
* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response
* Allow rollback on connect on any error across all connections
* fix shadow jutsu
* glorious/gk: nitters - adds in ws mock server
* linter: fix
* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.
* glorious: whooops
* gk: nits
* Leak issue and edge case
* Websocket: Add SendMessageReturnResponses
* whooooooopsie
* gk: nitssssss
* Update exchanges/stream/stream_match.go
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
* Update exchanges/stream/stream_match_test.go
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
* linter: appease the linter gods
* gk: nits
* gk: drain brain
* started
* more changes before merge match pr
* gateio: still building out
* gateio: finish spot
* fix up tests in gateio
* Add tests for stream package
* rm unused field
* glorious: nits
* rn files, specifically set function names to asset and offload routing to websocket type.
* linter: fix
* glorious: nits
* add counter and update gateio
* fix collision issue
* Update exchanges/stream/websocket.go
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* glorious: nits
* add tests
* linter: fix
* After merge
* Add error connection info
* upgrade to upstream merge
* Fix edge case where it does not reconnect made by an already closed connection
* stream coverage
* glorious: nits
* glorious: nits removed asset error handling in stream package
* linter: fix
* rm block
* Add basic readme
* fix asset enabled flush cycle for multi connection
* spella: fix
* linter: fix
* Add glorious suggestions, fix some race thing
* reinstate name before any routine gets spawned
* stop on error in mock tests
* glorious: nits
* glorious: nits found in CI build
* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.
* mutex across shutdown and connect for protection
* lint: fix
* test time withoffset, reinstate stop
* fix whoops
* const trafficCheckInterval; rm testmain
* y
* fix lint
* bump time check window
* stream: fix intermittant test failures while testing routines and remove code that is not needed.
* spells
* cant do what I did
* protect race due to routine.
* update testURL
* use mock websocket connection instead of test URL's
* linter: fix
* remove url because its throwing errors on CI builds
* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.
* remove another superfluous url thats not really set up for this
* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly
* linter: fixerino uperino
* glorious: panix
* linter: things
* whoops
* dont need to make consecutive Unix() calls
* websocket: fix potential panic on error and no responses and adding waitForResponses
* rm json parser and handle in json package instead
* linter: fix
* linter: fix again
* * change field name OutboundRequestSignature to WrapperDefinedConnectionSignature for agnostic inbound and outbound connections.
* change method name GetOutboundConnection to GetConnection for agnostic inbound and outbound connections.
* drop outbound field map for improved performance just using a range and field check (less complex as well)
* change field name connections to connectionToWrapper for better clarity
* spells and magic and wands
* glorious: nits
* comparable check for signature
* mv err var
* glorious: nits and stuff
* attempt to fix race
* glorious: nits
* gk: nits; engine log cleanup
* gk: nits; OCD
* gk: nits; move function change file names
* gk: nits; 🚀
* gk: nits; convert variadic function and message inspection to interface and include a specific function for that handling so as to not need nil on every call
* gk: nits; continued
* gk: engine nits; rm loaded exchange
* gk: nits; drop WebsocketLoginResponse
* stream: Add match method EnsureMatchWithData
* gk: nits; rn Inspect to IsFinal
* gk: nits; rn to MessageFilter
* linter: fix
* gateio: update rate limit definitions (cherry-pick)
* Add test and missing
* Shared REST rate limit definitions with Websocket service, set lookup item to nil for systems that do not require rate limiting; add glorious nit
* integrate rate limits for websocket trading spot
* bitstamp: fix issue
* glorious: nits
* ch name and commentary
* fix bug add test
* rm a thing
* fix test
* Update engine/engine.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* thrasher: nits
* Update exchanges/stream/stream_match_test.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Update exchanges/stream/stream_match_test.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* GK: nits rn websocket functions
* explicit function names for single to multi outbound orders
* linter: fix
---------
Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* exchanges: Fix gateio/coinbase test failures
* OKX: Fix TestGetAssetsFromInstrumentTypeOrID race
* GateIO: Add/improve comments
* GateIO: Rid additional API call for FetchTradablePairs and provide additional context for test
* GateIO: Prompt test reviewers to take action if BTC settlement is supported again
