141 Commits

Author SHA1 Message Date
Gareth Kirwan
5d6755b76e linters: Exclude govet shadow check on exchange instances (#2097)
* Linters: Exclude govet shadow check on e in tests

* Linters: Remove nolint rule for new(Exchange) in tests

Replay with:
```
perl -pi -e 's{(\se\s:=\s.*?)\s*//nolint:govet // Intentional shadow.*}{$1}' **/*_test.go
```
2025-10-29 09:30:23 +11:00
Gareth Kirwan
73e200e4e7 accounts: Move to instance methods, fix races and isolate tests (#1923)
* Bybit: Fix race in TestUpdateAccountInfo and  TestWSHandleData

* DriveBy rename TestWSHandleData
* This doesn't address running with -race=2+ due to the singleton

* Accounts: Add account.GetService()

* exchange: Assertify TestSetupDefaults

* Exchanges: Add account.Service override for testing

* Exchanges: Remove duplicate IsWebsocketEnabled test from TestSetupDefaults

* Dispatch: Replace nil checks with NilGuard

* Engine: Remove deprecated printAccountHoldingsChangeSummary

* Dispatcher: Add EnsureRunning method

* Accounts: Move singleton accounts service to exchange Accounts

* Move singleton accounts service to exchange Accounts

This maintains the concept of a global store, whilst allowing exchanges
to override it when needed, particularly for testing.

APIServer:

* Remove getAllActiveAccounts from apiserver

Deprecated apiserver only thing using this, so remove it instead of
updating it

* Update comment for UpdateAccountBalances everywhere

* Docs: Add punctuation to function comments

* Bybit: Coverage for wsProcessWalletPushData Save
2025-10-28 13:52:45 +11:00
Scott
85403fe801 exchange/order/limits: Migrate to new package and integrate with exchanges (#1860)
* move limits, transition to key gen

* rollout NewExchangePairAssetKey everywhere

* test improvements

* self-review fixes

* ok, lets go

* fix merge issue

* slower value func,assertify,drop IsValidPairString

* remove binance reference for backtesting test

* Redundant nil checks removed due to redundancy

* Update order_test.go

* Move limits back into /exchanges/

* puts limits in a different box again

* SHAZBERT SPECIAL SUGGESTIONS

* Update gateio_wrapper.go

* fixes all build issues

* Many niteroos!

* something has gone awry

* bugfix

* gk's everywhere nits

* lint

* extra lint

* re-remove IsValidPairString

* lint fix

* standardise test

* revert some bads

* dupe rm

* another revert 360 mcgee

* un-in-revertify

* Update exchange/order/limits/levels_test.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* fix

* Update exchanges/binance/binance_test.go

HERE'S HOPING GITHUB FORMATS THIS CORRECTLY!

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* update text

* rn func, same line err gk4202000

---------

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
2025-08-26 12:30:21 +10:00
Copilot
7879633c4a exchanges: Remove self-explanatory comments from test functions (#2004)
* Initial plan

* Remove self-explanatory comments from websocket test functions

Co-authored-by: thrasher- <4685270+thrasher-@users.noreply.github.com>

* Remove remaining self-explanatory comments from test functions

Co-authored-by: thrasher- <4685270+thrasher-@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thrasher- <4685270+thrasher-@users.noreply.github.com>
2025-08-19 12:01:04 +10:00
Samuael A.
3f534a15f1 cmd/exchange_template, exchanges: Update templates and propogate to exchanges (#1777)
* Added TimeInForce type and updated related files

* Linter issue fix and minor coinbasepro type update

* Bitrex consts update

* added unit test and minor changes in bittrex

* Unit tests update

* Fix minor linter issues

* Update TestStringToTimeInForce unit test

* Exchange test template change

* A different approach

* fix conflict with gateio timeInForce

* minor exchange template update

* Minor fix to test_files template

* Update order tests

* Complete updating the order unit tests

* Updating exchange wrapper and test template files

* update kucoin and deribit wrapper to match the time in force change

* minor comment update

* fix time-in-force related test errors

* linter issue fix

* ADD_NEW_EXCHANGE documentation update

* time in force constants, functions and unit tests update

* shift tif policies to TimeInForce

* Update time-in-force, related functions, and unit tests

* fix linter issue and time-in-force processing

* added a good till crossing tif value

* order type fix and fix related tim-in-force entries

* update time-in-force unmarshaling and unit test

* consistency guideline added

* fix time-in-force error in gateio

* linter issue fix

* update based on review comments

* add unit test and fix missing issues

* minor fix and added benchmark unit test

* change GTT to GTC for limit

* fix linter issue

* added time-in-force value to place order param

* fix minor issues based on review comment and move tif code to separate files

* update on exchanges linked to time-in-force

* resolve missing review comments

* minor linter issues fix

* added time-in-force handler and update timeInForce parametered endpoint

* minor fixes based on review

* nits fix

* update based on review

* linter fix

* rm getTimeInForce func and minor change to time-in-force

* minor change

* update based on review comments

* wrappers and time-in-force calling approach

* minor change

* update gateio string to timeInForce conversion and unit test

* update exchange template

* update wrapper template file

* policy comments, and template files update

* rename all exchange types name to Exchange

* update on template files and template generation

* templates and generation code and other updates

* linter issue fix

* added subscriptions and websocket templates

* update ADD_NEW_EXCHANGE.md with recent binance functions and implementations

* rename template files and update unit tests

* minor template and unit test fix

* rename templates and fix on unit tests

* update on template files and documentation

* removed unnecessary tag fix and update templates

* fix Add_NEW_EXCHANGE.md doc file

* formatting, comments, and error checks update on template files

* rename exchange receivers to e and ex for consistency

* rename unit test exchange receiver and minor updates

* linter issues fix

* fix deribit issue and minor style update

* fix test issues caused by receiver change

* raname local variables exchange declaration variables

* update templates comments

* update templates and related comments

* renamed ex to e

* update template comments

* toggle WS to false to improve coverage

* template comments update

* added test coverage to Ws enabled and minor changes

---------

Co-authored-by: Samuel Reid <43227667+cranktakular@users.noreply.github.com>
2025-07-17 10:46:36 +10:00
Adrian Gallagher
3cc9a2b9e0 exchanges: Refactor time handling and other minor improvements (#1948)
* exchanges: Refactor time handling and other minor improvements

- Updated Kraken wrapper to utilise new time handling methods.
- Simplified Kucoin types by removing unnecessary structures and using direct JSON unmarshalling.
- Improved websocket handling in Kucoin to directly parse candlestick data.
- Modified Lbank types to use the new time representation.
- Adjusted Poloniex wrapper and types to utilise the new time handling.
- Updated Yobit types and wrapper to reflect changes in time representation.
- Introduced DateTime type for better handling of specific time formats.
- Added tests for DateTime unmarshalling to ensure correctness.
- Rid UTC().Unix and UTC().UnixMilli as it's not needed
- Correct Huobi timestamp usage for some endpoints.
- Rid RFC3339 time parsing since Go does that automatically.

* exchanges: Refactor JSON unmarshalling for various types and improve test coverage

* linter: Update error message in TestGetKlines

* refactor: Simplify JSON unmarshalling in MovementHistory and improve test assertions in GetKlines

* refactor: Improve JSON unmarshalling for channel name and clarify comment in wsProcessOpenOrders

* refactor: Update time handling in Huobi types to use types.Time for createdAt fields and relax GetLiquidationOrders test

* refactor: Move wsTicker, wsSpread, wsTrades, and wsCandle types to kraken_types.go for better organistion

* refactor: Add validation for underlying parameter in GetExpirationTime and update tests
2025-07-01 09:11:55 +10:00
Adrian Gallagher
ebcbfab358 exchanges: Improve TestMain usage (#1946)
* exchanges: Improve TestMain usage

* exchanges: Further cleanups

* exchanges/kucoin: update TestProcessOrderbook to use test context

* refactor: rename SetRunning to SetRunningURL for clarity across exchanges
2025-06-25 13:02:47 +10:00
Ryan O'Hara-Reid
3e80f1b9e5 websocket/exchanges: populate context before multi connection upgrade (#1933)
* websocket/exchanges: populate context before multi connection upgrade

* fix test

* linter: fix

* gk: dial

* gk: nits rm param names

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2025-06-17 13:43:00 +10:00
Ryan O'Hara-Reid
2958e64afe orderbook: change Base struct name to Book (#1914)
* orderbook: change Base struct name to Snapshot

* linter: fix

* Update exchanges/exchange.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/depth.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/orderbook_types.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Snapshot -> Book

* Tranche(s) -> Level(s)

* Tranche(s) -> Level(s)

* rm tranche ref

* linter: fix

* linter: rides again

* update tests

* Update exchange/websocket/buffer/buffer.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update backtester/eventhandlers/exchange/slippage/slippage.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchange/websocket/buffer/buffer.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update exchange/websocket/buffer/buffer.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update exchanges/orderbook/orderbook_types.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update exchanges/orderbook/orderbook_types.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* fixup tests

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/orderbook_types.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/orderbook/orderbook_types.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* gk: nits and rm stuff that is not needed

* Update exchanges/orderbook/orderbook_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* gk: nits

---------

Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2025-06-16 17:09:25 +10:00
Adrian Gallagher
d5ba674fc4 codebase: Cleanup various things (#1935)
* codebase: Rid base64/hex to string common funcs

* codebase: Rid local scope variable usage and other improvements

* codebase: Refactor currency pair usage across multiple exchanges

- Updated HitBTC tests to use the new currency pair format.
- Modified Kraken futures types to use currency.Pair instead of string for Symbol.
- Adjusted Kraken wrapper methods to handle currency pairs correctly.
- Refined OKX tests and types to utilize currency.Pair for instrument IDs.
- Enhanced Poloniex tests to consistently use predefined currency pairs.
- Streamlined order and orderbook tests to replace string pairs with currency.NewBTCUSD().
- Improved Yobit tests to utilize a standardized currency pair format.
- Updated validator wrapper to use currency pairs directly instead of string conversions.

* codebase: Use types.Number where possible

* refactor: update PayoutFee type to types.Number for consistency

* Refactor: Remove crypto functions to use standard library and other minor changes

- Removed custom crypto functions for SHA256, SHA512, and MD5 from the common/crypto package.
- Replaced usages of removed functions with standard library implementations in various files including:
  - cmd/websocket_client/main.go
  - engine/apiserver.go
  - exchanges/kraken/kraken.go
  - exchanges/lbank/lbank.go
  - exchanges/okx/okx_business_websocket.go
  - exchanges/kucoin/kucoin_websocket.go
  - gctscript/vm/vm.go
- Updated tests to reflect changes in the crypto functions.
- Renamed several functions for clarity, particularly in the context of order book updates across multiple exchanges.

* refactor: replace assert with require for consistency in test assertions

* refactor: Improve Binance futures candlestick test, standardise orderbook update function names and improve test parallelism

* refactor: Replace require.Len with require.Equal for better output in TestGetFuturesKlineData
2025-06-12 14:12:36 +10:00
Adrian Gallagher
19b8957f3f codebase: Replace !errors.Is(err, target) with testify (#1931)
* tests: Replace !errors.Is(err, target) with testify equivalents

* codebase: Manual !errors.Is(err, target) replacements

* typo: Replace errMisMatchedEvent with errMismatchedEvent

* tests: Enhance error messages for better output

* tests: Refactor error assertions in various test cases to use require and improve clarity

* misc linter: Fix assert should wording

* tests: Simplify assertions in TestCreateSignals for clarity and conciseness

* tests: Enhance assertion message in TestCreateSignals
2025-06-10 16:29:57 +10:00
Adrian Gallagher
a5b638bfb7 GHA: Add additional checks for common issues (#1922)
* GHA, tests: Add additional checks for common issues

These checks include:
- Ensuring that all testify funcs use their formatted variants (e.g., `assert.Equalf(t, expected, actual)` instead of `assert.Equal(t, expected, actual)`).
- Replacing `%s` with %q
- Enforcing consistent usage of should/must wording for testify assert/require messages

* Add support for checking backticked string format specifiers and fix issues

* tests: Fix error comparisons

* tests: Replace errors.Is(err, nil) usage with testify and automate check

* refactor: Rename ExtractPort to ExtractPortOrDefault

* tests: Replace assert with require for error handling in multiple test files

* tests: Replace assert with require for error handling and improve assertions in data tests

* tests: Fix typo in assertion message for StreamVol test

* OKX: Fix GetOpenInterestAndVolumeStrike test with instrument selection and improved assertions

* OKX: Revert intentional error check

* Improve error message for expiry time check in GetOpenInterestAndVolumeStrike test
2025-05-28 12:26:51 +10:00
Gareth Kirwan
3caa149d8e Tests: Use currency.NewBTCUSD and NewBTCUSDT (#1895)
* Tests: Use currency.NewUSD and NewUSDT

Simple refactor to use the provided shortcut methods

* Github: Add CI check to ensure NewPair not used

Add a step to ensure NewPair(BTC, USD*) isn't used
2025-05-07 11:32:06 +10:00
Adrian Gallagher
bea16af380 golangci-lint: Enable usetesting and unused linters (#1893)
* golangci-lint: Enable usetesting and unused linters

* tests: Improve assertions in various test cases for clarity and accuracy

* tests: Enhance error assertions in TestExecuteStrategyFromFile for improved clarity

* tests: Update assertions for improved clarity and accuracy

* tests: Replace assert with require for task count checks

* config/versions/v7: Replace context.Background() with t.Context()

* Bithumb: Centralise and consoliate testPair, relax UpdateTickers check

with some glorious Doom Eternal music

* Bithumb: Use UpdatePairsOnce and update remaining pair string

* Bithumb: Add UpdatePairsOnce to TestUpdateTickers
2025-05-01 14:44:29 +10:00
Gareth Kirwan
c2d876d8b0 Exchanges: Remove WalletAddress (#1882)
* Exchanges: Remove cancel order walletAddress

* Order: Refactor TestMatchFilter

TestMatchFilter had inconsistent testing of empty values, and was
painful when a field was removed due to index methodology.
This should provide equal test clarity, but improve maintainability and
improve coverage on empty values.
2025-04-30 17:10:36 +10:00
Ryan O'Hara-Reid
cad7586e98 exchange/websocket, gateio: Rename/export package again, add websocket request functions for futures trading (#1603)
* gateio: Add multi asset websocket support WIP.

* meow

* Add tests and shenanigans

* integrate flushing and for enabling/disabling pairs from rpc shenanigans

* some changes

* linter: fixes strikes again.

* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.

* Add subscription tests (state functional)

* glorious:nits + proxy handling

* Spelling

* linter: fixerino

* instead of nil, dont do nil.

* clean up nils

* cya nils

* don't need to set URL or check if its running

* stream match update

* update tests

* linter: fix

* glorious: nits + handle context cancellations

* stop ping handler routine leak

* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response

* Allow rollback on connect on any error across all connections

* fix shadow jutsu

* glorious/gk: nitters - adds in ws mock server

* linter: fix

* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.

* glorious: whooops

* gk: nits

* Leak issue and edge case

* Websocket: Add SendMessageReturnResponses

* whooooooopsie

* gk: nitssssss

* Update exchanges/stream/stream_match.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/stream/stream_match_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* linter: appease the linter gods

* gk: nits

* gk: drain brain

* started

* more changes before merge match pr

* gateio: still building out

* gateio: finish spot

* fix up tests in gateio

* Add tests for stream package

* rm unused field

* glorious: nits

* rn files, specifically set function names to asset and offload routing to websocket type.

* linter: fix

* Add futures websocket request support

* gateio: integrate with IBOTExchange (cherry pick my nose)

* linter: fix

* glorious: nits

* add counter and update gateio

* fix collision issue

* Update exchanges/stream/websocket.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* add tests

* linter: fix

* After merge

* Add error connection info

* upgrade to upstream merge

* Fix edge case where it does not reconnect made by an already closed connection

* stream coverage

* glorious: nits

* glorious: nits removed asset error handling in stream package

* linter: fix

* rm block

* Add basic readme

* fix asset enabled flush cycle for multi connection

* spella: fix

* linter: fix

* Add glorious suggestions, fix some race thing

* reinstate name before any routine gets spawned

* stop on error in mock tests

* glorious: nits

* Set correct price

* glorious: nits found in CI build

* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.

* mutex across shutdown and connect for protection

* lint: fix

* test time withoffset, reinstate stop

* fix whoops

* const trafficCheckInterval; rm testmain

* y

* fix lint

* bump time check window

* stream: fix intermittant test failures while testing routines and remove code that is not needed.

* spells

* cant do what I did

* protect race due to routine.

* update testURL

* use mock websocket connection instead of test URL's

* linter: fix

* remove url because its throwing errors on CI builds

* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.

* remove another superfluous url thats not really set up for this

* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly

* linter: fixerino uperino

* fix ID bug, why I do this, I don't know.

* glorious: panix

* linter: things

* whoops

* dont need to make consecutive Unix() calls

* websocket: fix potential panic on error and no responses and adding waitForResponses

* rm json parser and handle in json package instead

* in favour of json package unmarshalling

* linter: fix

* linter: fix again

* * change field name OutboundRequestSignature to WrapperDefinedConnectionSignature for agnostic inbound and outbound connections.
* change method name GetOutboundConnection to GetConnection for agnostic inbound and outbound connections.
* drop outbound field map for improved performance just using a range and field check (less complex as well)
* change field name connections to connectionToWrapper for better clarity

* spells and magic and wands

* glorious: nits

* comparable check for signature

* mv err var

* glorious: nits and stuff

* attempt to fix race

* glorious: nits

* gk: nits; engine log cleanup

* gk: nits; OCD

* gk: nits; move function change file names

* gk: nits; 🚀

* gk: nits; convert variadic function and message inspection to interface and include a specific function for that handling so as to not need nil on every call

* gk: nits; continued

* gk: engine nits; rm loaded exchange

* gk: nits; drop WebsocketLoginResponse

* stream: Add match method EnsureMatchWithData

* gk: nits; rn Inspect to IsFinal

* gk: nits; rn to MessageFilter

* linter: fix

* gateio: update rate limit definitions (cherry-pick)

* Add test and missing

* Shared REST rate limit definitions with Websocket service, set lookup item to nil for systems that do not require rate limiting; add glorious nit

* integrate rate limits for websocket trading spot

* conform to match upstream changes

* standardise names to upstream style

* fix wrapper standards test when sending a auth request through a websocket connection

* whoops

* Update exchanges/gateio/gateio_types.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* linter: fix

* linter: overload

* whoops

* spelling fixes on recent merge

* glorious: nits

* linter: fix?

* glorious: nits

* gk: assert errors touched

* gk: unexport derive functions

* gk: nitssssssss

* fix test

* gk: nitters v1

* gk: http status

* gk/nits: Add getAssetFromFuturesPair

* gk: nits single response when submitting

* gk: new pair with delimiter in tests

* gk: param update slice to slice of pointers

* gk: add asset type in params, includes t.Context() for tests

* linter: fix

* linter: fix

* fix merge whoopsie

* glorious: nits

* gk: nit

* shift over to websocket package error

* internal/exchange/websocket -> exchange/websocket

* PEAK OCD!

* appease the OCD gods

* thrasher: nits

---------

Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2025-04-11 16:47:33 +10:00
Gareth Kirwan
b4e45e9a1b Websocket: Restructure files and types (#1859)
* Websocket: Rename stream package

* Websocket: Rename Websocket to Manager

* Websocket: Replace explicit errs with common.NilGuard

* Websocket: Move websocket_types.go to types.go

* Websocket: Minor field comment and alignment in types

* Webosocket: Rename WebsocketConnection to Connection

* Alphapoint: Make gorilla ws import explicit

Just to avoid confusion with our own packages.

* Websocket: Move stream_match to match

* Websocket: Move websocket_connection to connection

* Websocket: Move websocket.go to manager.go

* Websocket: Break out all subscription methods into subscriptions.go

* Websocket: Move connection type into its file

* Websocket: Remove PositionUpdated

Type is not used anywhere

* Kraken: Use local constant for pong

Was the only use of websocket.Pong and doesn't really feel right to
represent kraken's api resp in one of our packages

* Websocket: Move connection sub-types to connection package

* Websocket: Move manager types into manager

* Websocket: Move ConnectionWrapper into manager

* Websocket: Move websocket_test to manager_test

* Websocket: Privatise connectionWrapper

* Websocket: Remaining types into types.go

These really belong somewhere else mostly, but this will do for now

* Websocket: Tidy up connection method vars

* Gofumpt: Moving package imports around

* Websocket: Rename errDuplicateConnectionSetup

* Websocket: Fix duplicate import of gws

* Websocket: Fix gofumpt -extra

* Websocket: Standardise import of gws across other pkgs

* Kraken: Remove unused sub conf consts

These were replaced by the generic Levels and Depth fields on all subs

* Websocket: Privitise ConnectioWrapper fields

* Websocket: inline single use var WebsocketNotAuthenticatedUsingRest

* Websocket: Move documentation to template

* Bithumb: Assertify TestWsHandleData
2025-04-10 16:25:02 +10:00
Adrian Gallagher
f69abcd0ae Bitfinex: Fix intermittent TestGenerateSubscriptions and ticker FRRAmountAvailable failures (#1865)
* Bitfinex: Fix intermittent TestGenerateSubscriptions and ticker FRRAmountAvailable failures

* Bitfinex: Add comment for FRRAmountAvailable
2025-04-01 17:05:34 +11:00
Adrian Gallagher
4651af5767 modernise: Run new gopls modernise tool against the codebase and fix minor issues (#1826)
* modernise: Run new gopls modernise tool against codebase

* Address shazbert's nits

* apichecker, gctcli: Simplify HTML scraping functions and improve depth limit handling

* refactor: Create minSyncInterval const and update order book limit handling for binance and binanceUS

* refactor: Various slice usage improvements and rename TODO

* tranches: Revert deleteByID changes due to performance decrease

Shazbert was a F1 driver in a past lifetime 🏎️

* tranches: Simply retrieve copy

Thanks to shazbert

* documentation: Sort contributors list by contributions

* tranches: Remove deadcode in deleteByID
2025-03-21 09:17:10 +11:00
Adrian Gallagher
f5faca2eb2 linter: Enable gofumpt and run against codebase (#1848)
* linter: Enable gofumpt and run against codebase

* Address shazbert's nits

* gofumpt: Fix issues after rebase
2025-03-18 10:23:16 +11:00
Gareth Kirwan
16d2d9f35a Config: AssetEnabled upgrade (#1735)
* Config: Move assetEnabled upgrade to Version management

* Assets: Do not error on asset not enabled, or disabled

This became more messy with Disabling something that's defaulted to
disabled.
Taking an idealogical stance against erroring that what you want to have
done is already done.

* CurrencyManager: Set AssetEnabled when StorePairs(enabled)

* RPCServer: Fix tests expecting StoreAssetPairFormat to enable the asset

Also assertifies

* Bitfinex: Fix tests for MarginFunding subs

* GCTWrapper: Improve TestMain clarity

* BTSE: Add futures to testconfig

* Exchanges: Rename StoreAssetPairStore

Previously we were calling it "Format", but accepting everything from
the PairStore.
We were also defaulting to turning the Asset on.

Now callers need to get their AssetEnabled set as they want it, so
there's no magic

This change also moves responsibility for error wrapping outside to the
caller.

* Config: AssetEnabled upgrade should respect assetTypes

Previously we ignored the field and just turned on everything.
I think that was because we couldn't get at the old value.
In either case, we have the option to do better, and respect the
assetEnabled value

* Config: Improve exchange config version upgrade error messages
2025-03-17 21:47:37 +11:00
Bea
7fa2592e31 Kraken: Fix sending trades to the websocket DataHandler (#1813)
* Send trades down the DataHandler

When the tradeFeed is enabled, send the trades down the DataHandler
Add TestWSProcessTrades

* Update assertions

* Add check against null references in slices

* Add an error for ws parsing field to common

* Update kraken websocket ProcessTrades

Send individual trades down the DataHandler
Send multiple trades in test
Test error if the trade length is too short
Nits

* Fix nits
2025-03-04 17:06:07 +11:00
Ryan O'Hara-Reid
e99adca86f encoding/json: Add custom JSON package with build tag support for Sonic (#1623)
* tag optional sonic and allow full library conversion

* Add workflow and disallow arm and darwin usage

* Add basic hotswap benchmark

* linter: fix

* use bash

* linter: fix?

* Fix whoopsie, add to make file, also add mention in features list.

* test enforcement

* actually read documentation see if this works

* linter: fix

* linter: fix

* sonic: bump tagged version

* encoding/json: drop build tag arch and os filters

* encoding/json: consolidate tests

* encoding/json: log build tag usage

* rm superfluous builds

* glorious/nits: add template change and regen docs

* glorious/nits: update commentary on nolint directive

* glorious/nits: rm init func and log results in main.go

* Test to actually pull flag in

* linter: fix

* thrasher: nits

* gk: nits 4 goflags goooooooooo!

* gk: nits rn

* make sonic default json implementation

* screen 386

* linter: fix

* Add commentary

* glorious: nits Makefile not working

* gk: nits

* gk: nits whoops

* whoopsirino

* mention 32bit systems won't be sonic

* gk: super-duper nit of extremes

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2025-02-20 16:05:55 +11:00
Ryan O'Hara-Reid
08e015a125 exchanges/wrappers: Refactor fetch orderbook/ticker/account info funcs (#1440)
* acrost: Pull thread, examine

* fix tests

* linter

* fix_linter

* revert rm ctx param to limit breakages when merging usptream

* linter fix

* Add in priority update grouping so that tests pass

* Update cmd/exchange_wrapper_standards/exchange_wrapper_standards_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious nits

* fixed spelling

* whoopsie

* aanother whoops

* glorious: NITTERS!

* glorious: further nitters

* srry linter gods

* glorious: nits continued

* sub test p ara lel

* drop main t.Parallel

* fix whoops

* wrappertests: use context with cancel (test)

* linter: fix

* ensure primary execution

* kucoin test fix

* revert standards test changes and bypass non critical errors

* rm single override

* wrap exported error for accounts

* thrasher: nits ch name

* gk: nits

* gk: nits FetchTickerCached -> GetCachedTicker

* gk: nits rn FetchOrderbookCached -> GetCachedOrderbook

* gk: nits rn FetchAccountInfoCached -> GetCachedAccountInfo

* linter: fix

* gk: nits

* thrasher: nitters 1

* thrasher: nitters tmpls

* gk: nitter

---------

Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2025-02-19 10:47:10 +11:00
Gareth Kirwan
5463e359bc Bitfinex: Fix websocket trade processing (#1754)
* Bitfinex: Fix WS trade processing

* Add handling for funding trades

Fixes #1746

* Linter: Disable shadow linting for err

It's been a year, and I'm still getting caught out by govet demanding I
don't shadow a var I was deliberately shadowing.
Made worse by an increase in clashes with stylecheck when they both want
opposite things on the same line.

* Bitfinex: Move websocket constants to websocket file

* Bitfinex: Rename channel consts

* Bitfinex: Send individual WS trades down the DataHandler
2025-02-17 13:20:14 +11:00
Gareth Kirwan
ba77c9946d Bitfinex: Add subscription configuration and templating (#1597)
* Bitfinex: Correct comment about R0 OB

* Bitfinex: Test config updates

* Bitfinex: Add missing assets to configtest

* Bitfinex: Rename GenerateDefaultSubscriptions

* Bitfinex: Add Subscription configuration

* Subscriptions: Document panic in templates
2024-10-18 09:23:15 +11:00
Ryan O'Hara-Reid
ac731ce283 websocket/gateio: Support multi connection management and integrate with GateIO (#1580)
* gateio: Add multi asset websocket support WIP.

* meow

* Add tests and shenanigans

* integrate flushing and for enabling/disabling pairs from rpc shenanigans

* some changes

* linter: fixes strikes again.

* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.

* Add subscription tests (state functional)

* glorious:nits + proxy handling

* Spelling

* linter: fixerino

* instead of nil, dont do nil.

* clean up nils

* cya nils

* don't need to set URL or check if its running

* stop ping handler routine leak

* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response

* Allow rollback on connect on any error across all connections

* fix shadow jutsu

* glorious/gk: nitters - adds in ws mock server

* linter: fix

* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.

* gk: nits

* Leak issue and edge case

* gk: nits

* gk: drain brain

* glorious: nits

* Update exchanges/stream/websocket.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* add tests

* linter: fix

* After merge

* Add error connection info

* Fix edge case where it does not reconnect made by an already closed connection

* stream coverage

* glorious: nits

* glorious: nits removed asset error handling in stream package

* linter: fix

* rm block

* Add basic readme

* fix asset enabled flush cycle for multi connection

* spella: fix

* linter: fix

* Add glorious suggestions, fix some race thing

* reinstate name before any routine gets spawned

* stop on error in mock tests

* glorious: nits

* glorious: nits found in CI build

* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.

* mutex across shutdown and connect for protection

* lint: fix

* test time withoffset, reinstate stop

* fix whoops

* const trafficCheckInterval; rm testmain

* y

* fix lint

* bump time check window

* stream: fix intermittant test failures while testing routines and remove code that is not needed.

* spells

* cant do what I did

* protect race due to routine.

* update testURL

* use mock websocket connection instead of test URL's

* linter: fix

* remove url because its throwing errors on CI builds

* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.

* remove another superfluous url thats not really set up for this

* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly

* linter: fixerino uperino

* glorious: panix

* linter: things

* whoops

* defer lock and use functions that don't require locking in SetProxyAddress

* lint: fix

* thrasher: nits

---------

Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2024-10-10 15:09:52 +11:00
Ryan O'Hara-Reid
17c2ef2ec7 stream/match: Reduce complexity and limit locking when match occurs (#1581)
* stream match update

* update tests

* linter: fix

* glorious: nits + handle context cancellations

* glorious: whooops

* Websocket: Add SendMessageReturnResponses

* whooooooopsie

* gk: nitssssss

* Update exchanges/stream/stream_match.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/stream/stream_match_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* linter: appease the linter gods

* glorious: nits

* glorious: nits

* Update exchanges/stream/stream_match_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2024-08-19 10:35:46 +10:00
Adrian Gallagher
225429bda6 CI/build: Update Go version, linters and fix minor issues (#1612)
* CI/build: Update Go version, linters and fix minor issues

* linters: Add intrange, copyloopvar, additional go vet linters to match gopls and fix issues
2024-08-16 17:41:11 +10:00
Gareth Kirwan
0becfbd0a6 Kraken: Fix TestGetOpenInterest (#1611)
* Kraken: Fix TestGetOpenInterest

We see daily failures on OpenInterest for Kraken.
This fix assumes that the issue might be related to volume of ETHUSD
open interest, and switches the single pair test to XBTUSD instead

Also isolates the test from others, since we're changing stored pairs
and we might be colliding on the global k

* Kraken: Handle Errors field in futures response
2024-08-16 14:47:32 +10:00
Gareth Kirwan
1199f38546 subscriptions: Encapsulate, replace Pair with Pairs and refactor; improve exchange support
* Websocket: Use ErrSubscribedAlready

instead of errChannelAlreadySubscribed

* Subscriptions: Replace Pair with Pairs

Given that some subscriptions have multiple pairs, support that as the
standard.

* Docs: Update subscriptions in add new exch

* RPC: Update Subscription Pairs

* Linter: Disable testifylint.Len

We deliberately use Equal over Len to avoid spamming the contents of large Slices

* Websocket: Add suffix to state consts

* Binance: Subscription Pairs support

* Bitfinex: Subscription Pairs support

* Bithumb: Subscription Pairs support

* Bitmex: Subscription Pairs support

* Bitstamp: Subscription Pairs support

* BTCMarkets: Subscription Pairs support

* BTSE: Subscription Pairs support

* Coinbase: Subscription Pairs support

* Coinut: Subscription Pairs support

* GateIO: Subscription Pairs support

* Gemini: Subscription Pairs support and improvement

* Hitbtc: Subscription Pairs support

* Huboi: Subscription Pairs support

* Kucoin: Subscription Pairs support

* Okcoin: Subscription Pairs support

* Poloniex: Subscription Pairs support

* Kraken: Add subscription Pairs support

Note: This is a naieve implementation because we want to rebase the
kraken websocket rewrite on top of this

* Bybit: Subscription Pairs support

* Okx: Subscription Pairs support

* Bitmex: Subsription configuration

* Fixes unauthenticated websocket left as CanUseAuth
* Fixes auth subs happening privately

* CoinbasePro: Subscription Configuration

* Consolidate ProductIDs when all subscriptions are for the same list

* Websocket: Log actual sent message when Verbose

* Subscriptions: Improve clarity of which key is which in Match

* Subscriptions: Lint fix for HugeParam

* Subscriptions: Add AddPairs and move keys from test

* Subscriptions: Simplify subscription keys and add key types

* Subscriptions: Add List.GroupPairs Rename sub.AddPairs

* Subscription: Fix ExactKey not matching 0 pairs

* Subscriptions: Remove unused IdentityKey and HasPairKey

* Subscriptions: Fix GetKey test

* Subscriptions: Test coverage improvements

* Websocket: Change State on Add/Remove

* Subscriptions: Improve error context

* Subscriptions: Fix Enable: false subs not ignored

* Bitfinex: Fix WsAuth test failing on DataHandler

DataHandler is eaten by dataMonitor now, so we need to use ToRoutine

* Deribit: Subscription Pairs support

* Websocket: Accept nil lists for checkSubscriptions

If the user passes in a nil (implicitly empty) list, we would not panic.
Therefore the burden of correctness about that data lies with them.
The list of subscriptions is empty, and that's okay, and possibly
convenient

* Websocket: Add context to NilPointer errors

* Subscriptions: Add context to nil errors

* Exchange: Fix error expectations in UnsubToWSChans
2024-06-07 11:54:08 +10:00
Gareth Kirwan
7d1eecfa7e Tests: Various race fixes and move TestFixtureToDataHandler (#1534)
* Tests: Move and simplify TestFixtureToDataHandler

* Currency: Fix PairsManager.Load breaking matcher

* Tests: Add multi-instance cache to UpdatePairsOnce

* Kraken: Fix TestUpdateTickers race error

Calling StorePairs on global instance can lead to race

* Bitfinex: Fix TestUpdateTickers racing intermittently

* Currency: Fix concurrent access to PM formats

* Currency: Fix SupportsAsset implementation

This should delegate entirely to PairManager's IsAssetSupported

* Okx: Fix PM intrusion, rm GetPairFromInstrumentID

* Exchange: Fix SetGlobalPairsManager to set asset enabled

* Bitflyer: Fix race on set TestGetCurrURL

TestGetCurrencyTradeURL would fail sometimes due to sequencing of
enabling futures but not having pairs for it.

* Tests: Simplify usage pattern for FixtureToDH
2024-05-16 14:09:26 +10:00
Ryan O'Hara-Reid
4cd4fb06b4 orderbook: Refactor package structure for simplicity and efficiency (#1465)
* initial purge and benchmarks proof before rn overhaul

* rn LinkedList -> Tranche(s) and purge references

* roll out acrost exchanges

* linterino

* rn silly billy label

* linter strikes AAAAAGAIN!

* fix some things

* rm comment

* Add actual comparison from master to branch benchmark for sorting algorithms

* lower case via git mv YAAY!

* drop code

* convert type name

* glorious: nits

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2024-05-14 15:51:34 +10:00
Scott
f1ff951199 engine/exchanges: Add GetCurrencyTradeURL wrapper func/gRPC endpoint (#1521)
* beginning the concept

* expands testing and implementations

* test standardisation, expansion

* more

* finish remainder, add rpc func

* lint

* grammar? I barely know her!

* wow i forgot something wow wow wow wow

* rm SendPayload

* enFixio!

* improve binance long-dated support

* update test design, update wrapper funcs

* adds kraken futures, adds bybit support

* fixes SIMPLE bugs

* s is for sucks

* fixed option url x2

* fixed silly test
2024-05-03 17:01:17 +10:00
Gareth Kirwan
3e4b9becfe Tests: Abstract UpdatePairsOnce (#1503) 2024-03-15 16:37:06 +11:00
Gareth Kirwan
52c6b3bf0b Websocket: Various refactors and test improvements (#1466)
* Websocket: Remove IsInit and simplify SetProxyAddress

IsInit was basically the same as IsConnected.
Any time Connect was called both would be set to true.
Any time we had a disconnect they'd both be set to false
Shutdown() incorrectly didn't setInit(false)

SetProxyAddress simplified to only reconnect a connected Websocket.
Any other state means it hasn't been Connected, or it's about to
reconnect anyway.
There's no handling for IsConnecting previously, either, so I've wrapped
that behind the main mutex.

* Websocket: Expand and Assertify tests

* Websocket: Simplify state transistions

* Websocket: Simplify Connecting/Connected state

* Websocket: Tests and errors for websocket

* Websocket: Make WebsocketNotEnabled a real error

This allows for testing and avoids the repetition.
If each returned error is a error.New() you can never use errors.Is()

* Websocket: Add more testable errors

* Websocket: Improve GenerateMessageID test

Testing just the last id doesn't feel very robust

* Websocket: Protect Setup() from races

* Websocket: Use atomics instead of mutex

This was spurred by looking at the setState call in trafficMonitor and
the effect on blocking and efficiency.
With the new atomic types in Go 1.19, and the small types in use here,
atomics should be safe for our usage. bools should be truly atomic,
and uint32 is atomic when the accepted value range is less than one byte/uint8 since
that can be written atomicly by concurrent processors.
Maybe that's not even a factor any more, however we don't even have to worry enough to check.

* Websocket: Fix and simplify traffic monitor

trafficMonitor had a check throttle at the end of the for loop to stop it just gobbling the (blocking) trafficAlert channel non-stop.
That makes sense, except that nothing is sent to the trafficAlert channel if there's no listener.
So that means that it's out by one second on the trafficAlert, because any traffic received during the pause is doesn't try to send a traffic alert.

The unstopped timer is deliberately leaked for later GC when shutdown.
It won't delay/block anything, and it's a trivial memory leak during an infrequent event.

Deliberately Choosing to recreate the timer each time instead of using Stop, drain and reset

* Websocket: Split traficMonitor test on behaviours

* Websocket: Remove trafficMonitor connected status

trafficMonitor does not need to set the connection to be connected.
Connect() does that. Anything after that should result in a full
shutdown and restart. It can't and shouldn't become connected
unexpectedly, and this is most likely a race anyway.

Also dropped trafficCheckInterval to 100ms to mitigate races of traffic
alerts being buffered for too long.

* Websocket: Set disconnected earlier in Shutdown

This caused a possible race where state is still connected, but we start
to trigger interested actors via ShutdownC and Wait.
They may check state and then call Shutdown again, such as
trafficMonitor

* Websocket: Wait 5s for slow tests to pass traffic draining

Keep getting failures upstream on test rigs.
Think they can be very contended, so this pushes the boundary right out
to 5s
2024-02-23 18:39:25 +11:00
Gareth Kirwan
d7818ea956 Exchanges: Remove bespoke pair upgrade handling and abstract Start/Run (#1424)
* Exchanges: Remove Pair upgrade handling

Now redundant behind #1401. These paths should never be met.

Several legacy coin upgrade paths being deprecated as well: ZUSD and CNY
Expecting any users with bad config from 3+ years ago would have to
reset anyway.

Also: At the time the intention of this was to upgrade the config
format.
However now, instead, it'd mostly serve to reset enabled pairs if
there's a config mistake, which doesn't feel right.

* Kraken: Fix typo in Kraken type struct

* Exchanges: Abstract exchange Start() and Run()

* Exchanges: Add test for abstracted Start

* Exchanges: Move Start to Bootstrap

* Simplify waitgroup usage
* Add call to exchange.Bootstrap to allow overide or supplementation

* Exchanges: Concurrent common bootstap actions

* Gateio: Remove incorrect Run in test

* GateIO: Fix pair dependencies in tests

This ensures that the pairs are initialised no more than needed and
kind-of just-in-time.
Better pattern might be to use a function to get these pairs when we
need them.

* Exchanges: Complete UpdatePairs before ExecLims

If we're going to update pairs, it needs to complete before we check for
limits to avoid errors on old pairs

* Exchanges: Remove Start and Run from tmpl

Since they're replaced by bootstrap now and shouldn't need customisation
normally

* Alphapoint: Move Start to Bootstrap

* GateIO: Fix linter shadow var
2024-01-31 19:29:36 +11:00
Gareth Kirwan
23c82bead4 Bitfinex: Fix panic on TestUpdateTickers (#1454)
* Bitfinex: Handle Errors in tickers without panic

* Bitfinex: Use TypeAssertError for tickers

* Bitfinex: Refactor and improve ticker handling

* Unify Ticker response handling
* Simplify/Humanify errors in parsing ticker responses
* Remove polymorphic response handling for < 10 fields, seems [antiquated according to docs](https://docs.bitfinex.com/reference/rest-public-ticker)
* Add test coverage for tickers

* Bitfinex: Ignore resp format errs in GetTickers

We're still getting:
`received 'invalid ticker response format for tALT2612:USD field BidSize
from [100 <nil> 100 <nil> <nil> <nil> <nil> <nil> <nil> <nil>]`
too frequently right now.

Considered:
* Special casing tALT2612:*; However if it's happening with this curr
  it'll probably happen again
* Warning about the error; However it'd just be persistent,
  unactionable and annoying  noise in the logs of a running server

So the conclusion is to just silently ignore it

* Bitfinex: Accept locked market in TestUpdateTicker
2024-01-25 16:56:08 +11:00
Gareth Kirwan
e007f69f7c exchanges/websocket: Implement subscription configuration (#1394)
* Websockets: Move Subscription to its own package

This allows the small type to be imported from both `config` and from
`stream` without an import cycle, so we don't have to repeat ourselves

* Subs: Renamed Currency to Pair

This was being mis-used through much of the code, and since we're
already touching everything, we might as well fix it

* Websockets: Add Subscription configuration

* Binance: Add subscription configuration

* Kucoin: Subscription configuration

* Simplify GenerateDefaultSubs
* Improve TestGenSubs coverage
* Test Candle Sub generation
* Support Candle intervals
* Full responsibility for formatting Channel name on GenerateDefaultSubs
  OR consumer of Subscribe
* Simplify generatePayloads as a result
* Fix test coverage of asset types in processMarketSnapshot

* Exchanges: Abstract ParallelChanOp

* Tests: Generic ws mock instances

* Kucoin: Fix intermittent conflict in test currs

Use isolated test instance for `TestGetOpenInterest`.

`TestGetOpenInterest` would occassionally change pairs before
GenerateDefault Subs.
2024-01-24 15:54:07 +11:00
Gareth Kirwan
e866c8f08d Bitfinex: Fix UpdateTickers (#1427)
* Bitfinex: Fix TestUpdateTickers on ICE

Accept 5% failure rate of available pairs not working as tickers.

When Bitfinex [delisted ICE](https://www.bitfinex.com/posts/990) it's still coming back in all pub:conf and pub:info listings:
```
❯ curl -s https://api.bitfinex.com/v2/conf/pub:info:pair | jq -c '.[0][] | select(.[0] | test("(BTC|ICE|ZIL)USD")) '                                                                   ["BTCUSD",[null,null,null,"0.00006","2000.0",null,null,null,0.1,0.05,null,null]]
["ICEUSD",[null,null,null,"4.0","25000.0",null,null,null,null,null,null,null]]
["ZILUSD",[null,null,null,"40.0","1500000.0",null,null,null,null,null,null,null]]
```
_( I included ZIL to show a tradable pair without Margin fields )_
There's absolutely no sign it's not a tradable pair _until_ you ask for a ticker for it:
```
❯ curl -s https://api.bitfinex.com/v2/ticker/tICEUSD                                                                                                                                   ["error",10020,"symbol: invalid"]⏎
❯ curl -s https://api.bitfinex.com/v2/ticker/tBTCUSD                                                                                                                                   [42854,11.8920918,42855,12.71095442,-290,-0.00672292,42846,725.08132142,43288,41850]⏎
```

* fixup! Bitfinex: Fix TestUpdateTickers on ICE

* fixup! Bitfinex: Fix TestUpdateTickers on ICE

* Bitfinex: Fix UpdateTickers stopping on first error
2023-12-28 15:41:41 +11:00
Gareth Kirwan
f9437dbd08 Bitfinex: Websocket subscription improvements (#1353)
* Websockets: Add keys to websocket subscriptions

* This switches all RO uses of the mutex to use a RLock method.
* The mutex used for discrete field access has had scope drift from
  name 'connectionMutex' so rename to more appropriate fieldsMutex
* The mutex used for Set/CanUseAuthEndpoints moves from the
  subscriptions endpoint to the fieldsMutex
* Add GetSubscription by key
* Expose stream.Matcher type

* Bitfinex: Subscribe and Unsubscribe atomicly

* Fix Auth failures ignored
* This change makes it so that Subscribe and Unsubscribe wait for success
** Tells the DataHandler about errors
** Errors are returned to consumers
* Subscribes concurrently to the channels
* It also simplifies the chanId to stream mapping
* Removes unable to locate chanID: %d errors which are just noise
* Paves the way for unified channelSubscription id handling
* Adds support for subId for Book subscriptions, which is more robust

* Vastly simplifies what we need to test TestWsSubscribedResponse
This test was working to ensure that the various fancy key parsing
mechanisms all worked. Now that we use subId, we just need a thorough
test of that
* Expose Match.Set in order to capture websocket incoming data
Can't see another way of doing this. Doesn't seem too bad

* Allow tests to run with auth or WS
These flags made it difficult to run the tests whilst working on
websockets

* Enable API auth and WS in testconfig
This change minimises the changes requires for a full test run against
live endpoints, so that new contributors have a clearer testing path.
I cannot see any reason to turn WS off and Auth endpoints off when we're
not going to run API tests without Creds being set, and we're not going
to do live fire tests without canManipulateRealOrders

* TestWsSubscribe and various fixes
** Enables the websocket for live non-authed integration tests by default
** Adds an integration test for subscriptions
** Changes the Ws tests to respect canManipulateRealOrders
** Uses WsConnect instead of setupWs; fixes seqNo config not sent for WS tests
** Allows api creds to live in config/testdata.json which might be
  less likely to accidentally commit, and less obtrusive

* Bitfinex: Support period and timeframe for Candles

* Fixes manual Subscribe() symbol or key formatting
* Unifies handling of params for DefaultSubscriptions and manual
  subsrciptions

* Bitfinex: Handle conf and info WS channel events

* Bitfinex: Better tests for subscriptions

* fixup! Websockets: Add keys to websocket subscriptions

* fixup! Bitfinex: Subscribe and Unsubscribe atomicly

* fixup! Websockets: Add keys to websocket subscriptions

* Websockets: Add Pending subscription status

Add a status tracker so that Sub/Unsub can prevent duplicates,
and also fixes when first message comes before we have added the sub
to the tracker

* Websockets: Add State instead of pending

This change allows more clarity about the current state and
checks for specifically already Unsubing

* Bitfinex: Fix first sub message maybe lost

The only link we have between a sub req and the sub resp is the subID.
And the only link we have between a sub message and the sub is the chanID.
We can't derive a link using Pair or anything else.

This meant that by sending the resp and its chanID down the IncomingData
channel, we allowed the channel reader to maybe process the next
message, the first message on the channel, before the runtime executed
the switch back to subscribeToChan waiting on the chan.

To fix this, we key initially on subId.(string), and then replace it
with chanId.(int64) when we have it *inside* the wsHandleData so we
know we've procedurally handled it before the next message.

subscribeToChan is then free to remove the subId keyed Sub regardless of
error or not

If there's an error, we don't need to inline handling because there
won't be any second update.

Expands test coverage to make sure those subId keyed subscriptions are
removed.

* Websocket: Validate state in SetChanState

* fixup! Bitfinex: Fix first sub message maybe lost

* Websockets: Rename RemoveUnsuccessfulSubs

Implementation doesn't imply Unsuccessful or need to.
This change supports the registering of Pending subs

* Bitfinex: Fix race in Tests
2023-11-02 12:10:43 +11:00
Bea
2799f40401 Bitfinex: Fix websocket panic when seqNo is not sent (#1356) 2023-10-09 17:22:06 +11:00
Gareth Kirwan
1212b6750e OrderManager: Fix race condition in submit with ws (#1336)
* OrderManager: Fix race condition in submit with ws

If the ws sees the order before processSubmittedOrder then it will have
assigned it an internal order id already and added it to the store.
Don't treat that as an error. Instead just use the newer ws details

* OrderManager: Fix error comparisson

Should always use errors.Is when possible

* Tests: Simplify btcusd test pair declaration

* OrderManager: Improve test readability

* OrderManager: Add orderstore.getByDetail test

* Return a fresh pointer from orderstore.getByDetail

This protects the order.Details in the store from direct access.
The use-case was to allow the returned objects to be references so that
future changes to them would be reflected.
However we're not ready yet to allow people to touch the orders
directly, because they're not protected directly by a mutex, and nothing
would stop consumers contaminating the integrity of the data.

We can revisit this topic later atomicly, but it's definitely tangental
to the cause of action for PR #1336.

* Fix GetByDetail tests to assert a new pointer

* OrderManager: Avoid possible lock races

This fix internalises the getByDetail because the implication of moving
lock ownership out of exists/getByDetail to consumers breaks the order
store struct encapsulation in a way we really don't want to.
It's also more efficient

* Fix spelling mistake

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* OrderManager: Fix TestSubmitOrder... description

* OrderManager: Improve clarity of comment

* OrderManager: Capitalise error message

On failure to add to orderstore, capitalise the error message

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

---------

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-10-06 09:21:49 +11:00
Gareth Kirwan
ade2d9c5d2 Bitfinex: Resubscribe orderbook after checksum err (#1303)
* Bitfinex: Resubscribe orderbook after checksum err

* Bitfinex: chanForSub return only first match
2023-09-14 15:38:01 +10:00
Gareth Kirwan
6ab4c27d1c Bitfinex: Refactor websocket wsUpdate handling and various improvements (#1317)
* Bitfinex: Fix cancel/update order WS ack not seen

Fixes #1288

* Bitfinex: Fix ws Unsubscribe and Resubscribe

Unsubscribe needed to use the channel id.
Resubscribe needs to have the original subscription params.

* Bitfinex: Fix ws Trades Fees on te

The ws channel for authenticated Trades sends two types of update:
* te, Trade Executed
* tu, Trade Execution Update

Only the second one contains fee information.
[See the docs](https://docs.bitfinex.com/reference/ws-auth-trades)

This commit fixes:
`exchange Bitfinex websocket error - unable to type assert trade fee`
after an executed market trade on the te update

* Bitfinex: Fix error on ws auth ok

This fixes:
`Bitfinex Could not find an existing channel subscription: account Pair:
ChannelID: 0`
It's not clear from history why we'd want to store a reference to the
ubiquitous 0 channel like this, but it's definitely wrong, and anything
that attempts to get channel information about 0 chan needs to be fixed
anyway.

* Bitfinex: Refactor wsUpdate handling

This commit doesn't break out all the sub-updater, but attempts to do
something about the unmanagable size of ws update handling

* Binfinex: Fix linter issue on chanId casing

* Bitfinex: Fix linter outdent complaint

* Bitfinex: Fix linter issues on test

* Bitfinex: Fix TestWsTradingPairSnapshot chan lookup

* Bitfinex: Remove unnecessary WsAddSubs in test

* Bitfinex: Fix TestWsSubscribedResponse chan

* Bitfinex: Throw a specific error for bad event

* Bitfinex: WS Type assertions for positionSnapshots

* Bitfinex: tradeUpdate type assertion

* Bitfinex: Reinstate default subscriptions

* Bitfinex: Assert chan assetType is the same

* Bitfinex: Lowercase error string

* Bitfinex: Refactor WS eventType/chanId handling

* Bitfinex: Fix linter issues

* Bitfinex: Fix delimiter for pairs with more than 6 chars

* Bitfinex: Fix WS handling of subscribed symbols

This simplifies the handling of subscription symbols.
Now that we know the channel up front from handling the subscribed
response we can limit the parsing forms needed

* Bitfinex: Placate the linter

* Bitfinex: Disable margin assets for WS

Margin WS Currently not fully implemented and causes subscription collisions with spot

* Bitfinex: Fix parsing of 4 part funding keys

This improves overall handling and errors on a few current assumptions
about key structure

* Bitfinex: Linter fixes

* Bitfinex: Remove key parsing from assetPairFromSymbol

* Bitfinex: Use native error wrapping

* Bitfinex: Skip disabled assets in default ws subs
2023-09-01 16:46:00 +10:00
Bea
0cae2ad804 Bitfinex: Implement UpdateOrderExecutionLimits and tests (#1269)
* Bitfinex: Order Execution Limits and Testing

* Bitfinex: added proposed changes

* Bitfinex: lint fixes- removed the unnecessary whitespace and replaced the value with indexing

* Bitfinex: Moved data len check before the string conversion, moved the error for the array conversion before the info len check

* Bitfinex: Change GetSiteInfoConfigData to return slice, add tests

* Bitfinex: Fixed lint issue by preallocating pairs
2023-07-31 17:56:45 +10:00
Scott
fcc5ad4551 exchanges/qa: Add exchange wrapper testing suite (#1159)
* initial concept of a nice validation tester for exchanges

* adds some datahandler design

* expand testing

* more tests and fixes

* minor end of day fix for bithumb

* fixes implementation issues

* more test coverage and improvements, but not sure if i should continue

* fix more wrapper implementations

* adds error type, more fixes

* changes signature, fixes implementations

* fixes more wrapper implementations

* one more bit

* more cleanup

* WOW things work?

* lintle 1/1337

* mini bump

* fixes all linting

* neaten

* GetOrderInfo+ asset pair fixes+improvements

* adds new websocket test

* expand ws testing

* fix bug, expand tests, improve implementation

* code coverage of a lot of new codes

* fixes everything

* reverts accidental changes

* minor fixes from reviewing code

* removes Bitfinex cancelBatchOrder implementation

* fixes dumb baby typo for babies

* mini nit fixes

* so many nits to address

* addresses all the nits

* Titlecase

* switcheroo

* removes websocket testing for now

* fix appveyor, minor test fix

* fixes typo, re-kindles killed kode

* skip binance wrapper tests when running CI

* expired context, huobi okx fixes

* kodespull

* fix ordering

* time fix because why not

* fix exmo, others

* hopefully this fixes all of my life's problems

* last thing today

* huobi, more like hypotrophy

* golangci-lint, more like mypooroldknee-splint

* fix huobi times by removing them

* should fix okx currency issues

* blocks the application

* adds last little contingency for pairs

* addresses most nits and new problems

* lovely fixed before seeing why okx sucks

* fixes issues with okx websocket

* the classic receieieivaier

* lintle

* adds test and fixes existing tests

* expands error handling messages during setup

* fixes dumb okx bugs introduced

* quick fix for lint and exmo

* fixes nixes

* fix exmo deposit issue

* lint

* fixes issue with extra asset runs missing

* fix surprise race

* all the lint and merge fixes

* fixes surprise bugs in OKx

* fixes issues with times and chains

* fixing all the merge stuff

* merge fix

* rm logs and a panic potential

* lovely lint lament

* an easy demonstration of scenario, but not of initial purpose

* put it in the bin

* Revert "put it in the bin"

This reverts commit 15c6490f713233d43f10957367fcbf18e3818bdd.

* re-add after immediate error popup

* fix mini poor test design

* okx okay

* merge fixes

* fixes issues discovered in lovely test

* I FORGOT TO COMMIT THIS

* nit fixaroonaboo

* forgoetten test fix

* revert old okx asset intrument work

* fixes

* revert problems I didnt understand. update bybit

* fix merge bugs

* test cleanup

* further improvements

* reshuffle and lint

* rm redundant CI_TEST by rm the CI_TEST field that is redundant

* path fix

* move to its own section, dont run on 32 bit + appveyor

* lint

* fix lbank

* address nits

* let it rip

* fix failing test time range

* niteroo boogaloo

* mod tidy, use common.SimpleTimeFormat
2023-07-03 11:09:43 +10:00
Ryan O'Hara-Reid
d086c45b0a Bitfinex: filter security asset type from spot trading pairs (#1191)
* bitfinex: filter security pairs from spot trading

* asset: add securities type

* bitfinex: remove print

* linter: fix

* gomod tidy

* glorious: nits

* bitfinex: just some orderbook fun and added a pass through for bitfinex funding rate when rate == 0

* bitfinex: move update function from testmain to localised test

* test: fix up

* glorious:nits

* bitfinex; fix casing test

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-06-05 11:53:07 +10:00
Samuael A
3eac6d12bd exchanges: Update GateIO exchange to V4 (#1058)
* Adding Public Endpoints and test functions

* Adding public endpoints and test functions

* Adding private spot endpoints

* Adding private endpoints and corresponding tests for margin

* Adding Margin Private endpoints

* Adding cross margin and flash swap endpoints

* Adding futures private endpoints

* Adding futures private endpoints and corresponding tests

* Adding Options and SubAccount endpoints and their unit tests

* Adding Wrapper functions

* Complete wrapper functions and corresponding unit test functions

* Fixing wrapper issues and adding websocket functions

* Update of Spot websocket and adding futures websocket handlers

* completed futures WS push data endpoints

* Completed Options websocket endpoints

* Adding websocket support for delivery futures and slight update on endpoint funcs

* Added Delivery websocket support and fix linter issues

* Update on Unit tests

* fix slight currency format error

* Fix slight endpoint tempos

* Update on conditional statements and unit tests issues

* fixing slight tempos

* Slight model and websocket data push method change

* Fix unit test tempos and updating models

* Fix on code structures and update on unit tests

* Slight code fix

* Remove print statements

* Update on tradable pairs fetch eps

* Fix websocket tempos

* Adding types to websocket routine manager

* Fix slight issues

* Slight fixes

* Updating wrapper funcs and models

* Slight update

* Update on test

* Update on tradable pairs

* update conditional statements

* Fixing slight issues

* Updating unit tests

* Minor fixes depending review comments

* Remove redundant method declaration

* Adding missing intervals

* Updating fetch tradable pairs

* update tradable pairs issues

* Addressing small tempos

* Slight fix on ticker

* Minor Fixes

* Minor review comment fixes

* Unit test and minor code updates

* Slight code updates

* Minor updates depending review comments

* Fixes

* Updating incoming message matcher

* Fix missing merge issue

* Fix minor wrapper issues

* Updating ratelimit and other issues

* Updating endpoint models and adding missing eps

* Update on code structure and models

* Minor codespell fixes

* Minor update on models

* fix unit test panic

* Minor race fix

* Fix issues in generating signature and unit tests

* Minor update on wrapper and unit tests

* Minor fix on wrapper

* Mini linter issues fix

* Minor fix

* endpoint fixes and slight update

* Minor fixes

* Updating exchange functions and unit tests

* Unit test and wrapper updates

* Remove options candlestick support

* Minor unit test and wrapper fix

* Unit test update

* minor fix on unit test and wrapper

* endpoints constants name change

* Add minor wrapper issues

* endpoint constants update

* endpoint url updates

* Updating subscriptions

* fixing dual mode endpoint methods

* minor fix

* rm small tempo

* Update on websocket orderbook handling

* Orderbook and currency pair update

* fix linter and test issues

* minor helper function update

* Fix wrapper coverage and wrapper issues

* delete unused variables

* Minor fix on ReadData() call

* separating websocket handlers

* separating websocket handlers

* Minor fix on enabled pair

* minor fix

* check instrument availability in spot

* create a separate subscriber for sake of multiple websocket connection

* linter fix

* minor websocket and gateio endpoints fix

* fix nil pointer exception

* minor fixes

* spelling fix decerializes -> deserializes

* fix Bitfinex unit test issues

* minor unknown currency pair labling fix

* minor currency pair handling fix

* slight update on GetDepositAddress wrapper unit test

* setting max request job to 200

* fixing numerical and timestamp type convert

* fix value overflow error

* change method of parsing orderbook price

* unifying timestamp conversion types to gateioTime

---------

Co-authored-by: Samuael Adnew <samuaelad@Samuaels-MacBook-Air.local>
2023-05-30 14:03:53 +10:00
Ryan O'Hara-Reid
b20cf75d13 sharedtestvalues: Add helper functions and implement throughout exchange tests (#1163)
* exchanges/sharedtestvalues: implement new functions to handle test skipping and announcements for standardising.

* exchanges: fin test impl.

* linter: fixes

* exchange_template: fix test

* allocate so it doesn't make a panic at the disco

* glorious: nits

* glorious: nits

* Update exchanges/sharedtestvalues/sharedtestvalues.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* Update exchanges/sharedtestvalues/sharedtestvalues.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* linter: fix

* linter: shhhh

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2023-04-28 13:05:42 +10:00