Commit Graph

194 Commits

Author SHA1 Message Date
Adrian Gallagher
a5b638bfb7 GHA: Add additional checks for common issues (#1922)
* GHA, tests: Add additional checks for common issues

These checks include:
- Ensuring that all testify funcs use their formatted variants (e.g., `assert.Equalf(t, expected, actual)` instead of `assert.Equal(t, expected, actual)`).
- Replacing `%s` with %q
- Enforcing consistent usage of should/must wording for testify assert/require messages

* Add support for checking backticked string format specifiers and fix issues

* tests: Fix error comparisons

* tests: Replace errors.Is(err, nil) usage with testify and automate check

* refactor: Rename ExtractPort to ExtractPortOrDefault

* tests: Replace assert with require for error handling in multiple test files

* tests: Replace assert with require for error handling and improve assertions in data tests

* tests: Fix typo in assertion message for StreamVol test

* OKX: Fix GetOpenInterestAndVolumeStrike test with instrument selection and improved assertions

* OKX: Revert intentional error check

* Improve error message for expiry time check in GetOpenInterestAndVolumeStrike test
2025-05-28 12:26:51 +10:00
Adrian Gallagher
9f659e71e3 exchanges: Fix race, improve error handling in ValidateKline and assertify related tests (#1919)
* exchanges: Fix race, improve error handling in ValidateKline and assertify related tests

* exchanges: Refactor ValidateKline error handling and enhance related tests

* refactor: Fix spaghetti bolognese if else error logic

* exchanges: Simplify error assertion in TestGetKlineRequest
2025-05-23 17:30:11 +10:00
Ryan O'Hara-Reid
cad7586e98 exchange/websocket, gateio: Rename/export package again, add websocket request functions for futures trading (#1603)
* gateio: Add multi asset websocket support WIP.

* meow

* Add tests and shenanigans

* integrate flushing and for enabling/disabling pairs from rpc shenanigans

* some changes

* linter: fixes strikes again.

* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.

* Add subscription tests (state functional)

* glorious:nits + proxy handling

* Spelling

* linter: fixerino

* instead of nil, dont do nil.

* clean up nils

* cya nils

* don't need to set URL or check if its running

* stream match update

* update tests

* linter: fix

* glorious: nits + handle context cancellations

* stop ping handler routine leak

* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response

* Allow rollback on connect on any error across all connections

* fix shadow jutsu

* glorious/gk: nitters - adds in ws mock server

* linter: fix

* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.

* glorious: whooops

* gk: nits

* Leak issue and edge case

* Websocket: Add SendMessageReturnResponses

* whooooooopsie

* gk: nitssssss

* Update exchanges/stream/stream_match.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/stream/stream_match_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* linter: appease the linter gods

* gk: nits

* gk: drain brain

* started

* more changes before merge match pr

* gateio: still building out

* gateio: finish spot

* fix up tests in gateio

* Add tests for stream package

* rm unused field

* glorious: nits

* rn files, specifically set function names to asset and offload routing to websocket type.

* linter: fix

* Add futures websocket request support

* gateio: integrate with IBOTExchange (cherry pick my nose)

* linter: fix

* glorious: nits

* add counter and update gateio

* fix collision issue

* Update exchanges/stream/websocket.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* add tests

* linter: fix

* After merge

* Add error connection info

* upgrade to upstream merge

* Fix edge case where it does not reconnect made by an already closed connection

* stream coverage

* glorious: nits

* glorious: nits removed asset error handling in stream package

* linter: fix

* rm block

* Add basic readme

* fix asset enabled flush cycle for multi connection

* spella: fix

* linter: fix

* Add glorious suggestions, fix some race thing

* reinstate name before any routine gets spawned

* stop on error in mock tests

* glorious: nits

* Set correct price

* glorious: nits found in CI build

* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.

* mutex across shutdown and connect for protection

* lint: fix

* test time withoffset, reinstate stop

* fix whoops

* const trafficCheckInterval; rm testmain

* y

* fix lint

* bump time check window

* stream: fix intermittant test failures while testing routines and remove code that is not needed.

* spells

* cant do what I did

* protect race due to routine.

* update testURL

* use mock websocket connection instead of test URL's

* linter: fix

* remove url because its throwing errors on CI builds

* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.

* remove another superfluous url thats not really set up for this

* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly

* linter: fixerino uperino

* fix ID bug, why I do this, I don't know.

* glorious: panix

* linter: things

* whoops

* dont need to make consecutive Unix() calls

* websocket: fix potential panic on error and no responses and adding waitForResponses

* rm json parser and handle in json package instead

* in favour of json package unmarshalling

* linter: fix

* linter: fix again

* * change field name OutboundRequestSignature to WrapperDefinedConnectionSignature for agnostic inbound and outbound connections.
* change method name GetOutboundConnection to GetConnection for agnostic inbound and outbound connections.
* drop outbound field map for improved performance just using a range and field check (less complex as well)
* change field name connections to connectionToWrapper for better clarity

* spells and magic and wands

* glorious: nits

* comparable check for signature

* mv err var

* glorious: nits and stuff

* attempt to fix race

* glorious: nits

* gk: nits; engine log cleanup

* gk: nits; OCD

* gk: nits; move function change file names

* gk: nits; 🚀

* gk: nits; convert variadic function and message inspection to interface and include a specific function for that handling so as to not need nil on every call

* gk: nits; continued

* gk: engine nits; rm loaded exchange

* gk: nits; drop WebsocketLoginResponse

* stream: Add match method EnsureMatchWithData

* gk: nits; rn Inspect to IsFinal

* gk: nits; rn to MessageFilter

* linter: fix

* gateio: update rate limit definitions (cherry-pick)

* Add test and missing

* Shared REST rate limit definitions with Websocket service, set lookup item to nil for systems that do not require rate limiting; add glorious nit

* integrate rate limits for websocket trading spot

* conform to match upstream changes

* standardise names to upstream style

* fix wrapper standards test when sending a auth request through a websocket connection

* whoops

* Update exchanges/gateio/gateio_types.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* linter: fix

* linter: overload

* whoops

* spelling fixes on recent merge

* glorious: nits

* linter: fix?

* glorious: nits

* gk: assert errors touched

* gk: unexport derive functions

* gk: nitssssssss

* fix test

* gk: nitters v1

* gk: http status

* gk/nits: Add getAssetFromFuturesPair

* gk: nits single response when submitting

* gk: new pair with delimiter in tests

* gk: param update slice to slice of pointers

* gk: add asset type in params, includes t.Context() for tests

* linter: fix

* linter: fix

* fix merge whoopsie

* glorious: nits

* gk: nit

* shift over to websocket package error

* internal/exchange/websocket -> exchange/websocket

* PEAK OCD!

* appease the OCD gods

* thrasher: nits

---------

Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2025-04-11 16:47:33 +10:00
Gareth Kirwan
b4e45e9a1b Websocket: Restructure files and types (#1859)
* Websocket: Rename stream package

* Websocket: Rename Websocket to Manager

* Websocket: Replace explicit errs with common.NilGuard

* Websocket: Move websocket_types.go to types.go

* Websocket: Minor field comment and alignment in types

* Webosocket: Rename WebsocketConnection to Connection

* Alphapoint: Make gorilla ws import explicit

Just to avoid confusion with our own packages.

* Websocket: Move stream_match to match

* Websocket: Move websocket_connection to connection

* Websocket: Move websocket.go to manager.go

* Websocket: Break out all subscription methods into subscriptions.go

* Websocket: Move connection type into its file

* Websocket: Remove PositionUpdated

Type is not used anywhere

* Kraken: Use local constant for pong

Was the only use of websocket.Pong and doesn't really feel right to
represent kraken's api resp in one of our packages

* Websocket: Move connection sub-types to connection package

* Websocket: Move manager types into manager

* Websocket: Move ConnectionWrapper into manager

* Websocket: Move websocket_test to manager_test

* Websocket: Privatise connectionWrapper

* Websocket: Remaining types into types.go

These really belong somewhere else mostly, but this will do for now

* Websocket: Tidy up connection method vars

* Gofumpt: Moving package imports around

* Websocket: Rename errDuplicateConnectionSetup

* Websocket: Fix duplicate import of gws

* Websocket: Fix gofumpt -extra

* Websocket: Standardise import of gws across other pkgs

* Kraken: Remove unused sub conf consts

These were replaced by the generic Levels and Depth fields on all subs

* Websocket: Privitise ConnectioWrapper fields

* Websocket: inline single use var WebsocketNotAuthenticatedUsingRest

* Websocket: Move documentation to template

* Bithumb: Assertify TestWsHandleData
2025-04-10 16:25:02 +10:00
Adrian Gallagher
4651af5767 modernise: Run new gopls modernise tool against the codebase and fix minor issues (#1826)
* modernise: Run new gopls modernise tool against codebase

* Address shazbert's nits

* apichecker, gctcli: Simplify HTML scraping functions and improve depth limit handling

* refactor: Create minSyncInterval const and update order book limit handling for binance and binanceUS

* refactor: Various slice usage improvements and rename TODO

* tranches: Revert deleteByID changes due to performance decrease

Shazbert was a F1 driver in a past lifetime 🏎️

* tranches: Simply retrieve copy

Thanks to shazbert

* documentation: Sort contributors list by contributions

* tranches: Remove deadcode in deleteByID
2025-03-21 09:17:10 +11:00
Gareth Kirwan
16d2d9f35a Config: AssetEnabled upgrade (#1735)
* Config: Move assetEnabled upgrade to Version management

* Assets: Do not error on asset not enabled, or disabled

This became more messy with Disabling something that's defaulted to
disabled.
Taking an idealogical stance against erroring that what you want to have
done is already done.

* CurrencyManager: Set AssetEnabled when StorePairs(enabled)

* RPCServer: Fix tests expecting StoreAssetPairFormat to enable the asset

Also assertifies

* Bitfinex: Fix tests for MarginFunding subs

* GCTWrapper: Improve TestMain clarity

* BTSE: Add futures to testconfig

* Exchanges: Rename StoreAssetPairStore

Previously we were calling it "Format", but accepting everything from
the PairStore.
We were also defaulting to turning the Asset on.

Now callers need to get their AssetEnabled set as they want it, so
there's no magic

This change also moves responsibility for error wrapping outside to the
caller.

* Config: AssetEnabled upgrade should respect assetTypes

Previously we ignored the field and just turned on everything.
I think that was because we couldn't get at the old value.
In either case, we have the option to do better, and respect the
assetEnabled value

* Config: Improve exchange config version upgrade error messages
2025-03-17 21:47:37 +11:00
Adrian Gallagher
d64d56f77c build/ci: Update Go to v1.24, golangci-lint to v1.64.6 and fix issues (#1804)
* build/ci: Update Go to v1.24, golangci-lint to v1.64.5 and fix issues

* Address shazbert's nitters

* linter/config: Fix new linter issue and use versionSize const

* Address gk's nitters and fix additional linter issue after rebase

* Address glorious nits

* staticcheck: Fix additional linter issues after upgrading to Go 1.24.1 and golangci-lint v1.64.6

Also addresses nits

* Improve testing, assertify usage and use common.ErrParsingWSField

* TestCreateNewStrategy: Replace must > should wording
2025-03-10 16:33:55 +11:00
Jun
7a6d6cc002 trade: remove exchangeName for AddTradesToBuffer (#1820)
Signed-off-by: Ye Sijun <junnplus@gmail.com>
2025-03-06 12:50:03 +11:00
Ryan O'Hara-Reid
08e015a125 exchanges/wrappers: Refactor fetch orderbook/ticker/account info funcs (#1440)
* acrost: Pull thread, examine

* fix tests

* linter

* fix_linter

* revert rm ctx param to limit breakages when merging usptream

* linter fix

* Add in priority update grouping so that tests pass

* Update cmd/exchange_wrapper_standards/exchange_wrapper_standards_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious nits

* fixed spelling

* whoopsie

* aanother whoops

* glorious: NITTERS!

* glorious: further nitters

* srry linter gods

* glorious: nits continued

* sub test p ara lel

* drop main t.Parallel

* fix whoops

* wrappertests: use context with cancel (test)

* linter: fix

* ensure primary execution

* kucoin test fix

* revert standards test changes and bypass non critical errors

* rm single override

* wrap exported error for accounts

* thrasher: nits ch name

* gk: nits

* gk: nits FetchTickerCached -> GetCachedTicker

* gk: nits rn FetchOrderbookCached -> GetCachedOrderbook

* gk: nits rn FetchAccountInfoCached -> GetCachedAccountInfo

* linter: fix

* gk: nits

* thrasher: nitters 1

* thrasher: nitters tmpls

* gk: nitter

---------

Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2025-02-19 10:47:10 +11:00
Ryan O'Hara-Reid
50448ec6a0 websocket/gateio: Add request functions for websocket multi-connection [SPOT] (#1598)
* gateio: Add multi asset websocket support WIP.

* meow

* Add tests and shenanigans

* integrate flushing and for enabling/disabling pairs from rpc shenanigans

* some changes

* linter: fixes strikes again.

* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.

* Add subscription tests (state functional)

* glorious:nits + proxy handling

* Spelling

* linter: fixerino

* instead of nil, dont do nil.

* clean up nils

* cya nils

* don't need to set URL or check if its running

* stream match update

* update tests

* linter: fix

* glorious: nits + handle context cancellations

* stop ping handler routine leak

* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response

* Allow rollback on connect on any error across all connections

* fix shadow jutsu

* glorious/gk: nitters - adds in ws mock server

* linter: fix

* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.

* glorious: whooops

* gk: nits

* Leak issue and edge case

* Websocket: Add SendMessageReturnResponses

* whooooooopsie

* gk: nitssssss

* Update exchanges/stream/stream_match.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* Update exchanges/stream/stream_match_test.go

Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>

* linter: appease the linter gods

* gk: nits

* gk: drain brain

* started

* more changes before merge match pr

* gateio: still building out

* gateio: finish spot

* fix up tests in gateio

* Add tests for stream package

* rm unused field

* glorious: nits

* rn files, specifically set function names to asset and offload routing to websocket type.

* linter: fix

* glorious: nits

* add counter and update gateio

* fix collision issue

* Update exchanges/stream/websocket.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* add tests

* linter: fix

* After merge

* Add error connection info

* upgrade to upstream merge

* Fix edge case where it does not reconnect made by an already closed connection

* stream coverage

* glorious: nits

* glorious: nits removed asset error handling in stream package

* linter: fix

* rm block

* Add basic readme

* fix asset enabled flush cycle for multi connection

* spella: fix

* linter: fix

* Add glorious suggestions, fix some race thing

* reinstate name before any routine gets spawned

* stop on error in mock tests

* glorious: nits

* glorious: nits found in CI build

* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.

* mutex across shutdown and connect for protection

* lint: fix

* test time withoffset, reinstate stop

* fix whoops

* const trafficCheckInterval; rm testmain

* y

* fix lint

* bump time check window

* stream: fix intermittant test failures while testing routines and remove code that is not needed.

* spells

* cant do what I did

* protect race due to routine.

* update testURL

* use mock websocket connection instead of test URL's

* linter: fix

* remove url because its throwing errors on CI builds

* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.

* remove another superfluous url thats not really set up for this

* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly

* linter: fixerino uperino

* glorious: panix

* linter: things

* whoops

* dont need to make consecutive Unix() calls

* websocket: fix potential panic on error and no responses and adding waitForResponses

* rm json parser and handle in json package instead

* linter: fix

* linter: fix again

* * change field name OutboundRequestSignature to WrapperDefinedConnectionSignature for agnostic inbound and outbound connections.
* change method name GetOutboundConnection to GetConnection for agnostic inbound and outbound connections.
* drop outbound field map for improved performance just using a range and field check (less complex as well)
* change field name connections to connectionToWrapper for better clarity

* spells and magic and wands

* glorious: nits

* comparable check for signature

* mv err var

* glorious: nits and stuff

* attempt to fix race

* glorious: nits

* gk: nits; engine log cleanup

* gk: nits; OCD

* gk: nits; move function change file names

* gk: nits; 🚀

* gk: nits; convert variadic function and message inspection to interface and include a specific function for that handling so as to not need nil on every call

* gk: nits; continued

* gk: engine nits; rm loaded exchange

* gk: nits; drop WebsocketLoginResponse

* stream: Add match method EnsureMatchWithData

* gk: nits; rn Inspect to IsFinal

* gk: nits; rn to MessageFilter

* linter: fix

* gateio: update rate limit definitions (cherry-pick)

* Add test and missing

* Shared REST rate limit definitions with Websocket service, set lookup item to nil for systems that do not require rate limiting; add glorious nit

* integrate rate limits for websocket trading spot

* bitstamp: fix issue

* glorious: nits

* ch name and commentary

* fix bug add test

* rm a thing

* fix test

* Update engine/engine.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* thrasher: nits

* Update exchanges/stream/stream_match_test.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update exchanges/stream/stream_match_test.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* GK: nits rn websocket functions

* explicit function names for single to multi outbound orders

* linter: fix

---------

Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Gareth Kirwan <gbjkirwan@gmail.com>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2024-12-20 13:50:31 +11:00
Gareth Kirwan
4d36ea4943 Bitmex: Fix handling index records in WS trade stream (#1685)
Fixes handling for Size == 0 index records sent to trade stream

fixes #1684
2024-10-25 19:57:34 +11:00
Gareth Kirwan
2232340d49 Bitmex: Subscription templating (#1586)
* Exchanges: Allow empty batches in ParallelChanOp

In keeping with both common.Batch and "It's not my responsibility",
ParallelChanOp should just do nothing when given an empty list (and
implicitly an empty batch size.
Whatever it's going to do, it'll delegate to common.Batch,
and this allows us to just inline calls:
```
	return common.AppendError(
		b.ParallelChanOp(subs.Public(), func(l subscription.List) error { return b.manageSubs(wsSubscribeMethod, l, wsPublicStream) }, len(subs)),
		b.ParallelChanOp(subs.Private(), func(l subscription.List) error { return b.manageSubs(wsSubscribeMethod, l, wsPrivateStream) }, len(subs)),
	)
```

* Bitmex: Test config updates

* Bitmex: Sub Templating

* Bitmex: Enable websocket for tests

* Bitmex: Handle subscription errors

This switches to multiplexing so that we know which errors belong to
which stream, particularly for the auth attempt

* Bitmex: Fix ws order side err going to data stream

Shouldn't fall into classification error if it's actually a parsing
error
2024-10-22 17:21:35 +11:00
Ryan O'Hara-Reid
ac731ce283 websocket/gateio: Support multi connection management and integrate with GateIO (#1580)
* gateio: Add multi asset websocket support WIP.

* meow

* Add tests and shenanigans

* integrate flushing and for enabling/disabling pairs from rpc shenanigans

* some changes

* linter: fixes strikes again.

* Change name ConnectionAssociation -> ConnectionCandidate for better clarity on purpose. Change connections map to point to candidate to track subscriptions for future dynamic connections holder and drop struct ConnectionDetails.

* Add subscription tests (state functional)

* glorious:nits + proxy handling

* Spelling

* linter: fixerino

* instead of nil, dont do nil.

* clean up nils

* cya nils

* don't need to set URL or check if its running

* stop ping handler routine leak

* * Fix bug where reader routine on error that is not a disconnection error but websocket frame error or anything really makes the reader routine return and then connection never cycles and the buffer gets filled.
* Handle reconnection via an errors.Is check which is simpler and in that scope allow for quick disconnect reconnect without waiting for connection cycle.
* Dial now uses code from DialContext but just calls context.Background()
* Don't allow reader to return on parse binary response error. Just output error and return a non nil response

* Allow rollback on connect on any error across all connections

* fix shadow jutsu

* glorious/gk: nitters - adds in ws mock server

* linter: fix

* fix deadlock on connection as the previous channel had no reader and would hang connection reader for eternity.

* gk: nits

* Leak issue and edge case

* gk: nits

* gk: drain brain

* glorious: nits

* Update exchanges/stream/websocket.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* add tests

* linter: fix

* After merge

* Add error connection info

* Fix edge case where it does not reconnect made by an already closed connection

* stream coverage

* glorious: nits

* glorious: nits removed asset error handling in stream package

* linter: fix

* rm block

* Add basic readme

* fix asset enabled flush cycle for multi connection

* spella: fix

* linter: fix

* Add glorious suggestions, fix some race thing

* reinstate name before any routine gets spawned

* stop on error in mock tests

* glorious: nits

* glorious: nits found in CI build

* Add test for drain, bumped wait times as there seems to be something happening on macos CI builds, used context.WithTimeout because its instant.

* mutex across shutdown and connect for protection

* lint: fix

* test time withoffset, reinstate stop

* fix whoops

* const trafficCheckInterval; rm testmain

* y

* fix lint

* bump time check window

* stream: fix intermittant test failures while testing routines and remove code that is not needed.

* spells

* cant do what I did

* protect race due to routine.

* update testURL

* use mock websocket connection instead of test URL's

* linter: fix

* remove url because its throwing errors on CI builds

* connections drop all the time, don't need to worry about not being able to echo back ws data as it can be easily reviewed _test file side.

* remove another superfluous url thats not really set up for this

* spawn overwatch routine when there is no errors, inline checker instead of waiting for a time period, add sleep inline with echo handler as this is really quick and wanted to ensure that latency is handing correctly

* linter: fixerino uperino

* glorious: panix

* linter: things

* whoops

* defer lock and use functions that don't require locking in SetProxyAddress

* lint: fix

* thrasher: nits

---------

Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2024-10-10 15:09:52 +11:00
Gareth Kirwan
c2dfb37efd Subscriptions: Add List.Public, List.Enabled and List.Private (#1629) 2024-10-10 14:48:16 +11:00
Adrian Gallagher
225429bda6 CI/build: Update Go version, linters and fix minor issues (#1612)
* CI/build: Update Go version, linters and fix minor issues

* linters: Add intrange, copyloopvar, additional go vet linters to match gopls and fix issues
2024-08-16 17:41:11 +10:00
Gareth Kirwan
b41fe27684 Kucoin: Add subscription templating and various fixes (#1579)
* Currency: Variadic Pairs.Add

This version of Pairs.Add is simpler and [more
performant](https://gist.github.com/gbjk/06a1fc1832d04ee41213ca518938cf74)

Behavioural difference: If there's nothing to add, the same slice is
returned unaltered. This seems like good sauce

* Currency: Variadic Remove

* Common: Add Batch function

* Common: Add common.SortStrings for stringers

* Subscriptions: Add batching to templates

* Subscriptions: Sort list of pairs

* Kucoin: Switch to sub templating

* Kucoin: Simplify channel prefix usage

* Kucoin: Fix race on fetchedFuturesOrderbook

* Subscriptions: Filter AssetPairs

Now only the assetPairs relevant to the subscription are in the context

* Subscriptions: Respect subscription Pairs

* Subscriptions: Trim AssetSeparator early

We want to trim before checking for "AssetSeparator vs All" because a
template should be allowed to reuse a range template and generate just one trailing AssetSeparator
whilst using a specific Asset

* Kucoin: Fix empty margin asset added

* Kucoin: Add Subscription batching

Turns out that contary to the documentation, kucoin supports batching of
all symbols and currencies

* Kucoin: Fix checkSubscriptions and coverage

* Subscriptions: Simplify error checking

This reduces the complexity of error checking to just be "do we get the
correct numbers".

Fixes Asset.All with only one asset erroring on xpandPairs, because we
trimmed the only asset separator, and then errored that we're not
xpanding Assets and the asset on the sub is asset.All

This use-case conflicted with commit 6bbd546d74, which required:
```
Subscriptions: Trim AssetSeparator early

We want to trim before checking for "AssetSeparator vs All" because a
template should be allowed to reuse a range template and generate just one trailing AssetSeparator
whilst using a specific Asset
```

Now we set up the assets earlier, and we remove the check for xpandAssets, since the number of asset lines matching is all that matters.

I've removed the asset tests for this, but they were correctly erroring
on the number of asset lines instead.

Everything hits coverage, as well.

* Kucoin: Remove deprecated fundingBook endpoint

* BTCMarkets: Use common.Batch
2024-08-09 12:33:15 +10:00
Gareth Kirwan
c601575c66 subscriptions: Add templating support and integrate with Binance (#1568)
* Subscriptions: Add List.AssetPairs

* Subscriptions: Add Template and QualifiedChannel

These fields separate the concept of what the channel is from the
qualified resource name

* Subscriptions: Add List.SetStates()

* Subscriptions: Add List.QualifiedChannels

* Subscriptions: Rename testsubs.EqualLists

* Binance: Switch to ExpandTemplates

* Binance: Update ConfigTest format

* Subscriptions: Test Coverage improvements

* Subscriptions: Reenterant List.ExpandTemplates

* Subscriptions: Move templates from subscriptions to exchanges

* Binance: Inline subscription template and improvements
2024-07-09 15:53:00 +10:00
Ryan O'Hara-Reid
48349bc3bb protocol/order: adds additional fields for trading requirements (#1552)
* Add in initial handling for quote/base currency deployment requirements

* include client order ID checking

* glorious: suggestions

* spell and fix

* linter/context/test

* rework tests and order side specific requirements

* linter

* mend panic at the disco

* mending more panics at the disco

* anudda fix brudda

* glorious: NITTTTTT BOOOOOMB

* leftover things and stuff

* whoops

* tie in gateio

* glorious: nit fixes life and everything.

* thrasher: nits

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2024-07-03 16:07:23 +10:00
Gareth Kirwan
1199f38546 subscriptions: Encapsulate, replace Pair with Pairs and refactor; improve exchange support
* Websocket: Use ErrSubscribedAlready

instead of errChannelAlreadySubscribed

* Subscriptions: Replace Pair with Pairs

Given that some subscriptions have multiple pairs, support that as the
standard.

* Docs: Update subscriptions in add new exch

* RPC: Update Subscription Pairs

* Linter: Disable testifylint.Len

We deliberately use Equal over Len to avoid spamming the contents of large Slices

* Websocket: Add suffix to state consts

* Binance: Subscription Pairs support

* Bitfinex: Subscription Pairs support

* Bithumb: Subscription Pairs support

* Bitmex: Subscription Pairs support

* Bitstamp: Subscription Pairs support

* BTCMarkets: Subscription Pairs support

* BTSE: Subscription Pairs support

* Coinbase: Subscription Pairs support

* Coinut: Subscription Pairs support

* GateIO: Subscription Pairs support

* Gemini: Subscription Pairs support and improvement

* Hitbtc: Subscription Pairs support

* Huboi: Subscription Pairs support

* Kucoin: Subscription Pairs support

* Okcoin: Subscription Pairs support

* Poloniex: Subscription Pairs support

* Kraken: Add subscription Pairs support

Note: This is a naieve implementation because we want to rebase the
kraken websocket rewrite on top of this

* Bybit: Subscription Pairs support

* Okx: Subscription Pairs support

* Bitmex: Subsription configuration

* Fixes unauthenticated websocket left as CanUseAuth
* Fixes auth subs happening privately

* CoinbasePro: Subscription Configuration

* Consolidate ProductIDs when all subscriptions are for the same list

* Websocket: Log actual sent message when Verbose

* Subscriptions: Improve clarity of which key is which in Match

* Subscriptions: Lint fix for HugeParam

* Subscriptions: Add AddPairs and move keys from test

* Subscriptions: Simplify subscription keys and add key types

* Subscriptions: Add List.GroupPairs Rename sub.AddPairs

* Subscription: Fix ExactKey not matching 0 pairs

* Subscriptions: Remove unused IdentityKey and HasPairKey

* Subscriptions: Fix GetKey test

* Subscriptions: Test coverage improvements

* Websocket: Change State on Add/Remove

* Subscriptions: Improve error context

* Subscriptions: Fix Enable: false subs not ignored

* Bitfinex: Fix WsAuth test failing on DataHandler

DataHandler is eaten by dataMonitor now, so we need to use ToRoutine

* Deribit: Subscription Pairs support

* Websocket: Accept nil lists for checkSubscriptions

If the user passes in a nil (implicitly empty) list, we would not panic.
Therefore the burden of correctness about that data lies with them.
The list of subscriptions is empty, and that's okay, and possibly
convenient

* Websocket: Add context to NilPointer errors

* Subscriptions: Add context to nil errors

* Exchange: Fix error expectations in UnsubToWSChans
2024-06-07 11:54:08 +10:00
Gareth Kirwan
7d1eecfa7e Tests: Various race fixes and move TestFixtureToDataHandler (#1534)
* Tests: Move and simplify TestFixtureToDataHandler

* Currency: Fix PairsManager.Load breaking matcher

* Tests: Add multi-instance cache to UpdatePairsOnce

* Kraken: Fix TestUpdateTickers race error

Calling StorePairs on global instance can lead to race

* Bitfinex: Fix TestUpdateTickers racing intermittently

* Currency: Fix concurrent access to PM formats

* Currency: Fix SupportsAsset implementation

This should delegate entirely to PairManager's IsAssetSupported

* Okx: Fix PM intrusion, rm GetPairFromInstrumentID

* Exchange: Fix SetGlobalPairsManager to set asset enabled

* Bitflyer: Fix race on set TestGetCurrURL

TestGetCurrencyTradeURL would fail sometimes due to sequencing of
enabling futures but not having pairs for it.

* Tests: Simplify usage pattern for FixtureToDH
2024-05-16 14:09:26 +10:00
Scott
f1ff951199 engine/exchanges: Add GetCurrencyTradeURL wrapper func/gRPC endpoint (#1521)
* beginning the concept

* expands testing and implementations

* test standardisation, expansion

* more

* finish remainder, add rpc func

* lint

* grammar? I barely know her!

* wow i forgot something wow wow wow wow

* rm SendPayload

* enFixio!

* improve binance long-dated support

* update test design, update wrapper funcs

* adds kraken futures, adds bybit support

* fixes SIMPLE bugs

* s is for sucks

* fixed option url x2

* fixed silly test
2024-05-03 17:01:17 +10:00
Ryan O'Hara-Reid
9657a570dd exchanges: shift GetDefaultConfig wrapper function to exchange.go (#1472)
* Shift wrapper function GetDefaultConfig to exchange.Base method definition, to ensure set defaults doesn't get called twice and to reduce code

* rm alphapoint bootstrap method as is defined as exchange.Base method

* add tests

* glorious: make it a function and make it IBOTEXCHANGE

---------

Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
2024-04-12 16:15:43 +10:00
Gareth Kirwan
d7818ea956 Exchanges: Remove bespoke pair upgrade handling and abstract Start/Run (#1424)
* Exchanges: Remove Pair upgrade handling

Now redundant behind #1401. These paths should never be met.

Several legacy coin upgrade paths being deprecated as well: ZUSD and CNY
Expecting any users with bad config from 3+ years ago would have to
reset anyway.

Also: At the time the intention of this was to upgrade the config
format.
However now, instead, it'd mostly serve to reset enabled pairs if
there's a config mistake, which doesn't feel right.

* Kraken: Fix typo in Kraken type struct

* Exchanges: Abstract exchange Start() and Run()

* Exchanges: Add test for abstracted Start

* Exchanges: Move Start to Bootstrap

* Simplify waitgroup usage
* Add call to exchange.Bootstrap to allow overide or supplementation

* Exchanges: Concurrent common bootstap actions

* Gateio: Remove incorrect Run in test

* GateIO: Fix pair dependencies in tests

This ensures that the pairs are initialised no more than needed and
kind-of just-in-time.
Better pattern might be to use a function to get these pairs when we
need them.

* Exchanges: Complete UpdatePairs before ExecLims

If we're going to update pairs, it needs to complete before we check for
limits to avoid errors on old pairs

* Exchanges: Remove Start and Run from tmpl

Since they're replaced by bootstrap now and shouldn't need customisation
normally

* Alphapoint: Move Start to Bootstrap

* GateIO: Fix linter shadow var
2024-01-31 19:29:36 +11:00
Gareth Kirwan
e007f69f7c exchanges/websocket: Implement subscription configuration (#1394)
* Websockets: Move Subscription to its own package

This allows the small type to be imported from both `config` and from
`stream` without an import cycle, so we don't have to repeat ourselves

* Subs: Renamed Currency to Pair

This was being mis-used through much of the code, and since we're
already touching everything, we might as well fix it

* Websockets: Add Subscription configuration

* Binance: Add subscription configuration

* Kucoin: Subscription configuration

* Simplify GenerateDefaultSubs
* Improve TestGenSubs coverage
* Test Candle Sub generation
* Support Candle intervals
* Full responsibility for formatting Channel name on GenerateDefaultSubs
  OR consumer of Subscribe
* Simplify generatePayloads as a result
* Fix test coverage of asset types in processMarketSnapshot

* Exchanges: Abstract ParallelChanOp

* Tests: Generic ws mock instances

* Kucoin: Fix intermittent conflict in test currs

Use isolated test instance for `TestGetOpenInterest`.

`TestGetOpenInterest` would occassionally change pairs before
GenerateDefault Subs.
2024-01-24 15:54:07 +11:00
Scott
b71bf1f3d1 exchanges/futures: Implement open interest (#1417)
* adds open interest to exchanges

* ADDS TESTING YEAH

* New endpoints, BTSE, RPCS, cached

* slight design change, begin gateio

You will need to get cached for
each exchange that supports it

* gateio, huobi, rpc

* fix up kraken, cache retrieval

* okx, gateio

* finalising all implementations and tests

* definitely my final ever commit on this

* Well, well, well

* final v2

* quick fix of bug

* test coverage, assert notempty, test helper

Added a new testhelper for currency
management because its very annoying
in a parallel test setting which wastes
so much space otherwise

* minimises REST requests for Open Interest

* types.Number merge misses

* Minimises Kraken REST calls

* len change, value -> pointer receiver

* further fixup

* fixes gateio, batch calculates open interest

* single gateio, lint const fixes

* rejig and more thorough oi for huobi

* formatting expansion

* minor fix for handling expiring contracts

* rm unused Binance strings

* add bybit support, fix bybit issues

* oopsie doopsie, dont look at my whoopsie

* Fix issue, remove feature

* move an irrelevant function for the pr

* mini bybit upgrades

* fixes cli request bug
2024-01-12 15:27:35 +11:00
Ryan O'Hara-Reid
a13a6c3248 Kucoin: Add order execution limits (#1369)
* cmd/exchange_template: Add wrapper function

* exchanges: force UpdateOrderExecutionLimits to wrappers as this is important for order deployment

* kucoin: Add spot order execution limits

* linter: fix

* glorious: nits

* kucoin: change from name to symbol for correct fee fetching and order deployment

* WHAAAAAAAT says Vitalik

* kucoin: Add futures limit support, fix insertion of non margin tradable pairs, update naming and add comments.

* kucoin: implement master branch changes

* Update exchanges/kucoin/kucoin_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* kucoin/test: update commentary

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2023-11-14 11:34:38 +11:00
Scott
70690d9a04 futures: Implement GetLatestFundingRates across exchanges (#1339)
* adds funding rate implementations and improvements

* merge fixes x1

* lint

* kucoin funding rates func make

* migrate sync-manager to keys

* some kucoin work

* adds some kucoin wrapper funcs

* ehhh, todo

* kucoin position

* start of orders

* adds the kucoin tests yay

* multiplier

* nits, EWS includes order limits

* NotYetImplemented, IsPerp improvements, cleaning

* lint, test fix, huobi time

* fixes issues, improves testing

* fixes linters I WRECKED

* local lint but remote lint, lint, lint, lint

* fixes err

* skip CI

* lint

* Supported rates, binance endpoints

* fixes weird mocktest problems

* no, CZ is invalid

* fixes some new EWS test errors
2023-11-03 11:01:32 +11:00
Ryan O'Hara-Reid
ceef7a14e0 currency: Adds matching lookup table built from available pairs (#1312)
* currency: Add pair matching update (cherry-pick)

* exchange/currency: Add tests and update func

* linter fix, also if using json unmarshal functionality stop usage of string conversion without delimiter

* gemini: fix test

* currency/manager: potential optimisation

* exchanges: purge derive from wrapper cases and add warning comment

* glorious: nits

* glorious: nits

* linter: fix

* glorious: nits

* whoops

* whoops

* glorious: nits continued

* glorious: diff THANKS!

* hitbtc: fix update tradable pairs strings splitting. continue if not enabled tickers update pair.

* glorious: nits

* linter: fix

* Update exchanges/exmo/exmo_wrapper.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* bitstamp: fix test when 32 biterinos architecturinos

* capture more strings for speed

* swapsies because whos running 32bit \0/?

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2023-10-18 11:57:27 +11:00
Ryan O'Hara-Reid
233a65a778 exchange: refactor credentials.go from PR #1320 (#1360)
* refactor credentials

* glorious: nits

* shazbert: nits

---------

Co-authored-by: romanornr <romanornr@gmail.com>
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-10-10 10:17:07 +11:00
Scott
91d699be9d maps: expansion of Key concept (#1349)
* moves everything to use single map keys, also breaks

* full rollout

* tests

* fix a little bug

* minor test fixups

* Fix Key use

* rm 🔑 from 🔑 struct name
2023-10-04 10:19:41 +11:00
Scott
7f0faf7850 futures: add GetFuturesContractDetails wrapper function (#1274)
* all in a days work

* cleanup

* cleanup for real, also stop it binance.json

* minor coverage

* adds gateio to the slurry

* cleanup of types

* verbose verbose verbose verbose verbose verbose

* fixes huobi parsing issue

* fix bybit contract identification

* cleanup

* merge fixes

* addresses many big problems raised by SHAZ

* tracking errors and fixes

* funding rate if avail, fixes currency formatting

* Addresses nits and sneaks in extra fixes

* lint

* minor fixes after rebase

* better contract splitter for currencies like T-USDT

* forgot to add the exchange name like a fool

* merge fixes x1

* kucoin, direction, contract size

* rn direction, fix kucoin time

* WHOOPS

* Update exchanges/kucoin/kucoin_wrapper.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* misdirection

---------

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-10-03 16:04:45 +11:00
Scott
5f2f6f884b Binance,Okx: Add Leverage, MarginType, Positions and CollateralMode support (#1220)
* init

* surprise train commit

* basic distinctions

* the terms of binance are confusing

* renames and introduction of allocatedMargin

* add new margin funcs

* pulling out wires

* implement proper getposition stuff

* bad coding day

* investigate order manager next

* a broken mess, but a progressing one

* finally completes some usdtmargined stuff

* coinMfutures eludes me

* expand to okx

* imports fix

* completes okx wrapper implementations

* cleans and polishes before rpc implementations

* rpc setup, order manager features, exch features

* more rpc, collateral and margin things

* mini test

* looking at rpc response, expansion of features

* reorganising before the storm

* changing how futures requests work

* cleanup and tests of cli usage

* remove silly client side logic

* cleanup

* collateral package, typo fix, margin err, rpc derive

* uses convert.StringToFloat ONLY ON STRUCTS FROM THIS PR

* fix binance order history bug

* niteroos

* adds new funcs to exchange standards testing

* more post merge fixes

* fix binance

* replace simepletimeformat

* fix for merge

* merge fixes

* micro fixes

* order side now required for leverage

* fix up the rest

* global -> portfolio collateral

* Update exchanges/collateral/collateral_test.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* adds fields and todos

* rm field redundancy

* lint fix oopsie daisy

* fixes panic, expands error and cli explanations (sorry shaz)

* ensures casing is appropriate for underlying

* Adds a shiny TODO

---------

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-09-26 16:16:31 +10:00
Ryan O'Hara-Reid
1adbc99526 exchanges: improves GetDefaultConfig method (#1245)
* exchanges: Add function to get standard config

* exchanges: add tests (cherry-pick here and above)

* after pick stuff

* cleanup

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-09-18 11:45:02 +10:00
Scott
471f4f21c4 Binance,OKx: Implement fetching funding rates (#1239)
* adds basic groundwork for rates on binance

* more into rates on binance

* rm redudant redundancy, add payments

* mini commit before merging with testnet ability branch

* changes function signature and fixes resulting build

* gets billing data too

* funding rates package, features use, testnet reimpl

* new endpoint, refinements and tests

* cli fix, rpc impl, testing, payments

* fixups from looking at code

* typo fix

* niteroos

* merge fixes

* adds test, fixes cli issues

* woah nelly
2023-07-26 14:25:43 +10:00
Scott
fcc5ad4551 exchanges/qa: Add exchange wrapper testing suite (#1159)
* initial concept of a nice validation tester for exchanges

* adds some datahandler design

* expand testing

* more tests and fixes

* minor end of day fix for bithumb

* fixes implementation issues

* more test coverage and improvements, but not sure if i should continue

* fix more wrapper implementations

* adds error type, more fixes

* changes signature, fixes implementations

* fixes more wrapper implementations

* one more bit

* more cleanup

* WOW things work?

* lintle 1/1337

* mini bump

* fixes all linting

* neaten

* GetOrderInfo+ asset pair fixes+improvements

* adds new websocket test

* expand ws testing

* fix bug, expand tests, improve implementation

* code coverage of a lot of new codes

* fixes everything

* reverts accidental changes

* minor fixes from reviewing code

* removes Bitfinex cancelBatchOrder implementation

* fixes dumb baby typo for babies

* mini nit fixes

* so many nits to address

* addresses all the nits

* Titlecase

* switcheroo

* removes websocket testing for now

* fix appveyor, minor test fix

* fixes typo, re-kindles killed kode

* skip binance wrapper tests when running CI

* expired context, huobi okx fixes

* kodespull

* fix ordering

* time fix because why not

* fix exmo, others

* hopefully this fixes all of my life's problems

* last thing today

* huobi, more like hypotrophy

* golangci-lint, more like mypooroldknee-splint

* fix huobi times by removing them

* should fix okx currency issues

* blocks the application

* adds last little contingency for pairs

* addresses most nits and new problems

* lovely fixed before seeing why okx sucks

* fixes issues with okx websocket

* the classic receieieivaier

* lintle

* adds test and fixes existing tests

* expands error handling messages during setup

* fixes dumb okx bugs introduced

* quick fix for lint and exmo

* fixes nixes

* fix exmo deposit issue

* lint

* fixes issue with extra asset runs missing

* fix surprise race

* all the lint and merge fixes

* fixes surprise bugs in OKx

* fixes issues with times and chains

* fixing all the merge stuff

* merge fix

* rm logs and a panic potential

* lovely lint lament

* an easy demonstration of scenario, but not of initial purpose

* put it in the bin

* Revert "put it in the bin"

This reverts commit 15c6490f713233d43f10957367fcbf18e3818bdd.

* re-add after immediate error popup

* fix mini poor test design

* okx okay

* merge fixes

* fixes issues discovered in lovely test

* I FORGOT TO COMMIT THIS

* nit fixaroonaboo

* forgoetten test fix

* revert old okx asset intrument work

* fixes

* revert problems I didnt understand. update bybit

* fix merge bugs

* test cleanup

* further improvements

* reshuffle and lint

* rm redundant CI_TEST by rm the CI_TEST field that is redundant

* path fix

* move to its own section, dont run on 32 bit + appveyor

* lint

* fix lbank

* address nits

* let it rip

* fix failing test time range

* niteroo boogaloo

* mod tidy, use common.SimpleTimeFormat
2023-07-03 11:09:43 +10:00
Samuael A
969669cb0d linters: Bump golangci-lint version to v1.52.2 and fix issues (#1169)
* fix linter issues

* linter issue suppressing and fixing

* change linter version and linter issues fix

* Bump version
2023-04-27 10:13:46 +10:00
Ryan O'Hara-Reid
42475bf2b8 exchanges: add setTimeWindow boolean to GetKlineRequest param (#1160)
* exchanges: add setTimeWindow boolean to GetKlineRequest params to differentiate between a set time period return from endpoint.

* glorious: nits

* exchange: conjugation

* Update exchanges/exchange.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits and an assortment of differences

* exchanges: remove some comments

* glorious: nits

* cleanup

* tests: fix

* Update exchanges/hitbtc/hitbtc_wrapper.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* Update exchanges/kline/kline.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* Update exchanges/kline/kline_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* kline: fix test

* rm unused variables

* almost: nits

* glorious: nits

* linter: fix

* rm unused variable

* Refactored comment in the okex tests to ensure that it accurately reflects the variable name and the issue related to the time window, as requested by GloriousCode. The previous comment did not align with the identifier assigned to the property, which could cause confusion and misunderstanding among other programmers or stakeholders. The updated comment will improve the clarity and readability of the codebase and make it easier to understand the intended purpose of the associated variables. The change was made with the aim of improving the overall quality and maintainability of the code.

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2023-04-27 10:10:19 +10:00
Ryan O'Hara-Reid
d23898e63a engine: Adds shutdown method to exchange manager and unload all exchanges when engine is stopped (#1112)
* engine: shutdown and unload exchange when engine is stopped

* linter: fixes

* engine/exchMan: add nil check

* engine/exchanges: add shutdown method to exchanges, rm len check lock not needed, expanded code coverage, address some nits

* exchMan: report all failed shutdowns across exchanges, implement timer and monitoring routines.

* exchMan: improve shutdown sequence and aloc.

* further improvement

* exchman: log from warn to error

* websockconnection: Suppress error return when closure is caused by library

* linter: fix

* fix racies

* add note on why not parallel tests

* glorious: nits

* spelling kween

* thrasher: nits

* engine: change print of setting using reflection, I keep forgetting to implement this so program around forgetfulness

* engine/exchange_management: remove wait group and just rely on intermediary lock

* glorious: nits

* Update common/common.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update main.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-04-05 13:07:35 +10:00
Adrian Gallagher
7e08e483fb CI: Bump go version, linters and fix minor issues (#1130)
* CI: Bump go version, linters and fix minor issues

* Bump version, fix loop variables

* Revert

* Rid TODOs now that 1.51 has been released
2023-02-03 15:56:59 +11:00
Adrian Gallagher
a79e0d2b3e qa/spelling: Add Codespell support (#1121)
* Add codespell support

* Fix paths

* Add HTML files to exclusion list
2023-01-30 12:36:56 +11:00
Adrian Gallagher
c785ae73a7 exchanges: Remove FTX implementation and fix minor test issues (#1100)
* exchanges: Start removal of FTX

* Get tests happy again

* okx: improve logic and add basic coverage

* Fix linterino

* Round 2 plus rm useless assignment in test

* Fix exchange_wrapper_issues test error

* Fix nitters

* Address nitters
2023-01-25 16:40:04 +11:00
Scott
03a24b3ab1 Backtester: custom interval support (#1115)
* add backtester support

* Prevent live data custom candles, prevent nanosecond candles

* test coverage

* a more interesting rsi strategy result

* actual custom candle and proper strat date

* add test to old funk

* typos 🌞 🌞

* this was definitely worth failing linting for

* Adds stricter processing and adapts to it

* now compat with partial and absent candles

* test fixes, zb fixes

* fix more introduced bugeroos

* fix more introduced bugeroosx2

* linting for one space is so annoying

* addresseroos niteroos

* Update backtester/engine/setup.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-01-24 16:05:46 +11:00
Ryan O'Hara-Reid
83cfefa45c kline/exchanges: automatic creation of unsupported candle intervals (#1091)
* kline: Add builder and testing

* Ideas

* kline: deploy builder functionality across GCT

* exchanges: implement across gct

* exchanges: Add tests and fix implementations before kline package testing and veri.

* kline: Add tests and start to fix ConvertToNewInterval

* kline: fix ConvertToNewInterval add tests

* kline: complete overarching tests now on to exchanges

* kline: finish exchange tests and implement limits

* exchanges: more fixes

* linter: fix

* engine: fix tests

* kraken: fix recent trades and other fixes

* zb: fix tests

* bithumb: fix empty insertion

* kline: refactor/optimize CreateKline function

* kline: remove the mooos!

* kline: prealloc CalculateCandleDateRanges

* linter: fix

* exchanges: prealloc extended

* fix whoopsie

* reverse fix because this is a whoopsie

* okx: fix risidual issues

* linter: fix

* kline: initial nits from @gloriouscode

* kline: rename builder -> request and cascade change

* linter: fix + test

* kline: update forced alignment on start and end times when CreateKlineRequest is called.

* nits: more more more

* NITS: Addressed

* tests: fix race issue

* Update exchanges/kline/request.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* kline: add method AddPadding() to automatically fill in holes in kline.Request functionality and reject if missing data when converting

* kline: Add params start and end to addPadding() to insert blanks in between block

* kline: remove test comment code as it's not needed anymore

* kline: fix lint and test

* kline: sort slice without extra bool check every iteration

* okx: fix issues with timeing and candles and such from niterinos & address typo

* Update exchanges/kline/kline.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: niterinos

* Update exchanges/poloniex/poloniex_wrapper.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits now onto conflicts YAYA!!!

* Update exchanges/exchange_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits again

* thrasher: nitters

* thrasher: niterinos - adds partial flag for incomplete recent candles and fetching.

* kline: rm fmtizzle packageizzle

* glorious: nitters

* glorious: more niterinos

* fix last niterinos

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2023-01-17 16:22:33 +11:00
Scott
017cdf1384 Backtester: Live trading upgrades (#1023)
* Modifications for a smoother live run

* Fixes data appending

* Successfully allows multi-currency live trading. Adds multiple currencies to live DCA strategy

* Attempting to get cash and carry working

* Poor attempts at sorting out data and appending it properly with USD in mind

* =designs new live data handler

* Updates cash and carry strat to work

* adds test coverage. begins closeallpositions function

* Updates cash and carry to work live

* New kline.Event type. Cancels orders on close. Rn types

* =Fixes USD funding issue

* =fixes tests

* fixes tests AGAIN

* adds coverage to close all orders

* crummy tests, should override

* more tests

* more tests

* more coverage

* removes scourge of currency.Pair maps. More tests

* missed currency stuff

* Fixes USD data issue & collateral issue. Needs to close ALL orders

* Now triggers updates on the very first data entry

* All my problems are solved now????

* fixes tests, extends coverage

* there is some really funky candle stuff going on

* my brain is melting

* better shutdown management, fixes freezing bug

* fixes data duplication issues, adds retries to requests

* reduces logging, adds verbose options

* expands coverage over all new functionality

* fixes fun bug from curr == curr to curr.Equal(curr)

* fixes setup issues and tests

* starts adding external wallet amounts for funding

* more setup for assets

* setup live fund calcs and placing orders

* successfully performs automated cash and carry

* merge fixes

* funding properly set at all times

* fixes some bugs, need to address currencystatistics still

* adds 'appeneded' trait, attempts to fix some stats

* fixes stat bugs, adds cool new fetchfees feature

* fixes terrible processing bugs

* tightens realorder stats, sadly loses some live stats

* this actually sets everything correctly for bothcd ..cd ..cd ..cd ..cd ..!

* fix tests

* coverage

* beautiful new test coverage

* docs

* adds new fee getter delayer

* commits from the correct directory

* Lint

* adds verbose to fund manager

* Fix bug in t2b2 strat. Update dca live config. Docs

* go mod tidy

* update buf

* buf + test improvement

* Post merge fixes

* fixes surprise offset bug

* fix sizing restrictions for cash and carry

* fix server lints

* merge fixes

* test fixesss

* lintle fixles

* slowloris

* rn run to task, bug fixes, close all on close

* rpc lint and fixes

* bugfix: order manager not processing orders properly

* somewhat addresses nits

* absolutely broken end of day commit

* absolutely massive knockon effects from nits

* massive knockon effects continue

* fixes things

* address remaining nits

* jk now fixes things

* addresses the easier nits

* more nit fixers

* more niterinos addressederinos

* refactors holdings and does some nits

* so buf

* addresses some nits, fixes holdings bugs

* cleanup

* attempts to fix alert chans to prevent many chans waiting?

* terrible code, will revert

* to be reviewed in detail tomorrow

* Fixes up channel system

* smashes those nits

* fixes extra candles, fixes collateral bug, tests

* fixes data races, introduces reflection

* more checks n tests

* Fixes cash and carry issues. Fixes more cool bugs

* fixes ~typer~ typo

* replace spot strats from ftx to binance

* fixes all the tests I just destroyed

* removes example path, rm verbose

* 1) what 2) removes FTX references from the Backtester

* renamed, non-working strategies

* Removes FTX references almost as fast as sbf removes funds

* regen docs, add contrib names,sort contrib names

* fixes merge renamings

* Addresses nits. Fixes setting API credentials. Fixes Binance limit retrieval

* Fixes live order bugs with real orders and without

* Apply suggestions from code review

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update backtester/engine/live.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update backtester/engine/live.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* Update backtester/config/strategyconfigbuilder/main.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* updates docs

* even better docs

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-01-05 13:03:17 +11:00
E Sequeira
d7d4933f93 exchange: add SetRequester to exchange Base (#1095)
* add setRequester to exchange Base

* add tests

* fix lint

* Update exchange_test.go
2022-12-07 16:07:04 +11:00
Ryan O'Hara-Reid
f843b7d277 exchange: upgrade UpdatePair method (#991)
* exchange: upgrade update pair

* exchanges: Add enabled string matching and format handling if discrepency is found.

* linter: fixes

* bithumb: fix tests

* BTSE: api change fix ordering

* huobi: fix tests

* gloriousnits: stage 1

* gloriousnits: stage 2

* currency: more nits

* bitmex: add spot and process pairs before currency package call.

* bitmex: finished correct orderbook matching and other implementations

* linter: fix issue

* currency: Fix linter

* currency: segregate and protect pair store, update tests

* currency/manager: clean code, rm log output

* currency: Add store method and make sure formatting stays nil if not stored.

* gct: check errors

* engine/websocketroutineman: fix tests

* bybit: fix duplication bug

* huobi: fix test

* btse: fix tests?

* ob/buffer: fix tests

* Update currency/manager.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* glorious: nits strikes again.

* exchange: add bypassConfigFormatUpgrades to stop formatting

* GLORIOUS LINTER

* Update exchanges/bithumb/bithumb_wrapper.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* exchange: fix pair upgrade issue when duplications are in both avail and enabled pairs

* linter: fix shadow dec

* config: fix test

* Update currency/pair_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2022-09-16 08:59:27 +10:00
Scott
46cadd6f15 FTX: Funding rates, payments & stats + order manager tracking (#976)
* Adds basic PoC for calculating/retrieving position data

* A very unfortunate day of miscalculations

* Adds position summary and funding rate details to RPC

* Offline funding rate calculations

* More helpers, more stats, refining data, automated retrieval

* Adds new rpc server commands and attempts some organisation

* lower string, lower stress

* Adds ordermanager config. Fleshes outcli. Tracks positions automatically

* Adds new separation for funding payments/rates

* Combines funding rates and payments

* Fun test coverage

* ALL THE TESTS... I hope

* Fixes

* polishes ftx tests. improves perp check. Loops rates

* Final touches before nit attax

* buff 💪

* Stops NotYetImplemented spam with one simple trick!

* Some lovely little niteroos

* linteroo

* Clarifies a couple of errors to help narrow likely end user problems

* Fixes asset type bug, fixes closed position order return, fixes unset status bug

* Fixes order manager handling when no rates are available yet

* Continues on no funding rates instead. Removes err

* Don't show predicted rate if the time is zero

* Addresses scenario with no funding rate payments

* Bug fixes and commentary before updating maps to use *currency.Item

* Adds a pair key type

* Polishes pKey, fixes map order bug

* key is not a property in the event someone changes the base/quote

* Adds improvements to order processing...Breaks it all

* Shakes up the design of things by removing a function

* Fixes issues with order manager positions. Limits update range

* Fixes build issues. Identification of bad tests.

* Merges and fixes features from master and this branch

* buff linter 💪

* re-gen

* proto regen

* Addresses some nits. But not all of them.

* Fixes issue where funding rates weren't returned 🎉

* completes transition futures tracking to map[*currency.Item]map[*currency.Item]

* who did that? not me

* removes redundant check on account of being redundant and unnecessary

* so buf

* addresses nits: duplications, startTime, loops, go tidy, typos

* fixes minor mistakes

* fixes 🍣 🐻 changes to int64
2022-08-23 12:16:50 +10:00
Adrian Gallagher
68588560e3 CI: Bump go version, linters and fix minor issues (#1010)
* Bump golang, golangci-lint versions and fix issues

* Add -fno-stack-protector

* Fix AppVeyor golangci-lint ver

* Nitters

* Nitters round 2
2022-08-17 11:37:22 +10:00
Jaydeep Rajpurohit
247da918a8 exchanges: Add ByBit support (#887)
* few fixes and add ratelimiter

* adds test

* revert configtest.json changes

* configtest updated

* WIP: adds public endpoint support

* WIP: adds public endpoint support

* adds public endpoint support

* WIP: adds auth. endpoint support

* adds test for auth. endpoint

* fixes

* adds auth. endpoint support

* WIP: ws support

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* Testing

* Complete WS spot testing

* adds support for ws events

* minor change

* WIP: adds REST support for CoinMarginedFutures

* Fixes

* WIP: adds REST support for CoinMarginedFutures

* Fixes

* improvement in SPOT REST

* Typo fix

* WIP: add REST support for CMF Account API

* minor fixes

* WIP: add support for CMF conditional orders and few minor fixes

* complete support for CMF conditional orders

* adds support for public CMF endpoint

* adds support for CMF position API

* Complete REST CMF support

* WIP

* Testing REST CMF support

* Testing REST CMF support

* Testing REST CMF support completed

* WIP: add support for UMF

* completed non-auth UMF

* WIP: add support for REST Auth. UMF

* WIP: add support for REST Auth. UMF and some improvements

* WIP

* WIP

* WIP

* completed REST UMF

* renaming

* adds REST support for futures

* add testcases for UMF and some optimizations

* add testcases for futures

* Testing UMF, futures and its changes

* Fixes

* Fixes after testing

* WIP

* WIP

* WIP

* completed ws USDT futures support

* WIP: ws support for futures

* fixes in WS futures

* fixes in WS support

* roll back changes made for WS CMF, USDT and Futures

* fixes

* WIP

* WIP

* fixes

* Steps for new PR

* WIP

* WIP

* WIP

* WIP

* complete PR setup

* fixes for successfully running tests

* update in symbol for futures pair in test file

* WIP

* Fixes in test file and other minor fix

* fix testdata/configtest.json

* reset CONTRIBUTORS file

* review changes

* remove unwanted file

* remove redundant code

* improvisation

* adds comment for exported functions

* remove unwanted TODO and commented code

* fix

* improvisation

* fix

* defined errors

* improvisation

* improvisation

* improvisation

* updates test

* adds comment for exported types

* review changes

* review changes

* fix

* fixes

* Changes for making BYBIT compatible with existing code base

* Test file changes

* Changes for making BYBIT compatible with existing code base

* Changes for making BYBIT compatible with existing code base

* fix lint issues

* fix

* review changes

* review changes

* review changes

* review changes

* review changes

* review changes

* review changes

* review changes

* review changes

* review changes

* WIP

* add test cases for new API's

* minor improvements

* add missing API and their tests

* minor fixes

* add bybitTime

* add bybitTimeSec, bybitTimeMilliSec, bybitTimeNanoSec and necessary support

* fix GetTradeHistory function

* error handling

* test fixes

* add GetServerTime API

* adds GetHistoricCandlesExtended and review changes

* test fixes

* minor fix

* integrating CMF Bybit recent change log

* minor fixes

* adds extractCurrencyPair

* minor fixes

* minor fix

* review changes

* adds variable declaration of error

* review commit

* adds embeddable type in API response for all API and integrate it

* fixes

* adds authentication WS connection

* review changes

* review changes

* compatible changes

* adds asset to GetWithdrawalsHistory

* adds asset_type in rpc.proto

* adds asset argument in gctcli withdrawal request command

* improve error handling in exchange API error

* web socket fix

* review changes

* improvements

* improvements

* minor fix

* review changes

* fixing wrapper issues

* fixes

* fixes

* review changes

* add test cases

* fix for GetActiveOrders

* lint fixes

* fixes in websocket

* adds wrapper testcases

* adds wrapper testcases

* adds wrapper testcases

* fixes

* fix issue with GetHistoricCandlesExtended

* fix merge issues

* improving error reporting

* adds wrapper testcases and a minor fix

* gctrpc changes

* adds test cases
fixes in websocket

* review changes for ws

* review changes in WS

* fix gctrpc

* merge fixes

* review changes

* WIP

* updates pair in configs

* adds new asset USDCMarginedFutures

* adds URL const for USDCMarginedFutures

* adds API support

* minor fixes

* adds kline API

* minor fix

* adds API

* adds API

* adds API

* WIP

* WIP

* WIP

* adds support for USDC auth requests to SendAuthHTTPRequest

* adds SendUSDCAuthHTTPRequest

* run test and fix them

* rollback support added for Auth. USDC request inside SendAuthHTTPRequest

* adds API and test cases

* adds API and test cases

* adds APIs and test cases

* adds APIs

* adds rate limit for USDC

* adds USDCMarginedFutures to wrapper

* adds USDC testcases in wrapper and fix few issues

* minor test fixes

* minor test fixes

* fix lint issues

* WIP

* Merge changes

* minor fixes

* remove "else" and optimize

* review changes

* review changes

* review changes

* fix lint issue

* merge fix

* fix test

* fix templates and run them

* changes after merge

* review changes and improvements

* code improvement

* fixes with respect to changes in API response in documentation

* fixed review change in test

* adds check in CancelExistingOrder

* update exchange template

* review changes

* adds GetDepositAddress API

* WIP: adds GetOrderHistory

* complete GetOrderHistory

* fixes

* adds test case

* fixes and add WithdrawFund API

* WIP

* WIP

* updating all SendAuthHTTPRequest call

* adds WithdrawCryptocurrencyFunds

* update test cases

* fix lint issues

* fixes after merge

* adds GetAvailableTransferChains and few fixes

* minor fix in GetDepositAddress

* minor fix with WS ping/pong handling

* add ping handler for WS Auth.

* fix typo mistake

* update doc
2022-08-08 11:29:43 +10:00
Ryan O'Hara-Reid
663e753f52 account: segregate holdings by credentials for future multi-key management (#956)
* exchanges/account: shift credentials to account package and segregate funds to keys

* merge: fixes

* linter: fix

* Update exchanges/account/account.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits + protection for string panic

* glorious_suggestion: add method for matching keys

* linter: fix tests

* account: add protected method for credentials minimizing access, display full account details to rpc.

* linter: spelling kweeeeeeen

* accounts/portfolio: clean/check portfolio code and quickly check balances from change. Add protected method for future matching.

* accounts: theres no point in pointerising everything

* linter: ok pointerise this then...

* exchanges: fix regression add in little notes.

* glorious: nits

* Update exchanges/account/credentials.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* Update exchanges/account/credentials_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* Update exchanges/account/credentials_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

* gloriously: fix glorious glorious test gloriously

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2022-07-21 15:05:31 +10:00