Commit Graph

804 Commits

Author SHA1 Message Date
Gareth Kirwan
2b3c63c5b3 Exchanges: Convert bespoke Number types to types.Number (#1429)
* Kucoin: Rename WsSpotTicker => WsSnapshot

* Kucoin: Replace kucoinNumber => types.Number

* Okcoin: Convert to types.Number

* Gateio: Convert to types.Number

* Okx: Convert to types.Number
2023-12-28 14:55:23 +11:00
Gareth Kirwan
f05f24da8b Types: Switch convert.StringToFloat64 to types.Number (#1415)
* Types: Add Number type

* Types: Switch StringToFloat64 for Number

This change mostly just renames the type.
convert package and StringToFloat64 represent actions, not types,
and make it misleading to use outside of the API context,
especially when using it for a Float64ToString operation.

* Common: Remove StringToFloat64

Replaced by types.Number

* fixup! Types: Switch StringToFloat64 for Number

Second pass at Okx

* Spellcheck: Fix whitespace handling for okx line
2023-12-20 13:01:27 +11:00
Bea
5042bf9790 OKX: Change float64 to convert.StringtoFloat64 in okx_types 2023-12-20 09:55:20 +11:00
Gareth Kirwan
a9cdbd16f9 Orderbook: Fix test failures on arm64 (#1407)
* Orderbook: Fix test failures on different arch

Fixes floating point inaccuracies on different architectures
Moves the depth tests over to table driven tests

* Kline: Fix test failures on different arch
2023-12-19 15:13:46 +11:00
Gareth Kirwan
37b1121bbd BTSE: Fix duplicate pair errors on Million pairs (M_*) (#1401)
* BTSE: Fix duplicate error on Million pairs (M_*)

BTSE has listed Pitbull token with two symbols:
PIT-USD and M_PIT-USD for millons of PIT / USD.
The native token is not tradable, so we ignore them and
get a base of M_PIT because that's what later APIs will accept

* BTSE: Fix test errors on locked market

* Common: Improve AppendError and ExcludeError

This change switches from a stateful multiError to caring more about the
Unwrap() []error interface, the same as [go standard
lib](https://github.com/golang/go/blob/go1.21.4/src/errors/wrap.go#L54-L68)

Notably, if we implement Unwrap() []error and do NOT implement Is() then
we get free compatibility with the core functions.

The only distateful thing here is needing to deeply unwrap fmt.Errorf
errors, since they don't flatten. I can't see any way around that

* Pairs: Fix exchange config Pairs loading

When a pair string contained two punctuation runes, the first one is used,
and the configFormat is ignored.

This fix checks the list and corrects any with the wrong delimiter, or
errors if the format is inconsistent.

* BTSE: Fix all tickers retrieved by GetTicker

PR #764 introduced GetTickers, but it wasn't rolled out to BTSE.
This fix ensures that when one ticker is a locked market, the rest continue to
function. Particularly important if the locked market wasn't even
enabled anyway.

* Kucoin: Fix test config future pairs

* BTSE: Remove PIT tests; Token removed

BTSE have removed the PIT token pairs

All these changes stand, and this just removes the test

* ITBit: Fix fatal error on second run

This fix removes incorrect config pair delimiter, because it would be
re-inserted into config the first run, and then error the second time.

This delimiter doesn't match the config we have.
There's no implementation of fetching pairs, so what's in config files
now is all that matters

* Engine: Fix TestConfigAllJsonResponse

* Clarity of non-matching json improved
* Handling for fixing pair delimiters
2023-12-19 14:40:13 +11:00
Gareth Kirwan
8ce30a8375 Okx: Fix GetBlockTrades and enhance test coverage (#1416)
* Okx: Remove UpdateTradablePairs from TestMain

When running individual tests we do not want to be forced to update
trading pairs.
It also does not seem to be required, since disabling UpdateTradingPairs
entirely doesn't stop any tests from passing.

If it is required we can enact this pattern:
```
func TestUpdateTradablePairs(t *testing.T) {
    t.Parallel()
    updatePairsOnce(t)
}
var updatePairsGuard sync.Once
func updatePairsOnce(tb testing.TB) {
	tb.Helper()
	updatePairsGuard.Do(func() {
		err := ok.UpdateTradablePairs(context.Background(), true)
		assert.NoError(tb, err, "UpdateTradablePairs should not error")
	})
```

* Okx: Fix GetBlockTrades url

Endpoint was wrong

Not convinced we should prefix with market vs public, but not
committing a driveby

* Okx: Expand test coverage for GetBlockTrades

* Okx: Fix GetPublicBlockTrades

Was marked as an authenticated API call, and it's not.
Also renames it because without the context of "Block" (or RFQ) it's
misleading.
Adds tests.

* Okx: Fix GetPublicBlockTrades to be public

Expand coverage of GetBlockTrade and GetPublicBlockTrades
2023-12-14 15:53:39 +11:00
Gareth Kirwan
32f08f82f4 Kucoin: Fix unmarshalling of FuturesTicker resp (#1414)
* Kucoin: Fix unmarshalling of FuturesTicker resp

* Fix several string float64 fields not unmarshalling
* Fix time field misnamed; Make a time.Time
* Use order.Side for sides
* Add tests

This change opts to expose float64 instead of StringToFloat64 because
there's no wrapper to convert the type over, and it's not a websocket.
Essentially choosing to not expose internal convert types for methods
called directly.

Possible we'll eventually harmonise this with a types.Number or
something.

* Kucoin: FuturesRealTimeTicker => FuturesTicker

Kucoin only call it RealTime in the first sentence of their docs. None of the API names, endpoints or any thing mention RealTime.
This is just a ticker

* Tests: Improve clarity of failed push data

* Kucoin: Move PushData to a fixture file

This avoids both the linter errors, and the bloat.
Loses the name of the "test", but I think that's okay
2023-12-12 14:57:32 +11:00
Gareth Kirwan
0620ee034d Okx: Fix GetInsuranceFundInformation "" resp (#1405)
* Okx: Fix GetInsuranceFundInformation "" resp

When amount is "" this was erroring

* Test: Standardise configtest.json

Simple jq formatting of configtest
git diff -w or github ignore whitespaces should show no changes.
Any conflicts, just accept --ours and reformat it. Or not :o)
2023-11-28 09:38:37 +11:00
Ryan O'Hara-Reid
f23cc004fd Gateio: Implement GetHistoricalFundingRates wrapper func (#1385)
* adds funding rate implementations and improvements

* merge fixes x1

* lint

* kucoin funding rates func make

* migrate sync-manager to keys

* some kucoin work

* adds some kucoin wrapper funcs

* ehhh, todo

* kucoin position

* start of orders

* adds the kucoin tests yay

* multiplier

* nits, EWS includes order limits

* NotYetImplemented, IsPerp improvements, cleaning

* lint, test fix, huobi time

* fixes issues, improves testing

* fixes linters I WRECKED

* local lint but remote lint, lint, lint, lint

* fixes err

* skip CI

* lint

* Supported rates, binance endpoints

* fixes weird mocktest problems

* no, CZ is invalid

* fixes some new EWS test errors

* gateio: Add func GetHistoricalFundingRates

* glorious: nits

* rm: print call

* glorious: nits fix test

---------

Co-authored-by: Scott <scott.grant@thrasher.io>
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-11-22 10:52:26 +11:00
Adrian Gallagher
0fd433e865 exchanges/bittrex,kucoin: Remove exchange implementation and fix minor test issue (#1403)
* exchanges/Bittrex: Remove exchange implementation

* Kucoin: Fix TestProcessMarketSnapshot after pair removal update

* Kucoin: Fix race due to duplicate setupWS call

Unleash your inner Max Verstappen

* Kucoin: Actually test spot/margin market snapshot replication
2023-11-22 10:20:09 +11:00
Ryan O'Hara-Reid
929957cbd6 kucoin: Fix fetching fee rates for multiple pairs (#1370)
* kucoin: Fix fetching fee rates for multiple pairs

* Update exchanges/kucoin/kucoin_test.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* NITS: WOW

* glorious: nits

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-11-21 12:45:08 +11:00
Bea
06024fcb56 Kucoin: Fix ProcessMarketSnapshot and add a test (#1392)
* Kucoin: Fix ProcessMarketSnapshot and add a test

* Kucoin: ProcessMarketSnapshot: move the check before the ticker

* Kucoin: remove time.sleep from the test, add if statement to processmarketSnapshot

* Kucoin: ProcessMarketSnapshot to send margin, spot and both margin and spot pair data

* Kucoin: range over listOfAssetsCurrencyPairEnabledFor which returns a slice

* Kucoin: linter fix and pointers placed

* Kucoin: removed AssetWebsocketSupport and ku.CurrencyPairs.IsAssetEnabled, linter error fixed, comment amended
2023-11-21 11:28:38 +11:00
Ryan O'Hara-Reid
f65a582cd6 kucoin: fix spot order unmarshal bug (#1380)
* kucoin: fix unmarshal bug

* kucoin: Add time in force handling

* kucoin: fix test

* thrasher: nits

* kucoin_test: shift skip check down for coverage, rm market testing, change buy price to reduce chance of instant match

* kucoin: fix nits and force usage of uuid to rm conflicts

---------

Co-authored-by: shazbert <ryan.oharareid@thrasher.io>
2023-11-14 15:07:35 +11:00
Ryan O'Hara-Reid
a13a6c3248 Kucoin: Add order execution limits (#1369)
* cmd/exchange_template: Add wrapper function

* exchanges: force UpdateOrderExecutionLimits to wrappers as this is important for order deployment

* kucoin: Add spot order execution limits

* linter: fix

* glorious: nits

* kucoin: change from name to symbol for correct fee fetching and order deployment

* WHAAAAAAAT says Vitalik

* kucoin: Add futures limit support, fix insertion of non margin tradable pairs, update naming and add comments.

* kucoin: implement master branch changes

* Update exchanges/kucoin/kucoin_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* kucoin/test: update commentary

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2023-11-14 11:34:38 +11:00
Bea
340558b8ea Bitfinex: replace errTypeAssert with common.GetTypeAssertError (#1386)
* Bitfinex: Replace errTypeAssert with common.GetTypeAssertError

* Bitfinex/common.GetTypeAssertError description to match the documentation

* Bitfinex/Correct typos, fix type in finalResp error

* Bitfinex: Comments amended from.Symbol to .Data

* Bitfinex: fix the typo

* Bitfinex: fix the comments
2023-11-14 11:25:10 +11:00
Scott
70690d9a04 futures: Implement GetLatestFundingRates across exchanges (#1339)
* adds funding rate implementations and improvements

* merge fixes x1

* lint

* kucoin funding rates func make

* migrate sync-manager to keys

* some kucoin work

* adds some kucoin wrapper funcs

* ehhh, todo

* kucoin position

* start of orders

* adds the kucoin tests yay

* multiplier

* nits, EWS includes order limits

* NotYetImplemented, IsPerp improvements, cleaning

* lint, test fix, huobi time

* fixes issues, improves testing

* fixes linters I WRECKED

* local lint but remote lint, lint, lint, lint

* fixes err

* skip CI

* lint

* Supported rates, binance endpoints

* fixes weird mocktest problems

* no, CZ is invalid

* fixes some new EWS test errors
2023-11-03 11:01:32 +11:00
Gareth Kirwan
f9437dbd08 Bitfinex: Websocket subscription improvements (#1353)
* Websockets: Add keys to websocket subscriptions

* This switches all RO uses of the mutex to use a RLock method.
* The mutex used for discrete field access has had scope drift from
  name 'connectionMutex' so rename to more appropriate fieldsMutex
* The mutex used for Set/CanUseAuthEndpoints moves from the
  subscriptions endpoint to the fieldsMutex
* Add GetSubscription by key
* Expose stream.Matcher type

* Bitfinex: Subscribe and Unsubscribe atomicly

* Fix Auth failures ignored
* This change makes it so that Subscribe and Unsubscribe wait for success
** Tells the DataHandler about errors
** Errors are returned to consumers
* Subscribes concurrently to the channels
* It also simplifies the chanId to stream mapping
* Removes unable to locate chanID: %d errors which are just noise
* Paves the way for unified channelSubscription id handling
* Adds support for subId for Book subscriptions, which is more robust

* Vastly simplifies what we need to test TestWsSubscribedResponse
This test was working to ensure that the various fancy key parsing
mechanisms all worked. Now that we use subId, we just need a thorough
test of that
* Expose Match.Set in order to capture websocket incoming data
Can't see another way of doing this. Doesn't seem too bad

* Allow tests to run with auth or WS
These flags made it difficult to run the tests whilst working on
websockets

* Enable API auth and WS in testconfig
This change minimises the changes requires for a full test run against
live endpoints, so that new contributors have a clearer testing path.
I cannot see any reason to turn WS off and Auth endpoints off when we're
not going to run API tests without Creds being set, and we're not going
to do live fire tests without canManipulateRealOrders

* TestWsSubscribe and various fixes
** Enables the websocket for live non-authed integration tests by default
** Adds an integration test for subscriptions
** Changes the Ws tests to respect canManipulateRealOrders
** Uses WsConnect instead of setupWs; fixes seqNo config not sent for WS tests
** Allows api creds to live in config/testdata.json which might be
  less likely to accidentally commit, and less obtrusive

* Bitfinex: Support period and timeframe for Candles

* Fixes manual Subscribe() symbol or key formatting
* Unifies handling of params for DefaultSubscriptions and manual
  subsrciptions

* Bitfinex: Handle conf and info WS channel events

* Bitfinex: Better tests for subscriptions

* fixup! Websockets: Add keys to websocket subscriptions

* fixup! Bitfinex: Subscribe and Unsubscribe atomicly

* fixup! Websockets: Add keys to websocket subscriptions

* Websockets: Add Pending subscription status

Add a status tracker so that Sub/Unsub can prevent duplicates,
and also fixes when first message comes before we have added the sub
to the tracker

* Websockets: Add State instead of pending

This change allows more clarity about the current state and
checks for specifically already Unsubing

* Bitfinex: Fix first sub message maybe lost

The only link we have between a sub req and the sub resp is the subID.
And the only link we have between a sub message and the sub is the chanID.
We can't derive a link using Pair or anything else.

This meant that by sending the resp and its chanID down the IncomingData
channel, we allowed the channel reader to maybe process the next
message, the first message on the channel, before the runtime executed
the switch back to subscribeToChan waiting on the chan.

To fix this, we key initially on subId.(string), and then replace it
with chanId.(int64) when we have it *inside* the wsHandleData so we
know we've procedurally handled it before the next message.

subscribeToChan is then free to remove the subId keyed Sub regardless of
error or not

If there's an error, we don't need to inline handling because there
won't be any second update.

Expands test coverage to make sure those subId keyed subscriptions are
removed.

* Websocket: Validate state in SetChanState

* fixup! Bitfinex: Fix first sub message maybe lost

* Websockets: Rename RemoveUnsuccessfulSubs

Implementation doesn't imply Unsuccessful or need to.
This change supports the registering of Pending subs

* Bitfinex: Fix race in Tests
2023-11-02 12:10:43 +11:00
Scott
2ea3083468 exchanges/kraken,bittrex,gemini: Resolve Kraken panic, lint corrections, Bittrex batch tickers, set Gemini order limits and update tradable pairs (#1372)
* fix kraken, batch bittrex, fix lint

* surprise gemini!

* thought this happened automatically

* fix before shazbert sees

* fixes annoying atoi bug

* rm futures from gemini

* lint

* bittrex UpdatedAt, gemini Limits, stats relook

* STATS used HARDEN!(improve stats package)

* Whoopsies in your Daisies

* rm RWMutex, json stringeroo

* fixes additional index issues 😆 😭
2023-10-23 18:06:25 +11:00
Ryan O'Hara-Reid
ceef7a14e0 currency: Adds matching lookup table built from available pairs (#1312)
* currency: Add pair matching update (cherry-pick)

* exchange/currency: Add tests and update func

* linter fix, also if using json unmarshal functionality stop usage of string conversion without delimiter

* gemini: fix test

* currency/manager: potential optimisation

* exchanges: purge derive from wrapper cases and add warning comment

* glorious: nits

* glorious: nits

* linter: fix

* glorious: nits

* whoops

* whoops

* glorious: nits continued

* glorious: diff THANKS!

* hitbtc: fix update tradable pairs strings splitting. continue if not enabled tickers update pair.

* glorious: nits

* linter: fix

* Update exchanges/exmo/exmo_wrapper.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* bitstamp: fix test when 32 biterinos architecturinos

* capture more strings for speed

* swapsies because whos running 32bit \0/?

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2023-10-18 11:57:27 +11:00
Ryan O'Hara-Reid
51e2e42a19 orderbook: Add optional orderbook.Item string fields for potential checksum calculations (#1354)
* orderbook: Add optional orderbook.Item string fields for potential checksum calculations.

* glorious: nits

* glorious: nits

* thrasher: nits

* glorious: nits

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-10-13 16:38:49 +11:00
Ryan O'Hara-Reid
773441d5a7 Okx: Integrate websocket book5 processing and add configurable max subscriptions per connection (#1275)
* okx: books 5 (cherry-pick)

* okx: shift types to types file, remove commented code and updated field name to better reflect pushed type

* linter: fix

* remove slowness

* * Introduce function checksubscriptions and shift check of subscriptions to internal websocket package
* Shift Max websocket connection int to Websocket setup (temp) for this use case only.

* glorious: nits

* linter: fix

* websocket: don't try and subscribed with nothing to subscribe to.

* Update exchanges/stream/websocket_test.go

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* glorious: nits

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2023-10-13 15:54:49 +11:00
Bea
859c4512fb Bitstamp: Add new trading fees API endpoint; refine fee calcuations and test coverage (#1289)
* Bitstamp: WIP fixing trading fees

* Bitstamp/amended the TestGetFee test and currected the getTradingFee function

* Bitstamp: TestGetFee implemented for maker and taker

* Bitfinex: added a wrapped error

* Bitstamp: GetTradingFee and test updated, fetched from the API

Progresses #1271

* Bitstamp: minor changes- whitespace removal and comment added

* Bitstamp: fixed lint issues in TestGetBalance and TestGetTransactions

* Bitstamp:Typo in types TradingFee comment

* Bitstamp: GetAccountTradingFees at view the fees on all pairs

	modified:   exchanges/bitstamp/bitstamp.go

* Bitstamp: returning the TradingFee info instead of just MakerTakerFees

* Bitstamp:TestGetAccountTradingFees + TestGetAccountTradingFees added,data structure amended to match the outcome

* Bitstamp:error and a test for empty pair added in GetAccountTradingFee

* Bitstamp: RM the whitespace linter error
2023-10-13 15:34:21 +11:00
Gareth Kirwan
8430509807 Okx: Websocket order channel fixes (#1346)
* Okx: Fix WS order fields

* Fixes float64 with string annotation erroring on empty strings:
Okx Order Push Data error json: invalid use of ,string struct tag, trying to unmarshal "" into float64
Specifically this came from px field from a market order
* Switch to convert.StringToFloat64 instead of okxNumericalValue
* Fix typo in Notional* field names; Ironically prevented them from erroring

* Okx: Add tests for first order fields

* Okx: CID and maybe set WS order Filled time

* Tests: Set TestFixtureToDataHandler to t.Helper

* Orders: Add UnmarshalJSON to order.Side

* Okx: Fix FillTime not parsed for PendingOrder

* Okx: Switch to order.Side Unmarshal throughout

* Okx: Add Fee and FeeAsset to order processing

* Okx: Fix WS order.Detail amounts and Test

This fixes Amount vs QuoteAmount for market sells where tgtCcy is
quote_ccy

* Add comment to order.Side.UnmarshalJSON

* Okx: Replace PendingOrderItem Unmarshal with local types

* Okx: string type for WS order reduceOnly

Note: Not yet in unit tests, since it's not part of the spot tests I was
originally fixing. I'll circle back to adding full test support for
Reduce only and deleveraging positions.

* Okx: Fix TestOrderPushData Amount

We were expecting 0 when we're given a quoteAmount
In reality, we'll calculate the size from the price

* Okx: Fix order and remAmount in wsOrders

Improved handling for Float64 issues and boundaries when the order is
fully executed but not yet marked as Filled

* Fix ErrSideIsInvalid in tests
2023-10-13 15:18:55 +11:00
Ryan O'Hara-Reid
233a65a778 exchange: refactor credentials.go from PR #1320 (#1360)
* refactor credentials

* glorious: nits

* shazbert: nits

---------

Co-authored-by: romanornr <romanornr@gmail.com>
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-10-10 10:17:07 +11:00
Bea
2799f40401 Bitfinex: Fix websocket panic when seqNo is not sent (#1356) 2023-10-09 17:22:06 +11:00
Gareth Kirwan
1212b6750e OrderManager: Fix race condition in submit with ws (#1336)
* OrderManager: Fix race condition in submit with ws

If the ws sees the order before processSubmittedOrder then it will have
assigned it an internal order id already and added it to the store.
Don't treat that as an error. Instead just use the newer ws details

* OrderManager: Fix error comparisson

Should always use errors.Is when possible

* Tests: Simplify btcusd test pair declaration

* OrderManager: Improve test readability

* OrderManager: Add orderstore.getByDetail test

* Return a fresh pointer from orderstore.getByDetail

This protects the order.Details in the store from direct access.
The use-case was to allow the returned objects to be references so that
future changes to them would be reflected.
However we're not ready yet to allow people to touch the orders
directly, because they're not protected directly by a mutex, and nothing
would stop consumers contaminating the integrity of the data.

We can revisit this topic later atomicly, but it's definitely tangental
to the cause of action for PR #1336.

* Fix GetByDetail tests to assert a new pointer

* OrderManager: Avoid possible lock races

This fix internalises the getByDetail because the implication of moving
lock ownership out of exists/getByDetail to consumers breaks the order
store struct encapsulation in a way we really don't want to.
It's also more efficient

* Fix spelling mistake

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* OrderManager: Fix TestSubmitOrder... description

* OrderManager: Improve clarity of comment

* OrderManager: Capitalise error message

On failure to add to orderstore, capitalise the error message

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

---------

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-10-06 09:21:49 +11:00
Gareth Kirwan
d86da76b1f Bitstamp: Add auth channel subscription handling (#1333)
* Bitstamp: Add auth channel subscription handling

* Bitstamp: Avoid searching for asset type

We've hardcoded asset.Spot in order to find the pair.
Looking the asset up from the pair makes no sense.

* Bitstamp: Add type for wsOrders

* Bitstamp: Working test of Generic DataHandler

* Bitstamp: WS Order chan tests and remove type

orderType could be derived from status == New and Buy & price == 9+e9 or Sell & price == 0
But it would only be true for the first update and it really doesn't feel worth the risk
Consumers are going to have to merge to original request anyway

* Bitstamp: Linter fixes

* Kraken: Switch to shared fixture test

* Bitstamp: Fix lint on TestFixtureToDataHandler

* Engine: Add Clone for PairsManager

go-vet highlighted that the mutex here is a value when we copied the
PairsManager in a test.
Options to fix:
* Add a deep clone method to PairsManager
* Add a shallow clone method with a disclaimer
* Make the mutex a pointer
* Make the PairsManager itself a pointer

Options 3 and 4 are too invasive to justify changing at this point.
There's an inherent risk of PM being passed by value, but govet should
catch the copylock.
There's more risk in changing everything to use a pointer at this stage.

* Engine: Fix linter again, ironically

* Bitstamp: Rename OHLC const

* Bitstamp: Minor fixes to syntax

* Bitstamp: Simplify chanSymb=>pair

* Bitstamp: Still process order updates without ID

If there's a ClientOrderID we'll still process the order.
It doesn't seem likely we'd have this happen, but if it does we still
want consumers to get something.

* Bitstamp: Replace Clone with Lock methods

* Engine: Expose PairsManager's Mutex

Makes more sense than wrapping functions

* Bitstamp: Fix linter copylock (again)

* fixup! Engine: Expose PairsManager's Mutex

Omit Mutex from Json

* fixup! Bitstamp: Add auth channel subscription handling

Remove unused wsAuthToken

* Bitstamp: Simplify OrderData Unmarshal

* Bitstamp: Remove unused contexts

I added these following best practices, but the reality is that when/if
we get context awareness in GCT, there will be a lot more to fix and
this will be a drop in the ocean anyway.

* Bitstamp: Only call handleWSOrder for  MyOrders

* Bitstamp: Avoid allocating again in handleWSOrder

* CurrencyPairs: Remove public mutex

Simplified to a Load method to avoid making mutex public

* Tests: Improve test readability and clarity

* Bitstamp: Wrap errWSPairParsingError

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>

* Bitstamp: FetchWSAuth mock and live test

---------

Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
2023-10-05 16:13:05 +11:00
Scott
91d699be9d maps: expansion of Key concept (#1349)
* moves everything to use single map keys, also breaks

* full rollout

* tests

* fix a little bug

* minor test fixups

* Fix Key use

* rm 🔑 from 🔑 struct name
2023-10-04 10:19:41 +11:00
Ryan O'Hara-Reid
033a72b61a orderbook: fix race condition when function scope is in error (#1357)
* orderbook: fix race condition when in error

* Whoops
2023-10-03 16:08:31 +11:00
Ryan O'Hara-Reid
ecf0a5bcb3 spellcheck: fix and update (#1359)
* spellcheck: fix and update

* spellcheck: add exclude line for gateio_types
2023-10-03 16:07:03 +11:00
Scott
7f0faf7850 futures: add GetFuturesContractDetails wrapper function (#1274)
* all in a days work

* cleanup

* cleanup for real, also stop it binance.json

* minor coverage

* adds gateio to the slurry

* cleanup of types

* verbose verbose verbose verbose verbose verbose

* fixes huobi parsing issue

* fix bybit contract identification

* cleanup

* merge fixes

* addresses many big problems raised by SHAZ

* tracking errors and fixes

* funding rate if avail, fixes currency formatting

* Addresses nits and sneaks in extra fixes

* lint

* minor fixes after rebase

* better contract splitter for currencies like T-USDT

* forgot to add the exchange name like a fool

* merge fixes x1

* kucoin, direction, contract size

* rn direction, fix kucoin time

* WHOOPS

* Update exchanges/kucoin/kucoin_wrapper.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* misdirection

---------

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-10-03 16:04:45 +11:00
Gareth Kirwan
08c27afded Okx: Various websocket fixes (#1335)
* Okx: Remove redundant asset types from conversion

* Okx: Fix list of private channels for subscription

* Okx: Fix ws order update Acc and Rem amounts

The amounts aren't treated as deltas by anything receiving them
upstream, e.g. orderManager or wsRoutineManager.
So we should treat them the same as the REST implementation

* Okx: Add context to stream unmarshall errors

I encountered a series of errors with unmarshalling from okx and have no
idea what the messages which weren't valid json actually were.
This fix adds the raw message for context to the error

* Okx: Reduce ping interval

Seeing occassional disconnects and suspect dropping the ping interval
down might help

* Fix ClientSupplier and ClientSupplied inconsist

This simply removes the Supplie. part in favour of the simpler name used
in binance. Seems safe since its already inconsistent and not populated
in at least ws order handlin

* Okx: Fix ClientOrderID ignored in ws orders

* Okx: Standardise Rfq abbrev; Fix RfqIDs

* Okx: Fix RfqIds outside of ClientRfqIds

* Okx: Simplify TestUpdateOrderExecutionLimits

* Okx: Fix panic on no avgPrice and QuoteAmount
2023-10-03 11:01:36 +11:00
Samuael A
6105071114 exchanges: Add Kucoin support (#1102)
* init

* updates config

* wrapper configuration

* updates exchange readme

* adds SendAuthHTTPRequest and SendHTTPRequest

* adds ratelimit file

* adds test case and minor fixes

* improve error handling

* update testcases and improve GetSymbols API

* adds SPOT API's

* minor fix

* WIP

* WIP

* adds test case

* adds check in test case

* fixes in Auth. HTTP

* improvements

* adds trade, kline support and testcases

* adds SPOT API and testcases for same

* adds SPOT API and testcases

* adds SPOT API and testcase

* WIP

* adds API's

* adds API's

* adds test cases

* adds comment to exported data types

* adds API and test cases

* adds API

* adds API

* rearrange functions

* WIP: adds API

* adds API for Post Order SPOT

* adds API and few fixes

* fixes

* WIP

* WIP

* add PostBulkOrder API and its test case

* fix issues

* adds cancel order APIs and test cases for same

* add minor test fixes

* add API

* adds API

* fixes

* add API

* adds API and test cases

* fix test

* adds API

* adds test

* fix test

* adds API and test

* adds deposit API and test cases

* WIP

* adds API and test cases

* WIP

* WIP

* add public future API and test cases

* WIP

* remove v2 API and replace them with v1

* update test cases

* adds future order API and test cases

* adds futures order API

* adds API

* add API and test cases

* adds API and test cases

* adds API and test cases

* adds API and test cases

* Adding wrapper functions

* Fix on wrapper function

* Adding websocket support

* Complete addressing WS push datas

* Adding spot push data unit tests

* adding futures websocket push data handlers

* Adding futures websocket push data handlers

* Added unit tests

* Updating unit tests

* Updating wrapper and unit test functions

* Adding missing wrapper functions and code cleaning up

* Resolved linter issues

* Fixing websocket issues

* Fixing websocket issues

* Slight fix on config_example file

* Minor update

* Basic nits updates

* Fix minor linter issues

* Minor update

* Minor unit test update

* Minor unit test update

* Code update and linter issues fix

* Removed unnecessary type conversion codes

* Monor update based on review comment

* Fix based on review comments

* Adding rate-limiter

* Websocket update and overall minor fixes

* Removed IsAssetTypeEnabled method implementation

* Fix connection and formatting issues

* Updating orderbook issues

* Very minor label fix

* Minor error returning fix

* code cleaning up and minor spelling fix

* Updates on unit test

* Update on unit tests and slight code structure

* unit test update

* orderbook update and minor fix

* fix on race

* Mini linter fix

* fix minor parameter and unit test issues

* handler funcs and models update

* Fixing websocket and unit test issues

* order side string for active orders

* Fix on websocket and unit tests

* Minor type changes

* Minor Orderbook fix and unit test update

* Small fix on orderbook

* Updating orderbook functionality

* FIx on websocket orderbook handlers

* Small update on kucoin websocket

* fix missed review comments

* fix based on review comments

* Updating websocket orderbook and fixing unit tests

* Minor fixes

* unit test update

* Updating unit test according to enabled asset type

* toggle canManipulateRealOrders const

* Unit test update

* Fix minor issues

* minor fix

* documentation fix

* wrapper coverage and unused params fix

* testing and minor changes

* documentation, websocket and unit test update

* minor linter fix

* Websocket spot/margin subscription update

* minor ticker update fix

* minor fixes on endpoints

* timestamp and number convert method and unit tests

* timestamp convert minor update

* minor type and conversion fix

* create a common timestamp convert and fix minor issues

* linter and ticker fix

* Updating unit tests and order placing endpoint methods

* Added a pairs check

* Fix config test error

* rm unused error variable

* Fix source of linter issue

* code update: convert, wrapper and websocket fix

* minor code update

* Websocket code and unit tests update

* Websocket ticker ask/bid type change and small error msg fix

* docs update

* fix: websocket orderbook handling

* change orderbook channel to marketOrderbookLevel2Channels and fix websocket orderbook update

* Minor func rename and reciever change

* Minor orderbook unit test issue fix

* comment: about why we used a random delimiter '-' for futures

* update config files and FetchTradablePair func for futures pairs

* futures config pairs update

* remove ConnextionMonitorDelay from websocket setup

* fix on types and futures pair conversion

* updating config pairs

* change NewPairFromString to DeriveFrom

* unit tests update

* unit tests update

* Added TickerBatching

* added GetStandardConfig to GetDefaultConfig

---------

Co-authored-by: Jaydeep Rajpurohit <jaydeeppurohit1996@gmail.com>
2023-09-27 15:09:38 +10:00
Scott
5f2f6f884b Binance,Okx: Add Leverage, MarginType, Positions and CollateralMode support (#1220)
* init

* surprise train commit

* basic distinctions

* the terms of binance are confusing

* renames and introduction of allocatedMargin

* add new margin funcs

* pulling out wires

* implement proper getposition stuff

* bad coding day

* investigate order manager next

* a broken mess, but a progressing one

* finally completes some usdtmargined stuff

* coinMfutures eludes me

* expand to okx

* imports fix

* completes okx wrapper implementations

* cleans and polishes before rpc implementations

* rpc setup, order manager features, exch features

* more rpc, collateral and margin things

* mini test

* looking at rpc response, expansion of features

* reorganising before the storm

* changing how futures requests work

* cleanup and tests of cli usage

* remove silly client side logic

* cleanup

* collateral package, typo fix, margin err, rpc derive

* uses convert.StringToFloat ONLY ON STRUCTS FROM THIS PR

* fix binance order history bug

* niteroos

* adds new funcs to exchange standards testing

* more post merge fixes

* fix binance

* replace simepletimeformat

* fix for merge

* merge fixes

* micro fixes

* order side now required for leverage

* fix up the rest

* global -> portfolio collateral

* Update exchanges/collateral/collateral_test.go

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>

* adds fields and todos

* rm field redundancy

* lint fix oopsie daisy

* fixes panic, expands error and cli explanations (sorry shaz)

* ensures casing is appropriate for underlying

* Adds a shiny TODO

---------

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-09-26 16:16:31 +10:00
Ryan O'Hara-Reid
8b8d96c612 gate.io: Enforce the use of 'convert.StringToFloat64' and permit its use in outbound requests (#1308)
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-09-18 17:15:27 +10:00
Bea
78fe6d5e79 Bitfinex: Ensure websocket configuration is sent with every connection (#1350)
* Bitfinex fix Websocket configured with every subscription

* Bitfinex rm whitespace, consolidate ConfigureWS
2023-09-18 17:13:55 +10:00
Ryan O'Hara-Reid
1adbc99526 exchanges: improves GetDefaultConfig method (#1245)
* exchanges: Add function to get standard config

* exchanges: add tests (cherry-pick here and above)

* after pick stuff

* cleanup

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-09-18 11:45:02 +10:00
Gareth Kirwan
ade2d9c5d2 Bitfinex: Resubscribe orderbook after checksum err (#1303)
* Bitfinex: Resubscribe orderbook after checksum err

* Bitfinex: chanForSub return only first match
2023-09-14 15:38:01 +10:00
Gareth Kirwan
c8ec22fe92 websocket: Relay disconnect errors to subscribers (#1347)
* WS: Relay disconnect errors to subscribers

Subscribers probably care when the WS got disconncted.
Tell them and expose a method to test the error for matching

* Fix linter error
2023-09-14 10:14:04 +10:00
Gareth Kirwan
51f73300bb Bitfinex: Improve websocket type assertion checks for order processing (#1292)
* Bitfinex: Error if ws data id/cid is not a float64

* Bitfinex: Add annotation to WS id/cid assest errs
2023-09-07 15:43:38 +10:00
Ryan O'Hara-Reid
ad9de19d47 orderbook: Check assignment of time values and reject if not set (#1318)
* orderbook: Check assignment of time values and reject if not set.

* linter: fix

* buffer: additional linter winter fixter

* Implement through pending exchanges

* finished push

* linty: minty

* gomod: tidy

* thrasher: nits

* glorious: nits

* orderbook: purge type now in favour of external call allocation

* orderbook: push last param

* orderbook: only 1 unlock call is needed

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-09-07 11:00:16 +10:00
Gareth Kirwan
20143886ca Kraken: Fix WS Auth endpoints not re-enabled (#1334)
When Kraken has a disconnect or failure that sets SetCanUseAuthenticatedEndpoints(false),
it's never re-enabled when the websocket is reconnected
That means all subsequent requests would fall back to rest
2023-09-07 10:59:30 +10:00
Gareth Kirwan
253b9a5049 Kraken: Fix websocket order updates (#1327)
* Kraken: Fix error on WS update without status

* Kraken: Fix parsing of WS updates without Desc

* Kraken: Fix WS market order handling

* Kraken: Fix tests in parallel using DataHandler

* WebsocketManager: Fix order summary printing

* Kraken: Fix parallel tests race with DataHandler

* Kraken: Issue a classification err on asset 404

* Kraken: Switch to testify and close fixture
2023-09-05 13:22:34 +10:00
Adrian Gallagher
578dd0bafe Binance: Add crypto/flexible loan endpoints (#1331)
* Binance: Add crypto/flexible loan endpoints

* Address nitters

* Address shazbert's nitters
2023-09-01 16:46:53 +10:00
Gareth Kirwan
6ab4c27d1c Bitfinex: Refactor websocket wsUpdate handling and various improvements (#1317)
* Bitfinex: Fix cancel/update order WS ack not seen

Fixes #1288

* Bitfinex: Fix ws Unsubscribe and Resubscribe

Unsubscribe needed to use the channel id.
Resubscribe needs to have the original subscription params.

* Bitfinex: Fix ws Trades Fees on te

The ws channel for authenticated Trades sends two types of update:
* te, Trade Executed
* tu, Trade Execution Update

Only the second one contains fee information.
[See the docs](https://docs.bitfinex.com/reference/ws-auth-trades)

This commit fixes:
`exchange Bitfinex websocket error - unable to type assert trade fee`
after an executed market trade on the te update

* Bitfinex: Fix error on ws auth ok

This fixes:
`Bitfinex Could not find an existing channel subscription: account Pair:
ChannelID: 0`
It's not clear from history why we'd want to store a reference to the
ubiquitous 0 channel like this, but it's definitely wrong, and anything
that attempts to get channel information about 0 chan needs to be fixed
anyway.

* Bitfinex: Refactor wsUpdate handling

This commit doesn't break out all the sub-updater, but attempts to do
something about the unmanagable size of ws update handling

* Binfinex: Fix linter issue on chanId casing

* Bitfinex: Fix linter outdent complaint

* Bitfinex: Fix linter issues on test

* Bitfinex: Fix TestWsTradingPairSnapshot chan lookup

* Bitfinex: Remove unnecessary WsAddSubs in test

* Bitfinex: Fix TestWsSubscribedResponse chan

* Bitfinex: Throw a specific error for bad event

* Bitfinex: WS Type assertions for positionSnapshots

* Bitfinex: tradeUpdate type assertion

* Bitfinex: Reinstate default subscriptions

* Bitfinex: Assert chan assetType is the same

* Bitfinex: Lowercase error string

* Bitfinex: Refactor WS eventType/chanId handling

* Bitfinex: Fix linter issues

* Bitfinex: Fix delimiter for pairs with more than 6 chars

* Bitfinex: Fix WS handling of subscribed symbols

This simplifies the handling of subscription symbols.
Now that we know the channel up front from handling the subscribed
response we can limit the parsing forms needed

* Bitfinex: Placate the linter

* Bitfinex: Disable margin assets for WS

Margin WS Currently not fully implemented and causes subscription collisions with spot

* Bitfinex: Fix parsing of 4 part funding keys

This improves overall handling and errors on a few current assumptions
about key structure

* Bitfinex: Linter fixes

* Bitfinex: Remove key parsing from assetPairFromSymbol

* Bitfinex: Use native error wrapping

* Bitfinex: Skip disabled assets in default ws subs
2023-09-01 16:46:00 +10:00
Gareth Kirwan
9a0f261211 Orderbook: Add GetDepth to Base (#1328)
* Orderbook: Add GetDepth to Base

Base.GetDepth returns the concrete book of which Base is a copy
This is probably useful for immutably monitoring orderbook health and state
whereas FetchOrderbook would trigger a refresh.

* Orderbook: Reword GetDepth comment

* Orderbook: Add test for Base.GetDepth
2023-08-29 15:49:24 +10:00
Gareth Kirwan
ec312ca0d3 Bitstamp: Add websocket heartbeat (#1329)
This prevents the frequent (5x / hour) disconn/reconns we're seeing with a
10s or even 20s traffic timeout.
I'd like to base the interval on the traffic timeout / 2, but that's
non-trivial right now, and 8s isn't an excessive default
2023-08-29 09:22:50 +10:00
Ryan O'Hara-Reid
c5240153f9 orderbook: Add GetTranches and GetPair methods to Depth type (#1324)
* orderbook: Add GetTranches and GetPair methods to Depth type (cherry-pick)

* glorious: nits

* linter: fix

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-08-22 13:29:17 +10:00
Samuael A
9c83231696 exchanges: Update OKCoin V5 exchange support (#1206)
* starting public endpoints

* Adding public endpoints

* added public spot market endpoints

* websocket subscriptions updates

* websocket push data handlers completing

* linter fix

* Added funding private endpoints

* Adding authenticated account endpoints

* Added fiat and OTC-RFQ authenticated endpoints

* trading authenticated endpoints

* completing trade endpoints and add public wrapper endpoints

* Authenticated wrapper functions and corresponding unit test

* Adding authenticated websocket endpoint and fixing wrapper functions

* Documentation and exchange websocket update

* update websocket orderbook checksum handling

* linter issues fix and unit test update

* remove invalid orderbook endpoint and unit test

* Documentation, handlers, and model types update

* minot fix

* Minor fixes

* Updating unit tests and added missing endpoints

* Add missing credential check

* Minor unit test fixes

* fix minor linter issue

* add snaphot test unit test

* Fix on update checksum and documentation update

* update exchange, add UpdateOrderExecutionLimits, and update documentation

* Minor fix on tickers fetching

* Minor websocket fix and smaill unit tests

* Minor websocket and naming fixes

* uncomment default channels

* Fix type and unit test issues

* websocket channels and data handling update

* Update Advanced-Algo websocket handling and minor fixes

* documentation and minor code fixes

* Fix name changes

* documentation contribution update

* intervalToString method update

* fix exchange_wrapper_standard tests

* Fix minor issues based on exchange_wrapper_standards_test

* Fix wrapper extended candlestick check

* websocket orders fetching error check method update

* Exchange name check and change

* docs: Add missing contributors

---------

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2023-08-22 09:44:39 +10:00
Ryan O'Hara-Reid
a440fab4f3 stream/buffer: Reduces map lookups by using key struct (#1309)
* stream/buffer: Adds key map optimisation (cherry-pick)

* stream/buffer: Add buffer to DataHandler intermediary. Add field InitialUpdate bool to toggle when first update is seen for initial sync.

* whoops

* buffer: Add difference benchmarks for reference

* glorious: nits (reverting out of context changes)

* RM unused error that will be used later

* purge: benchmark

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2023-08-21 16:28:38 +10:00