Commit Graph

14 Commits

Author SHA1 Message Date
Scott
3a66e99899 Authenticated Websocket support (#315)
* Improves subscribing by not allowing duplicates. Adds bitmex auth support

* Adds coinbase pro support. Partial BTCC support. Adds WebsocketAuthenticatedEndpointsSupported websocket feature. Adds GateIO support

* Adds Coinut support

* Moves Coinut WS types to file. Implements Gemini's secure WS endpoint

* Adds HitBTC ws authenticated support. Fixes var names

* Adds huobi and hadax authenticated websocket support

* Adds auth to okgroup (okex, okcoin). Fixes some linting

* Adds Poloniex support

* Adds ZB support

* Adds proper bitmex support

* Improves bitfinex support, improves websocket functionality definitions

* Fixes coinbasepro auth

* Tests all endpoints

* go formatting, importing, linting run

* Adds wrapper supports

* General clean up. Data race destruction

* Improves testing on all exchanges except ZB

* Fixes ZB hashing, parsing and tests

* minor nits before someone else sees them <_<

* Fixes some nits pertaining to variable usage, comments, typos and rate limiting

* Addresses nits regarding types and test responses where applicable

* fmt import

* Fixes linting issues

* No longer returns an error on failure to authenticate, just logs. Adds new AuthenticatedWebsocketAPISupport config value to allow a user to seperate auth from REST and WS. Prevents WS auth if AuthenticatedWebsocketAPISupport is false, adds additional login check 'CanUseAuthenticatedEndpoints' for when login only occurs once (not per request). Removes unnecessary time.Sleeps from code. Moves WS auth error logic to auth function so that wrappers can get involved in all the auth fun. New-fandangled shared test package, used exclusively in testing, will be the store of all the constant boilerplate things like timeout values. Moves WS test setup function to only run once when there are multiple WS endpoint tests. Cleans up some struct types

* Increases test coverage with tests for config.areAuthenticatedCredentialsValid config.CheckExchangeConfigValues, exchange.SetAPIKeys, exchange.GetAuthenticatedAPISupport, exchange_websocket.CanUseAuthenticatedEndpoitns and exchange_websocket.SetCanUseAuthenticatedEndpoints. Adds b.Websocket.SetCanUseAuthenticatedEndpoints(false) when bitfinex fails to authenticate
Fixes a typo. gofmt and goimport

* Trim Test Typos

* Reformats various websocket types. Adds more specific error messaging to config.areAuthenticatedCredentialsValid
2019-06-19 13:19:01 +10:00
Scott
6c850e73e2 Websocket connection handling and subscription management (#297)
* Step one: Sets up  connection handler for websockets to always be connected until a shutdown event is received.
Sets up a vague subscription handler to ensure subscriptions are subscribed

* Adds support for resubscriptions for bitfinex, bitstamp, bitmex and btcc. Adds subscription params for special websocket subscription requirements. Removes subscription monitor from wait group so that it can exist despite a shutdown and continuously check

* Adds channel subscription support to bitmex, btse, coibasepro, coinut, gateio, gemini, hitbtc, huobi, hadax, kraken, okgroup, poloniex and zb

* Implements unsubscribe for bitfinex, btcc, btse, coinbasepro, gateio, gitbtc, huobi, hadax

* ManageSubscriptions now called from WSConnect and made private instead of inside individual exchanges. ManageSubscriptions can now unsubscribe. exchange_websocket_types.go now contains all exchange_websocket.go types to avoid clutter

* Adds it to websocket functionality so managesubscriptions will close when not supported

* Separates functions into testable functions to ensure logic works. Adds tests. Updates websocket setup to include verbosity (inherited from exchange). Adds no connection tolerance to fatal on failed reconnects

* More exchange_websocket tests. Updating to use pointers. Creation of equals func to make comparison easier

* Fixes okex, okcoin tests. Fixes race conditions. Removes pointer usage again.

* Adds subscribe and unsubscribe to wrappers

* Fixes deadlock. Fixes ws verbosity.

* Updates all exchanges to properly support subscription/connection feature. Also reintroduces race conditions....

* Moves connection varialbes to struct from package to allow each websocket to have their own reconnection checks. Neatens up logs

* Fixes lint/critic issues. Fixes tests. Removes unused function.

* Moves websocket ratelimiter to their own const variables. Fixes more race conditions with connecting variable

* Removes redundant subscribe functions. Ensuring only the exchange_websocket.go can manage subscriptions. Fixes debug logs to be verbose wrapped

* Fixes issue with slice copying. Re-adds okgroup default channels

* Adds nolint to append

* Adds comments and adds support for gateio auth request subscriptions

* Adds new test to ensure slices dont point to the same vars

* removes fatals. gofmt goimports

* more gofmts

* Addresses PR comments, removing empty and redundant lines

* Addresses PR comments. Ensures that writing to the websocket is single-threaded by adding a mutex to exchanges. Minimises wrapper code and moves subscription loops to exchange_websocket. Privatises ChannelsToSubscribe, Connecting properties and removeChannelToSubscribe func to prevent unnecessary tampering.

* Removes unused mutex. FMTS and IMPORTS

* Fixes request lock time change

* More specific logs

* Renames ws mutex. Fixes bitmex subscriptions. Increased gateio ratelimiter to 120ms. Removes ratelimiter from bitfinex, bitmex, bitstamp, btcc, btse, coibasepro, hitbtc, huobi, hadax, poloniex and zb

* changes recieved typo due to not being well received

* Fixes parsing issue with Huobi and hadax

* Fixes data race with more locks

* removes defer locks. fixes huobi/hadax verbose output

* Fixes double JSONEncode for coinut. Fixes verbose output for coinut

* gofmt,goimport for coinut

* Fixes issue where multiple connection monitors can spawn

* Removes defer exchange.WebsocketConn.Close() in defer handledata exit as connectionmonitor handles connections instead

* gofmt and go import

* More fmts
2019-05-16 16:39:16 +10:00
leilaes
cdab89a58a Replace a zero-length, non-nil slice with a nil slice (#299)
* Replace a zero-length, non-nil slice with a nil slice

* Update codelingo.yaml
2019-05-16 10:19:08 +10:00
Scott
107cf76373 Kraken websocket support (#264)
* Initial commit. Adds ticker, candle and trade, subscription support

* Adds support for spread and orderbooks

* Adds new currency pair delimiter ("/"), Adds dedicated websocket Connected channel handler, Updates Kraken websocket capability definition, Refines websocket tests to connect and disconnect without freezing, separates WebsocketUnsubscribeEventRequest ChannelID into its own struct WebsocketUnsubscribeByChannelIDEventRequest to prevent bad json WS requests, Adds asset type to orderbook, Kraken WS handles connection better

* Removes duplicate type, reverts config value

* Addresses error returns and changes writeToWebsocket to use byte array. Removes deferred funcs in tests. Increases test listening limit for rare cases

* Fixes verbose log. Rearranges WS Connect async ordering. Fixes DATA RACE. Fixes random okex tests. Ensures Kraken WS tests only connect once
2019-04-04 10:21:44 +11:00
Andrew
dc236c251e Optimisation: large structs/huge param fixes (part 2) (#262)
* updated golangci config to enable hugeparam linter

* ModifyOrder struct usage converted to a pointer

* OrderBook conversion to struct

* More conversion of large structs to pointers

* updated golangci config to enable hugeparam linter

* ModifyOrder struct usage converted to a pointer

* OrderBook conversion to struct

* More conversion of large structs to pointers

* disabled hugeParam check for golang again

* changed based on suggested feedback and fix for no default provider

* fixed typing
2019-03-26 15:40:46 +11:00
Ryan O'Hara-Reid
0990f9d118 Currency package update (#247)
* Initial currency overhaul before service system implementation

* Remove redundant currency string in orderbook.Base
Unexport lastupdated field in orderbook.Base as it was being instantiated multiple times
Add error handling for process orderbook

*  Remove redundant currency string in ticker.Price
 Unexport lastupdated field in ticker.Price
 Add error handling for process ticker function and fix tests

* Phase Two Update

* Update translations to use map type - thankyou to kempeng for spotting this

* Change pair method name from Display -> Format for better readability

* Fixes misspelling and tests

* Implement requested changes from GloriousCode

* Remove reduntant function and streamlined return in currency_translation.go

* Revert pair method naming conventions

* Change currency naming conventions

* Changed code type to exported Item type with underlying string to reduce complexity

* Added interim orderbook process method to orderbook.Base type

* Changed feebuilder struct field to currency.Pair

* Adds fall over system for backup fx providers

* deprecate function and children and fix linter issue with btcmarkets

* Fixed requested changes

* Fix bug and move mtx for rates

* Fixed after rebase oopsies

* Fix linter issues

* Fixes race conditions in testing functions

* Final phase coinmarketcap update

* fix linter issues

* Implement requested changes

* Adds configuration variables to increase/decrease time durations between updating currency file and fetching new currency rates

* Add a collection of tests to improve codecov

* After rebase oopsy fixes for btse

* Fix requested changes

* fix after rebase oopsies and add more efficient comparison checks within currency pair

* Fix linter issues
2019-03-19 11:49:05 +11:00
Scott
6a15ecc65c OKEX/OKCoin API combine via OKGroup (#252)
* Initial commit

* Successful authenticated request implementation.

* Removes data from okcoin, okex. Implements some account okgroup endpoints. Adds tests

* Finishes account API endpoint implementations.

* Implements and adds tests for the following okgroup v3 API endpoints: GetSpotTradingAccounts, GetSpotTradingAccountForCurrency, GetSpotBillDetailsForCurrency, PlaceSpotOrder, PlaceMultipleSpotOrders, CancelSpotOrder, CancelMultipleSpotOrders, GetSpotOrders, GetSpotOpenOrders, GetSpotOrder, GetSpotTransactionDetails, GetSpotTokenPairDetails, GetSpotOrderBook, GetSpotAllTokenPairsInformation, GetSpotAllTokenPairsInformationForCurrency, GetSpotFilledOrdersInformation, GetSpotMarketData

* Implements and adds tests for all margin api endpoints: GetMarginTradingAccounts, GetMarginTradingAccountsForCurrency, GetMarginBillDetails, GetMarginAccountSettings, GetMarginLoanHistory, OpenMarginLoan, RepayMarginLoan, PlaceMarginOrder, PlaceMultipleMarginOrders, CancelMarginOrder, CancelMultipleMarginOrders, GetMarginOrders, GetMarginOpenOrders, GetMarginOrder, GetMarginTransactionDetails. Simplifies some Spot endpoints combining ForCurrency funcs where possible

* Adds all 29 Futures endpoints with tests. Updates comments and naming conventions. Adds standard realordertest func. Adds ability to make public API requests. Adds test purpose comments

* Adds 29 endpoints for SWAP API support. Adds tests for each endpoint. Declares response variables in function declaration. Simplifies URL parameter formatting

* Adds all ETT endpoints with tests

* Creates OKCoin and OKEX struct types. Moves okgroup tests to okcoin and okex exchanges. Updates withdraw fee calculation. Updates exchange.go exchange declaration to point to new types. Streamlines wrapper tests. Begins websocket integration

* Rebase fixes

* Deletes okcoin_types.go, okcoin_wrapper.go, okex_types.go, okex_wrapper.go. Combines okex,okcoin wrappers in okgroup_wrapper.go. Removes boilerplate url.values with new request struct type parsing. Adds github.com/google/go-querystring to go modules. Replaces USDT with USD for OKCoin tests. Moves OKEX specific endpoints (futures, swap & ett) to okex.go. Fixes recieving receiving again

* Maps the wrapper

* Parses json into time.Time instead of string + conversion

* Renames websocket.SetEnabled to websocket.SetWsStatusAndConnection. Maps main spot websocket functions for okgroup. Adds some basic ws tests

* Updates websocket default URLS, adds checksum tests, removes setdefaults from okgroup, adds WebsocketDataWrapper to wrap all okgroup websocket data responses, handles spot, swap, index and futures ticker, candle, trade, orderbook, funding fee websocket responses. Partially implements checksum validation on orderbook data. Fixes all linting issues

* Handles the calculation of okgroup websocket checksums. Adds tests

* Now all orderbook checksums are validated. Cleans up implementations and removes invalid checksum calculator functions. Adds function to parse ordertype. Puts verbose logs behind verbose check

* Removes parallel from okgroup tests. Removes unused code. Adds GetWsChannelWithoutOrderType. Improves handling of WS data types. Adds types for more ws channels. Simplifies update orderbook handling

* updates btse func name

* linting

* Fixes websocket connection issue with tests. Removes test verbosity

* Updates checksum calculation to handle more than 7 decimal places. Adds rate limiters. Uses != "" instead of len > 0. Adds new test to handle checksum calculation with 8 decimal places.

* Removes logging. Fixes orderbook wrapper references

* Adds slightly more robust resubscribe func. Adds websocket tests that can detect websocket errors

* Fixes linting issues

* Adds new WS func IsConnected() to expose ws status. Tests protect against channel timeout

* Adds test comments. Fixes parallel issues for futures tests

* Fixes error output for wrapper function
2019-03-18 15:18:36 +11:00
Adrian Gallagher
7dcb1ab553 Migrate from gometalinter.v2 to golangci-lint (#249)
* Migrate from gometalinter.v2 to golangci-lint
2019-03-01 16:10:29 +11:00
Adrian Gallagher
5e5ca8a887 Linter fixes (#246)
Linter fixes
2019-02-05 16:26:04 +11:00
Adrian Gallagher
291e404a4a Makefile: add new recipes and linter features (#244)
* Makefile: add new recipes and linter features

* expand linter coverage and fix issues

* Update makefile

* address PR nitterinos
2019-01-31 14:53:24 +11:00
Ryan O'Hara-Reid
41415ca3b9 Websocket update increasing exchange coverage and bug fixes (#233)
Websocket update increasing exchange coverage and bug fixes
2019-01-23 14:23:11 +11:00
Andrew
d01e7bad72 Implement Logger (#228)
* Added new base logger

* updated example and test configs

* updated exchange helpers restful router & server

* logPath is now passed to the logger to remove dependency on common package

* updated everything besides exchanges to use new logger

* alphapoint to bitmex done

* updated bitmex bitstamp bittrex btcc and also performance changes to logger

* btcmarkets coinbase coinut exmo gateio wrappers updated

* gateio and gemini logger updated

* hitbtc huobi itbit & kraken updated

* All exchanges updatd

* return correct error for disabled websocket

* don't disconnect client on invalid json

* updated router internal logging

* log.Fatal to t.Error for tests

* Changed from fatal to error failure to set maxprocs

* output ANSI codes for everything but windows for now due to lack of windows support

* added error handling to logger and unit tests

* clear wording on print -> log.print

* added benchmark test

* cleaned up import sections

* Updated logger based on PR requests (added default config options on failure/setting errors)

* ah this should fix travici enc config issue

* Load entire config and clear out logging to hopefully fix travisci issue

* wording & test error handling

* fixed formatting issues based on feedback

* fixed formatting issues based on feedback

* changed CheckDir to use mkdirall instead of mkdir and other changes based on feedback
2019-01-08 21:56:22 +11:00
Adrian Gallagher
046e15a67e Periodic GoReportCard fixes 2018-12-27 15:10:41 +11:00
Ryan O'Hara-Reid
d3c2800fe0 Initial overhaul of websocket connection and feeds (#189)
* Initial overhaul of websocket connection and feeds
* Added proxy support
* Piped to routines.go

* Added new websocket file in exchanges
Refactored orderbook handling into exchange_websocket.go
Added better error responses for binance_websocket.go
General clean for binance_websocket.go

* General fixes - bitfinex_websocket.go
Refactored orderbook cache code - bitfinex_websocket.go
Removed fatal error with unhandled type - routines.go

* Added general improvements to bitmex_websocket.go
Refactored orderbook handling to exchange_websocket.go
Added variable in Item struct in orderbook.go for looking up orders by ID

* Fix issue when routines are blocked due to Data Handler not started
Updated traffic handler
General fixes for bitstamp_websocket.go

* General fixes for coinbasepro_websocket.go

* General fixes for coinut_websocket.go
Fixed error return in exchange_websocket.go

* Removed comments in coinut_wrapper.go
Refactor orderbook logic from hitbtc_websocket.go to exchange_websocket.go

* General fixes

* Removed comments
General fixes

* Updated routines.go

* After rebase fix

* Fixed update config pairs in okcoin.go

* fixed config currency issue in okcoin.go for okcoin China

* exchange_websocket.go
*Removed unused const dec
*Removed state change routine
*Improved trafficMonitor routine
*Increased verbosity for error returns
*Removed uneeded mutex locks

exchange_websocket_test.go
*Added new tests for websocket and orderbook updating

routines.go
*Removed string cased

* Fixed race conditions on sync.waitgroup in exchanges_websocket.go

* Changes variable name in config.go

* Removes unnecessary comment

* Removes indefinite lock on error return

* Removes unnecessary comment

* Adds support for BTCC websocket
Drops support for BTCC REST

* Rewords comment in exchange_websocket.go
Moves types to poloniex_types.go

* Moves types to coinut_types.go

* Removes uneeded range for accessing array variables for coinbase_websocket.go
Removes comments in coinut_types.go

* Adds verbosity flag to GCT
Suppresses verbose output from routines.go

* Fixes setting proxy for REST and Websocket per exchange
Upgrades error handling
Drops unused *url.Url variable in exchange type

* Adds test for setting proxy

* Fixes bug that closes connection due to incorrect timeout time through a proxy connection

* Clarify verbose flag message
2018-10-24 14:22:40 +11:00