5 Commits

Author	SHA1	Message	Date
Scott	b686cf2e0e	Feature: Websocket order handling (#446) ... * Initial changes, removing exchange name as an arg and puts it in the pointer struct. Adds case to ws routines * Adds CancelAllOrders func, adds GetByExchangeAndID. Adds modify handler in routines.go * initial poor attempts to have bitmex work with new datahandler handlers. fixes ordersides * bitmex Completes new order * Better bitmex handling, but not complete. Begins a gargantuan task of unifying order data structs. Sometimes an order update will contain lot's of information, so its best to be able to update all fields of our orders, rather than just an arbitrary subset. As a result, everything will be broken for the foreseeable future :glitch_crab: * Removes old order handler which did nothing. Updates order properties for everything everywhere - now consistent. Changes order status. Adds asset type and wallet address to all order types * Adds order updater to update only relevant fields since the object is generic, we don't know what fields are passed from what exchanges. Adds "lastupdated" field to order.Detail. Expands order cancellation for engine orders. * Ensures that new orders are added to the ordermanager's order store. Saaa many comments. Internalises orderStore get func. Adds internalOrderID to orderdetail and adds websocket support for it * Fixes a cancelAllOrders oopsie doopsie * Adds potential func to update orderdetails from an orderdetail struct. Unsure if will keep. * Begins btcmarkets implementation. Expands order "stringToOrder" funcs to allow for some more flexible string coversions. Removes order.Submit via websocket as it would cause unlimited order place issues :D * Finishes btc markets without testing * Adds untested ws auth func to btse * Finises btse, fixes btcmarkets bug * Adds coinbasepro support * Fixes a few more fields in coinbase pro and readds the extra subs * Begins work on coinbene. Plus theyve added a new ws connection yeee * Wasted a bunch of time adding support to an additional websocket that isn't needed ;_; Fixed a bug in coinbasepro. Fully kitted out coinbene support. Updates order types with all fields * Removes extra websocket connection ;_; * Finishes gemini. Fixes order side unknown * Adds okgroup support. Moves byte reading to another function to allow for unit testing. Updates routines to use pointers. Updates date update handling for order details * Finishes order data for okgroup websocket, but starts the STRANGE process of converting all other websocket endpoints to be a little less silly * Cleans up okroup websocket implementation. Fixes bug in Gemini * Adds poloniex support. Updates ws order handling * new bitmex support. Adds some tests now that its all in its own func. Fixes poloniex bug * Begins work on authenticated binance websocket * Attempts to track user data via binance websocket * Maybe finishes Binance websocket support * Begins adding test coverage to orders.go. Updates names of script properties to match updated * Begins an experiment with code coverage. Fixes more rebase issues * Completes orders coverage. Botches a few other things though. Fixes more scripting stuff * All tests in engine package pass * Adds some loevely routine tests * Moves ordermanager to test Bot ordermanager Adds lovely routine tests to ensure things that get sent to be handled the data handler are handled by the data handler by handling them * Replaces "wsHandleData" with "wsReadData" as that's what its going to do now. * Splits all wsHandleData into wsReadData and wsHandleData to allow for easy testing via sending []byte json examples to test proper functionality. Breaks so many tests * Fixes majority of test issues. But data races which are tough on the engine package * "Fixes" test by removing shutdown test. It interferes with too many things. Requires some thought * Tests all the binance websocket points * Adds better bitfinex websocket support. * Adds testing for bitfinex, bitstamp and btcmarkets. Fixes websocket bugs encountered * Adds BTSE ws tests. Fixes bugs in ws * Adds coinbase pro tests. Fixes any issues * Coinbene tests * Starts to handle coinut. Runs into a problem conceptually regarding websocket roundtrip and orders. Both events need to happen without impacting eachother/racing * Addresses a data race issue regarding websocket and bot order management submission - order submission locks at an earlier point to prevent routines.go from creating an order before order submission creates it. Updates rpcserver to use order management bot to submit orders. * Finishes the hectic coinut testing * Adds tests for gateio * Fixes rebase issues. Updates tests to work without being overloaded * Begins testing of gemini. fixes up minor issues * ginishes gemini tests and fixes * Adds hitbtc tests. Fixes all the many issues with hitbtc websocket * Adds remaining tests. Increases default test channel limit again * Begins work towards huobi tests * Finishes huobi tests * Fixed all mythical rebase adventures * Begins kraken transformation * Finishes kraken. Fixes coinbene leverage now that its changed * Begins okgroup testing * Adds okgroup ws tests * Does some poloniex * Fixes basic curreny issue by extracting to func * Begins redesign of poloniex websocket datahandling. Completes authenticated handling, now onto unauth * Finishes poloniex revision * Finishes ZB additions * Fixes data races * Fixes rebase issues. Fixes bad kraken logic * Fixes after reviewing code * lint everywhere * Fixes lingering lints * lint * Adds test coverage to order detail and modify updating * Fixes linting * Fixes huge int, fixes date tests * Adds GetByExchange, adds test for it. Protects fakepass echange. Renames DisplayQty to DisplayQuantity. Removes verbose. Adds some websocket properties. Updates bitmex asset type in test * Addresses timestamps, type abbreviations, verbosity. Expands binance kline switch cases. Updates some websocket capabilities. * Adds coverage to the stringToOrderType/Status functions introduced in PR * Minor fixes addressing some time, error text and use of StringDataCompareInsensitive * Introduces shiny new system which checks if there is an awaiting ID, if found, processes via wrapper method, else, goes through wsHandleData method. Removes weird locking system from wrapper/websocket data race. Updates bitfinex to properly handle websocket order requests and notifications * Moves fakePassingExchange to test_helper. Fixes some order side implementations for trades. Botches a new error type * Adds new error type to track and handle order classification errors separately * Fully fleshes out ClassificationError for all instances of status conversion. Even in order trades and some wrapper functions * Introduces common.SimpleTimeFormat for "2006-01-02 15:04:05". Fixes binance and bitfinex issues with auth endpoint use, map casting. Expands more order.ClassificationError usage. Fixes some more generic websocket response errors * Future proofs order updating by utilising asset types. Expands testing to accomodate. Adds shiny new time type. Expands wrapper websocket functionality definitions * minty linty * Broken end of day code addressing basic nits on comments, returns and currency conversion * Adds testing to btcmarkets websocket. Also updates websocket orderbook to use update instead * Fixes fun rebase fun fun so fun * Addresses minor nits regarding changed interface and comments * Creates new function `GetRequestFormattedPairAndAssetType` to retrieve a currency pair and asset type based on a string. It will iterate over enabled pairs and compare them to formatted pairs and then return that pair if found. * Fixes test * Adds a single line to the end of the file, because that would be really bad if it wasn't there * Updates fakepassexchange to not use params, updates test params, uses fatal in some tests where its important, updates order manager to have a rwmutex, removes some returns, improves ws key test for binance, updates properties to reflect their actual values, adds some more websocket properties * Addresses binance switch linting * Updates leverage property to int64 * Fixes what was broken	2020-03-03 13:32:14 +11:00
Ryan O'Hara-Reid	6d8ba0a96a	Initial implementation of HTTP mock testing framework (#310) ... * Initial implementation of HTTP mock testing framework Convert to VCR testing server. Segregate live testing via build tags. Converted Binance to VCR server Convert Bitstamp to VCR mocking tests Added VCR mock testing for localbitcoins * Add server generation for concurrent testing * Fix linter issues * Fix linter issue * fix race - potentially * revert auto assigning of host vals * Fix requested changes * Adds mock testing for ANX Switch to using TestMain functionality Added cron job usage for travis-ci to live testing Added appveyor scheduled build check for live testing * WOOPS * silly correction * Fixes fantastic linter issues * fixed another whoopsie * WOOO! * Adds gemini mock testing with additional fixes * Add docs and sharedvalue * Added tls using httptest package * Fixed issues * added explicit mock recording reference to error * Fix requested changes * strip port from mock files as they are not needed on tls server * Change incorrect names * fix requested changes * lbank update * Fix another issue * Updated readme	2019-08-23 15:20:02 +10:00
Scott	0fbf8b172a	Websocket orderbook buffering (#333) ... * Initial commit setting up a map orderbook system with a buffer. It will write to the buffer, sort apply to main orderbook and then process. * Moves namespaces again * Updates orderbook to use a sweet new WebsocketOrderbookUpdate type to handle all updates whether its using ID or not. So good. Adds many tests * Starting to implement orderbook update handling per exchange. Updates namespaces again. Hopefuylly will find a way to update via ID not timestamp, too many endpoints dont provide update timestamps * Changes orderbookbuffer to use BufferUpdate type instead of orderbook.Base to achieve more functionality and no need for type conversion functions. Updates tests * Updates all instances of ws.orderbook.Update. Simplifies some orderbook logic * Introduces toggleable buffer. Renames orderbooks. Completes implementation for everywhere but OKGroup due to hash calculation * Implements orderbook update for okgroup, but forgets about the orderbook hash checking * Fixes okgroup checksum calculation. Fixes linting issue. Removes redundant Kraken tests. * Introduces sorting toggle and separates from buffer toggle. Uses benchmarks to highlight performance gains * Fixes Gemini rate limit and parsing. Removes comments and fixes typos * Fixes bitfinex orderbook processing * Inbuilt sorting, minor fixes for websocket implementations. Improves test coverage * Adds surprise LakeBTC websocket support * Fixes data race * Fixes rebasing issues due to namespace movements * Addresses PR nits: moves folder namespace from ws to websocket. Removes line spaces in imports. Fixes lakebtc websocket returns and defer fucntions. Fixes comments * Adds poloniex orderook sorting support * Enables bitstamp and hitbtc orderbook sorting. Fixes poloniex's sorting * Renames namespaces and combines monitor and connection into wshandler. Removes unused SPOT const. Changes how orderbook stuff is loaded. It is done in startup with a setup. Removes exchange name from loadsnapshot as well * Removes the connection.go from rebasing issues. Removes error response from functions used in goroutines * Fixes test with exchange name output change * Fixes issues where copy and paste and replace all were used poorly	2019-08-13 09:32:59 +10:00
Scott	e209d85d0d	Websocket request-response correlation (#328) ... * Establishes new websocket functionality. Begins the creation of the websocket request * Adding a wrapper over gorilla websocket connect,send,receive to handle ID messages. Doesn't work * Successfully moved exchange_websocket into its own wshandler namespace. oof * Sets up ZB to use a round trip WS request system * Adds Kraken ID support to subscriptions. Renames duplicate func name UnsubscribeToChannels to RemoveSubscribedChannels. Adds some helper methods in the WebsocketConn to reduce duplicate code. Cleans up ZB implementation * Fixes double locking which caused no websocket data to be read. Fixes requestid for kraken subscriptions * Completes Huobi and Hadax implementation. Extends ZB error handling. Adds GZip support for reading messages * Adds HitBTC support. Adds GetCurrencies, GetSymbols, GetTrades WS funcs. Adds super fun new parameter to GenerateMessageID for Unix and UnixNano * Adds GateIO id support * Adds Coinut support. Prevents nil reference error in constatus when there isnt one * Standardises all Exchange websockets to use the wshandler websocket. Removes the wsRequestMtx as wshandler handles that now. Makes the Dialer a dialer, its not externally referenced that I can see. * Fixes issue with coinut implementation. Updates bitmex currencies. Removes redundant log messages which are used to log messages * Starts testing. Renames files * Adds tests for websocket connection * Reverts request.go change * Linting everything * Fixes rebase issue * Final changes. Fixes variable names, removes log.Debug, removes lines, rearranges test types, removes order correlation websocket type * Final final commit, fixing ZB issues. * Adds traffic alerts where missed. Changes empty struct pointer addresses to nil instead. Removes empty lines * Fixed string conversion * Fixes issue with ZB not sending success codes * Fixes issue with coinut processing due to nonce handling with subscriptions * Fixes issue where ZB test failure was not caught. Removes unnecessary error handling from other ZB tests * Removes unused interface * Renames wshandler.Init() to wshandler.Run() * Updates template file * Capitalises cryptocurrencies in struct. Moves websocketResponseCheckTimeout and websocketResponseMaxLimit into config options. Moves connection configuration to main exchange Setup (where appropriate). Reverts currencylastupdated ticks. Improves reader close error checking * Fixes two inconsistent websocket delay times * Creates a default variable for websocket ResponseMaxLimit and ResponseCheckTimeout, then applies it to setdefaults and all tests * Updates exchange template to set and use default websocket response limits	2019-08-07 15:15:01 +10:00
Scott	3a66e99899	Authenticated Websocket support (#315) ... * Improves subscribing by not allowing duplicates. Adds bitmex auth support * Adds coinbase pro support. Partial BTCC support. Adds WebsocketAuthenticatedEndpointsSupported websocket feature. Adds GateIO support * Adds Coinut support * Moves Coinut WS types to file. Implements Gemini's secure WS endpoint * Adds HitBTC ws authenticated support. Fixes var names * Adds huobi and hadax authenticated websocket support * Adds auth to okgroup (okex, okcoin). Fixes some linting * Adds Poloniex support * Adds ZB support * Adds proper bitmex support * Improves bitfinex support, improves websocket functionality definitions * Fixes coinbasepro auth * Tests all endpoints * go formatting, importing, linting run * Adds wrapper supports * General clean up. Data race destruction * Improves testing on all exchanges except ZB * Fixes ZB hashing, parsing and tests * minor nits before someone else sees them <_< * Fixes some nits pertaining to variable usage, comments, typos and rate limiting * Addresses nits regarding types and test responses where applicable * fmt import * Fixes linting issues * No longer returns an error on failure to authenticate, just logs. Adds new AuthenticatedWebsocketAPISupport config value to allow a user to seperate auth from REST and WS. Prevents WS auth if AuthenticatedWebsocketAPISupport is false, adds additional login check 'CanUseAuthenticatedEndpoints' for when login only occurs once (not per request). Removes unnecessary time.Sleeps from code. Moves WS auth error logic to auth function so that wrappers can get involved in all the auth fun. New-fandangled shared test package, used exclusively in testing, will be the store of all the constant boilerplate things like timeout values. Moves WS test setup function to only run once when there are multiple WS endpoint tests. Cleans up some struct types * Increases test coverage with tests for config.areAuthenticatedCredentialsValid config.CheckExchangeConfigValues, exchange.SetAPIKeys, exchange.GetAuthenticatedAPISupport, exchange_websocket.CanUseAuthenticatedEndpoitns and exchange_websocket.SetCanUseAuthenticatedEndpoints. Adds b.Websocket.SetCanUseAuthenticatedEndpoints(false) when bitfinex fails to authenticate Fixes a typo. gofmt and goimport * Trim Test Typos * Reformats various websocket types. Adds more specific error messaging to config.areAuthenticatedCredentialsValid	2019-06-19 13:19:01 +10:00