* initial update of currency system
* WIP progress
* Finish initial currency string error returns
* fix whoopsie testing for non https insecureinos
* Current WIP for getEnabledPairs check and error return
* WIP continued
* When getting enabled pairs throw error when item is not contained in available pairs list
* More updates -WIP
* Wip continued including potential interface
* Current WIP
* pairs manager pass
* drop asset string and just use the map key, plus return some errors and create more work for myself.
* clean and fixed a bug in currency.json, will not populate correctly without coinmarketcap api keys set.
* purge logger references after merge
* go mod tidy after merge
* Pointer change WIP
* fix some issues and added error returns to a few items (WIP)
* WIP
* Clean
* Fix some linter issues
* Fix more linter issues
* even more linters
* xtda nits
* revert pointer change and rm field
* Addr madcozbadd nits
* fix linter issues: shadow declarations
* Fix linter issues: gocritic huge things
* linter issue fix
* Addr nits
* flush go mod files
* after merge woops
* fix shadow dec
* Addr thrasher nits
* addr nits
* fix some issues
* more fixes
* RM println
* Addr glorious nits
* Add helper method for setting assets
* add missing format directive
* Addr nits
* Actually process new futures contracts -_- derp
* WIP for GRPC upgrade for pair management
* update config pairs
* finished disabling and enabling asset
* linked update of tradable pairs to cli
* fix oopsies
* defer writing to file on program termination for currency storage system
* update template
* don't add disabled asset items to initial sync
* Fix enable disabling a list of pairs and added in a slice error type so we can add whats allowable without throwing an error and return a report, also addressed some other nits
* WIP on getting a channel to unsub
* Wip track down unsubscribe bug and start creating streaming interface
* purge websocket orderbook object and centralised updating routine for orderbook
* general clean before interface implementation
* stage one connection interface WIP
* WIP
* repackage wshandler WIP
* find difference of subs and change signature of subscriber functions so we can batch subscriptions and unsubscription in exchanges that support it
* design change on mange subscription routine WIP
* integrated ZB with the new webosocket updates
* WIP - okex conversion
* integrate websocket upgrades for lakebtc, kraken, huobi, hitbtc, gateio, and WIP for coinbene
* integrate another range of exchanges for websocket update
* Added subscriber and unsubscriber methods to websocket functionality
* fixed tests WIP
* amalgamate cache setup with main websocket setup
* reinstate exported fields traffic alert and shutdownC to accommodate gemini and lakebtc implementations
* added in colon
* Updated websocket auth handling as they werent getting passed through. Added a setter method for websocket URL due to the Binance generated auth key/listenKey. Fixed bug which stopped reconnection.
* Fix subscribe candle bug
Fix time conversion in candle
Fix inititial candle history to datahandler
Include funding to orderbook handling
Include funding to trades
Reduce code duplication in sub and unsub functions
Added the ability to include funding currency websocket subscriptions
validated all channels and added more items todo list (Auth items)
* RM line
* bitstamp pass
* btcmarkets pass - still needs to implement unsubscriber functionality and pairs change test.
* Batch outgoing subscriptions and fix unsubscribe bug
* BTSE - bumped time to minute to reduce pinger calling by 75 calls per day. Fix authentication bug and add authentication pass into to-do. Batch outgoing subscription calls
* fix type field and batch outgoing subs and unsubs for coinbasepro
* Batch outgoing subs and unsubs
* Fixes bug when matching return from authentication
* Fix bug where params where being sent out of order due to map ,where depth items werent being subscribed too, where trying to subscribe to too many kline items caused error, where trying to get a nano secocond ID conflicted due to speed of generation.
* Add websocket capability for currency pair change by utilizing full channel subscription list in subscribe function.
* Add error handling
* Fix public: time conversions, subscription list, stopped pushing heartbeat to data handler, aggregated list of connections.
* hitBTC pass
* returned nil instead of error due to period null bids and asks updates coming through.
* Fix auth ping capture and reply. Added in interval handling for kline data. Added correct full trade data handling. Fix subscribe and unsubscribe.
* Fix when websocket auth conn and token generation fail we don't try and auth sub. Fix bug between auth and normal connection id generation and matching. Batch outgoing payloads to increase efficiency. Updated matching functions to utilise channels instead of waitgroups and go routines.
* RM debug output
* rm func to get shutdown channel
* Add unsubscriber functionality, added wsTicker type, removed return as this will impede data flow and cause reconnection when handling and processing data
* okgroup WIP
* *Added missing fields for websocket trades
*Fix bug processing kline interval
*Added fields for websocket ticker struct
*Fix auth bug
-Updated request and response matching param to interface so we can custom signature match. Stops auth subscribing before a reply is issued.
-Updated channel inclusion of pair fo auth subs as this was missing.
*Assortment of perfomance improvements
* poloniex pass
* send all trades to data handler, validated enabled and disable pairs
* initial clean
* centralised request matching mechanism
* websocket main improvements WIP
* WIP
* Websocket management via gctcli WIP
* GRPC expansion
* Updated GCTCLI with websocket url and proxy setting functionality which flushes connection
* Fix continuous spawning of routines bug on error with reconnection
* Addr linter issues
* fix subscription bug that I caused when I changed to a switch case
* fix linter issue
* fix woopsie
* End of day WIP
* Fix order submission REST, time conversion, order type conversion, orderID bugs
* fix gateio test and unsubscribe bug
* revert comment out code
* websocketAPI changed to to true in configtest.json
* fix race in gateio test
* End of day WIP for websocket tests.
* BugFix for binance when book isn't seeded. Updated websocket tests. Deprecated subscription manager. RM wrapper funcs.
* Added string title to exchange name as they are saved as lower case in type, reinstated verbose check in websocket.go
* Added verbosity check for setting websocket URL
* fix bug where the asset had a mind of its own
* purge dodgy coding
* Fix tests, drop blocking chan in websocket Dial function
* few more changes
* race condition fix for websocket tests.
* fix intermittent test failure due to underlying hash table storage
* Address madcuzbad nit
* RM superfluous printlines
* Add quick top example with paramater fields
* First pass Glorious nits
* As per madcozbad suggestion return error when enabled pair not found in full return map. Add test.
* addr madcozbadd nits
* as per glorious suggestion rm'd loadedJSON field
* adjusted ticker, added test and RM'd code that can never be executed
* Addr nits and add in crypto rand genration for ID's
* remove global channel declaration and rescoped as this was causing a lock
* as per glorious suggestion restructured return error for websocket
* addr glorious suggestions
* fix linter issues
* purge non-existent pair from testdata
* add side field to struct and parse
* addr glorious nits
* Add verbosity to error returns and logs and fix string parsing in GCTRPC
* fix speeling mistwake
* Adds websocket functionality check before flushing websocket connection
* Addr kraken panic and setting/flush websocket url stage one.
* added websocket url check before setting with tests
* Added in edge case test if by the time we call contains on available pairs it has been changed
* remove error return for func
* Continuation of tests
* continuation of tests
* Stop potential panic within pair creation function
* Implement changes to upstream
* rm sup comment
* fix bug when subscribing and unsubscribing. Also add in boolean to determine there are currencies that need to be flushed via set pairs via gctcli
* fix test
* Fix linter issues
* Fix tests
* turn websocket off in config example
* Fix issue where you cannot enable websocket when config is set to false, also added config websocket enable state saving
* Introduced err var for same error returns
* Add err var exchange base not found
* restructure function
* drop gctscript from generic response name
* drop managesub delay const as its not being used
* correctly implement websocket rate limiting for coinut
* remove quotations
* drop pair management check
* fix spelling
* return error in function to not update currency with unset role
* amalagamted enable/disable into set function and added in pairstore fetch function
* update error description
* rm function
* moved test function to sharedtestvals and move type to types.go
* append delimiter onto currency delimiter strings
* add test coverage
* rm functions as they are set as methods in base
* remove superfluous methods
* Fix issue that would occur when a subscription errored and not appending successful subs
* fix after rebase woopsie
* fix linter issues
* fix bug streamline code
* fix linter issues
* fix linter issues
* fix case where it should not change ID if set but append new
* fix whoopsie
* fix websocket tests
* fix readme, fix wrapper issues reporting template, go mod tidy
* add test coverage
* add test coverage and verified futures pariing
* add in futures bypass as its not currently supported on BTSE until API update and implementation
* removed downside/upside profit contract type as its no longer supported. Added in check in set config pairs to warn user of potential conflict and to manually remove or update.
* If asset enabled add pair and increase code coverage
* remove strings.title, set and fetch with strings.Lower but keep struct field exchangename unchanged. Streamline ticker and orderbook code.
* Add code coverage
* log error if setting default currency fails, add code coverage
* address glorious nits
* Addr xtda nits
* fix linter issues
* addr glorious nits
* xtda nits
* Addr glorious nits
* add subscription protection and removed a superfluous wait call
* fix test
* fix whoopsie
* addr xtda nits
* addr glorious nits
* Added asset types to subscriptions structs, also added in error handling for resubscription errors
* consolidated rpc returned type and added in sucessful strings
* dropped stream timing down to 100ms
* DOC changes
* proxy and url usage string additions
* WIP
* go mod tidy rides again
* Addr nits
* Addr nits, fix tests
* fix wording
* add in test case for currency matching
* Add byte length check on outbound websocket payload subscriptions
* addr thrasher nits
* Addr madcozbadd nits
* addr linter issues
* Addr glorious nits by amalgamating function into one mega amazing function.
* fix futures account subscription bug
* addr glorious nits and reinstated wg.Wait() checks
* changed string to currency delimiter string and setconnected by function
* renamed package to log to stop side import requirement
* reverted comment changes
* reverted comment changes
* one more reverting wording back to logger
* wording changes on comments
* drop common uuid v4 func and imported package as needed
* removed common functions regarding json marshal and unmarshal and used the json package directly. WRT unmarshal it was calling reflect and converted to string which is also checked in the JSON package so it was doing a double up, this will be a tiny gain as it was directly used in the requester package for all our outbound requests.
* add in string
* explicitly throw away return error value
* atleast return the error that websocket initialise returns
* return error when not connected
* fix comment
* Adds comments
* move package declarations
* drop append whenever we call supported
* remove unused import
* Change incorrect spelling
* fix tests
* fix go import issue
* Initial commit tearing down the websocket connection management. The purpose is to remove the traffic monitoring and dropping as syncer.go is a better manager
* Adds a readwrite mutex and helper functions to minimise inline lock/unlocks and prevent races
* Creates new WebsocketType struct to contain all parameters required. Deletes WebsocketReset. Utilises ReadMessageErrors channel for all websocket readmessages to analyse when an error returned is due to a disconnect
* Fixes issue with syncer trying to connect while connecting
* Simplifies initialisation function for websocket. Reconnects and resubscribes after disconnection
* Adds WebsocketTimeout config value to dictate when the websocket traffic monitor should die. Default to two minutes of no traffic activity. Increases test coverage and updates existing tests to work with new technologic. RE-ADDS TESTS I ACCIDENTALLY DELETED FROM PREVIOUS PR
* Removes snapshot override as its always necessary when considering reconnections. Increases test coverage. Re-adds tests that were ACCIDENTALLY DELETED. Removes unused websocket channels. Bug fix for traffic monitor to shutdown via goroutine instead of killing itself
* Fixes gateio bug for authentication errors when null. Adds little entry to syncer for when websocket is switched to rest and then back, you get a log notifying of the return. Fixes okgroup bug where ws message is sent on a disconnected ws, causing panic. Renames setConnectionStatus to setConnectedStatus. Puts connection monitor log behind verbose bool
* Fixes lingering races. Fixes bug where websocket was enabled whether you liked it or not. Removes demonstration test
* Fixes log message, renames unc, removes comments
* Fixes data race
* Removes verbosity, ensures shutdown sets connection status appropriately
* Removes go routine causing CPU spike. Stops timers properly and resets timers properly
* Renames `WsEnabled` to `Enabled`. Increases test coverage. Fixes typos. Handles unhandled errors
* The forgotten lint
* With using RWlocks, removes the channel nil check and relies on !w.IsConnected() to prevent a shutdown from recurring
* Removes extra closure step in the defer as it causes all the issues
* Prevents timer channel hangups. Minimises use of websocket Connect(). Expands disconnection error definition. Removes routine disconnection error handling. Ensures only one traffic monitor can ever be run. Renames subscriptionLock to subscriptionMutext for consistency
* Extends timeout to 30 seconds to cover for non-popular exchanges and non-popular currencies
* Updates test from rebase to use new websocket setup function
* Fixes test to ensure it tests what it says it does
* Establishes new websocket functionality. Begins the creation of the websocket request
* Adding a wrapper over gorilla websocket connect,send,receive to handle ID messages. Doesn't work
* Successfully moved exchange_websocket into its own wshandler namespace. oof
* Sets up ZB to use a round trip WS request system
* Adds Kraken ID support to subscriptions. Renames duplicate func name UnsubscribeToChannels to RemoveSubscribedChannels. Adds some helper methods in the WebsocketConn to reduce duplicate code. Cleans up ZB implementation
* Fixes double locking which caused no websocket data to be read. Fixes requestid for kraken subscriptions
* Completes Huobi and Hadax implementation. Extends ZB error handling. Adds GZip support for reading messages
* Adds HitBTC support. Adds GetCurrencies, GetSymbols, GetTrades WS funcs. Adds super fun new parameter to GenerateMessageID for Unix and UnixNano
* Adds GateIO id support
* Adds Coinut support. Prevents nil reference error in constatus when there isnt one
* Standardises all Exchange websockets to use the wshandler websocket. Removes the wsRequestMtx as wshandler handles that now. Makes the Dialer a dialer, its not externally referenced that I can see.
* Fixes issue with coinut implementation. Updates bitmex currencies. Removes redundant log messages which are used to log messages
* Starts testing. Renames files
* Adds tests for websocket connection
* Reverts request.go change
* Linting everything
* Fixes rebase issue
* Final changes. Fixes variable names, removes log.Debug, removes lines, rearranges test types, removes order correlation websocket type
* Final final commit, fixing ZB issues.
* Adds traffic alerts where missed. Changes empty struct pointer addresses to nil instead. Removes empty lines
* Fixed string conversion
* Fixes issue with ZB not sending success codes
* Fixes issue with coinut processing due to nonce handling with subscriptions
* Fixes issue where ZB test failure was not caught. Removes unnecessary error handling from other ZB tests
* Removes unused interface
* Renames wshandler.Init() to wshandler.Run()
* Updates template file
* Capitalises cryptocurrencies in struct. Moves websocketResponseCheckTimeout and websocketResponseMaxLimit into config options. Moves connection configuration to main exchange Setup (where appropriate). Reverts currencylastupdated ticks. Improves reader close error checking
* Fixes two inconsistent websocket delay times
* Creates a default variable for websocket ResponseMaxLimit and ResponseCheckTimeout, then applies it to setdefaults and all tests
* Updates exchange template to set and use default websocket response limits
* First pass at adding new logging system
* NewLogger
* NewLogger
* WIP
* silly bug fix
* :D removed files
* removed old logging interface
* added tests
* added tests
* Started to add new lines to all f calls
* Added subsystem log types
* Logger improvements
* Further performance improvements
* changes to logger and sublogger creation
* Renamed Logging types
* removed old print statement
* changes based on feedback
* moved sublogger types to own file
* :)
* added console as output type
* added get level command
* added get/set log level via grpc command
* added check for output being empty for migration support
* first pass at log rotation
* added log rotation
* :D derp fixed
* added tests
* changes based on feedback
* changed log type
* comments
* renamed file -> fileSettings
* typo fix
* changes based on feedback
* gofmt ran on additional files
* gofmt ran on additional files
* Added new base logger
* updated example and test configs
* updated exchange helpers restful router & server
* logPath is now passed to the logger to remove dependency on common package
* updated everything besides exchanges to use new logger
* alphapoint to bitmex done
* updated bitmex bitstamp bittrex btcc and also performance changes to logger
* btcmarkets coinbase coinut exmo gateio wrappers updated
* gateio and gemini logger updated
* hitbtc huobi itbit & kraken updated
* All exchanges updatd
* return correct error for disabled websocket
* don't disconnect client on invalid json
* updated router internal logging
* log.Fatal to t.Error for tests
* Changed from fatal to error failure to set maxprocs
* output ANSI codes for everything but windows for now due to lack of windows support
* added error handling to logger and unit tests
* clear wording on print -> log.print
* added benchmark test
* cleaned up import sections
* Updated logger based on PR requests (added default config options on failure/setting errors)
* ah this should fix travici enc config issue
* Load entire config and clear out logging to hopefully fix travisci issue
* wording & test error handling
* fixed formatting issues based on feedback
* fixed formatting issues based on feedback
* changed CheckDir to use mkdirall instead of mkdir and other changes based on feedback
* Initial overhaul of websocket connection and feeds
* Added proxy support
* Piped to routines.go
* Added new websocket file in exchanges
Refactored orderbook handling into exchange_websocket.go
Added better error responses for binance_websocket.go
General clean for binance_websocket.go
* General fixes - bitfinex_websocket.go
Refactored orderbook cache code - bitfinex_websocket.go
Removed fatal error with unhandled type - routines.go
* Added general improvements to bitmex_websocket.go
Refactored orderbook handling to exchange_websocket.go
Added variable in Item struct in orderbook.go for looking up orders by ID
* Fix issue when routines are blocked due to Data Handler not started
Updated traffic handler
General fixes for bitstamp_websocket.go
* General fixes for coinbasepro_websocket.go
* General fixes for coinut_websocket.go
Fixed error return in exchange_websocket.go
* Removed comments in coinut_wrapper.go
Refactor orderbook logic from hitbtc_websocket.go to exchange_websocket.go
* General fixes
* Removed comments
General fixes
* Updated routines.go
* After rebase fix
* Fixed update config pairs in okcoin.go
* fixed config currency issue in okcoin.go for okcoin China
* exchange_websocket.go
*Removed unused const dec
*Removed state change routine
*Improved trafficMonitor routine
*Increased verbosity for error returns
*Removed uneeded mutex locks
exchange_websocket_test.go
*Added new tests for websocket and orderbook updating
routines.go
*Removed string cased
* Fixed race conditions on sync.waitgroup in exchanges_websocket.go
* Changes variable name in config.go
* Removes unnecessary comment
* Removes indefinite lock on error return
* Removes unnecessary comment
* Adds support for BTCC websocket
Drops support for BTCC REST
* Rewords comment in exchange_websocket.go
Moves types to poloniex_types.go
* Moves types to coinut_types.go
* Removes uneeded range for accessing array variables for coinbase_websocket.go
Removes comments in coinut_types.go
* Adds verbosity flag to GCT
Suppresses verbose output from routines.go
* Fixes setting proxy for REST and Websocket per exchange
Upgrades error handling
Drops unused *url.Url variable in exchange type
* Adds test for setting proxy
* Fixes bug that closes connection due to incorrect timeout time through a proxy connection
* Clarify verbose flag message
