Commit Graph

38 Commits

Author SHA1 Message Date
Scott
242b02c382 (Engine) Bugfix: Unlocking an unlocked mutex PANIC + Increase dispatcher job capacity via commandline (#371)
* Removes lock unlock timer and instead sets unlocks between getting a nonce and sending a payload. Increases dispatch channel buffer to deal with len(enabledCurrencies) > ~100

* Adds additional comments to help explain the situation

* Fixes bug that could unlock mutex too early

* Fixes LIES where Gemini gets a nonce and then proceeds to declare it doesn't get a nonce causing an unrecoverable lock

* Fun new concept! The creation of a tested timed mutex. Unlocking an unlocked mutex cannot occur and response can be checked to verify whether the mutex was unlocked from timeout or command.

* Adds new cmd parameter "dispatchjobbuffer"

* Expands comments and renames benchmark. Makes `Timer` property private

* Happy little linters

* Renames jobBuffer and all related instances to jobs limit

* Tiny error message update

* Grammatical fix and setting dispatch.Start to use defaults
2019-10-29 14:00:45 +11:00
Scott
ccfcdf26aa Engine: Protocol Features, coverage, types, BTC markets websocket (#368)
* Attempts to update orderbook so it doesn't need to sort

* Reverts the ws ob stuff. Gets rid of sorting because it happens later. Adds some exchange features

* update existing feature lists. Expands list definition to match my emotions

* Adds bithumb bitmex and bitstamp. adds a couple more types

* Features for you, features for me, features for bittrex, btcmarkets, btse, coinbasepro, coinut, exmo, gateio and gemini

* Features for hitbtc, huobi, itbit, kraken, lakebtc, lbank, localbitcoins, okcoin, okex, poloniex, yobit, zb

* Who can forget good old alphapoint?

* Adds btcmarksets websocket :glitch_crab: fixes alphapoint features

* Adds extra data not in the documentation :/

* Replaces websocket features by using protocol features. However, it breaks it due to import cycles. I'm not sure what I'll do just yet

* Removes import cycle via duplicate structs.

* Increases coverage of config with `TestCheckCurrencyConfigValues`. Moves all currency pair package types into their own files or places it at the bottom of files if necessary

* Increase coverage in code.go

* One way of determining a test has failed, is when to it fails. Removed redundant explanation

* Increases code coverage of conversion

* Lint fixes

* Fixes orderbook tests

* Re-adds sorting because its important to still have the internal pre-processed orderbook to be representative of a real orderbook

* Secret lints that did not show up via Windows linting

* Adds protocol package to contain exchange features

* Fixes protocol implementation

* Fixes ws tests

* Addresses the following: Removes st-st-stutters in config types, changes GetAvailableForexProviders -> GetSupportedForexProviders, removes errors from tests where error is nil, removes orderbook setup when not necessary, removes import newlines, removes false bools from declaration, changes should of to should have

* imports and casing

* Fixes two more nil error checks
2019-10-22 10:56:20 +11:00
Scott
62a528a064 Bugfix/improvement: Orderbook/ticker update processing (#364)
* Greatly increases efficiency of web socket orderbook processing by minimising multiple map lookups. Changes buffer to use pointers. Ensures orderbook benchmarks are on equal footing and set correctly. Removes data race by not setting waitgroup adds inside go routine causing wait and add to happen simultaneously. Updates ticker and orderbook service to use a pointer var instead of map lookups when setting data.

* Removes misguided comment

* Removes waitgroups and goroutines for updating bids and asks for non-id orderbook updates via websocket. Updates benchmarks to be consistent
2019-10-04 16:24:29 +10:00
Ryan O'Hara-Reid
db317a2447 Engine: Dispatch service (#346)
* Added dispatch service

* Added orderbook streaming capabilities

* Assigned correct orderbook.base exchange name

* Fixed Requested niterinos
Add in cli orderbook QA tool to gctcli
Add exchange orderbook streaming

* Add ticker streaming support through dispatch package

* Added in some more info on error returns for orderbook.go

* fix linter issues

* Fix some issues

* Update

* Fix requested

* move dispatch out of exchanges folder to its own independant folder

* Fix requested

* change orderbook string to tickers

* Limit orderbooks to 50 and made dispatch system more stateless in operation

* lower cases for update/retrieve/sub exchange name

* Adds in asset validation and lower case conversion on cli

* Remove comment

* Moved timer to a higher scope so its not constantly initialised just reset per instance and removed returning unused channel on error

* Rm unused release function in dispatch.go
Reset timer and bleed buffered timer chan if needed in dispatch.go
Added in ticker.Stop() and timer.Stop() functions for worker routine return in dispatch.go
Index aggregated bid and ask functions for orderbook.go
Added in dummy slice for wsorderbook_test.go

* Moved drain to above Reset so potential race would not occur in dispatch.go
Fix various linter issues dispatch.go

* Fix some issues

* change to start/stop service, added in service state change via cli, updated logger

* fix requested

* Add worker amount init spawning

* fix linter issues

* Fix more linter issues

* More fixes

* Fix race issue on releasing pipe channel on a close after shutting down dispatcher system

* Moved all types to dispatch_types.go && remove panic

* Moved types into serperate file && improve test coverage

* RM unnecessary select case for draining channel && fixed error string

* Added orderbook_types file and improved code coverage

* gofmt file

* reinstated select cases on drain because I am silly

* Remove error for drop worker

* Added more test cases

* not checking error issue fix

* remove func causing race in test, this has required protection via an exported function

* set Gemini websocket orderbook exchange name
2019-10-03 09:47:37 +10:00
Adrian Gallagher
132d2d03c2 Merge branch 'master' into engine 2019-09-24 17:05:58 +10:00
Adrian Gallagher
bbdf532b16 Update Slack shared invite link 2019-09-24 17:01:32 +10:00
Scott
74d5bca03b engine: Ticker REST/Websocket improvements (#345)
* Adds extra properties to Websocket ticker. Adds new properties to binance and bitfinex, but doesn't test anything yet. Changes names and properties of ticker package to make more streamlined

* Adds support for coinbasepro, coinut, gateio, gitbtc, huobi, hadax, kraken, okex, okcoin. Adds quoteVolume

* Adds poloniex and ZB ticker datas

* Updates ANX, Binance, Bitfinex, Bistamp, Bittrex, BTCMarkets, BTSE, CoinbasePRo, Coinut, Exmo tickers. It looks like a whole bunch of stuff is wrong in how tickers are done though :/

* Updates tickers everywhere. Will revert batch ones

* Re-Preseves ticker batching

* Minor fixes to ticks and removal of comment

* Logging errors instead of returning mid loop. Adds bitfinex batch ticker processing. Fixes unrelated okgroup wallet bug

* Removes bad code I wrote preventing function from running if feature not enabled

* Fixes issue with bitmex and rebase issues

* Fixes bitmex iterator error, splits hitbtc ticker requests

* Fixes okgroup currency pair formatting. Updates okgroup to use ticker batching. Fixes okgroup ticker issues due to assetTypes. Fixes okgroup ws pinging. Fixes Kraken's currency pairs formatting. Reverts ANXs auto parsing back to strings because ANX json makes me cry. Minor property improvements for coinut, coinbasepro, btse, exmo. Protects wshandler manageSubscriptions() from running and returning an error when feature not supported

* Updates config example to reflect the underscore dash situation

* Fixes a config delimiter oopsie. Simplifies ANX wrapper ticker parsing. Fixes bittrex date parsing. Simplifies okcoin switch to if.

* Fixes super fun issue where kraken has updated their currency pair format and must add new delimiter support. Fixes super fun issue where okex has updated their currency pair format and must add new delimiter support. Fixes super fun issue where okcoin has updated their currency pair format and must add new delimiter support.

* Updates config example for kraken

* Adds lbank batch ticker support. Adds details to errors

* Updates FetchTradablePairs to use the config delimiter to prevent issues if the delimiter ever changes

* Fixes nil reference bug. Uses NAme not GetNAme

* Fixes hardcoded delimiter in Binance.  Expands bitfinex websocket ticker fields. Updates Bitstamp rate limits. Expands BTSE ticker data fields. Fixes typo in coibasepro. Expands Coinut ticker data. Renames currency to curr as it conflicts with package name. Expands GateIO websocket ticker data. Fixes ticker data implementation for huobi and huobi hadax. Reverts ticker map to string instead of assetType. Fixes real stupid bug I introduced which preveted subscriptions from running :glitch_crab: Adds Price ATH to ws ticker type. Adds quotevolume to yobit ticker. Uses delimiter in ZB rather than hardcoded field.

* Fixes bug in syncer where if the websocket is already connected, then UsingWebsocket is not set

* Updates broken tests

* Simplifies poloniex frozen check

* Updates configtest.json with new delimiters

* Renames shorthand properties of structs. Fixes delimiter referencing

* Fixes some bugs and nits around variable declaration, currency pairs and config upscaling

* Adds config upgrade path for okcoin and okex. Reverts configtest.json.

* Fixes okex futures currency formatting by no longer using global currency format. updates Run code to adapt. Removes BTSE value

* Adds ":" as a delimiter for when a delimiter only shows up SOMETIMES

* Adds support for optional delimiter
2019-09-02 18:05:09 +10:00
Adrian Gallagher
8ecd8a0c16 Merge branch 'master' into engine 2019-08-09 15:46:24 +10:00
Adrian Gallagher
2078ba907f Update URLs for transfer into org repo (#338)
* Update URLs for transfer into org repo

* Update codecov, travis and othe remaining links

* Update appveyor paths
2019-08-09 11:50:39 +10:00
Andrew
3de1d94e5f New logging system (#319)
* First pass at adding new logging system

* NewLogger

* NewLogger

* WIP

* silly bug fix

* :D removed files

* removed old logging interface

* added tests

* added tests

* Started to add new lines to all f calls

* Added subsystem log types

* Logger improvements

* Further performance improvements

* changes to logger and sublogger creation

* Renamed Logging types

* removed old print statement

* changes based on feedback

* moved sublogger types to own file

* :)

* added console as output type

* added get level command

* added get/set log level via grpc command

* added check for output being empty for migration support

* first pass at log rotation

* added log rotation

* :D derp fixed

* added tests

* changes based on feedback

* changed log type

* comments

* renamed file -> fileSettings

* typo fix

* changes based on feedback

* gofmt ran on additional files

* gofmt ran on additional files
2019-07-07 05:20:31 +10:00
Adrian Gallagher
4a41c5cff3 Merge branch 'master' into engine 2019-06-17 14:36:06 +10:00
Adrian Gallagher
67a58a10bd Fix ticker empty err and improve test/consistency for orderbook/ticke… (#317)
* Fix ticker empty err and improve test/consistency for orderbook/ticker pkgs
2019-06-17 13:43:13 +10:00
Ryan O'Hara-Reid
20c24601fb Asset update to fix minor stutter (#316) 2019-06-17 09:02:07 +10:00
Adrian Gallagher
e965e54e09 Split up common.go, file path fixes and much more 2019-06-04 17:04:32 +10:00
Adrian Gallagher
6ebafb0f33 Merge branch 'master' into engine 2019-06-04 12:25:40 +10:00
Adrian Gallagher
8048962b60 Remove BTCC exchange 2019-06-03 17:17:17 +10:00
Adrian Gallagher
f5914e8c10 Engine changes 2019-05-22 17:06:38 +10:00
Andrew
ca55f2f965 Optimisation: Large struct pointer conversion (final part) (#265)
Completes large struct pointer optomisations over the entire codebase and enables hugeParams linter by default
2019-04-04 15:31:49 +11:00
Ryan O'Hara-Reid
0990f9d118 Currency package update (#247)
* Initial currency overhaul before service system implementation

* Remove redundant currency string in orderbook.Base
Unexport lastupdated field in orderbook.Base as it was being instantiated multiple times
Add error handling for process orderbook

*  Remove redundant currency string in ticker.Price
 Unexport lastupdated field in ticker.Price
 Add error handling for process ticker function and fix tests

* Phase Two Update

* Update translations to use map type - thankyou to kempeng for spotting this

* Change pair method name from Display -> Format for better readability

* Fixes misspelling and tests

* Implement requested changes from GloriousCode

* Remove reduntant function and streamlined return in currency_translation.go

* Revert pair method naming conventions

* Change currency naming conventions

* Changed code type to exported Item type with underlying string to reduce complexity

* Added interim orderbook process method to orderbook.Base type

* Changed feebuilder struct field to currency.Pair

* Adds fall over system for backup fx providers

* deprecate function and children and fix linter issue with btcmarkets

* Fixed requested changes

* Fix bug and move mtx for rates

* Fixed after rebase oopsies

* Fix linter issues

* Fixes race conditions in testing functions

* Final phase coinmarketcap update

* fix linter issues

* Implement requested changes

* Adds configuration variables to increase/decrease time durations between updating currency file and fetching new currency rates

* Add a collection of tests to improve codecov

* After rebase oopsy fixes for btse

* Fix requested changes

* fix after rebase oopsies and add more efficient comparison checks within currency pair

* Fix linter issues
2019-03-19 11:49:05 +11:00
Adrian Gallagher
7dcb1ab553 Migrate from gometalinter.v2 to golangci-lint (#249)
* Migrate from gometalinter.v2 to golangci-lint
2019-03-01 16:10:29 +11:00
Adrian Gallagher
928a332220 Use official Slack join link instead of herokuapp 2019-02-12 13:45:43 +11:00
Adrian Gallagher
1f2516dbd0 ProcessTicker: added a condtion to check for existing tickers
In line with https://github.com/thrasher-/gocryptotrader/pull/197/

Thanks to @shazbert and @andreygrehov
2018-10-31 11:46:49 +11:00
Adrian Gallagher
96cbf90446 Remove non-needed getter functions for currency pairs
Fixes https://github.com/thrasher-/gocryptotrader/issues/180
2018-09-20 15:01:11 +10:00
Ryan O'Hara-Reid
3b8591bcc8 Updated documentation tool (#155)
* Updated documentation tool
Added templates
Updated documentation using tool

* Fixed incorrect version in web README

* Added new templates to tool.
Updated documents in templates across the code base.
Used tool to regenerate documentation.
2018-07-19 16:02:24 +10:00
Adrian Gallagher
e5b3ce8de8 Use WebsocketEvent in ws client and add timestamp to Tickers
Fixes https://github.com/thrasher-/gocryptotrader/issues/137
2018-07-13 11:20:29 +10:00
Ryan O'Hara-Reid
dc11f87b8b Updated README template files and regenerated documentation. (#125) 2018-05-14 10:03:17 +10:00
Adrian Gallagher
ac41a7cfad New features and bug fixes
- Modifications made to the request package. Planned improvements will be
sending requests on intervals, rate limiter back off support, dynamic tuning
and requests packaged into a request job group.
- Can modify each exchanges individual HTTP client (e.g timeout and
transport settings).
- Bot now uses an exchange config HTTP timeout value.
- Bot now uses a global HTTP timeout (configurable).
- Batched ticker request support for exchanges.
- Ticker and Orderbook fetching now are spanned accross multiple
go routines and regulated by a sync wait group.
- Fixes hack used to load exchanges, now uses a sync wait group.
- Ticker and Orderbook storage and fetching now uses mutex locks.
- New pair function for finding different pairs between two supplied
 pair arrays. This is used for currency pair updates for exchange which
support dynamic updating.
- Shows removal/additions of dynamic updates currencies.
2018-05-04 13:20:19 +10:00
Ryan O'Hara-Reid
9514754960 New documents generated by documentation tool. 2018-02-28 09:55:11 +11:00
Ryan O'Hara-Reid
dd00eba27e Applied fix to documentation.go for contribution HTML_URL.
Added period to documentation templates sentences.
Added logic to documentation.go to fix broken links for godoc in sub-packages.
Fix coding style to conform to golang idiomatic practice.
Applied fix to access main godocs in tools and test data.
Generated new documents using tools.
2018-02-20 11:56:25 +11:00
Ryan O'Hara-Reid
6cb17bb97e Added new base documentation using documentation tool 2018-02-20 10:43:59 +11:00
Adrian Gallagher
87633c2142 Link up orderbook websocket code, improve exchange test coverage and various other fixes 2017-09-14 15:07:59 +10:00
Adrian Gallagher
913c104d09 Finish off websocket ticker demo, update config files and fix tests 2017-09-14 15:04:38 +10:00
Ryan O'Hara-Reid
3e4fb1660d Fixed test race conditions 2017-07-31 11:44:12 +10:00
Adrian Gallagher
77ca9cc2b7 Various fixes 2017-04-21 21:35:26 +10:00
Adrian Gallagher
fb6d966007 Merge branch 'test'
Merge current test branch progress
2017-04-19 11:17:10 +10:00
Adrian Gallagher
d526e9f2a9 Add currency pair support 2017-04-19 00:16:53 +10:00
Ryan O'Hara-Reid
9a6484e113 Added ticker_test.go 2017-04-11 22:00:20 +10:00
Adrian Gallagher
aeb327c80c Package exchanges 2017-03-29 13:47:56 +11:00