Commit Graph

1058 Commits

Author SHA1 Message Date
Adrian Gallagher
132d2d03c2 Merge branch 'master' into engine 2019-09-24 17:05:58 +10:00
Adrian Gallagher
bbdf532b16 Update Slack shared invite link 2019-09-24 17:01:32 +10:00
Adrian Gallagher
7c49ada7f6 Add bash/zsh autocomplete files and minor fixes 2019-09-18 10:22:52 +10:00
Adrian Gallagher
09d642c381 Expand config test coverage 2019-09-16 17:15:38 +10:00
Adrian Gallagher
870fd6f4f3 Increase config test coverage 2019-09-13 16:53:41 +10:00
Adrian Gallagher
7f576d0d4f Merge branch 'master' into engine 2019-09-13 14:29:48 +10:00
Adrian Gallagher
b1ed894d99 Bump golangci-lint to v1.18.0 (#354)
* Bump golangci-lint to v0.18.0

* Bump AppVeyor golangci-lint to v1.18

For science

* Bump golangci deadline

* Fix notifications typo

* Fix grammar

* Update DisableNTPCheck test string comparison
2019-09-13 13:36:52 +10:00
Scott
3bcaa5db96 Fixes infinite loop (#355) 2019-09-13 13:01:28 +10:00
Adrian Gallagher
427f9c9d40 Expand config test coverage 2019-09-12 17:28:05 +10:00
Andrew
ba7e387643 (Engine) Gate-IO Websocket subscription pairs case sensitivity (#352)
* convert pairs to upper case as gateio websocket subscription is case sensative

* Update gateio_websocket.go
2019-09-12 15:00:24 +10:00
Adrian Gallagher
6579196c57 Daily engine changes
Expand config test coverage
2019-09-11 18:00:31 +10:00
Adrian Gallagher
e8b517ef0a Daily engine changes
1) Although gRPC does server side validation currently, validate basic things
on gctcli before relaying the request to the gRPC server
2) Make pair format consistent for the exchange sycner
3) Fix OKEX ticker failure due to thinking futures info is authenticated
4) Start filling out config tests
5) Extend timeout for golangci config so that AppVeyor has time to
complete (Travis is fine)
6) Add IsSupported exchange func for easy lookup
2019-09-10 17:07:00 +10:00
Scott
0824ee04c9 Engine: bugfix: OKEX Future swap panic (#350)
* Ensures that futures, index and swap currencies are properly set on OKEX startup with an existing engine config that hasn't been updated yet

* Removes package/variable CLASH
2019-09-10 10:53:25 +10:00
Adrian Gallagher
0fcf867697 Add support for get/set/rm of exchange pairs via gRPC 2019-09-06 11:47:58 +10:00
Adrian Gallagher
c041c990a9 Improve ticker batching and websocket pair consistency 2019-09-06 11:24:38 +10:00
Adrian Gallagher
2307834fb6 Minor fixes
1) Fixes go fmt issue
2) Removes TO-DO item for supported asset pairs for GetExchangeInfo API
3) Fixes useless select used for testing
2019-09-03 10:32:17 +10:00
Adrian Gallagher
939953211d Merge branch 'master' into engine 2019-09-03 08:57:21 +10:00
Adrian Gallagher
5bd80fc758 Merge branch 'engine' of https://github.com/thrasher-corp/gocryptotrader into engine 2019-09-02 18:05:49 +10:00
Scott
74d5bca03b engine: Ticker REST/Websocket improvements (#345)
* Adds extra properties to Websocket ticker. Adds new properties to binance and bitfinex, but doesn't test anything yet. Changes names and properties of ticker package to make more streamlined

* Adds support for coinbasepro, coinut, gateio, gitbtc, huobi, hadax, kraken, okex, okcoin. Adds quoteVolume

* Adds poloniex and ZB ticker datas

* Updates ANX, Binance, Bitfinex, Bistamp, Bittrex, BTCMarkets, BTSE, CoinbasePRo, Coinut, Exmo tickers. It looks like a whole bunch of stuff is wrong in how tickers are done though :/

* Updates tickers everywhere. Will revert batch ones

* Re-Preseves ticker batching

* Minor fixes to ticks and removal of comment

* Logging errors instead of returning mid loop. Adds bitfinex batch ticker processing. Fixes unrelated okgroup wallet bug

* Removes bad code I wrote preventing function from running if feature not enabled

* Fixes issue with bitmex and rebase issues

* Fixes bitmex iterator error, splits hitbtc ticker requests

* Fixes okgroup currency pair formatting. Updates okgroup to use ticker batching. Fixes okgroup ticker issues due to assetTypes. Fixes okgroup ws pinging. Fixes Kraken's currency pairs formatting. Reverts ANXs auto parsing back to strings because ANX json makes me cry. Minor property improvements for coinut, coinbasepro, btse, exmo. Protects wshandler manageSubscriptions() from running and returning an error when feature not supported

* Updates config example to reflect the underscore dash situation

* Fixes a config delimiter oopsie. Simplifies ANX wrapper ticker parsing. Fixes bittrex date parsing. Simplifies okcoin switch to if.

* Fixes super fun issue where kraken has updated their currency pair format and must add new delimiter support. Fixes super fun issue where okex has updated their currency pair format and must add new delimiter support. Fixes super fun issue where okcoin has updated their currency pair format and must add new delimiter support.

* Updates config example for kraken

* Adds lbank batch ticker support. Adds details to errors

* Updates FetchTradablePairs to use the config delimiter to prevent issues if the delimiter ever changes

* Fixes nil reference bug. Uses NAme not GetNAme

* Fixes hardcoded delimiter in Binance.  Expands bitfinex websocket ticker fields. Updates Bitstamp rate limits. Expands BTSE ticker data fields. Fixes typo in coibasepro. Expands Coinut ticker data. Renames currency to curr as it conflicts with package name. Expands GateIO websocket ticker data. Fixes ticker data implementation for huobi and huobi hadax. Reverts ticker map to string instead of assetType. Fixes real stupid bug I introduced which preveted subscriptions from running :glitch_crab: Adds Price ATH to ws ticker type. Adds quotevolume to yobit ticker. Uses delimiter in ZB rather than hardcoded field.

* Fixes bug in syncer where if the websocket is already connected, then UsingWebsocket is not set

* Updates broken tests

* Simplifies poloniex frozen check

* Updates configtest.json with new delimiters

* Renames shorthand properties of structs. Fixes delimiter referencing

* Fixes some bugs and nits around variable declaration, currency pairs and config upscaling

* Adds config upgrade path for okcoin and okex. Reverts configtest.json.

* Fixes okex futures currency formatting by no longer using global currency format. updates Run code to adapt. Removes BTSE value

* Adds ":" as a delimiter for when a delimiter only shows up SOMETIMES

* Adds support for optional delimiter
2019-09-02 18:05:09 +10:00
Andrew
833a866506 (Engine) Logger - fixes issues with multiwriter go routine (#347)
* Fixes goroutine leak and comments

* passed error down from newLogEvent to each sub logger to handle and display

* Update loggers.go
2019-09-02 09:58:23 +10:00
Adrian Gallagher
fb7149a372 BTSE orderbook 2019-08-27 12:06:52 +10:00
Adrian Gallagher
c7ca29a6d7 Remove unnecessary log.Fatals and suboptimal os.Exits (#343)
Engine fixes up the rest for exchange setup loading
2019-08-27 10:47:47 +10:00
Adrian Gallagher
5e9d13f7be Expand RetrieveConfigCurrencyPairs to support different asset types 2019-08-27 08:54:10 +10:00
Adrian Gallagher
c1ad7657b1 Periodic update of available pairs 2019-08-26 15:52:01 +10:00
Ryan O'Hara-Reid
c191dfc666 Bugfix: Bitstamp race condition and ANX format directive issue (#342)
* Fix issue for ANX live testing, incorrect format directive

* Used cpy of Balances as param for Bitstamp function, added space for ANX fatal error.

* Fix linter issue

* Drop struct field for balance and localised a variable in each individual getfee call as needed
2019-08-26 15:06:50 +10:00
Adrian Gallagher
f6afeee800 Minor improvements
1) gen_otp supports single use OTP secrets
2) improve database config check logic
3) reconnect websocket routine to engine (apart from the exchange pair syncer)
4) ImPrOvE CoNsIsTeNcY wItH LoG OuTpUt
2019-08-26 12:46:53 +10:00
Adrian Gallagher
c1f9e04015 Merge branch 'master' into engine 2019-08-23 22:21:01 +10:00
Adrian Gallagher
2dc813b5f3 Merge branch 'master' into engine 2019-08-23 21:59:01 +10:00
Ryan O'Hara-Reid
6d8ba0a96a Initial implementation of HTTP mock testing framework (#310)
* Initial implementation of HTTP mock testing framework

Convert to VCR testing server. Segregate live testing via build tags.

Converted Binance to VCR server

Convert Bitstamp to VCR mocking tests

Added VCR mock testing for localbitcoins

* Add server generation for concurrent testing

* Fix linter issues

* Fix linter issue

* fix race - potentially

* revert auto assigning of host vals

* Fix requested changes

* Adds mock testing for ANX
Switch to using TestMain functionality
Added cron job usage for travis-ci to live testing
Added appveyor scheduled build check for live testing

* WOOPS

* silly correction

* Fixes fantastic linter issues

* fixed another whoopsie

* WOOO!

* Adds gemini mock testing with additional fixes

* Add docs and sharedvalue

* Added tls using httptest package

* Fixed issues

* added explicit mock recording reference to error

* Fix requested changes

* strip port from mock files as they are not needed on tls server

* Change incorrect names

* fix requested changes

* lbank update

* Fix another issue

* Updated readme
2019-08-23 15:20:02 +10:00
Adam
a81ddead9e Add LBank exchange support (#327)
* wip

* Lbank support being added

* Lbank exchange linter issues fixed

* Removed the incomplete websocket stuff

* PR Requests completed

* PR request fixes

* Lbank Update

* Lbank Update

* Wrapper functions fixed, linter issues fixed

* Changed as per requested in PR

* Changed as per requested in PR

* Changed as per requested in PR

* Changed as per requested in PR

* PR Requests completed

* FINALLY DONE

* appveyor issues fixed

* Skip functionality for new tests

* Test functions fixed

* PR Fixes

* PR Fixes

* PR Fixes

* PR Fixes

* Final Changes

* Final Changes 2

* Final Changes 3

* Final Changes 4
2019-08-23 11:28:38 +10:00
Adrian Gallagher
94a784d63f Merge branch 'master' into engine 2019-08-22 18:32:57 +10:00
Adrian Gallagher
4f982dcdc9 BTSE: Fix endpoint issues after API update (#341) 2019-08-21 09:22:31 +10:00
Andrew
b3e8be2c75 NTP out of sync prompt fix if logger is disabled (#340)
* Add check to see if logger is enabled

* don't need an explicit true check here

* removed line
2019-08-21 09:22:14 +10:00
Andrew
0c76789b0d Database interface & auditing feature (#332)
* added audit manager

* Basic database DOA setup

* Added base config file

* added sqlite support and creation of schema

* added basic tests and config entry

* corrected issues of database is disabled

* fixed path for test

* WIP

* Added tests fixed config checking

* reverted files back to upstream

* reverted go.mod files

* no more test test test

* removed local testing details for psql

* hello

* added comments

* increased ping to 30 seconds

* renamed database table and added additional condition around test

* removed database test details

* goimport ran on all files

* WIP

* first attempt at migration

* fixes for migration system

* Migration system logger interface implemented

* fixes to print functions

* added write pooling pass

* gofmt :D

* formatted imports correctly

* removed old code

* added creation of migration

* gofmt

* :D Hello

*  🏎️

* maybe one day i will remember to revert go mod files

* checked err return condition correctly

* first changes for PR feedback

* code clean up

* protect Connected with RWmutex & event with mutex

* : D

* we can just pretend like it never happened

* MOved migrations back to source directory and added README

* readme formatting update

* Addd command line override for datadir

* use correct var when creating a migration and confirm folder is created

* Check if database version is newer than latest migration and also you know make migrations work.....

* uses filepath instead of manual path to use correct path seperator

* Add connection message and lower timeout

* Added support for sslmode for psql

* no longer force Close of database instead allow driver to maage

* Added closer func to test output

* sslmode added to example config
2019-08-20 16:35:06 +10:00
Ryan O'Hara-Reid
21172bf63f Bithumb fixes due to API change (#339)
* Bithumb fixes due to API change

* Fix requested

* Used suggestion
2019-08-20 14:28:46 +10:00
Scott
0fbf8b172a Websocket orderbook buffering (#333)
* Initial commit setting up a map orderbook system with a buffer. It will write to the buffer, sort apply to main orderbook and then process.

* Moves namespaces again

* Updates orderbook to use a sweet new WebsocketOrderbookUpdate type to handle all updates whether its using ID or not. So good. Adds many tests

* Starting to implement orderbook update handling per exchange. Updates namespaces again. Hopefuylly will find a way to update via ID not timestamp, too many endpoints dont provide update timestamps

* Changes orderbookbuffer to use BufferUpdate type instead of orderbook.Base to achieve more functionality and no need for type conversion functions. Updates tests

* Updates all instances of ws.orderbook.Update. Simplifies some orderbook logic

* Introduces toggleable buffer. Renames orderbooks. Completes implementation for everywhere but OKGroup due to hash calculation

* Implements orderbook update for okgroup, but forgets about the orderbook hash checking

* Fixes okgroup checksum calculation. Fixes linting issue. Removes redundant Kraken tests.

* Introduces sorting toggle and separates from buffer toggle. Uses benchmarks to highlight performance gains

* Fixes Gemini rate limit and parsing. Removes comments and fixes typos

* Fixes bitfinex orderbook processing

* Inbuilt sorting, minor fixes for websocket implementations. Improves test coverage

* Adds surprise LakeBTC websocket support

* Fixes data race

* Fixes rebasing issues due to namespace movements

* Addresses PR nits: moves folder namespace from ws to websocket. Removes line spaces in imports. Fixes lakebtc websocket returns and defer fucntions. Fixes comments

* Adds poloniex orderook sorting support

* Enables bitstamp and hitbtc orderbook sorting. Fixes poloniex's sorting

* Renames namespaces and combines monitor and connection into wshandler. Removes unused SPOT const. Changes how orderbook stuff is loaded. It is done in startup with a setup. Removes exchange name from loadsnapshot as well

* Removes the connection.go from rebasing issues. Removes error response from functions used in goroutines

* Fixes test with exchange name output change

* Fixes issues where copy and paste and replace all were used poorly
2019-08-13 09:32:59 +10:00
Adrian Gallagher
b178dd2c1d Linter fixes 2019-08-08 23:02:27 -07:00
Adrian Gallagher
8ecd8a0c16 Merge branch 'master' into engine 2019-08-09 15:46:24 +10:00
Adrian Gallagher
2078ba907f Update URLs for transfer into org repo (#338)
* Update URLs for transfer into org repo

* Update codecov, travis and othe remaining links

* Update appveyor paths
2019-08-09 11:50:39 +10:00
Scott
e209d85d0d Websocket request-response correlation (#328)
* Establishes new websocket functionality. Begins the creation of the websocket request

* Adding a wrapper over gorilla websocket connect,send,receive to handle ID messages. Doesn't work

* Successfully moved exchange_websocket into its own wshandler namespace. oof

* Sets up ZB to use a round trip WS request system

* Adds Kraken ID support to subscriptions. Renames duplicate func name UnsubscribeToChannels to RemoveSubscribedChannels. Adds some helper methods in the WebsocketConn to reduce duplicate code. Cleans up ZB implementation

* Fixes double locking which caused no websocket data to be read. Fixes requestid for kraken subscriptions

* Completes Huobi and Hadax implementation. Extends ZB error handling. Adds GZip support for reading messages

* Adds HitBTC support. Adds GetCurrencies, GetSymbols, GetTrades WS funcs. Adds super fun new parameter to GenerateMessageID for Unix and UnixNano

* Adds GateIO id support

* Adds Coinut support. Prevents nil reference error in constatus when there isnt one

* Standardises all Exchange websockets to use the wshandler websocket. Removes the wsRequestMtx as wshandler handles that now. Makes the Dialer a dialer, its not externally referenced that I can see.

* Fixes issue with coinut implementation. Updates bitmex currencies. Removes redundant log messages which are used to log messages

* Starts testing. Renames files

* Adds tests for websocket connection

* Reverts request.go change

* Linting everything

* Fixes rebase issue

* Final changes. Fixes variable names, removes log.Debug, removes lines, rearranges test types, removes order correlation websocket type

* Final final commit, fixing ZB issues.

* Adds traffic alerts where missed. Changes empty struct pointer addresses to nil instead. Removes empty lines

* Fixed string conversion

* Fixes issue with ZB not sending success codes

* Fixes issue with coinut processing due to nonce handling with subscriptions

* Fixes issue where ZB test failure was not caught. Removes unnecessary error handling from other ZB tests

* Removes unused interface

* Renames wshandler.Init() to wshandler.Run()

* Updates template file

* Capitalises cryptocurrencies in struct. Moves websocketResponseCheckTimeout and websocketResponseMaxLimit into config options. Moves connection configuration to main exchange Setup (where appropriate). Reverts currencylastupdated ticks. Improves reader close error checking

* Fixes two inconsistent websocket delay times

* Creates a default variable for websocket ResponseMaxLimit and ResponseCheckTimeout, then applies it to setdefaults and all tests

* Updates exchange template to set and use default websocket response limits
2019-08-07 15:15:01 +10:00
Adrian Gallagher
e8c9a9a2a4 Minor linter/test fixes after merging master 2019-08-06 16:08:52 +10:00
Adrian Gallagher
23206caffa Merge branch 'master' into engine 2019-08-06 16:05:35 +10:00
Adrian Gallagher
f061bfee20 Merge branch 'master' into engine 2019-08-06 15:26:29 +10:00
Luke Hamilton
6e70f0642a Refactor main.go to improve startup execution flow (#334)
Much easier to work on a project when you can clearly see its execution path.
2019-08-06 15:20:29 +10:00
Adrian Gallagher
20a5ff256e Update codelingo.yaml (#335)
* Update codelingo.yaml

* Import comment-first-word-when-empty again
2019-08-06 15:19:57 +10:00
Morten Sørensen
3f71050881 Bitmex compare initial orderbook snapsnot insensitive (#331) 2019-07-31 15:58:20 +10:00
Scott
eb07c7e495 Implements endpoint GetAggregatedBalance for Hadax too. (#330) 2019-07-19 09:00:54 +02:00
Yunseok
e0aea96a5c Add Huobi GetAggregatedBalance api (#329)
Add Huobi GetAggregatedBalance api that returns the balances of all the
sub-account aggregated.

See also
  - https://huobiapi.github.io/docs/spot/v1/en/#get-the-aggregated-balance-of-all-sub-accounts-of-the-current-user
2019-07-18 05:10:30 +02:00
DirectX
4222b327c6 Fixes to OKEX and Binance WS trades data mapping (#326)
* Fixes to OKEX and Binance WS trades data mapping

* Small improvement using semantic time.Milliscond unit

* gofmt -s file
2019-07-15 13:36:18 +10:00
Adrian Gallagher
db7aa5df65 Merge branch 'master' into engine 2019-07-07 06:06:30 +10:00