Commit Graph

31 Commits

Author SHA1 Message Date
Ryan O'Hara-Reid
22ff33cd54 Engine QA (#367)
* Improved error message when no config is set on startup

* Change inccorect error wording

* bump Bitfinex websocket orderbook return length to max

* temporary fix of incorrect orderbook updates, limit to bid and ask len of 100, will be extended later if needed

* Fixed issue in binance websocket that appended 0 volume bid/ask items

* Fix panic when unmarshalling an empty pair from config

* Add get pair asset method for exchange base
Fix Bitmex orderbook stream
Unbuffer Bitmex orderbook stream

* force syncer to update ticker instead of fetch, which allows a stream

* Fix websocket last price for coinbasepro

* fix websocket ticker for coinut

* Fix websocket orderbook stream Huobi

* increase orderbook depth REST for Huobi

* Fix websocket support and ensure data integrity

* Fix time parsing issue after error checks

* check error, only process enabled currency pairs, signal websocket data processing

* expanded websocket functionality for okgroup

* Add logic to not process zero length slice for orderbooks

* fix websocket ticker only updating enabled and individual book updates

* ZB fixes to order submission/retrieval/cancellation w/ general fixes

* Quiet unnecessary warning

* updated config entry values for REST and websocket (initial hack until I come up with a better solution for asset types)

* Ch GetName function to field access modifyer & rm useless code

* Add in error I missed

* Nits addressed

* some more fixes

* Turned kraken default websocket to true and some small changes

* fixes linter issues

* Ensured okgroup books and sent update through to datahandler. Zb update as well.

* Add test case to get asset type from pair

* Add test for pairs unmarshal

* Add testing and addressed nits

* FIX linter issue

* Addressed Gees nits

* Thanks glorious spotter

* more nitorinos

* Addres even more nits

* Add stringerino 4000

* Fix for panic cause by sort slice out of range, also nits addressed

* fix linter issues

* Changed from function to field access

* Changed from function to field access

* fix for orderbook update panic, removes quick fix - caused by sync item fetching through same protocol

* Add new test and update random generator

* pass in invalid string to future ob fetching, due to futures contract expire and a http 400 error is returned
2019-11-04 15:34:30 +11:00
Scott
242b02c382 (Engine) Bugfix: Unlocking an unlocked mutex PANIC + Increase dispatcher job capacity via commandline (#371)
* Removes lock unlock timer and instead sets unlocks between getting a nonce and sending a payload. Increases dispatch channel buffer to deal with len(enabledCurrencies) > ~100

* Adds additional comments to help explain the situation

* Fixes bug that could unlock mutex too early

* Fixes LIES where Gemini gets a nonce and then proceeds to declare it doesn't get a nonce causing an unrecoverable lock

* Fun new concept! The creation of a tested timed mutex. Unlocking an unlocked mutex cannot occur and response can be checked to verify whether the mutex was unlocked from timeout or command.

* Adds new cmd parameter "dispatchjobbuffer"

* Expands comments and renames benchmark. Makes `Timer` property private

* Happy little linters

* Renames jobBuffer and all related instances to jobs limit

* Tiny error message update

* Grammatical fix and setting dispatch.Start to use defaults
2019-10-29 14:00:45 +11:00
Adrian Gallagher
09d642c381 Expand config test coverage 2019-09-16 17:15:38 +10:00
Adrian Gallagher
2dc813b5f3 Merge branch 'master' into engine 2019-08-23 21:59:01 +10:00
Adrian Gallagher
8ecd8a0c16 Merge branch 'master' into engine 2019-08-09 15:46:24 +10:00
Adrian Gallagher
2078ba907f Update URLs for transfer into org repo (#338)
* Update URLs for transfer into org repo

* Update codecov, travis and othe remaining links

* Update appveyor paths
2019-08-09 11:50:39 +10:00
Andrew
3de1d94e5f New logging system (#319)
* First pass at adding new logging system

* NewLogger

* NewLogger

* WIP

* silly bug fix

* :D removed files

* removed old logging interface

* added tests

* added tests

* Started to add new lines to all f calls

* Added subsystem log types

* Logger improvements

* Further performance improvements

* changes to logger and sublogger creation

* Renamed Logging types

* removed old print statement

* changes based on feedback

* moved sublogger types to own file

* :)

* added console as output type

* added get level command

* added get/set log level via grpc command

* added check for output being empty for migration support

* first pass at log rotation

* added log rotation

* :D derp fixed

* added tests

* changes based on feedback

* changed log type

* comments

* renamed file -> fileSettings

* typo fix

* changes based on feedback

* gofmt ran on additional files

* gofmt ran on additional files
2019-07-07 05:20:31 +10:00
Adrian Gallagher
e965e54e09 Split up common.go, file path fixes and much more 2019-06-04 17:04:32 +10:00
Adrian Gallagher
f5914e8c10 Engine changes 2019-05-22 17:06:38 +10:00
Scott
6c850e73e2 Websocket connection handling and subscription management (#297)
* Step one: Sets up  connection handler for websockets to always be connected until a shutdown event is received.
Sets up a vague subscription handler to ensure subscriptions are subscribed

* Adds support for resubscriptions for bitfinex, bitstamp, bitmex and btcc. Adds subscription params for special websocket subscription requirements. Removes subscription monitor from wait group so that it can exist despite a shutdown and continuously check

* Adds channel subscription support to bitmex, btse, coibasepro, coinut, gateio, gemini, hitbtc, huobi, hadax, kraken, okgroup, poloniex and zb

* Implements unsubscribe for bitfinex, btcc, btse, coinbasepro, gateio, gitbtc, huobi, hadax

* ManageSubscriptions now called from WSConnect and made private instead of inside individual exchanges. ManageSubscriptions can now unsubscribe. exchange_websocket_types.go now contains all exchange_websocket.go types to avoid clutter

* Adds it to websocket functionality so managesubscriptions will close when not supported

* Separates functions into testable functions to ensure logic works. Adds tests. Updates websocket setup to include verbosity (inherited from exchange). Adds no connection tolerance to fatal on failed reconnects

* More exchange_websocket tests. Updating to use pointers. Creation of equals func to make comparison easier

* Fixes okex, okcoin tests. Fixes race conditions. Removes pointer usage again.

* Adds subscribe and unsubscribe to wrappers

* Fixes deadlock. Fixes ws verbosity.

* Updates all exchanges to properly support subscription/connection feature. Also reintroduces race conditions....

* Moves connection varialbes to struct from package to allow each websocket to have their own reconnection checks. Neatens up logs

* Fixes lint/critic issues. Fixes tests. Removes unused function.

* Moves websocket ratelimiter to their own const variables. Fixes more race conditions with connecting variable

* Removes redundant subscribe functions. Ensuring only the exchange_websocket.go can manage subscriptions. Fixes debug logs to be verbose wrapped

* Fixes issue with slice copying. Re-adds okgroup default channels

* Adds nolint to append

* Adds comments and adds support for gateio auth request subscriptions

* Adds new test to ensure slices dont point to the same vars

* removes fatals. gofmt goimports

* more gofmts

* Addresses PR comments, removing empty and redundant lines

* Addresses PR comments. Ensures that writing to the websocket is single-threaded by adding a mutex to exchanges. Minimises wrapper code and moves subscription loops to exchange_websocket. Privatises ChannelsToSubscribe, Connecting properties and removeChannelToSubscribe func to prevent unnecessary tampering.

* Removes unused mutex. FMTS and IMPORTS

* Fixes request lock time change

* More specific logs

* Renames ws mutex. Fixes bitmex subscriptions. Increased gateio ratelimiter to 120ms. Removes ratelimiter from bitfinex, bitmex, bitstamp, btcc, btse, coibasepro, hitbtc, huobi, hadax, poloniex and zb

* changes recieved typo due to not being well received

* Fixes parsing issue with Huobi and hadax

* Fixes data race with more locks

* removes defer locks. fixes huobi/hadax verbose output

* Fixes double JSONEncode for coinut. Fixes verbose output for coinut

* gofmt,goimport for coinut

* Fixes issue where multiple connection monitors can spawn

* Removes defer exchange.WebsocketConn.Close() in defer handledata exit as connectionmonitor handles connections instead

* gofmt and go import

* More fmts
2019-05-16 16:39:16 +10:00
Andrew
5bce701d43 Add httputil.DumpRequestOut & httputil.DumpResponse to SendPayload (#298)
* First pass adding DumpResponse

* WIP

* Added httpDebug flag to SendPayload

* anx -> coinbase converted

* HTTPDebug flag added to all exchanges

* Updated Alphapoint:

* reworded error message

* Corrected itbit and huobihadax sendpayload

* Removed setting of default value

* don't return body for response as we already get this in verbose mode

* body is now included based on verbose setting

* WIP

* Due to issues with DumpResponse and gzip decoding use raw response from exchange for body output Also adds gzip decoding back as bug fix for OK group exchange

* removed additional new line

* Added URL path to dumpbody display as well
2019-05-16 13:46:28 +10:00
Ryan O'Hara-Reid
35b94268e0 Update request.go to fix concurrency nonce issues (#285)
* Updates nonce generation to adhere to fifo channel buffer before request executes by routine

* removed unused variables, lns etc

* Fix requested changes and added in timer that disengages lock if out of scope error occurs

* Fixed woopsy daisy issue

* Add benchmark, reduce time in force to unlock before stack insertion, add nil check for edge case

* Remove unusued waitgroup field

* use return nonce.Value and method, rm redundant nonce code, fix tests.

* Fix linter issue: unnecessary conversion
2019-05-06 13:46:34 +10:00
Adrian Gallagher
9bdcc22ae1 Relax case sensitive string comparisons in various parts of GCT 2019-04-23 14:22:00 +10:00
Adrian Gallagher
7dcb1ab553 Migrate from gometalinter.v2 to golangci-lint (#249)
* Migrate from gometalinter.v2 to golangci-lint
2019-03-01 16:10:29 +11:00
Adrian Gallagher
e000a8d6c7 minor improvements and bugfixes
1) reduce request content-type mismatch ambiguity
2) remove duplicate wrapper function for withdraw permissions
3) update exchange_template tool to add missing/incorrect wrapper funcs
2019-02-27 15:31:56 +11:00
Adrian Gallagher
5e5ca8a887 Linter fixes (#246)
Linter fixes
2019-02-05 16:26:04 +11:00
Ryan O'Hara-Reid
82a622294c Coinmarketcap implementation (#243)
* Updates requester package to allow unpacking of zipped files and defaults to warn if no JSON is present

* Initial addition of coinmarketcap functionality

* fix requested changes

* Fix issue with displaying false positive in request.go && reorder plan list

* Rename CurrencyProvider -> CryptocurrencyProvider
Skip seeding currency data if not enabled
Rm line in main.go

* Update test procedures and relevant json files

* Fix const issue within config.go
2019-01-31 16:11:42 +11:00
Andrew
d01e7bad72 Implement Logger (#228)
* Added new base logger

* updated example and test configs

* updated exchange helpers restful router & server

* logPath is now passed to the logger to remove dependency on common package

* updated everything besides exchanges to use new logger

* alphapoint to bitmex done

* updated bitmex bitstamp bittrex btcc and also performance changes to logger

* btcmarkets coinbase coinut exmo gateio wrappers updated

* gateio and gemini logger updated

* hitbtc huobi itbit & kraken updated

* All exchanges updatd

* return correct error for disabled websocket

* don't disconnect client on invalid json

* updated router internal logging

* log.Fatal to t.Error for tests

* Changed from fatal to error failure to set maxprocs

* output ANSI codes for everything but windows for now due to lack of windows support

* added error handling to logger and unit tests

* clear wording on print -> log.print

* added benchmark test

* cleaned up import sections

* Updated logger based on PR requests (added default config options on failure/setting errors)

* ah this should fix travici enc config issue

* Load entire config and clear out logging to hopefully fix travisci issue

* wording & test error handling

* fixed formatting issues based on feedback

* fixed formatting issues based on feedback

* changed CheckDir to use mkdirall instead of mkdir and other changes based on feedback
2019-01-08 21:56:22 +11:00
Ryan O'Hara-Reid
7c3134f35b Wrapper update modify order (#222)
* Changed IBotExchange interface ModifyOrder function paramater and return value to exchange type for easier addition or retraction of variables.

* Function ModifyOrder for Binance not supported via API

* Change to unsupported function for exchange ZB

* Change to unsupported function for exchange Yobit

* Add modify order support for Poloniex

*  Change to unsupported function for exchange Okex

* Change to unsupported function for exchange Localbitcoins

* Change to unsupported function for exchange Liqui

* Change to unsupported function for exchange LakeBTC

* Change to unsupported function for exchange Kraken

* Change to unsupported function for exchange Itbit

* Change to unsupported function for exchange HuobiHadax

* Change to unsupported function for exchange Huobi

* Change to unsupported function for exchange HitBTC

* Change to unsupported function for exchange Gemini

* Change to unsupported function for exchange GateIO

* Change to unsupported function for exchange Exmo

* Change to unsupported function for exchange Coinut

* Change to unsupported function for exchange Coinbase

* Change to unsupported function for exchange BTCMarkets

* Change to unsupported function for exchange Bittrex

* Change to unsupported function for exchange Bitstamp

* Add modify order support for Bitmex

* Add verbose header information in request package

* Add modify order support for Bithumb exchange

* Change to unsupported function for exchange Bitflyer

* Change to unsupported function for exchange Bitfinex

*  Change to unsupported function for exchange ANX

* Change interface function signature

* Rm redundant code for authenticated requests in Bithumb

* Add error check if decimal values supplied for create or modifying an order on Bitmex

* Added test functions across the exchanges

* Rm comment for modify order on Alphapoint exchange

* Update tmpl file for exchange wrapper
2018-12-20 16:31:17 +11:00
Adrian Gallagher
9680cdea38 Limit HTTP request verbosity 2018-12-10 10:40:43 +11:00
Scott
458aab301e Cancel order wrapper wrapup (#214)
* Reimplements order cancellation for alphapoint, anx, binance, bitfinex, bithumb, bitmex, bitstamp, bittrex,  btcmarkets, coinbasepro, coinut, exmo, gateio, gemini, gitbtc, huobi, hadax, itbit, kraken, lakebtc, liqui, okcoin, okex, poloniex, wex, yobit and zb wrappers. Adds new order cancellation struct type. Updates old tests that pointed to the wrong unrenamed methods

* Sets up tests for all supported exchanges. request.DoRequest errors when response status is not 200

* Updates alphapoint, coinut, hitbtc, lakebtc cancel order implementations. Finishes testing

* Adds localbitcoin cancel order wrapper support

* Fixes tests and build issues. Adds WexIssue flag for tests

* Changes CancelOrder signature to only return error. Allows exchange to format currency pairs with delimiters
2018-11-30 16:20:34 +11:00
Ryan O'Hara-Reid
f6060ff1fc Requester update (#203)
* Adds upgrade to re-do request on client timeout.

* Updated readme with documentation tool.

* Add Requester variable for timeout retry
Improve tests
2018-11-05 12:14:54 +11:00
Ryan O'Hara-Reid
d3c2800fe0 Initial overhaul of websocket connection and feeds (#189)
* Initial overhaul of websocket connection and feeds
* Added proxy support
* Piped to routines.go

* Added new websocket file in exchanges
Refactored orderbook handling into exchange_websocket.go
Added better error responses for binance_websocket.go
General clean for binance_websocket.go

* General fixes - bitfinex_websocket.go
Refactored orderbook cache code - bitfinex_websocket.go
Removed fatal error with unhandled type - routines.go

* Added general improvements to bitmex_websocket.go
Refactored orderbook handling to exchange_websocket.go
Added variable in Item struct in orderbook.go for looking up orders by ID

* Fix issue when routines are blocked due to Data Handler not started
Updated traffic handler
General fixes for bitstamp_websocket.go

* General fixes for coinbasepro_websocket.go

* General fixes for coinut_websocket.go
Fixed error return in exchange_websocket.go

* Removed comments in coinut_wrapper.go
Refactor orderbook logic from hitbtc_websocket.go to exchange_websocket.go

* General fixes

* Removed comments
General fixes

* Updated routines.go

* After rebase fix

* Fixed update config pairs in okcoin.go

* fixed config currency issue in okcoin.go for okcoin China

* exchange_websocket.go
*Removed unused const dec
*Removed state change routine
*Improved trafficMonitor routine
*Increased verbosity for error returns
*Removed uneeded mutex locks

exchange_websocket_test.go
*Added new tests for websocket and orderbook updating

routines.go
*Removed string cased

* Fixed race conditions on sync.waitgroup in exchanges_websocket.go

* Changes variable name in config.go

* Removes unnecessary comment

* Removes indefinite lock on error return

* Removes unnecessary comment

* Adds support for BTCC websocket
Drops support for BTCC REST

* Rewords comment in exchange_websocket.go
Moves types to poloniex_types.go

* Moves types to coinut_types.go

* Removes uneeded range for accessing array variables for coinbase_websocket.go
Removes comments in coinut_types.go

* Adds verbosity flag to GCT
Suppresses verbose output from routines.go

* Fixes setting proxy for REST and Websocket per exchange
Upgrades error handling
Drops unused *url.Url variable in exchange type

* Adds test for setting proxy

* Fixes bug that closes connection due to incorrect timeout time through a proxy connection

* Clarify verbose flag message
2018-10-24 14:22:40 +11:00
Adrian Gallagher
a0de1b78a7 Support configurable HTTP user agent per exchange 2018-08-07 13:05:13 +10:00
Adrian Gallagher
c63f1b0ff6 Port from idoall's codebase (#161)
* 修复火币Post REST API方法不正确的问题,同时增加火币海带丝交易所

* add vendor folder

* 修改命名空间依赖

* 第一次提交分支

* 增加取消订单功能

* 修复binance.GetAccount方法

* 更新readme.md

* 增加 Gateio 交易所的支持,支持获取K线、支持的交易对、交易市场参数

* 替换HuobiHadax的参数

* 买/卖订单、取消订单

* OKEX 币币交易:增加获取用户信息,下订单,取消订单

* 测试ok kline

* 修复 Bitfinex 的 GetAccountInfo 方法

* 做一些不必要的删减

* 修复binfinex不返回错误的bug

* 统一我修改交易所的Kline获取方式

* Bitfinex 增加获取最新价格

* update main.go

* 更新GetSymbol方法

* 修改火币和海带丝的Kline编号ID类型

* 修改海带丝的默认配置大小写

* okex增加获取最新价格

*   调整okex的参数判断

* 调整比特儿的参数名称

* 修改火币、火币Hadax的参数全名

* 更新海带丝的配置名称

* 修改bintfinex的GetAccountInfo方法

* 去掉一行注释

* 支持zb交易所的部分功能

* 修复获取K线时没有设置参数的错误

* 增加 Binance 取消订单的方法,获取订单状态,获取所有打开的状态以及所有订单

* 修改获取深度和历史订单的数据

* 修改币安获取深度的参数

* 修改火币获取市场深度的参数

* 修改okex获取市场深度的参数

* 修改币安、OKex获取历史订单的参数

* 修复币安提交参数错误的问题

* merge upstrem

* merge后,调整一部分命名空间

* 修改ZB时间参数的命名方式

* 继续替换命名空间

* 命名空间的替换

* 继续命名空间的替换

* 测试

* Port code from idoall's PR

* Drop errors dep

* Start amending PR

* Fix commented code
* Translate text from Chinese to English (except for ZB). The reasning behind this is that it's a Chinese exchange and the structs are self explanatory in English, but would for other developers in China

* Translate Chinese text, basic formatting changes

* Remove commented lines and address feedback on PR
2018-08-04 08:30:20 +10:00
Adrian Gallagher
a5f51328d4 Improve request package and adjust bitfinex rate limiter
Fixes: https://github.com/thrasher-/gocryptotrader/issues/147
2018-07-17 16:24:41 +10:00
Adrian Gallagher
69acf88a7b Fix race condition in request 2018-07-13 11:19:11 +10:00
Frank
0478c55b45 fix huobib/binance api bug (#129) 2018-06-04 19:20:05 +10:00
Adrian Gallagher
ac41a7cfad New features and bug fixes
- Modifications made to the request package. Planned improvements will be
sending requests on intervals, rate limiter back off support, dynamic tuning
and requests packaged into a request job group.
- Can modify each exchanges individual HTTP client (e.g timeout and
transport settings).
- Bot now uses an exchange config HTTP timeout value.
- Bot now uses a global HTTP timeout (configurable).
- Batched ticker request support for exchanges.
- Ticker and Orderbook fetching now are spanned accross multiple
go routines and regulated by a sync wait group.
- Fixes hack used to load exchanges, now uses a sync wait group.
- Ticker and Orderbook storage and fetching now uses mutex locks.
- New pair function for finding different pairs between two supplied
 pair arrays. This is used for currency pair updates for exchange which
support dynamic updating.
- Shows removal/additions of dynamic updates currencies.
2018-05-04 13:20:19 +10:00
Adrian Gallagher
c7774708cd Request: Add additional method support for auth/unauth requests 2018-03-29 15:46:47 +11:00
Ryan O'Hara-Reid
7fc9d20fd7 Introduce request package and integrate with exchanges 2018-03-27 14:22:52 +11:00