Commit Graph

40 Commits

Author SHA1 Message Date
Adrian Gallagher
2dc813b5f3 Merge branch 'master' into engine 2019-08-23 21:59:01 +10:00
Adrian Gallagher
8ecd8a0c16 Merge branch 'master' into engine 2019-08-09 15:46:24 +10:00
Adrian Gallagher
2078ba907f Update URLs for transfer into org repo (#338)
* Update URLs for transfer into org repo

* Update codecov, travis and othe remaining links

* Update appveyor paths
2019-08-09 11:50:39 +10:00
Scott
e209d85d0d Websocket request-response correlation (#328)
* Establishes new websocket functionality. Begins the creation of the websocket request

* Adding a wrapper over gorilla websocket connect,send,receive to handle ID messages. Doesn't work

* Successfully moved exchange_websocket into its own wshandler namespace. oof

* Sets up ZB to use a round trip WS request system

* Adds Kraken ID support to subscriptions. Renames duplicate func name UnsubscribeToChannels to RemoveSubscribedChannels. Adds some helper methods in the WebsocketConn to reduce duplicate code. Cleans up ZB implementation

* Fixes double locking which caused no websocket data to be read. Fixes requestid for kraken subscriptions

* Completes Huobi and Hadax implementation. Extends ZB error handling. Adds GZip support for reading messages

* Adds HitBTC support. Adds GetCurrencies, GetSymbols, GetTrades WS funcs. Adds super fun new parameter to GenerateMessageID for Unix and UnixNano

* Adds GateIO id support

* Adds Coinut support. Prevents nil reference error in constatus when there isnt one

* Standardises all Exchange websockets to use the wshandler websocket. Removes the wsRequestMtx as wshandler handles that now. Makes the Dialer a dialer, its not externally referenced that I can see.

* Fixes issue with coinut implementation. Updates bitmex currencies. Removes redundant log messages which are used to log messages

* Starts testing. Renames files

* Adds tests for websocket connection

* Reverts request.go change

* Linting everything

* Fixes rebase issue

* Final changes. Fixes variable names, removes log.Debug, removes lines, rearranges test types, removes order correlation websocket type

* Final final commit, fixing ZB issues.

* Adds traffic alerts where missed. Changes empty struct pointer addresses to nil instead. Removes empty lines

* Fixed string conversion

* Fixes issue with ZB not sending success codes

* Fixes issue with coinut processing due to nonce handling with subscriptions

* Fixes issue where ZB test failure was not caught. Removes unnecessary error handling from other ZB tests

* Removes unused interface

* Renames wshandler.Init() to wshandler.Run()

* Updates template file

* Capitalises cryptocurrencies in struct. Moves websocketResponseCheckTimeout and websocketResponseMaxLimit into config options. Moves connection configuration to main exchange Setup (where appropriate). Reverts currencylastupdated ticks. Improves reader close error checking

* Fixes two inconsistent websocket delay times

* Creates a default variable for websocket ResponseMaxLimit and ResponseCheckTimeout, then applies it to setdefaults and all tests

* Updates exchange template to set and use default websocket response limits
2019-08-07 15:15:01 +10:00
Andrew
3de1d94e5f New logging system (#319)
* First pass at adding new logging system

* NewLogger

* NewLogger

* WIP

* silly bug fix

* :D removed files

* removed old logging interface

* added tests

* added tests

* Started to add new lines to all f calls

* Added subsystem log types

* Logger improvements

* Further performance improvements

* changes to logger and sublogger creation

* Renamed Logging types

* removed old print statement

* changes based on feedback

* moved sublogger types to own file

* :)

* added console as output type

* added get level command

* added get/set log level via grpc command

* added check for output being empty for migration support

* first pass at log rotation

* added log rotation

* :D derp fixed

* added tests

* changes based on feedback

* changed log type

* comments

* renamed file -> fileSettings

* typo fix

* changes based on feedback

* gofmt ran on additional files

* gofmt ran on additional files
2019-07-07 05:20:31 +10:00
Adrian Gallagher
f5914e8c10 Engine changes 2019-05-22 17:06:38 +10:00
Andrew
5bce701d43 Add httputil.DumpRequestOut & httputil.DumpResponse to SendPayload (#298)
* First pass adding DumpResponse

* WIP

* Added httpDebug flag to SendPayload

* anx -> coinbase converted

* HTTPDebug flag added to all exchanges

* Updated Alphapoint:

* reworded error message

* Corrected itbit and huobihadax sendpayload

* Removed setting of default value

* don't return body for response as we already get this in verbose mode

* body is now included based on verbose setting

* WIP

* Due to issues with DumpResponse and gzip decoding use raw response from exchange for body output Also adds gzip decoding back as bug fix for OK group exchange

* removed additional new line

* Added URL path to dumpbody display as well
2019-05-16 13:46:28 +10:00
leilaes
cdab89a58a Replace a zero-length, non-nil slice with a nil slice (#299)
* Replace a zero-length, non-nil slice with a nil slice

* Update codelingo.yaml
2019-05-16 10:19:08 +10:00
Ryan O'Hara-Reid
35b94268e0 Update request.go to fix concurrency nonce issues (#285)
* Updates nonce generation to adhere to fifo channel buffer before request executes by routine

* removed unused variables, lns etc

* Fix requested changes and added in timer that disengages lock if out of scope error occurs

* Fixed woopsy daisy issue

* Add benchmark, reduce time in force to unlock before stack insertion, add nil check for edge case

* Remove unusued waitgroup field

* use return nonce.Value and method, rm redundant nonce code, fix tests.

* Fix linter issue: unnecessary conversion
2019-05-06 13:46:34 +10:00
Scott
e56fc26d93 Offline worst case trade fees (#274)
* Really basic getSimulated fee function everywhere

* Worst case fees for all exchanges

* Adds tests, fixes comment spacing. Adds wrapper logic. Makes test api key var name consistent. Removes some okcoin ETT tests

* Removes redundant functions

* linting issues. Fixes introduces huobi issues

* More linting

* Stops trying to hide ETT problems, uses iota

* Skips ETT tests for now
2019-04-09 19:38:31 +10:00
Andrew
ca55f2f965 Optimisation: Large struct pointer conversion (final part) (#265)
Completes large struct pointer optomisations over the entire codebase and enables hugeParams linter by default
2019-04-04 15:31:49 +11:00
Andrew
76ac98830e optimisation: large structs/huge param fixes (#257)
* Updated SetAPIURL() & UpdateConfig() to use pointers

* Withdraw functions now take a pointer to withdrawrequest struct

* Fund Withdraw is now a pointer

* Moved GetFee functions to use a pointer due to large size of feebuilder struct

* OrderCancellation usage converted to pointer due to size of struct

* merged up upstream/master and changes made to OKgroup code

* GetOrdersRequest usage converted to a pointer

* okgroup pointer migration

* reverted golangci config back to master

* removed old anx benchmark typo
2019-03-21 16:36:06 +11:00
Ryan O'Hara-Reid
0990f9d118 Currency package update (#247)
* Initial currency overhaul before service system implementation

* Remove redundant currency string in orderbook.Base
Unexport lastupdated field in orderbook.Base as it was being instantiated multiple times
Add error handling for process orderbook

*  Remove redundant currency string in ticker.Price
 Unexport lastupdated field in ticker.Price
 Add error handling for process ticker function and fix tests

* Phase Two Update

* Update translations to use map type - thankyou to kempeng for spotting this

* Change pair method name from Display -> Format for better readability

* Fixes misspelling and tests

* Implement requested changes from GloriousCode

* Remove reduntant function and streamlined return in currency_translation.go

* Revert pair method naming conventions

* Change currency naming conventions

* Changed code type to exported Item type with underlying string to reduce complexity

* Added interim orderbook process method to orderbook.Base type

* Changed feebuilder struct field to currency.Pair

* Adds fall over system for backup fx providers

* deprecate function and children and fix linter issue with btcmarkets

* Fixed requested changes

* Fix bug and move mtx for rates

* Fixed after rebase oopsies

* Fix linter issues

* Fixes race conditions in testing functions

* Final phase coinmarketcap update

* fix linter issues

* Implement requested changes

* Adds configuration variables to increase/decrease time durations between updating currency file and fetching new currency rates

* Add a collection of tests to improve codecov

* After rebase oopsy fixes for btse

* Fix requested changes

* fix after rebase oopsies and add more efficient comparison checks within currency pair

* Fix linter issues
2019-03-19 11:49:05 +11:00
Adrian Gallagher
7dcb1ab553 Migrate from gometalinter.v2 to golangci-lint (#249)
* Migrate from gometalinter.v2 to golangci-lint
2019-03-01 16:10:29 +11:00
Adrian Gallagher
5e5ca8a887 Linter fixes (#246)
Linter fixes
2019-02-05 16:26:04 +11:00
Adrian Gallagher
291e404a4a Makefile: add new recipes and linter features (#244)
* Makefile: add new recipes and linter features

* expand linter coverage and fix issues

* Update makefile

* address PR nitterinos
2019-01-31 14:53:24 +11:00
Ryan O'Hara-Reid
41415ca3b9 Websocket update increasing exchange coverage and bug fixes (#233)
Websocket update increasing exchange coverage and bug fixes
2019-01-23 14:23:11 +11:00
Andrew
d01e7bad72 Implement Logger (#228)
* Added new base logger

* updated example and test configs

* updated exchange helpers restful router & server

* logPath is now passed to the logger to remove dependency on common package

* updated everything besides exchanges to use new logger

* alphapoint to bitmex done

* updated bitmex bitstamp bittrex btcc and also performance changes to logger

* btcmarkets coinbase coinut exmo gateio wrappers updated

* gateio and gemini logger updated

* hitbtc huobi itbit & kraken updated

* All exchanges updatd

* return correct error for disabled websocket

* don't disconnect client on invalid json

* updated router internal logging

* log.Fatal to t.Error for tests

* Changed from fatal to error failure to set maxprocs

* output ANSI codes for everything but windows for now due to lack of windows support

* added error handling to logger and unit tests

* clear wording on print -> log.print

* added benchmark test

* cleaned up import sections

* Updated logger based on PR requests (added default config options on failure/setting errors)

* ah this should fix travici enc config issue

* Load entire config and clear out logging to hopefully fix travisci issue

* wording & test error handling

* fixed formatting issues based on feedback

* fixed formatting issues based on feedback

* changed CheckDir to use mkdirall instead of mkdir and other changes based on feedback
2019-01-08 21:56:22 +11:00
Scott
ff6a84f0f1 Cancel all orders wrapper implementation (#217)
* Changes method signature for cancelling all orders (experitmental). Implements cancelAllOrders wrapper for alphapoint, anx, binance

* Implements cancel all wrapper for bitfinex, bitmex, bitstamp, bittrex, btcmarkets, coinbasepro and hilariously coinut

* Changes method signature to only use one OrderCancellation type. Adds support for Exmo, gateio, gemini, itbit, lakebtc

* Adds/updates support for hitbtc, huobi, hadax, itbit and kraken

* Adds support for liqui, localbitcoins, okcoin, poloniex, wex and yobit. Splits up open order methods for poloniex

* Adds bithumb, okex and zb support. BTCC for another PR

* Updates bitflyer, bithumb, bitmex, coinut, okex and zb cancelAllOrders method to cancel via enabled currency pairs rather than a singular currency

* Adds tests to all exchanges to test wrapper function CancelAllOrders

* Fixes OKEX and huobi, btcmarkets, kraken, okCoin cancel order implementations

* Fixes coinut, hitbtc and okex api for authenticated requests

* Fixes comment and spacing

* Changes the CancelAllOrders signature to return orderids and errors along with a generic error.

* Fixes OKEX delimiter

* Removes spacing and test verbosity

* Removes more spacing

* Removes space

* Fixes okex rebasing issue. Also makes the maps instead of assuming they just work
2018-12-14 15:53:26 +11:00
Ryan O'Hara-Reid
4ca3fd5b00 GetAccountInfo wrapper update (#220)
* Added untested [cloudflare issue] changes to accountinfo for ANX

* Add alphapoint comment for future implementation

* Adds GetAccountInfo for Binance

* Adds GetAccountInfo update for Bithumb

* Updates GetAccountInfo for GateIO.
Adds error handling feature for authenticated requests.

* Updates GetAccountInfo function for Huobi.
Adds function for getting account ID.

* Updates GetAccountInfo function
Adds GetAccountID function

* Updates GetAccountInfo [un-tested, no access to keys at this time]

* Updates GetAccountInfo for Kraken

* Updates GetAccountInfo func for OKEX

* Updates GetAccountInfo for exchange ZB

* Updates GetAccountInfo func for Bitmex

* Updates GetAccountInfo func for Coinut

* Updates GetAccountInfo for ANX exchange

* Fixes incorrect hold currency issue

* Fixes type name

* Fixes issue with unneeded code in wrapper for Bithumb

* Change strings to type symbol string

* Fixes nit for Gateio

* Fixes GetAccountInfo issue
Fixes SpotCancelOrder issue
2018-12-13 16:53:19 +11:00
Ryan O'Hara-Reid
92534249bf Remove unwanted wrapper name stuttering and standardise common error returns (#213) 2018-11-23 12:55:00 +11:00
Scott
baffb46300 Exchange withdraw capabilities (#199)
* Initial broken commit for updating exchanges to store data on whether withdrawals can be handled completely by the API

* Changes to use base exchange method to determine withdrawal permissions using bitshifting

* Adds withdrawal capabilities for alphapoint, anx, binance, bitfinex, bitflyer, bithumb, bitmex, bitstamp, bittrex exchanges

* Adds withdraw capabilities for btcmarkets, coinbasepro, coinut, exmo, gateio, gemini, hitbtc, huobi, hadax

* Adds withdraw capabilities for itbit, kraken, lakebtc, liqui, localbitcoins, okcoin, okex, poloniex, wex, yobit, zb

* Titillating tests & Wonderful wrappers

* Fixes typo, double checks all apis and corrects permissions

* Fixes gemini test

* Fixes incorrect log placement. Removes breaks

* Addresses PR comments. Changes readable function to: FormatWithdrawPermissions, adds new function 'GetWithdrawPermissions' and 'SupportsWithdrawPermissions'. Adds three functions to interface

* Removes unnecessary config_example.json additions

* Fixes bitwise comparisons. Adds more tests to highlight expected outcomes
2018-11-05 10:35:03 +11:00
Scott Grant
c1b2959606 Adds more international withdrawal fee details for bithumb, gemini, itbit and lakebtc 2018-11-01 12:03:10 +11:00
Scott
e4c443b901 Calculating each exchange's fees (#188)
* Adds some constants for fee types
Adds some fee calculation in an attempt to be generic
Adds fee stuff to Bittrex
Adds fee stuff to bitstamp

* Fixes bitstamp fee calculation

* Tests
Tests all scenarios for GetFeeByType

* Adds method to wrapper
Adds err to response
Checks for err

* Adds support for Bittrex fees

* Adds maker/taker dynamic to fees
Updates tests
Adds bitmex fee support
Removes unused switch case scenarios to not waste space

* Adds bithumb support for fee calculation

* Adds Bitfinex fee support
Adds list of currencies as const strings
Sets up bitflyer

* Fixes arguments

* Greatly expands symbols
Adds Binance fee calculation support
Cleans up previous exchanges

* Fixes errors for fee calculations

* Adds ANX fee support

* Adds btcc fee support
Adds alphapoint fee wrapper support
Renames method to match "enum"
Uses symbols in tests, not inline strings

* Adds support for BTCMarkets fee calculation
Adds new method to retrieve fee amount from BTCMarkets
Adds new fee type struct: FeeBuilder
Updates ANX and BTCMarkets to use new FeeBuilder type struct
Standardises the tests to run when it comes to fee calculation

* Migrates all existing exchange fee to use new feebuilder type struct
Uses standard testing model

* Fixes unit tests

* Updates maker taker fees in test config

* Removes parallel from fee testing

* Removes more parallel from tests

* Adds coinbasepro fee support

* Adds Coinut fee support

* Adds Exmo fee support
Adds maker fee support to coinut
Introduces a type for fees and bank transfers to prevent random strings being used

* Adds partial bitflyer support
Moves bitflyer to feeBuilder struct

* Adds gateio fee support

* Adds Gemini fee support

* Adds hitbtc fee support

* Adds huobi fee support

* Adds HuobiHadax fee support

* Adds itbit fee support

* Adds partial kraken fee support with trading fees

* Finishes basic Kraken fee support

* Adds basic LakeBTC fee support

* Adds basic liqui fee support

* Adds localbitcoins fee support.......

* Adds basic okcoin fee support

* Adds simple OKEX fee support
Adds many new currency symbols
Fixes liqui's fees

* Adds poloniex fee support

* Adds fee support for Yobit

* Adds WEX fee support

* Adds ZB fee support

* Removes bad reference

* Improves accuracy of variable name

* trading fee method names are now consistent

(cherry picked from commit 21c82e8b90cae590cfd73d365d7be39e1a00e973)

* Fixes rebasing issues

* Fixes issues from rebase
Removes "IsTaker" as IsMaker bool can imply taker
Updates tests to actually work.

* Adds a zero to the test

* Fixes bitfinex api endpoints and fixes fee calculations

* Updates btcmarkets trading fee calculation

* Verifies tests with apis for all exchanges except coinbasepro, itbit and bitflyer
Removes taker fee test as taker is default

* Removes redundant all exchange wrapper error checks due to the error checks being redundant

* Addresses review comments:
- Renames variables
- Changes how functions return data
- Fixes typo
2018-10-29 12:32:46 +11:00
Ryan O'Hara-Reid
d3c2800fe0 Initial overhaul of websocket connection and feeds (#189)
* Initial overhaul of websocket connection and feeds
* Added proxy support
* Piped to routines.go

* Added new websocket file in exchanges
Refactored orderbook handling into exchange_websocket.go
Added better error responses for binance_websocket.go
General clean for binance_websocket.go

* General fixes - bitfinex_websocket.go
Refactored orderbook cache code - bitfinex_websocket.go
Removed fatal error with unhandled type - routines.go

* Added general improvements to bitmex_websocket.go
Refactored orderbook handling to exchange_websocket.go
Added variable in Item struct in orderbook.go for looking up orders by ID

* Fix issue when routines are blocked due to Data Handler not started
Updated traffic handler
General fixes for bitstamp_websocket.go

* General fixes for coinbasepro_websocket.go

* General fixes for coinut_websocket.go
Fixed error return in exchange_websocket.go

* Removed comments in coinut_wrapper.go
Refactor orderbook logic from hitbtc_websocket.go to exchange_websocket.go

* General fixes

* Removed comments
General fixes

* Updated routines.go

* After rebase fix

* Fixed update config pairs in okcoin.go

* fixed config currency issue in okcoin.go for okcoin China

* exchange_websocket.go
*Removed unused const dec
*Removed state change routine
*Improved trafficMonitor routine
*Increased verbosity for error returns
*Removed uneeded mutex locks

exchange_websocket_test.go
*Added new tests for websocket and orderbook updating

routines.go
*Removed string cased

* Fixed race conditions on sync.waitgroup in exchanges_websocket.go

* Changes variable name in config.go

* Removes unnecessary comment

* Removes indefinite lock on error return

* Removes unnecessary comment

* Adds support for BTCC websocket
Drops support for BTCC REST

* Rewords comment in exchange_websocket.go
Moves types to poloniex_types.go

* Moves types to coinut_types.go

* Removes uneeded range for accessing array variables for coinbase_websocket.go
Removes comments in coinut_types.go

* Adds verbosity flag to GCT
Suppresses verbose output from routines.go

* Fixes setting proxy for REST and Websocket per exchange
Upgrades error handling
Drops unused *url.Url variable in exchange type

* Adds test for setting proxy

* Fixes bug that closes connection due to incorrect timeout time through a proxy connection

* Clarify verbose flag message
2018-10-24 14:22:40 +11:00
soxipy
fb4e2d1452 localbitcoins fixes (#177)
* General LocalBitcoin fixes

* Added override variables to config for exchange packages to allow different API URL's
2018-08-27 14:19:29 +10:00
Adrian Gallagher
a0de1b78a7 Support configurable HTTP user agent per exchange 2018-08-07 13:05:13 +10:00
Adrian Gallagher
ac41a7cfad New features and bug fixes
- Modifications made to the request package. Planned improvements will be
sending requests on intervals, rate limiter back off support, dynamic tuning
and requests packaged into a request job group.
- Can modify each exchanges individual HTTP client (e.g timeout and
transport settings).
- Bot now uses an exchange config HTTP timeout value.
- Bot now uses a global HTTP timeout (configurable).
- Batched ticker request support for exchanges.
- Ticker and Orderbook fetching now are spanned accross multiple
go routines and regulated by a sync wait group.
- Fixes hack used to load exchanges, now uses a sync wait group.
- Ticker and Orderbook storage and fetching now uses mutex locks.
- New pair function for finding different pairs between two supplied
 pair arrays. This is used for currency pair updates for exchange which
support dynamic updating.
- Shows removal/additions of dynamic updates currencies.
2018-05-04 13:20:19 +10:00
Ryan O'Hara-Reid
7fc9d20fd7 Introduce request package and integrate with exchanges 2018-03-27 14:22:52 +11:00
Adrian Gallagher
52dfddbb18 Feature: Add support to check whether an exchange supports automatic currency pair updates and if they don't, show a warning if the last currency pair update is >= 30 days
Also fix race condition in config get/set functions
2018-03-27 12:53:46 +11:00
Adrian Gallagher
55c8b02d1d Fix misspells 2017-10-04 18:15:01 +11:00
Ryan O'Hara-Reid
79a1911c93 In the common package added JSONDecode error check. Added verbosity in SendHTTPGetRequest. Updated Nonce package function. Fixed issues in ItBit package and expanded test coverage. 2017-09-18 14:47:14 +10:00
Adrian Gallagher
913c104d09 Finish off websocket ticker demo, update config files and fix tests 2017-09-14 15:04:38 +10:00
Adrian Gallagher
55ea1fe434 Add generalised functions for handling exchange enabled and available currencies 2017-08-25 15:54:45 +10:00
Adrian Gallagher
f3c1f4880d Add nonce package for exchanges 2017-08-21 15:57:41 +10:00
Adrian Gallagher
4eaa9d0ec9 Prevent authenticated calls when authenticated API support is disabled 2017-08-21 13:10:57 +10:00
Adrian Gallagher
c7399ce69b Fix various issues for goreport 2017-08-10 09:15:46 +10:00
Ryan O'Hara-Reid
de48b62731 Fixed stutter issues from exchange.go 2017-07-31 11:45:12 +10:00
Ryan O'Hara-Reid
00d2024e5a Changed const naming convention assoc. with common.go 2017-07-31 11:42:56 +10:00
Adrian Gallagher
7aff04e4c6 Standardise filenames 2017-03-30 14:05:32 +11:00