Commit Graph

23 Commits

Author SHA1 Message Date
Andrew
4a736fb335 (Exchange) Add GetHistoricCandles() & GetHistoricCandlesEx() support to exchanges (#479)
* implemented binance and bitfinex GetHistoricCandles wrapper methods)

* coinbene supported added

* after and before clean up

* gateio wrapper completed

* merged upstream/master

* Added bsaic KlineIntervalSupported() method

* Converted binance fixed test

* WIP

* new KlineConvertToExchangeStandardString method added

* end of day WIP

* WIP

* end of day WIP started migration of trade history

* added kline support to hitbtc huobi lbank

* added exchangehistory to all supported exchanges started work on coinbase 300 candles/request method

* end of day WIP

* removed unused ta and misc changes to flag ready for review

* yobit cleanup

* revert coinbase changES

* general code clean up and added zb support

* poloniex support added

* renamed method to FormatExchangeKlineInterval other misc fixes

* linter fixes

* linter fixes

* removed verbose

* fixed poloniex test coverage

* revert poloniex mock data

* regenerated poloniex mock data

* a very verbose clean up

* binance mock clean up

* removed unneeded t.Log()

* setting verbose to true to debug CI issue

* first pass changes addressed

* common.ErrNotYetImplemented implemented :D

* comments added

* WIP-addressed exchange requests and reverted previous GetExchangeHistory changes

* WIP-addressed exchange requests and reverted previous GetExchangeHistory changes

* increased test coverage added kraken support

* OKGroup support completed started work on address GetExchangeHistory feedback and migrating to own PR under https://github.com/xtda/gocryptotrader/tree/exchange_history

* convert zb ratelimits

* gofmt run on okcoin

* increased delay on rate limit

* gofmt package

* fixed panic with coinbene and bithumb if conversion fails

* very broken end of day WIP

* added support for GetHistoricCandlesEx to coinbase and binance

* gofmt package

* coinbase, btcmarkets, zb ex wrapper function added

* added all exchange support for ex regenerated mock data

* update bithumb to return wrapper method

* gofmt package

* end of day started work on changes

* reworked test coverage added okgroup support general fixes/change requests addressed

* Added OneMonth

* limit checks on supportedexchanges

* reverted getexchangehistory

* reworked binance tesT

* added workaround for kraken panic

* renamed command to extended removed interval check on non-implemented commands

* added wrapperconfig back

* increased test coverage for FormatExchangeKlineInterval

* WIP

* increased test coverage for FormatExchangeKlineInterval bitfinex/gateio/huobi

* linter fixes

* zb kraken lbank coinbene btcmarkets support added

* removed verbose

* OK group support for other asset types added

* swapped margin to use spot endpoint

* index support added test coverage added for asset types

* added asset type to okcoin test

* gofmt

* add asset to extended method

* removed verbose

* add support for coinbene swap increase test coverage

* removed verbose

* small clean up of okgroup wrapper functions

* verbose to troubleshoot CI issues

* removed verbose

* added error check reverted coinbasechanges

* readme updated

* removed unused start/finish started work on decoupling api requests from kline package

* restructured coinbene, bithumb methods, added bitstamp support

* kraken time fix

* BTCMarkets restructure

* typo fix

* removed test for futures due to contact changing

* added start/end date to extended method over range

* converted to assettranslator

* removed verbose

* removed invalid char

* reverted incorrectly removed return

* added import

* further template updates

* macos hates my keyboard :D

* misc canges

* x -> i

* removed verbose

* updated fixCasing to allocate var before checks

* removed time conversion

* sort all outgoing kline candles

* fixCasing fix

* after/before checks added

* added parallel to test

* logic check on BTCmarkets

* removed unused param, used correct iterator

* converted HitBTC to use time.Time

* add iszero false check to candle times

* updated resultlimit to 5000

* new line added

* added comment to exported const

* use configured ratelimit

* fixed pair for test

* panic fixed WIP on fixCasing

* fixCasing rework, started work on readme docs

* enable rate limiter for wrapper issues tool

* docs updated

* removed err from return and formatted currency

* updated Yobit supported status

* Updated HitBTC to use onehour candles due to test exeuction times

* added further details to gctcli output

* added link to docs

* added link to tempalte

* disable FTX websocket in config_example

* fix poloneix

* regenerated poloniex mock data

* removed recording flag
2020-07-08 10:51:54 +10:00
Adam
c2c200cd1b (Exchange) Add FTX exchange support with implementation tutorial (#495)
* initial

* wip

* brokenwip

* broken wipzzzz

* more functions

brokenwip

NO API KEYS

* broken wip

* WIP

* wip

* WIP

work in progress

* WIP

* WIP

* wip

* more wip

* wip

* ws wip

* broken wip

* adding new functions for websocket to work

* trying to fix websocket issues

* websocket bug fix wip

* broken websocket implementation

* WS unauth functions + brokenWS auth func

* authentication problems

* authentication problems fixed

* data handling for websocket

* websocket completed

* remove verbose

* minor error fix changes and testing

* reorganising variable declarations and minor errors fixed

* enabled exchanges updated

* enabled exchanges fixed

* remove keys

* glorious nits

* xdta n shazzy nitzzz

* shazzy n thrasher nitz

* nitz wip

* broken wip

* apichecker donee n make code better

* apichecker donee n make code better

* OB update

* wip

* wip

* all nitz done

* merge conflicts

* go mod tidy

* merge conflicts

* PLEASE merge conflicts

* new funcs added n binanceapi check update

NO APIKEYS

* basic tests

* linter fixs

* linter fixs

* remove verbose

* test errors fixed

* remove comented code

* minor changes

* some tests fixed

no apikeys

* documentation work

* documentation

* wip

* ryan nitz

* nits addressed

* unnecessary conversion

* no fail

* remove verbose

* type field checking

* broken

* websocket nits fixed

* some thangs

* remove verbose

* fix function

* linter issues

* test error fixed

* nits

* bumperino fixed

* very small change

* nits

* errors fixing

* errors fixing retry

* linters

* thrasher glorious nits

* more changes

* changes

* 2 more changes to be addressed

* 2 more changes to be addressed

* issues addressed

* whip

* changes

* missed change

* changes

* currency issues

* changes

* unsaved

* int64

* HUGE

* HUGE

* NO NITS PLS

* no more

* YES

* :

* changes

* PLEASE

* n another one

* thanks guys

* ill believe in god if this ever ends

* :D
2020-07-02 10:38:50 +10:00
Adrian Gallagher
8afee0b4b2 Miscellaneous bug fixes (#513)
* Various bug fixes

* Deadlink, cleanup plus bug fixes

* Various Kraken fixes

* Add convert func for decimal unix timestamps

* Convert all test times to UTC

* Kraken: Make assets a pointer to prevent excessive copying

* Docker slash fix

* Address nits plus bump ITBit last checked pairs timestamp

* Set pairs to enabled pairs when getting active orders

* Use asset translator for UpdateAccountInfo and more checks for the exchange template tool

* Address MadCozBadd's nits

* Make exchange var 2 chars

* Make program more user friendly

* Project wide comment checks and exchName check

* Fix Huobi indexing bug and use correct pair formatting

* Address nits + readme change
2020-06-03 12:48:36 +10:00
David Ackroyd
56e535001c Implement Request Retry and Backoff (#491)
Refactoring the timeout retries into a more general 'retry policy' with
 support for retrying on HTTP 429 (Too Many Requests) and other responses
 with a `Retry-After` header

The delay between requests is controlled by a combination of a 'backoff'
 (currently only a simple linear backoff), and honouring the
 `Retry-After` value (longest delay wins)

This makes the 'rate limiter' an optional argument as well, removing the
 use of `nil` when one isn't supplied

Signed-off-by: David Ackroyd <daveo.ackroyd@gmail.com>
2020-05-05 13:12:29 +10:00
Adam
7dd3f13c8d Exchange API modification monitor (#404)
* WIP

* Variable Name Changes and other PR Changes

* Name Changes and Error Checks

* WIP

* Some new functionality added and nits fixed

* WIP on Nits

* More trello functionality added and some PR changes

* Minor changes and linter fixes

* Errors Fixed and some PR Changes

* PR Changes

* WIP

* WIP

* Addition of Config

* WIP of Readme file

* Readme WIP

* Addition of ReadMe

* Linter Issues fixed

* Minor Errors fixed

* fixed test timeout issues and added api keys check for tests

* Errors fixed

* Nits WIP

* Nits fixed

* More efficient mutex usage

* Consts cleaned up

* Readme file fixed

* linter issues fixed

* format issues:

* readme updates and nits

* remove backup file and address nits

* backup file removed for real

* OkexOkcoin updates and linter fixes

* errors fixed

* Nits

* errors fixed

* formatting issues

* race conditions fixed

* files fixed

* go mod n go sum

* WIP adding rate limits

* Added rate limits

* Remove test vars

* go mod tidy

* improving readme file and 2digits for trello auto update

* Added space

* removed space

* addressing nits

* documentation update

* more nits

* board id check

* linter fixes

* test race fixed

* test file match

* errors fixed

* nitz

* LOWERCASE flags

* more nitz

* broken wip

* flag addition fixed

* error fixed

* documentation updated for adding exchange using flags

* something

* add comment

* nitz fixed

* pr changes

* added gemini exchange

* testfile saving changes

* nitz

* fixing done

* pr changes

* nits

* WIP

* minor error fix

* wip

* wip

* WIP

* pr changes

* pr changes

* pr changes

* nits

* nits fixed

* t.Fail removed

* nitz

* disable lakeBTC

* contributors fixed

contributors fixed

ALL THE NITZ

OMG

* OMGGG

ALL THE NITZ

OMG

* name change errors fixed

* linter issues fixed

* linterzz

* Added disabling feature for exchanges

* bug fix

* small feature added

* fixed

bug fix:

no keys

* nits

* test

* test2

* test 3

* linter fixed

* testing

* testing

* remove logs

* OMG

* bugfix

* cyam88888

* nits

* error handled

* minor nits

fixed createnewcheck issue

fixed createnewcheck issue

* functionality change

some functionality changes

no api keys 4u :D

* trello names thro config

* err check
2020-04-01 11:15:10 +11:00
Andrew
b26ec86d43 Withdraw additional functionality (validation/submission/tracking) (#409)
* reworked request struct and exchange response started work on validation system

* removed import cycle until work around

* Added intial withdraw support via CLI added

* Added Crypto command to gctcli

* moved var declartion to single line

* Test updates for binance and anx

* All exchange tests have been updated test coverage added to validate

* First pass at adding withdrawl select from database

* started adding basic lru cache system

* Added basic LRU cache including Add Get Remove Contains ContainsOrAdd Clear

* wording changes on comments

* removed exported var's in strut as they are not required

* Added README

* README updates

* corrected ID on commands

* rm line :D

* merged in origin/cache

* linter fixes (gofmt)

* Added basic cache lookup to events

* swapped to mutex over rwmutex updated comments

* unexported getNewest & getOldest

* unexported getNewest & getOldest

* Updated comments and cited references in source

* updated comments

* WIP

* Migrated exchange WithdrawFiat wrapper to new struct response

* Migrated exchange WithdrawFiat wrapper to new struct response

* started work on bank management

* Added exchange level banking details back with migration to banking package

* Removed broken tests for now

* Added validation to bank accounts

* removed duplicate bank details from withdraw struct

* Test coverage increased

* gofmt

* merged upstream/master with clean up

* First pass at adding command line linking to gctcli

* added validation for crypto address, added gctcli support to retreive previous withdrawal requests

* general cleanup

* general cleanup

* reordered imports

* Increased test coverage moved to database sublogger

* Pass incorrect currency no longer return error from c.CheckBankAccountConfig

* remove TestMain() for now as other tests in this package will need to be reworked

* Happy little race car

* Reverted to upstream tests

* Added test covarege for validation method, corrected response on cli protobuf

* query clean up and removal of duplicated code

* cleaned up queries into singlem ethod increased test coverage

* Migrated international fund withdraw to new exchange response and added cache size override

* Migrated international fund withdraw to new exchange response and added cache size override

* Extended gctcli commands

* lowered default cache to 25

* small code clean up

* added get by date method

* add returned error

* cli commands cleaing return error on nil results to fix out of bounds

* merged write & read helpers into one for test coverage and increased engine/withdraw test coverage

* gofmt

* Added test coverage for valid ID

* removed unused param

* converted to use timestamp package from protobuf

* gofmt

* use built in RFC3339 timestamp

* remove setting of CreatedAt & UpdatedAt and allow ORm to take care of it

* also use ptype on byid

* code flow improvements

* remove inverse conditional check and linters run

* removed test data

* removed comment

* removed comment

* also write failures to database for auditing

* converted to use default time for start & end

* Default to time.Now() minus 30 days

* Default to time.Now() minus 30 days

* small code clean up

* fixed missing semicolon on migrations, code clean up

* updated sqlite migrations

* Added additonal check for exchange level bank account if global is not found

* case sensativity fix for currency names

* use correct compare

* test coverage fixed

* removed space

* return pointer to banking.Account

* return pointer to banking.Account

* added else check back to validate()

* Added empty string as default to migration over NULL due to retrivial of data
2020-02-26 15:45:13 +11:00
Andrew
7fccc03164 (Logger) Rename package to log (#444)
* renamed package to log to stop side import requirement

* reverted comment changes

* reverted comment changes

* one more reverting wording back to logger

* wording changes on comments
2020-02-12 18:09:56 +11:00
Andrew
75ac5ee791 (Exchange Interface) Convert Fetch & Update orderbook/ticker methods to return pointers (#398)
* moved order and ticker fetching to return a pointer

* return nil instead of empty struct

* fixed incorrect nil

* general cleanup
2019-12-17 15:54:09 +11:00
Adrian Gallagher
44aa1e306c Fix exchange_template missing import (#401) 2019-12-17 15:53:11 +11:00
Andrew Jackson
9178dbcc61 removed withdraw from struct name: 2019-12-13 08:36:07 +11:00
Andrew Jackson
a91d147781 Moved withdraw structs into own package 2019-12-12 14:16:45 +11:00
Adrian Gallagher
63191ce3ec Engine QA (#381)
* 1) Update Dockerfile/docker-compose.yml
2) Remove inline strings for buy/sell/test pairs
3) Remove dangerous order submission values
4) Fix consistency with audit_events (all other spec files use
CamelCase)
5) Update web websocket endpoint
6) Fix main param set (and induce dryrun mode on specific command line
params)

* Engine QA

Link up exchange syncer to cmd params, disarm market selling bombs and fix OKEX endpoints

* Fix linter issue after merge

* Engine QA changes

Template updates
Wrapper code cleanup
Disarmed order bombs
Documentation updates

* Daily engine QA

Bitstamp improvements
Spelling mistakes
Add Coinbene exchange to support list
Protect API authenticated calls for Coinbene/LBank

* Engine QA changes

Fix exchange_wrapper_coverage tool
Add SupportsAsset to exchange interface
Fix inline string usage and add BCH withdrawal support

* Engine QA

Fix Bitstamp types
Inform user of errors when parsing time accross the codebase
Change time parsing warnings to errors (as they are)
Update markdown docs [with linter fixes]

* Engine QA changes

1) Add test for dryrunParamInteraction
2) Disarm OKCoin/OKEX bombs if someone accidently sets canManipulateRealOrders to true and runs all package tests
3) Actually check exchange setup errors for BTSE and Coinbene, plus address this in the wrapper template
4) Hardcode missing/non-retrievable contributors and bump the contributors
5) Convert numbers/strings to meaningful types in Bitstamp and OKEX
6) If WS is supported for the exchange wrapper template, preset authWebsocketSupport var

* Fix the shadow people

* Link the SyncContinuously paramerino

* Also show SyncContinuously in engine.PrintSettings

* Address nitterinos and use correct filepath for logs

* Bitstamp: Extract ALL THE APM

* Fix additional nitterinos

* Fix time parsing error for Bittrex
2019-11-22 16:07:30 +11:00
lozdog245
52e2686b9e Change exchanges usage of GetName to Name (#378)
* Change exchanges usage of GetName to Name

* Changed GetName to Name
2019-11-20 09:50:14 +11:00
Ryan O'Hara-Reid
63a9e9fcb1 Engine QA - Order update (#372)
Engine QA - Order update
2019-10-30 14:06:48 +11:00
Scott
ccfcdf26aa Engine: Protocol Features, coverage, types, BTC markets websocket (#368)
* Attempts to update orderbook so it doesn't need to sort

* Reverts the ws ob stuff. Gets rid of sorting because it happens later. Adds some exchange features

* update existing feature lists. Expands list definition to match my emotions

* Adds bithumb bitmex and bitstamp. adds a couple more types

* Features for you, features for me, features for bittrex, btcmarkets, btse, coinbasepro, coinut, exmo, gateio and gemini

* Features for hitbtc, huobi, itbit, kraken, lakebtc, lbank, localbitcoins, okcoin, okex, poloniex, yobit, zb

* Who can forget good old alphapoint?

* Adds btcmarksets websocket :glitch_crab: fixes alphapoint features

* Adds extra data not in the documentation :/

* Replaces websocket features by using protocol features. However, it breaks it due to import cycles. I'm not sure what I'll do just yet

* Removes import cycle via duplicate structs.

* Increases coverage of config with `TestCheckCurrencyConfigValues`. Moves all currency pair package types into their own files or places it at the bottom of files if necessary

* Increase coverage in code.go

* One way of determining a test has failed, is when to it fails. Removed redundant explanation

* Increases code coverage of conversion

* Lint fixes

* Fixes orderbook tests

* Re-adds sorting because its important to still have the internal pre-processed orderbook to be representative of a real orderbook

* Secret lints that did not show up via Windows linting

* Adds protocol package to contain exchange features

* Fixes protocol implementation

* Fixes ws tests

* Addresses the following: Removes st-st-stutters in config types, changes GetAvailableForexProviders -> GetSupportedForexProviders, removes errors from tests where error is nil, removes orderbook setup when not necessary, removes import newlines, removes false bools from declaration, changes should of to should have

* imports and casing

* Fixes two more nil error checks
2019-10-22 10:56:20 +11:00
Ryan O'Hara-Reid
e2d57540a6 Config overwrite bugfix (#363)
* Fix bug where on parsing an alternate new config it will overwrite main config.json in gct dir

* Stop movement of config.json file from root dir when a new config is parsed in

* Stop overiding config.json at gct dir with new config.json from root directory

* RM LN :D

* Fix bug where promptforconfig in config_encryption.go overwrites default config
Ensure periphery command packages do not interact or save over configuration
Ensure tests to not save over or change current testdata/config
2019-09-27 16:03:41 +10:00
Adrian Gallagher
94a784d63f Merge branch 'master' into engine 2019-08-22 18:32:57 +10:00
Adrian Gallagher
8ecd8a0c16 Merge branch 'master' into engine 2019-08-09 15:46:24 +10:00
Andrew
3de1d94e5f New logging system (#319)
* First pass at adding new logging system

* NewLogger

* NewLogger

* WIP

* silly bug fix

* :D removed files

* removed old logging interface

* added tests

* added tests

* Started to add new lines to all f calls

* Added subsystem log types

* Logger improvements

* Further performance improvements

* changes to logger and sublogger creation

* Renamed Logging types

* removed old print statement

* changes based on feedback

* moved sublogger types to own file

* :)

* added console as output type

* added get level command

* added get/set log level via grpc command

* added check for output being empty for migration support

* first pass at log rotation

* added log rotation

* :D derp fixed

* added tests

* changes based on feedback

* changed log type

* comments

* renamed file -> fileSettings

* typo fix

* changes based on feedback

* gofmt ran on additional files

* gofmt ran on additional files
2019-07-07 05:20:31 +10:00
Adrian Gallagher
9ff4471a45 Merge branch 'master' into engine 2019-06-21 18:10:55 +10:00
Ryan O'Hara-Reid
20c24601fb Asset update to fix minor stutter (#316) 2019-06-17 09:02:07 +10:00
Adrian Gallagher
e965e54e09 Split up common.go, file path fixes and much more 2019-06-04 17:04:32 +10:00
Adrian Gallagher
f5914e8c10 Engine changes 2019-05-22 17:06:38 +10:00