Commit Graph

472 Commits

Author SHA1 Message Date
David Ackroyd
0440f14179 Binance: Fix Orderbook Management (#506)
With the incorrect implementation currently, orderbooks have incorrect
 entries in them which never get removed. This results in an
 unsynchronised state, having issues such as bid prices greater than the
 lowest ask prices

The websocket depth updates must be subscribed to before getting a
 snapshot. Any updates from the websocket that have earlier update ids
 must be discarded.

https://binance-docs.github.io/apidocs/spot/en/#how-to-manage-a-local-order-book-correctly
Signed-off-by: David Ackroyd <daveo.ackroyd@gmail.com>
2020-05-18 16:16:09 +10:00
David Ackroyd
802d265d56 Binance: Update NewOrder API (#507)
Including new fields for request/response and allowing test orders.

Signed-off-by: David Ackroyd <daveo.ackroyd@gmail.com>
2020-05-18 15:42:50 +10:00
David Ackroyd
56e535001c Implement Request Retry and Backoff (#491)
Refactoring the timeout retries into a more general 'retry policy' with
 support for retrying on HTTP 429 (Too Many Requests) and other responses
 with a `Retry-After` header

The delay between requests is controlled by a combination of a 'backoff'
 (currently only a simple linear backoff), and honouring the
 `Retry-After` value (longest delay wins)

This makes the 'rate limiter' an optional argument as well, removing the
 use of `nil` when one isn't supplied

Signed-off-by: David Ackroyd <daveo.ackroyd@gmail.com>
2020-05-05 13:12:29 +10:00
Andrew
70615279bd (GCTScript) Add technical analysis support via script bindings (#467)
* added mfi and example

* renamed to moving average

* converted to array return type and added obv and mfi

* started work on test coverage

* test coverage added for rsi & mfi

* test coverage added for all indicators removed go mod replace moved to append helper method

* moved all indicators to new appendTo and increased test coverage

* added additional test and bumped go-talib to latest commi

* go.mod update

* linter fixes

* go mod clean up

* small fixes

* reverted changes from previous attempt to rework as data is still incorrect now passing full OHLCV data back to script binding

* testing new structure of passing full ohlcv data

* started linking ohlcv to gctscript

* OHCLV link up completed reworking passing back to indicators started

* OHCLV link up completed reworking passing back to indicators started

* added test coverage for tofloat

* linter fixes (gofmt)

* removed unused value

* improved test coverage

* added correct detection for 1w added ParseInterval test coverage moved OHCLV string to const

* removed unused value

* first round of changes addressed

* all indicators have been split with packages named after each indicator and a new calculate() method added

* linters

* fixed tests

* added check to check ta is running in validator for uploading

* Added test data for OHLCV testing new indicator interface for wrapper

* typed const to float64

* reworked validator data to generate previous timestamps

* rewored macd to return slice of array

* adding bbands linking and example

* why didn't this pick it up before :D

* bumped up total number of modules for test

* moved parseIndicator to exchange added comments

* test coverage added for ParseMAType & ParseIndicatorSelector

* gofmt

* WIP changes

* updated tests for bbands  & obv bumped to latest go-talib

* move multiple use strong to const

* reverted rpc.pb.go to master

* added 4w option

* removed selector from obv as unneeded

* improved test coverage and reworked all indicator methods on how they pass errors back

* order incoming OHCLV data

* revert go.mod

* removed verbose toggles

* added spot asset type

* removed 4w as its unused/uncommon

* renamed

* reworked further tests

* converted all examples to use coinbasepro for consistency

* updated all date ranges to 2019 + 6 months

* backported binance OHLCV wrapper from #479

* removed o

* rounded numbers

* chnage requests addressed and attempt to fix MACD... today has been really unproctive code wise :D

* Migrated to gct-ta library

* Corrected test import

* wording changes on test

* removed TA lib from go.mod

* PR changes addressed

Removed parallel running from tests due to slight possibility in very extreme cases TestExecution might not be set to the expected value and will cause lower test coverage

* removed pkg folder

* bumped gct-ta version

* gct-ta version bump
2020-04-24 15:36:49 +10:00
David Ackroyd
342b2680d1 Binance: Fix Request Rate Limits (#483)
Request types / variations contribute different weights towards the limit
 Binance enforces. These can be considerably more than 1 per request,
 which results in the server side limits being hit, producing 429 and 418
 responses and bans
2020-04-20 22:57:28 +10:00
David Ackroyd
c0d2ac5e51 Binance: replace deprecated /v1 API calls (#482)
Several /v1 API calls have been deprecated, and will be removed shortly.
 Moving these across to their /v3 equivalents

Capturing of the klines request via the recorder was failing, as it did
 not handle the structure where the klines response in an array of
 mixed primitive arrays. Excluding primitive values from the exclusion
 checks addresses this
2020-04-17 08:59:06 +10:00
Adrian Gallagher
0d787bc259 Bump golangci-lint to v1.24.0, linter fixes and general code improvements (#478)
* Bump golangci-lint version, update Go version deps and generic code improvements

* Fix wesbocket resp nil check and zip closures

* Update pprof path
2020-04-09 13:07:32 +10:00
Adam
7dd3f13c8d Exchange API modification monitor (#404)
* WIP

* Variable Name Changes and other PR Changes

* Name Changes and Error Checks

* WIP

* Some new functionality added and nits fixed

* WIP on Nits

* More trello functionality added and some PR changes

* Minor changes and linter fixes

* Errors Fixed and some PR Changes

* PR Changes

* WIP

* WIP

* Addition of Config

* WIP of Readme file

* Readme WIP

* Addition of ReadMe

* Linter Issues fixed

* Minor Errors fixed

* fixed test timeout issues and added api keys check for tests

* Errors fixed

* Nits WIP

* Nits fixed

* More efficient mutex usage

* Consts cleaned up

* Readme file fixed

* linter issues fixed

* format issues:

* readme updates and nits

* remove backup file and address nits

* backup file removed for real

* OkexOkcoin updates and linter fixes

* errors fixed

* Nits

* errors fixed

* formatting issues

* race conditions fixed

* files fixed

* go mod n go sum

* WIP adding rate limits

* Added rate limits

* Remove test vars

* go mod tidy

* improving readme file and 2digits for trello auto update

* Added space

* removed space

* addressing nits

* documentation update

* more nits

* board id check

* linter fixes

* test race fixed

* test file match

* errors fixed

* nitz

* LOWERCASE flags

* more nitz

* broken wip

* flag addition fixed

* error fixed

* documentation updated for adding exchange using flags

* something

* add comment

* nitz fixed

* pr changes

* added gemini exchange

* testfile saving changes

* nitz

* fixing done

* pr changes

* nits

* WIP

* minor error fix

* wip

* wip

* WIP

* pr changes

* pr changes

* pr changes

* nits

* nits fixed

* t.Fail removed

* nitz

* disable lakeBTC

* contributors fixed

contributors fixed

ALL THE NITZ

OMG

* OMGGG

ALL THE NITZ

OMG

* name change errors fixed

* linter issues fixed

* linterzz

* Added disabling feature for exchanges

* bug fix

* small feature added

* fixed

bug fix:

no keys

* nits

* test

* test2

* test 3

* linter fixed

* testing

* testing

* remove logs

* OMG

* bugfix

* cyam88888

* nits

* error handled

* minor nits

fixed createnewcheck issue

fixed createnewcheck issue

* functionality change

some functionality changes

no api keys 4u :D

* trello names thro config

* err check
2020-04-01 11:15:10 +11:00
Scott
4cb2f4603a Bugfix: Fixes OKCoin websocket port (#475) 2020-04-01 09:27:24 +11:00
Scott
74cb7558cc (FIX) Update ZB API URLs (#474)
* Updates websocket URL

* Minor test changes, URL changes too

* verbosity purposity prepostery

* Fixes issue where websocket data passed through the round trip was invalid

* Adds a test protection
2020-03-27 13:19:21 +11:00
Andrew
47cf874911 (BTC Markets) Add support for GetHistoricCandles wrapper (#472)
* reworked btcmarkets GetMarketCandles()

* pass utc back
2020-03-17 10:02:13 +11:00
Ryan O'Hara-Reid
2c7e531c5c Initial kline trade converter && restructure wrapper functions (#454)
* Initial kline trade converter && restructure wrapper function

* Addr nits

* fix linter issues

* fix requested

* fix after merge interface issue with fakepassingexchange

* consistentizations

* Addr glorious nits

* Added in explicit interval strings for gctcli client (ease of use)

* rm value stutter

* Addr nits

* update protobuf and push regen

* go mod tidy

* change description of usage for granularity
2020-03-16 10:31:07 +11:00
Allie
80f4cf1784 Bug-fixes: #459 #460 (#461)
* Fix bitmex order params

* Fix insecure endpoint check

* comment on Title funcs

* tests for Title func

* fix order_test

Co-authored-by: Allie c( ⁰ 〰 ⁰ )੭ <88z@pm.me>
2020-03-09 09:20:33 +11:00
Adrian Gallagher
3fae5a0924 Implementation of Exchange GetOrder through GRPC for Coinbase (#453)
* fix 440: SubmitOrder and CB-ACCESS-TIMESTAMP value

* removed gctcli executable and added it to gitignore

* implemented GetOrderInfo

* getOrder also has trades attached to an order

* getOrder also has trades attached to an order

* fix 440: SubmitOrder and CB-ACCESS-TIMESTAMP value

* removed gctcli executable and added it to gitignore

* fix 440: SubmitOrder and CB-ACCESS-TIMESTAMP value

* removed gctcli executable and added it to gitignore

* minor

* go lint fixes

* removed clutter

* fixed Amount, Executed Amount, Remaining Amount

* removed default value declarations

* removed useless nil chceck

* iteration rewritten go style from java style

* returning error if GetFills fails

* returning error if GetOrderInfo fails

* ExecutedAmont value is FilledSize

* added TODO to fix asset type

* chnages after merging #446

* range without copy value, using index

* removed useless slice initialisation

* couple of nits from code review

* fix indentation

* Update rpc.proto

Fixes indentation

Co-authored-by: Adrian Gallagher <thrasher@addictionsoftware.com>
2020-03-05 10:19:58 +11:00
Adrian Gallagher
ba07c86af9 Bitfinex: Implement leaderboard API (#458)
* Bitfinex: Implement leaderboard API

* Expand test coverage

* fix grammer
2020-03-05 08:40:23 +11:00
Scott
b686cf2e0e Feature: Websocket order handling (#446)
* Initial changes, removing exchange name as an arg and puts it in the pointer struct. Adds case to ws routines

* Adds CancelAllOrders func, adds GetByExchangeAndID. Adds modify handler in routines.go

* initial poor attempts to have bitmex work with new datahandler handlers. fixes ordersides

* bitmex Completes new order

* Better bitmex handling, but not complete. Begins a gargantuan task of unifying order data structs. Sometimes an order update will contain lot's of information, so its best to be able to update all fields of our orders, rather than just an arbitrary subset. As a result, everything will be broken for the foreseeable future :glitch_crab:

* Removes old order handler which did nothing. Updates order properties for everything everywhere - now consistent. Changes order status. Adds asset type and wallet address to all order types

* Adds order updater to update only relevant fields since the object is generic, we don't know what fields are passed from what exchanges. Adds "lastupdated" field to order.Detail. Expands order cancellation for engine orders.

* Ensures that new orders are added to the ordermanager's order store. Saaa many comments. Internalises orderStore get func. Adds internalOrderID to orderdetail and adds websocket support for it

* Fixes a cancelAllOrders oopsie doopsie

* Adds potential func to update orderdetails from an orderdetail struct. Unsure if will keep.

* Begins btcmarkets implementation. Expands order "stringToOrder" funcs to allow for some more flexible string coversions. Removes order.Submit via websocket as it would cause unlimited order place issues :D

* Finishes btc markets without testing

* Adds untested ws auth func to btse

* Finises btse, fixes btcmarkets bug

* Adds coinbasepro support

* Fixes a few more fields in coinbase pro and readds the extra subs

* Begins work on coinbene. Plus theyve added a new ws connection yeee

* Wasted a bunch of time adding support to an additional websocket that isn't needed ;_; Fixed a bug in coinbasepro. Fully kitted out coinbene support. Updates order types with all fields

* Removes extra websocket connection ;_;

* Finishes gemini. Fixes order side unknown

* Adds okgroup support. Moves byte reading to another function to allow for unit testing. Updates routines to use pointers. Updates date update handling for order details

* Finishes order data for okgroup websocket, but starts the STRANGE process of converting all other websocket endpoints to be a little less silly

* Cleans up okroup websocket implementation. Fixes bug in Gemini

* Adds poloniex support. Updates ws order handling

* new bitmex support. Adds some tests now that its all in its own func. Fixes poloniex bug

* Begins work on authenticated binance websocket

* Attempts to track user data via binance websocket

* Maybe finishes Binance websocket support

* Begins adding test coverage to orders.go. Updates names of script properties to match updated

* Begins an experiment with code coverage. Fixes more rebase issues

* Completes orders coverage. Botches a few other things though. Fixes more scripting stuff

* All tests in engine package pass

* Adds some loevely routine tests

* Moves ordermanager to test Bot ordermanager
Adds lovely routine tests to ensure things that get sent to be handled the data handler are handled by the data handler by handling them

* Replaces "wsHandleData" with "wsReadData" as that's what its going to do now.

* Splits all wsHandleData into wsReadData and wsHandleData to allow for easy testing via sending []byte json examples to test proper functionality. Breaks so many tests

* Fixes majority of test issues. But data races which are tough on the engine package

* "Fixes" test by removing shutdown test. It interferes with too many things. Requires some thought

* Tests all the binance websocket points

* Adds better bitfinex websocket support.

* Adds testing for bitfinex, bitstamp and btcmarkets. Fixes websocket bugs encountered

* Adds BTSE ws tests. Fixes bugs in ws

* Adds coinbase pro tests. Fixes any issues

* Coinbene tests

* Starts to handle coinut. Runs into a problem conceptually regarding websocket roundtrip and orders. Both events need to happen without impacting eachother/racing

* Addresses a data race issue regarding websocket and bot order management submission - order submission locks at an earlier point to prevent routines.go from creating an order before order submission creates it. Updates rpcserver to use order management bot to submit orders.

* Finishes the hectic coinut testing

* Adds tests for gateio

* Fixes rebase issues. Updates tests to work without being overloaded

* Begins testing of gemini. fixes up minor issues

* ginishes gemini tests and fixes

* Adds hitbtc tests. Fixes all the many issues with hitbtc websocket

* Adds remaining tests. Increases default test channel limit again

* Begins work towards huobi tests

* Finishes huobi tests

* Fixed all mythical rebase adventures

* Begins kraken transformation

* Finishes kraken. Fixes coinbene leverage now that its changed

* Begins okgroup testing

* Adds okgroup ws tests

* Does some poloniex

* Fixes basic curreny issue by extracting to func

* Begins redesign of poloniex websocket datahandling. Completes authenticated handling, now onto unauth

* Finishes poloniex revision

* Finishes ZB additions

* Fixes data races

* Fixes rebase issues. Fixes bad kraken logic

* Fixes after reviewing code

* lint everywhere

* Fixes lingering lints

* lint

* Adds test coverage to order detail and modify updating

* Fixes linting

* Fixes huge int, fixes date tests

* Adds GetByExchange, adds test for it. Protects fakepass echange. Renames DisplayQty to DisplayQuantity. Removes verbose. Adds some websocket properties.  Updates bitmex asset type in test

* Addresses timestamps, type abbreviations, verbosity. Expands binance kline switch cases. Updates some websocket capabilities.

* Adds coverage to the stringToOrderType/Status functions introduced in PR

* Minor fixes addressing some time, error text and use of StringDataCompareInsensitive

* Introduces shiny new system which checks if there is an awaiting ID, if found, processes via wrapper method, else, goes through wsHandleData method. Removes weird locking system from wrapper/websocket data race. Updates bitfinex to properly handle websocket order requests and notifications

* Moves fakePassingExchange to test_helper. Fixes some order side implementations for trades. Botches a new error type

* Adds new error type to track and handle order classification errors separately

* Fully fleshes out ClassificationError for all instances of status conversion. Even in order trades and some wrapper functions

* Introduces common.SimpleTimeFormat for "2006-01-02 15:04:05". Fixes binance and bitfinex issues with auth endpoint use, map casting. Expands more order.ClassificationError usage. Fixes some more generic websocket response errors

* Future proofs order updating by utilising asset types. Expands testing to accomodate. Adds shiny new time type. Expands wrapper websocket functionality definitions

* minty linty

* Broken end of day code addressing basic nits on comments, returns and currency conversion

* Adds testing to btcmarkets websocket. Also updates websocket orderbook to use update instead

* Fixes fun rebase fun fun so fun

* Addresses minor nits regarding changed interface and comments

* Creates new function `GetRequestFormattedPairAndAssetType` to retrieve a currency pair and asset type based on a string. It will iterate over enabled pairs and compare them to formatted pairs and then return that pair if found.

* Fixes test

* Adds a single line to the end of the file, because that would be really bad if it wasn't there

* Updates fakepassexchange to not use params, updates test params, uses fatal in some tests where its important, updates order manager to have a rwmutex, removes some returns, improves ws key test for binance, updates properties to reflect their actual values, adds some more websocket properties

* Addresses binance switch linting

* Updates leverage property to int64

* Fixes what was broken
2020-03-03 13:32:14 +11:00
Adrian Gallagher
31fe74a380 Fix Coinut nonce issue based on new API nonce rule (#456) 2020-02-28 16:31:52 +11:00
Andrew
b26ec86d43 Withdraw additional functionality (validation/submission/tracking) (#409)
* reworked request struct and exchange response started work on validation system

* removed import cycle until work around

* Added intial withdraw support via CLI added

* Added Crypto command to gctcli

* moved var declartion to single line

* Test updates for binance and anx

* All exchange tests have been updated test coverage added to validate

* First pass at adding withdrawl select from database

* started adding basic lru cache system

* Added basic LRU cache including Add Get Remove Contains ContainsOrAdd Clear

* wording changes on comments

* removed exported var's in strut as they are not required

* Added README

* README updates

* corrected ID on commands

* rm line :D

* merged in origin/cache

* linter fixes (gofmt)

* Added basic cache lookup to events

* swapped to mutex over rwmutex updated comments

* unexported getNewest & getOldest

* unexported getNewest & getOldest

* Updated comments and cited references in source

* updated comments

* WIP

* Migrated exchange WithdrawFiat wrapper to new struct response

* Migrated exchange WithdrawFiat wrapper to new struct response

* started work on bank management

* Added exchange level banking details back with migration to banking package

* Removed broken tests for now

* Added validation to bank accounts

* removed duplicate bank details from withdraw struct

* Test coverage increased

* gofmt

* merged upstream/master with clean up

* First pass at adding command line linking to gctcli

* added validation for crypto address, added gctcli support to retreive previous withdrawal requests

* general cleanup

* general cleanup

* reordered imports

* Increased test coverage moved to database sublogger

* Pass incorrect currency no longer return error from c.CheckBankAccountConfig

* remove TestMain() for now as other tests in this package will need to be reworked

* Happy little race car

* Reverted to upstream tests

* Added test covarege for validation method, corrected response on cli protobuf

* query clean up and removal of duplicated code

* cleaned up queries into singlem ethod increased test coverage

* Migrated international fund withdraw to new exchange response and added cache size override

* Migrated international fund withdraw to new exchange response and added cache size override

* Extended gctcli commands

* lowered default cache to 25

* small code clean up

* added get by date method

* add returned error

* cli commands cleaing return error on nil results to fix out of bounds

* merged write & read helpers into one for test coverage and increased engine/withdraw test coverage

* gofmt

* Added test coverage for valid ID

* removed unused param

* converted to use timestamp package from protobuf

* gofmt

* use built in RFC3339 timestamp

* remove setting of CreatedAt & UpdatedAt and allow ORm to take care of it

* also use ptype on byid

* code flow improvements

* remove inverse conditional check and linters run

* removed test data

* removed comment

* removed comment

* also write failures to database for auditing

* converted to use default time for start & end

* Default to time.Now() minus 30 days

* Default to time.Now() minus 30 days

* small code clean up

* fixed missing semicolon on migrations, code clean up

* updated sqlite migrations

* Added additonal check for exchange level bank account if global is not found

* case sensativity fix for currency names

* use correct compare

* test coverage fixed

* removed space

* return pointer to banking.Account

* return pointer to banking.Account

* added else check back to validate()

* Added empty string as default to migration over NULL due to retrivial of data
2020-02-26 15:45:13 +11:00
Christian Achilli
def3cdf226 fix 440: SubmitOrder and CB-ACCESS-TIMESTAMP value (#441)
* fix 440: SubmitOrder and CB-ACCESS-TIMESTAMP value

* removed gctcli executable and added it to gitignore

* removed nonce from conibasepro send payload

* fixed typo in executable name

* Bump github.com/grpc-ecosystem/grpc-gateway from 1.12.2 to 1.13.0 (#442)

Bumps [github.com/grpc-ecosystem/grpc-gateway](https://github.com/grpc-ecosystem/grpc-gateway) from 1.12.2 to 1.13.0.
- [Release notes](https://github.com/grpc-ecosystem/grpc-gateway/releases)
- [Changelog](https://github.com/grpc-ecosystem/grpc-gateway/blob/master/CHANGELOG.md)
- [Commits](https://github.com/grpc-ecosystem/grpc-gateway/compare/v1.12.2...v1.13.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Minor general cleanup and bug fixes (#443)

* Bugfix, remove non-needed code and cleanup some code

* Run go mod tidy

* Remove non-needed test and fix tautological err

* Fix Huobi interim var reference

* (Logger) Rename package to log (#444)

* renamed package to log to stop side import requirement

* reverted comment changes

* reverted comment changes

* one more reverting wording back to logger

* wording changes on comments

* Bump github.com/gorilla/mux from 1.7.3 to 1.7.4 (#445)

Bumps [github.com/gorilla/mux](https://github.com/gorilla/mux) from 1.7.3 to 1.7.4.
- [Release notes](https://github.com/gorilla/mux/releases)
- [Commits](https://github.com/gorilla/mux/compare/v1.7.3...v1.7.4)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* (QOL) Improve test setup for logger, improve test coverage for database (#447)

* Improved test setup for logger, improved test coverage for database

* removed some new lines

* add new line

* removed database config detailS

* removed lines

* code cleanup

* Bump github.com/d5/tengo/v2 from 2.0.2 to 2.0.4 (#449)

Bumps [github.com/d5/tengo/v2](https://github.com/d5/tengo) from 2.0.2 to 2.0.4.
- [Release notes](https://github.com/d5/tengo/releases)
- [Changelog](https://github.com/d5/tengo/blob/master/.goreleaser.yml)
- [Commits](https://github.com/d5/tengo/compare/v2.0.2...v2.0.4)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Bump github.com/d5/tengo/v2 from 2.0.4 to 2.0.5 (#450)

Bumps [github.com/d5/tengo/v2](https://github.com/d5/tengo) from 2.0.4 to 2.0.5.
- [Release notes](https://github.com/d5/tengo/releases)
- [Changelog](https://github.com/d5/tengo/blob/master/.goreleaser.yml)
- [Commits](https://github.com/d5/tengo/compare/v2.0.4...v2.0.5)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Adrian Gallagher <thrasher@addictionsoftware.com>
Co-authored-by: Andrew <andrew@disvelop.net>
2020-02-25 09:30:44 +11:00
Andrew
7fccc03164 (Logger) Rename package to log (#444)
* renamed package to log to stop side import requirement

* reverted comment changes

* reverted comment changes

* one more reverting wording back to logger

* wording changes on comments
2020-02-12 18:09:56 +11:00
Adrian Gallagher
168c966441 Minor general cleanup and bug fixes (#443)
* Bugfix, remove non-needed code and cleanup some code

* Run go mod tidy

* Remove non-needed test and fix tautological err

* Fix Huobi interim var reference
2020-02-12 16:52:05 +11:00
Adrian Gallagher
b949388994 General engine improvements (#437)
* Add exchange manager to engine

* Several improvements for engine and friends

1) New file.Exists func
2) gRPC TLS cert expiration date check and regeneration
3) New donation var for use across the codebase
4) Use Go log package until the logger is initialised

* Add cert tests and create dir tree if it doesn't exist for file.Write

* Link up donation address to documentation tool plus minor adjustments

* Fix remaining donation addrs

* Move non-needed reload exchange funcs

* Revert accidental config_example.json changes 🕯️

* Use go logger for logging until the logger has initiliased, otherwise no output will be seen

* Link up portfolio delay val and other fixes

* Run go mod tidy after dependabot PR

* Address nitterinos
2020-02-06 12:32:01 +11:00
Ryan O'Hara-Reid
0a84c5d97a Request package update & rate limit system expansion (#413)
* Initial rework of rework of requester - WIP

* Implementing and checking rate limits - WIP

* implemented coinbene rate limiting shenanigans

* add in remaining WIP

* fixy

* use authenticated rate limit

* drop ceiling as this can be done with a counter later

* add functionality to struct

* purge config options for rate limiting so as to keep things minimal

* prepare futures and swap rate limiting for implementation

* Address linter issues

* Addressed nits, fixed race

* fix linter issue

* remove global var as this was only setting when newrequester was called

* moved rate limit functionality into its own file

* Update Bitfinex with correct rate limit and test endpoints (WIP)

* finish off bitfinex adjustments

* fixes

* fix linter issues

* slowed rate for coinbasepro

* drop rate limit for huobi as the doc times have intermittent 429 issues.

* Set MACOSX_DEPLOYMENT_TARGET to remove linking warning

* Addr Thrasher nits

* Addr glorious nits

* unexport do request function

* fixed nitorinos

* Fixed something I missed

* move disabled rate limiter into loadexchange and use interface functionality

* Add temp quick fix
2020-02-06 11:44:28 +11:00
Adrian Gallagher
b8dfa443d3 Huobi: Add deposit address and withdraw quota API support (#433)
* Add Huobi deposit and withdraw quota API support
Plus perform come code deduplication

* Go mod tidy
2020-02-03 12:44:46 +11:00
Ryan O'Hara-Reid
a32d16e1f5 Expose auth validator functionality for wrapper (#416)
* expose auth validator functionality for wrapper

* Add REST validation after keys set, package account types for future syncing

* Add transient error checking for initial creddemtial validation

* fix command types

* Addressed nits from glorious person

* Amalgamate body within error when not between 2xx status, added btcmarket specific auth error check

* nit fix for glorious person

* Format fix

* removed unused code

* check transient first then validate if its an exchange specific authentication error, all others will be disregarded

* Addressed glorious nits

* Addressed glorious nits

* Moved account processing to updateaccountinfo func and added in fetch account info

* Add GRPC Account streaming (NOTE: could not complete until sync item added)

* RM exchange check

* Address xtda nits

* RM comment code

* Fix linter issues

* used most recent protoc version

* lbank linter issues fixed

* Addressed nits and changed len check to range in for loops

* Fixed timeout issue

* thrasher nits addressed

* add string holdings
2020-01-31 12:09:24 +11:00
Adrian Gallagher
01e2ab1499 GCTCLI: Destination var/arg check usage fixes (#421)
* GCTCLI param and linter fixes

* Linter fixes

* Add more basic validation and address codelingo nits

* Add arg number support to cancelOrder and more validity checks
2020-01-29 16:27:06 +11:00
Christian Achilli
5ac5ec8fc1 Add OHLC retrieval func (GetHistoricCandles) to all exchanges and expose it as a wrapper func (#414)
* initial wiring to providegethistoricalcandles

* initial wiring to providegethistoricalcandles

* initial wiring to providegethistoricalcandles

* gethistriccandles work from cli using hard coded inputs

* gethistoriccandles RPC service and CLI working fine for coinbasepro

* fixed unit test

* input check on grpc for gethistoriccandles

* updated deps

* fixed the return value when a method is not yet implemented

* code review: fixed CLI input check and int32->int64

* code review: handling wrong exchange name

* added check on granularity and allowing start and end being empty

* code review: removed currency2

* code review: dependency reverted

* improved func comment

* typo in func comment

* get historic values tests

* unit tests for get historical rates on coinbasepro

* using time format time.RFC3339

* names to camel case and improved comments

* test cleanup

* changed to camel case

* added InArray tests

* dropped not needed string time

* enforced use of int64

* fixed make check

* cleaned up code organisation to be consistent

* fixed Travis remarks

* more Travis remarks

* added comments

* regenerated proto files after merge

* linter fix
2020-01-24 16:10:33 +11:00
Andrew
f6fd94ea69 Engine: Scripting support (#383)
* WIP

* updated appveyor and increased deadline 5 seconds due to increased linters being added

* revert files to upstream/engine

* WIP

* WIP

* mod file changes

* added script manager

* Added manager/and cli interfaces to scripting

* Added script task handler

* WIP - Added timer/repeat support and fleshed out wrapper further

* autoload support added + WIP

* WIP commit

* added account balance info

* btc markets temp work around

* WIP - merged with upstream for new order package BTC Markets responses broken

* Cancel order wrapper WIP

*  order wrapper update

* Added test coverage for VM

* moved to map for VM List shutdown of all VM now handled added gctcli commands for list and stop of running scripts

* added override to load/execute for path

* fixed incorrect channel shutdown added further test coverage and restructured gctcli commands into sub commands

* increased test coverage for packages

* Added docs cleaned up tests and example scripts

* Test coverage increased for module/gct/exchange package

* windows fixes

* merged upstream/engine

* WIP

* logger fixes - removed pointer to bool check removed duplicate test check for logger

* remove unused mutex

* added inital upload support

* fix linter issues for go-fmt

* added zip support for uploading and added base for fund withdrawing

* changed error return types and also log errors, fix zip path issue

* improved error outputs and code flow

* pairs response fix added protobuf defs for stop all and list all

* added stop all running scripts general clean up and moved across to OrderManager

* linter fixes (gofmt)

* added list all command

* rewrote zip handler to be cleaner also fixed file overwrite on upload

* added query command reworked tests

* added further error checking to compileandrun corrected use of pointers for accountinfo

* bumped tengo version

* Removed named returns reworded log messages removed unused falseptr

* WIP

* Added virutal machine limit improved config options

* added model for script event added upload validation

* script_event table has been completed, tests for wrapper functions implemented

* README updates

* reverted changes opened new PR to move withdraw struct outs

* intial work on adding withdraw support after merger of withdraw package

* started work on examples

* Added crypto withdraw support

* fix switch case assignment and gofmt project

* Reworking Fiat withdraw request pending #402

* removed double pointer call

* added withdraw support for fiat  currencies

* added tests for withdraw methods increased readme

* removed local tengo require and also fix linter issues

* Added default log size const added basic test for invalid script execution

* First pass at moving wrapper to validator package to allow proper validation of uploaded scripts

* Added script details to README added config test added test for no file extension

* moved tests to const and fixed incorrect pathing

* added test coverage to withdraw package

* corrected file close handling

* point to included configtest.json

* extended validator support when a script is uploaded

* Bug fix on bool logic

* Added mutex

* Don't create autit events on test execution

* reverted common to master

* moved file rename to unix timestamp format

* converted logger enabled back to pointer as i need nilness check also moved scriptid to text over blob

* started work on autoload add/remove support

* First round of PR fixes (mostly commented exports)

* Moved GCTScript load to last, removed unneeded error from cleanup()

* Comment clairty for AuitEventID

* added autoload add/remove command to cli

* added tests for autoload

* Test updates for Exchanges

* linter fixes (gofmt)

* Removed double check of engine pointer

* remove possible nil pointer on GetSpecificTicker

* Fixed not closing file handler on write that causes archive removal to fail

* file handler Close clean ups

* corrected spelling on error return and return invalid name n autoload

* moved strings to cosnt moved bool pointer creation to convert package

* new zip extractor added

* Validation has been added to archive uploads

* removed shadow var on err

* added ok check to conversion

* converted condition check

* basic test for zip extract added

* new zip handler

* reverted back to old atomic loading system

* removed shadow err

* lets add a new line

* added space to error return

* command line toggle for script now works properly

* readme updated

* set configLoaded to true

* check for configLoaded condition

* added mutex to allow for multiple access on virtual machine increased test coverage disable script manager if scripting is disabled

* linked up to enable/disablesubsystem commands

* added start/stop example to readme

* reworked logic on test as check should be done on Load()

* updated to tengo v2

* linters

* lower time on ntp client to stop slippage

* remove all fails if any fail validtion from an archive

* remove vm from list if timer is invalid

* removed shadow on err

* remove config creation from NTPCheck test

* WIP testing DB changes

* add unique constraint

* WIP: created has many model

* linters run

* basic sqlite3 support added for new database format

* linters run

* Added test coverage for script repo

* removed unused print

* updated env vars for CI instances

* updated env vars for CI instances

* Updated test packages

* Test updates for postgresql

* removed invalid tests from postgres

* remove duplication of struct and improved code flow

* general cleanup

* wording changes on log output

* use databasemgr logger and add support for autoload without file extension

* corrected test naming

* return correct error

* return correct error again version 82

* store scriptdata on creation

* Hello

* Errorln -> Errorf

* Removed unused vars

* Read me updates

* testing without parallel

* comment on exported type

* added nil check against VM for test

* add debugging information

* gofmt

* remove verbose and data sent to channel

* Added debug information

* linter fixes (gofmt)

* remove unused CompileAndRun() call

* test sleep to see if issue is timing related

* semi-concurrent map fixes

* one day i will run gofmt or setup precommit hooks

* new line :D

* increased test coverage

* added correct sleep time

* Moved over to sync map

* linter fixes (gofmt)

* goimports

* moved VM related methods to vm.go

* new line at end of file

* trying increased timeout on golangci-lint for appveyor

* add debugging information

* removed timeout

* reworked timeout logic

* linter fixes (gofmt)

* increased test coverage

* increased test coverage

* one day i will run gofmt or setup precommit hooks

* removed unused exchange test

* increased golangci-lint timeout

* Added nil check on shutdown and test coverage for it lowered timeout back to 1:30

* reworked ID system

* removed script hash as it was unused

* added comments on exported methods and read me update

* reorder code

* removed to atomic.value for test execution flag

* increased test coverage

* move add further up execution

* point to correct script file
2020-01-23 13:54:04 +11:00
Vazha
4c33a0738a Bittrex - changed type of field "Id" in DepositHistory struct (#419)
* Bittrex GetDepositHistory method return new DepositHistory struct type

* Id field of DepositHistory type changed to int64

* check exchange (exist and enabled) before calling methods on, to avoid program crashes

* Update bittrex_types.go
2020-01-22 11:30:14 +11:00
Scott
4b9de0fdd0 LintFix: Bittrex properties and tests (#418)
* Updates id property. Changes int to int64. Updates tests

* Adds some happy little spaces
2020-01-16 09:56:09 +11:00
Vazha
838b378c21 Bittrex GetDepositHistory method return new DepositHistory struct type (#417) 2020-01-16 08:10:07 +11:00
Scott
44ac3586a0 Bugfix: Websocket ping/pong improvements (#406)
* Renames func. Creates new func to setup pinghander either via gorilla style or our own

* Cleans up all ping pong handlers.......

* Clears up issues, makes naming a bit better

* Adds tests

* Adds ping support to binance

* Cleans up ping pongs and adds a comment

* Cleans up waitgroup stuff.

* DISCREETLY cleans up woeful function

* Fixes Kraken ping message type. Removes unnecessary test property. Adds `if err == websocket.ErrCloseSent {` to ping func

* +1 for +v
2020-01-03 15:47:46 +11:00
Adrian Gallagher
4e05ad41e3 Exchanges: Remove ANX from codebase (#408)
* Remove ANX from codebase

* Doc changes
2020-01-03 14:12:52 +11:00
Ryan O'Hara-Reid
98a277a4c3 Bugfixes: Bitfinex websocket, ZB market response and portfolio (#397)
* bug fix for websocket orderbook processing

* Fix more panics

* fix linter issue

* kick panic can down the road

* temp fix for issue with a 404 returned error as chainz.cryptoid dropped eth support

* Address nits and fixed orderbook updating

* Fix trade data, rm'd event time from struct

* fix time conversion for huobi

* Actually process kline data and fix time stamps

* btse time conversion fix and RM log, as it seems that the gain is reflecting transaction side. Drop ticker fetching support because there does not seem to be support on docs. And added trade fetching support.

* revert huobi println

* Adressed suggestion

* rm unnecessary assignment

* rm unnecessary check and assign

* fix conversion mishap

* fix currency conversion bug

* update websocket logging

* RM websocket type which stops conversion and copy

* fix linter issue, add in unknown side type
2019-12-19 13:40:30 +11:00
Adrian Gallagher
467d8d91a2 Coinbene: Add swap endpoints and code cleanup/bugfixes (#403)
* Start Coinbene SWAP implementation

* Flesh out more API endpoints

* Code cleanup

* Add more endpoints, bug fixes and order validation checks

* More endpoints, tests and bugfixes

* Remove omitempty for enabled/available pairs
2019-12-18 17:36:07 +11:00
Andrew
e37512e94f (BTC Markets) Ticker batching support (#399)
* moved to getticekers endpoint

* add support for tickerbatching

* const and things

* moved openOrders to bool

* gofmt and clean up test logs

* revert canManipulateRealOrders to false
2019-12-18 10:56:02 +11:00
Andrew
40122b3094 (CLI) Fix getorders command currency pair filtering (#400)
* add currency pair filtering to getorders

* long live the Queen

* fixed wording on tests
2019-12-18 09:12:58 +11:00
Andrew
75ac5ee791 (Exchange Interface) Convert Fetch & Update orderbook/ticker methods to return pointers (#398)
* moved order and ticker fetching to return a pointer

* return nil instead of empty struct

* fixed incorrect nil

* general cleanup
2019-12-17 15:54:09 +11:00
Andrew
a727beeb15 (BTC Markets): Wrapper SubmitOrder parameter order fix & IsOrderPlaced condition correction (#394)
* corrected param order Side -> Type, also corrected condition check for IsOrderPlaced

* send open status for GetActiveOrders

* GetActiveOrder() changes to include OrderID and status matching

* BTC Markets batch order limit fixes & SplitStringSliceByLimit  method

BTC markets batch end points have limits (20 for cancel 50 for query) adds new method SplitStringSliceByLimit in common to split a slice by limit and return slice of slice

* rm line :D

* Added test for SplitStringSliceByLimit and moved to const

* ntp client reworked to not return error if no valid time servers are found but default to system

* clean up

* new line added

* use TimeMgr sublogger and wording correction on output

* Moved to DialTimeout() & Removed SetDeadline call

* removed line

* added setdeadline fix

* goimport file

* removed unused error from NTPClient as we now default to system time if no server can be reached

* Added checks for number overflows

* converted to uint as you should not be passing a negative number in

* Increased test cases for NTPClient

* Removed Helper call as no longer outputting any data from function

* removed unused param
2019-12-16 14:47:54 +11:00
Adrian Gallagher
25e290563c Exchange: Pair formatting/loading/SetDefaults improvements (#396)
* Improve exchange pair loading/setting/formatting

* Update configs

* Link up forceUpdate for OKEX

* Fix linter issues

* Rearrange exchPS

* Fix typerino

* Config updates
2019-12-13 13:10:17 +11:00
Andrew Jackson
9178dbcc61 removed withdraw from struct name: 2019-12-13 08:36:07 +11:00
Andrew Jackson
b8e8c25f78 Renamed structs to remove package name 2019-12-13 08:26:55 +11:00
Andrew Jackson
db25bb1374 removed empty file 2019-12-12 14:17:02 +11:00
Andrew Jackson
a91d147781 Moved withdraw structs into own package 2019-12-12 14:16:45 +11:00
lozdog245
8c30505d46 String concatenation/conversion/formatting improvements (#393)
* Magic strings

* Comment, format and builder fixes
2019-12-09 11:44:01 +11:00
Scott
11a68a9bb7 Utilising authenticated websocket functions in exchange wrappers (#384)
* Basic concept commit

* Initial changes to support bitfinex v2. Reverts linter changes as they suck. Exports bitfinex ws types

* Adds ticker, trade and orderbook support

* Candles sub that returns no data COMPLETE

* Adds authenticated ws support

* Adds the barebones endpoints to support

* Adds more endpoints

* Even more endpoints

* minicommit to switch and test

* All the interactive types

* Adds support for simultaneous connections. Updates tests. Nothing is working

* Successfully adds place order. Moves all authenticated endpoints to new switch case

* Cancel order and modify order

* Cancel all orders, cancel multi orders

* Finalises implementation. Uses testMain

* Adds WS wrapper support for some funcs

* Fixing rebasing issues

* Replaces use of currency as a variable. Updates a lot of coinut websocket auth endpoint stuff

* Fixes some fun for loops with GetEnabledPairs

* Fixes tests impacted by currency var change

* Adds coinut support for WS functions. Replaces `order` vars with `ord`. Fixes some for loops too. Removes verbose from bitfinex

* So many panics

* I'm fixing a hole, where the panics get in, and stops my mind from wandering, where it will go

* Moves func `CanUseAuthenticatedWebsocketEndpoint` to Websocket package as it fits better. Adds test coverage of `CanUseAuthenticatedWebsocketEndpoint`

* Finishes up all of coinuts ws implementations.

* GateIO implementation

* Adds some helper funcs for types, sides and status. Adds support for huobi. Removes unnecessary type

* Adds forgotten huobi endpoint

* Fixes cancel order endpoint

* go hates my formatting and so do I

* The process to get authenticated kraken websocket to work. Uses testmain. Adds new auth channel, auth subscriptions, auth data handling. Not working yet

* Finishes open orders handling

* Mini update for status only updates

* Fixes some kraken points

* Finishes WS kraken since it doesn't work

* Unfinished commit, cleaning up types

* Finishes the const replacing

* Fixes extra GetNAmes after rebase

* An end to the cleanup. testmain for gateio

* Adds ZB support

* Bitfinex cleanup. Renamed func

* Testmain-47s for everyone!!! yayaaaaaaa

* Adds kraken websocket wrapper support

* Fixes rebase issues

* Fixes tests from rebase

* Adds test for conversion. Fixes for loop. Updates test order pricing. Fixes some poor made tests. Adds proper error handling for ws responses instead of logging them. Fixed issue where commented code ruined kraken ws.

* Fixes secret linting issues. Prioritises bitfinex channelID responses over authorised

* Fixes sloppy error/var declarations

* Fixes crazy bad logic where submit order errors weren't really considered. Parralols alphapoint/alphapoint_test.go. Removes buffer for multi-websocket comms channel.

* Removal of inline string and removal of redundant nil checkerinos

* Fixes err checks. Checks whether float has decimal. Fixes append. Drops omitempties. Parallel to some tests. Moves var declarations

* Replaces my lazy sprintfs with strconv.FormatInt(time.Now().Unix(), 10)

* Adds shiny new FullyMatched bool. Fixes coinbene buy sell consts

* Fixes oopsie with coinbene const replacement

* Fixes currency issue

* Cleans up new places that use JSONDecode

* Fixes huge panic bug from string int conversion. Adds large testtable for strings to order types

* Fixes some more strconversion issues. Fixes table test var usage. Changes mapperino name

* Added some new scenarios for number splitting

* Fixes lint issues

* negative num fix

* Typo fix

* Accuracy warning comment
2019-12-04 14:16:23 +11:00
Adam
a33ddcfa0a Engine: BTC Markets V3 Updates (#385)
* Broken WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

* Errors Fixed

* PR Changes

* PR Changes

* PR Changes

* PR Changes

* PR Changes

* PR Changes

* PR Changes

* PR Changes

* PR Changes

* Offline Fees Fixed

* MarketCandles fixed and constants added

* t.log deleted

* Broken WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

WIP

* Errors Fixed

* PR Changes

* PR Changes

* PR Changes

* MarketCandles fixed and constants added

* t.log deleted

* Added BSB and Order Status

* WIP

* Websocket and BatchPlaceCancelOrder and other minor nits fixed

* Linter Issues Fixed

* Function Name Change

* Replacing b.GetMarkets with b.GetEnabledPairs

* Pagination param changes and PlaceCancelBatch changes

* Merge Conflicts WIP

* Linter Issue Fixed

* optional params fixed
2019-12-04 11:53:35 +11:00
Ryan O'Hara-Reid
0c5d75b22c (Engine) Variety of engine updates (#390)
* drop common uuid v4 func and imported package as needed

* removed common functions regarding json marshal and unmarshal and used the json package directly. WRT unmarshal it was calling reflect and converted to string which is also checked in the JSON package so it was doing a double up, this will be a tiny gain as it was directly used in the requester package for all our outbound requests.

* add in string

* explicitly throw away return error value

* atleast return the error that websocket initialise returns

* return error when not connected

* fix comment

* Adds comments

* move package declarations

* drop append whenever we call supported

* remove unused import

* Change incorrect spelling

* fix tests

* fix go import issue
2019-12-03 10:06:08 +11:00
Ryan O'Hara-Reid
24bddcc090 Add directories to exclusion list && change default repo string (#387)
* Add directories to exclusion list && change default repo string

* Add in .idea folder to directory exclusion list

* cleaned

* Added support for calling the tool outside of its file

* fix formatting

* changed strings
2019-11-29 11:14:19 +11:00
Adrian Gallagher
d636882310 Use sync.Mutex instead of sync/atomic for 32 bit systems (#388)
Change CLRF -> LF
2019-11-29 10:38:44 +11:00