Commit Graph

150 Commits

Author SHA1 Message Date
Andrew
b26ec86d43 Withdraw additional functionality (validation/submission/tracking) (#409)
* reworked request struct and exchange response started work on validation system

* removed import cycle until work around

* Added intial withdraw support via CLI added

* Added Crypto command to gctcli

* moved var declartion to single line

* Test updates for binance and anx

* All exchange tests have been updated test coverage added to validate

* First pass at adding withdrawl select from database

* started adding basic lru cache system

* Added basic LRU cache including Add Get Remove Contains ContainsOrAdd Clear

* wording changes on comments

* removed exported var's in strut as they are not required

* Added README

* README updates

* corrected ID on commands

* rm line :D

* merged in origin/cache

* linter fixes (gofmt)

* Added basic cache lookup to events

* swapped to mutex over rwmutex updated comments

* unexported getNewest & getOldest

* unexported getNewest & getOldest

* Updated comments and cited references in source

* updated comments

* WIP

* Migrated exchange WithdrawFiat wrapper to new struct response

* Migrated exchange WithdrawFiat wrapper to new struct response

* started work on bank management

* Added exchange level banking details back with migration to banking package

* Removed broken tests for now

* Added validation to bank accounts

* removed duplicate bank details from withdraw struct

* Test coverage increased

* gofmt

* merged upstream/master with clean up

* First pass at adding command line linking to gctcli

* added validation for crypto address, added gctcli support to retreive previous withdrawal requests

* general cleanup

* general cleanup

* reordered imports

* Increased test coverage moved to database sublogger

* Pass incorrect currency no longer return error from c.CheckBankAccountConfig

* remove TestMain() for now as other tests in this package will need to be reworked

* Happy little race car

* Reverted to upstream tests

* Added test covarege for validation method, corrected response on cli protobuf

* query clean up and removal of duplicated code

* cleaned up queries into singlem ethod increased test coverage

* Migrated international fund withdraw to new exchange response and added cache size override

* Migrated international fund withdraw to new exchange response and added cache size override

* Extended gctcli commands

* lowered default cache to 25

* small code clean up

* added get by date method

* add returned error

* cli commands cleaing return error on nil results to fix out of bounds

* merged write & read helpers into one for test coverage and increased engine/withdraw test coverage

* gofmt

* Added test coverage for valid ID

* removed unused param

* converted to use timestamp package from protobuf

* gofmt

* use built in RFC3339 timestamp

* remove setting of CreatedAt & UpdatedAt and allow ORm to take care of it

* also use ptype on byid

* code flow improvements

* remove inverse conditional check and linters run

* removed test data

* removed comment

* removed comment

* also write failures to database for auditing

* converted to use default time for start & end

* Default to time.Now() minus 30 days

* Default to time.Now() minus 30 days

* small code clean up

* fixed missing semicolon on migrations, code clean up

* updated sqlite migrations

* Added additonal check for exchange level bank account if global is not found

* case sensativity fix for currency names

* use correct compare

* test coverage fixed

* removed space

* return pointer to banking.Account

* return pointer to banking.Account

* added else check back to validate()

* Added empty string as default to migration over NULL due to retrivial of data
2020-02-26 15:45:13 +11:00
Andrew
7fccc03164 (Logger) Rename package to log (#444)
* renamed package to log to stop side import requirement

* reverted comment changes

* reverted comment changes

* one more reverting wording back to logger

* wording changes on comments
2020-02-12 18:09:56 +11:00
Adrian Gallagher
b949388994 General engine improvements (#437)
* Add exchange manager to engine

* Several improvements for engine and friends

1) New file.Exists func
2) gRPC TLS cert expiration date check and regeneration
3) New donation var for use across the codebase
4) Use Go log package until the logger is initialised

* Add cert tests and create dir tree if it doesn't exist for file.Write

* Link up donation address to documentation tool plus minor adjustments

* Fix remaining donation addrs

* Move non-needed reload exchange funcs

* Revert accidental config_example.json changes 🕯️

* Use go logger for logging until the logger has initiliased, otherwise no output will be seen

* Link up portfolio delay val and other fixes

* Run go mod tidy after dependabot PR

* Address nitterinos
2020-02-06 12:32:01 +11:00
Ryan O'Hara-Reid
0a84c5d97a Request package update & rate limit system expansion (#413)
* Initial rework of rework of requester - WIP

* Implementing and checking rate limits - WIP

* implemented coinbene rate limiting shenanigans

* add in remaining WIP

* fixy

* use authenticated rate limit

* drop ceiling as this can be done with a counter later

* add functionality to struct

* purge config options for rate limiting so as to keep things minimal

* prepare futures and swap rate limiting for implementation

* Address linter issues

* Addressed nits, fixed race

* fix linter issue

* remove global var as this was only setting when newrequester was called

* moved rate limit functionality into its own file

* Update Bitfinex with correct rate limit and test endpoints (WIP)

* finish off bitfinex adjustments

* fixes

* fix linter issues

* slowed rate for coinbasepro

* drop rate limit for huobi as the doc times have intermittent 429 issues.

* Set MACOSX_DEPLOYMENT_TARGET to remove linking warning

* Addr Thrasher nits

* Addr glorious nits

* unexport do request function

* fixed nitorinos

* Fixed something I missed

* move disabled rate limiter into loadexchange and use interface functionality

* Add temp quick fix
2020-02-06 11:44:28 +11:00
Andrew
f6fd94ea69 Engine: Scripting support (#383)
* WIP

* updated appveyor and increased deadline 5 seconds due to increased linters being added

* revert files to upstream/engine

* WIP

* WIP

* mod file changes

* added script manager

* Added manager/and cli interfaces to scripting

* Added script task handler

* WIP - Added timer/repeat support and fleshed out wrapper further

* autoload support added + WIP

* WIP commit

* added account balance info

* btc markets temp work around

* WIP - merged with upstream for new order package BTC Markets responses broken

* Cancel order wrapper WIP

*  order wrapper update

* Added test coverage for VM

* moved to map for VM List shutdown of all VM now handled added gctcli commands for list and stop of running scripts

* added override to load/execute for path

* fixed incorrect channel shutdown added further test coverage and restructured gctcli commands into sub commands

* increased test coverage for packages

* Added docs cleaned up tests and example scripts

* Test coverage increased for module/gct/exchange package

* windows fixes

* merged upstream/engine

* WIP

* logger fixes - removed pointer to bool check removed duplicate test check for logger

* remove unused mutex

* added inital upload support

* fix linter issues for go-fmt

* added zip support for uploading and added base for fund withdrawing

* changed error return types and also log errors, fix zip path issue

* improved error outputs and code flow

* pairs response fix added protobuf defs for stop all and list all

* added stop all running scripts general clean up and moved across to OrderManager

* linter fixes (gofmt)

* added list all command

* rewrote zip handler to be cleaner also fixed file overwrite on upload

* added query command reworked tests

* added further error checking to compileandrun corrected use of pointers for accountinfo

* bumped tengo version

* Removed named returns reworded log messages removed unused falseptr

* WIP

* Added virutal machine limit improved config options

* added model for script event added upload validation

* script_event table has been completed, tests for wrapper functions implemented

* README updates

* reverted changes opened new PR to move withdraw struct outs

* intial work on adding withdraw support after merger of withdraw package

* started work on examples

* Added crypto withdraw support

* fix switch case assignment and gofmt project

* Reworking Fiat withdraw request pending #402

* removed double pointer call

* added withdraw support for fiat  currencies

* added tests for withdraw methods increased readme

* removed local tengo require and also fix linter issues

* Added default log size const added basic test for invalid script execution

* First pass at moving wrapper to validator package to allow proper validation of uploaded scripts

* Added script details to README added config test added test for no file extension

* moved tests to const and fixed incorrect pathing

* added test coverage to withdraw package

* corrected file close handling

* point to included configtest.json

* extended validator support when a script is uploaded

* Bug fix on bool logic

* Added mutex

* Don't create autit events on test execution

* reverted common to master

* moved file rename to unix timestamp format

* converted logger enabled back to pointer as i need nilness check also moved scriptid to text over blob

* started work on autoload add/remove support

* First round of PR fixes (mostly commented exports)

* Moved GCTScript load to last, removed unneeded error from cleanup()

* Comment clairty for AuitEventID

* added autoload add/remove command to cli

* added tests for autoload

* Test updates for Exchanges

* linter fixes (gofmt)

* Removed double check of engine pointer

* remove possible nil pointer on GetSpecificTicker

* Fixed not closing file handler on write that causes archive removal to fail

* file handler Close clean ups

* corrected spelling on error return and return invalid name n autoload

* moved strings to cosnt moved bool pointer creation to convert package

* new zip extractor added

* Validation has been added to archive uploads

* removed shadow var on err

* added ok check to conversion

* converted condition check

* basic test for zip extract added

* new zip handler

* reverted back to old atomic loading system

* removed shadow err

* lets add a new line

* added space to error return

* command line toggle for script now works properly

* readme updated

* set configLoaded to true

* check for configLoaded condition

* added mutex to allow for multiple access on virtual machine increased test coverage disable script manager if scripting is disabled

* linked up to enable/disablesubsystem commands

* added start/stop example to readme

* reworked logic on test as check should be done on Load()

* updated to tengo v2

* linters

* lower time on ntp client to stop slippage

* remove all fails if any fail validtion from an archive

* remove vm from list if timer is invalid

* removed shadow on err

* remove config creation from NTPCheck test

* WIP testing DB changes

* add unique constraint

* WIP: created has many model

* linters run

* basic sqlite3 support added for new database format

* linters run

* Added test coverage for script repo

* removed unused print

* updated env vars for CI instances

* updated env vars for CI instances

* Updated test packages

* Test updates for postgresql

* removed invalid tests from postgres

* remove duplication of struct and improved code flow

* general cleanup

* wording changes on log output

* use databasemgr logger and add support for autoload without file extension

* corrected test naming

* return correct error

* return correct error again version 82

* store scriptdata on creation

* Hello

* Errorln -> Errorf

* Removed unused vars

* Read me updates

* testing without parallel

* comment on exported type

* added nil check against VM for test

* add debugging information

* gofmt

* remove verbose and data sent to channel

* Added debug information

* linter fixes (gofmt)

* remove unused CompileAndRun() call

* test sleep to see if issue is timing related

* semi-concurrent map fixes

* one day i will run gofmt or setup precommit hooks

* new line :D

* increased test coverage

* added correct sleep time

* Moved over to sync map

* linter fixes (gofmt)

* goimports

* moved VM related methods to vm.go

* new line at end of file

* trying increased timeout on golangci-lint for appveyor

* add debugging information

* removed timeout

* reworked timeout logic

* linter fixes (gofmt)

* increased test coverage

* increased test coverage

* one day i will run gofmt or setup precommit hooks

* removed unused exchange test

* increased golangci-lint timeout

* Added nil check on shutdown and test coverage for it lowered timeout back to 1:30

* reworked ID system

* removed script hash as it was unused

* added comments on exported methods and read me update

* reorder code

* removed to atomic.value for test execution flag

* increased test coverage

* move add further up execution

* point to correct script file
2020-01-23 13:54:04 +11:00
Adrian Gallagher
7a90aecf6f Bugfix: Introduces a new config.DefaultFilePath func (#415)
* Introduces a new config.DefaultFilePath func

* FiX GrAmMeRiNo
2020-01-16 08:10:25 +11:00
Adrian Gallagher
17a786536d Engine pre-merge changes (#392)
* Engine pre merge changes

* Remove redundant "seconds"
2019-12-04 16:25:55 +11:00
Adrian Gallagher
63191ce3ec Engine QA (#381)
* 1) Update Dockerfile/docker-compose.yml
2) Remove inline strings for buy/sell/test pairs
3) Remove dangerous order submission values
4) Fix consistency with audit_events (all other spec files use
CamelCase)
5) Update web websocket endpoint
6) Fix main param set (and induce dryrun mode on specific command line
params)

* Engine QA

Link up exchange syncer to cmd params, disarm market selling bombs and fix OKEX endpoints

* Fix linter issue after merge

* Engine QA changes

Template updates
Wrapper code cleanup
Disarmed order bombs
Documentation updates

* Daily engine QA

Bitstamp improvements
Spelling mistakes
Add Coinbene exchange to support list
Protect API authenticated calls for Coinbene/LBank

* Engine QA changes

Fix exchange_wrapper_coverage tool
Add SupportsAsset to exchange interface
Fix inline string usage and add BCH withdrawal support

* Engine QA

Fix Bitstamp types
Inform user of errors when parsing time accross the codebase
Change time parsing warnings to errors (as they are)
Update markdown docs [with linter fixes]

* Engine QA changes

1) Add test for dryrunParamInteraction
2) Disarm OKCoin/OKEX bombs if someone accidently sets canManipulateRealOrders to true and runs all package tests
3) Actually check exchange setup errors for BTSE and Coinbene, plus address this in the wrapper template
4) Hardcode missing/non-retrievable contributors and bump the contributors
5) Convert numbers/strings to meaningful types in Bitstamp and OKEX
6) If WS is supported for the exchange wrapper template, preset authWebsocketSupport var

* Fix the shadow people

* Link the SyncContinuously paramerino

* Also show SyncContinuously in engine.PrintSettings

* Address nitterinos and use correct filepath for logs

* Bitstamp: Extract ALL THE APM

* Fix additional nitterinos

* Fix time parsing error for Bittrex
2019-11-22 16:07:30 +11:00
Scott
242b02c382 (Engine) Bugfix: Unlocking an unlocked mutex PANIC + Increase dispatcher job capacity via commandline (#371)
* Removes lock unlock timer and instead sets unlocks between getting a nonce and sending a payload. Increases dispatch channel buffer to deal with len(enabledCurrencies) > ~100

* Adds additional comments to help explain the situation

* Fixes bug that could unlock mutex too early

* Fixes LIES where Gemini gets a nonce and then proceeds to declare it doesn't get a nonce causing an unrecoverable lock

* Fun new concept! The creation of a tested timed mutex. Unlocking an unlocked mutex cannot occur and response can be checked to verify whether the mutex was unlocked from timeout or command.

* Adds new cmd parameter "dispatchjobbuffer"

* Expands comments and renames benchmark. Makes `Timer` property private

* Happy little linters

* Renames jobBuffer and all related instances to jobs limit

* Tiny error message update

* Grammatical fix and setting dispatch.Start to use defaults
2019-10-29 14:00:45 +11:00
Adrian Gallagher
596be31b6a dispatcher: Use int32 for atomic operations to prevent crash on ARM 32bit systems (#370)
See https://github.com/golang/go/issues/599
2019-10-23 11:30:40 +11:00
Andrew
92147cdc5f (Engine): Database system improvements (#358)
* Migrated to goose & sqlboiler

* create tests with sqlboiler

* code clean up

* Added gct -> sqlboiler config gen

* dropped pgx support

* dropped pgx support because who needs connection pools

* reenable sqlite audit tests

* first pass of migration changes

* stuff is broken :D

* sqlboiler :D

* end of date commit

* Added comments code clean up

* revert go module files back to upstream

* bug fix

* pushed go.mod update to use correc goose version

* renamed sqlite to sqlite3 for consistency across codebase and PR feedback changes

* makefile updates

* things are broken end of day commit

* added postgresql test

* use correct database name

* travis fixes for env vars

* travis fixes for env vars

* test fixes

* run migration on test setup

* test adding postgres support to appveyor

* Skip tests on appveyor due to issues with missing binaries

* oh yeah i have to support windows don't i

* bumped goose version up

* add postgres to osx

* fix travis config as osx does not support services move spin up to before_script

* added PGDATA path fix

* pass PG_DATA to pg_ctl

* added initdb to before install

* fixes to wording and bumps up goose version

* who needs ssl anyway

* moved ssl to correct section :D

* bumped goose version up

* unbreak travis

* unbreak travis

* fix if database is disabled in config

* move strings to consts

* converted more strings to const

* improvements to sqlboiler mmodel gen

* Added contrib\sqlboiler file

* sqlboiler windows contrib fixes

* bumped goose version up

* :D whoops

* further fixes to sql models

* further fixes to sql models

* database type fix for config gen

* README update

* go.mod clean up

* added config details for appveyor

* appveyor ordering fix

* force psql9.6

* appveyor config changes

* all the environmen vars

* model changes for psql

* model changes for psql

* sqlite model fixes

* attempt at osx fix

* added error check for migration

* typos and check against goose error instead of string :D

* updated sqlboiler commit id

* bump sqlboiler version again

* set decimal package to @0bb1631

* readme and makefile updates

* bump goose version update readme and add override flag to config gen

* README typo fix and lowered inserts in test down to 20 as we are only testing that inserts work running 200 was unnecessary

* added gctcli command for audit event

* Added debug output toggle to config added both postgres & sqlite support to gctcli command

* Wording changes on errors

* set sqlite to 1 connection to stop locke database issues

* Usage update for order

* README updates with config examples

* go.mod/sum tidy

* removed lines in import second

* removed lines in imports

* convert local time to utc for database and display output

* go mod clean up and error checking to time

* renamed all packages to sqlite3

* added windows command output for sql model gen

* time conversion fix

* time conversion on gctcli
2019-10-08 15:28:31 +11:00
Adrian Gallagher
8ca1e5c36f Fix utils.AdjustGoMaxProcs
Since Go 1.5, Go will use the total number of logical processers that the
system has available. Caveats to this are if someone has set the
GOMAXPROCS env var set or wish to limit usage of the number of logical processers
between a range from 1 to NumCPUs
2019-10-03 10:45:59 +10:00
Ryan O'Hara-Reid
db317a2447 Engine: Dispatch service (#346)
* Added dispatch service

* Added orderbook streaming capabilities

* Assigned correct orderbook.base exchange name

* Fixed Requested niterinos
Add in cli orderbook QA tool to gctcli
Add exchange orderbook streaming

* Add ticker streaming support through dispatch package

* Added in some more info on error returns for orderbook.go

* fix linter issues

* Fix some issues

* Update

* Fix requested

* move dispatch out of exchanges folder to its own independant folder

* Fix requested

* change orderbook string to tickers

* Limit orderbooks to 50 and made dispatch system more stateless in operation

* lower cases for update/retrieve/sub exchange name

* Adds in asset validation and lower case conversion on cli

* Remove comment

* Moved timer to a higher scope so its not constantly initialised just reset per instance and removed returning unused channel on error

* Rm unused release function in dispatch.go
Reset timer and bleed buffered timer chan if needed in dispatch.go
Added in ticker.Stop() and timer.Stop() functions for worker routine return in dispatch.go
Index aggregated bid and ask functions for orderbook.go
Added in dummy slice for wsorderbook_test.go

* Moved drain to above Reset so potential race would not occur in dispatch.go
Fix various linter issues dispatch.go

* Fix some issues

* change to start/stop service, added in service state change via cli, updated logger

* fix requested

* Add worker amount init spawning

* fix linter issues

* Fix more linter issues

* More fixes

* Fix race issue on releasing pipe channel on a close after shutting down dispatcher system

* Moved all types to dispatch_types.go && remove panic

* Moved types into serperate file && improve test coverage

* RM unnecessary select case for draining channel && fixed error string

* Added orderbook_types file and improved code coverage

* gofmt file

* reinstated select cases on drain because I am silly

* Remove error for drop worker

* Added more test cases

* not checking error issue fix

* remove func causing race in test, this has required protection via an exported function

* set Gemini websocket orderbook exchange name
2019-10-03 09:47:37 +10:00
Ryan O'Hara-Reid
e2d57540a6 Config overwrite bugfix (#363)
* Fix bug where on parsing an alternate new config it will overwrite main config.json in gct dir

* Stop movement of config.json file from root dir when a new config is parsed in

* Stop overiding config.json at gct dir with new config.json from root directory

* RM LN :D

* Fix bug where promptforconfig in config_encryption.go overwrites default config
Ensure periphery command packages do not interact or save over configuration
Ensure tests to not save over or change current testdata/config
2019-09-27 16:03:41 +10:00
Adrian Gallagher
6bdbe236c0 Rid GCT os.Exit's in packages outside of main 2019-09-27 15:18:54 +10:00
Adrian Gallagher
e8b517ef0a Daily engine changes
1) Although gRPC does server side validation currently, validate basic things
on gctcli before relaying the request to the gRPC server
2) Make pair format consistent for the exchange sycner
3) Fix OKEX ticker failure due to thinking futures info is authenticated
4) Start filling out config tests
5) Extend timeout for golangci config so that AppVeyor has time to
complete (Travis is fine)
6) Add IsSupported exchange func for easy lookup
2019-09-10 17:07:00 +10:00
Andrew
0c76789b0d Database interface & auditing feature (#332)
* added audit manager

* Basic database DOA setup

* Added base config file

* added sqlite support and creation of schema

* added basic tests and config entry

* corrected issues of database is disabled

* fixed path for test

* WIP

* Added tests fixed config checking

* reverted files back to upstream

* reverted go.mod files

* no more test test test

* removed local testing details for psql

* hello

* added comments

* increased ping to 30 seconds

* renamed database table and added additional condition around test

* removed database test details

* goimport ran on all files

* WIP

* first attempt at migration

* fixes for migration system

* Migration system logger interface implemented

* fixes to print functions

* added write pooling pass

* gofmt :D

* formatted imports correctly

* removed old code

* added creation of migration

* gofmt

* :D Hello

*  🏎️

* maybe one day i will remember to revert go mod files

* checked err return condition correctly

* first changes for PR feedback

* code clean up

* protect Connected with RWmutex & event with mutex

* : D

* we can just pretend like it never happened

* MOved migrations back to source directory and added README

* readme formatting update

* Addd command line override for datadir

* use correct var when creating a migration and confirm folder is created

* Check if database version is newer than latest migration and also you know make migrations work.....

* uses filepath instead of manual path to use correct path seperator

* Add connection message and lower timeout

* Added support for sslmode for psql

* no longer force Close of database instead allow driver to maage

* Added closer func to test output

* sslmode added to example config
2019-08-20 16:35:06 +10:00
Adrian Gallagher
8ecd8a0c16 Merge branch 'master' into engine 2019-08-09 15:46:24 +10:00
Adrian Gallagher
2078ba907f Update URLs for transfer into org repo (#338)
* Update URLs for transfer into org repo

* Update codecov, travis and othe remaining links

* Update appveyor paths
2019-08-09 11:50:39 +10:00
Luke Hamilton
6e70f0642a Refactor main.go to improve startup execution flow (#334)
Much easier to work on a project when you can clearly see its execution path.
2019-08-06 15:20:29 +10:00
Andrew
3de1d94e5f New logging system (#319)
* First pass at adding new logging system

* NewLogger

* NewLogger

* WIP

* silly bug fix

* :D removed files

* removed old logging interface

* added tests

* added tests

* Started to add new lines to all f calls

* Added subsystem log types

* Logger improvements

* Further performance improvements

* changes to logger and sublogger creation

* Renamed Logging types

* removed old print statement

* changes based on feedback

* moved sublogger types to own file

* :)

* added console as output type

* added get level command

* added get/set log level via grpc command

* added check for output being empty for migration support

* first pass at log rotation

* added log rotation

* :D derp fixed

* added tests

* changes based on feedback

* changed log type

* comments

* renamed file -> fileSettings

* typo fix

* changes based on feedback

* gofmt ran on additional files

* gofmt ran on additional files
2019-07-07 05:20:31 +10:00
Adrian Gallagher
1daaa66830 Engine changes
Add addr helpers (will be split off into own package)
Engine status updates (log and data dir display)
Use GetPairFormat for various exchanges instead of calling the config
QA fixes
Implement GCTRPC exchange deposit address handling
2019-06-24 17:38:11 +10:00
Adrian Gallagher
bd8dc47c38 daily progress build 2019-06-06 17:20:40 +10:00
Adrian Gallagher
f5914e8c10 Engine changes 2019-05-22 17:06:38 +10:00
Ryan O'Hara-Reid
8279a036c2 Connection checker update (#294)
* Adds logic to ensure nothing proceeds a connectivity check and adds tests

* Adds waitgroup to verify monitor routine startup before shutdown is called thus eliminating race condition

* fix func desc and test func naming convention

* Add invalid address check

* Fix race condition for okgroup on wait group when instant shutdown called

* Fix fmt issue with gofmt
2019-05-13 12:02:39 +10:00
Ryan O'Hara-Reid
6e2cba566f Add in routine that checks internet connectivity for bot services (#287)
* Add in routine that checks internet connectivity for bot services

* Packaged connection checker
2019-05-02 10:14:42 +10:00
Andrew Jackson
c449568f6d Fix incorrect variable usage (err -> errNTP) 2019-04-18 10:23:19 +10:00
Andrew
071f4f68a8 New NTP Client (#277)
* WIP

* Added check for time out of sync

* merged upstream/master

* added tests

* Increased configuration options for NTPclient and test coverage

* removed unneeded config save at end of ntp update

* Added test for empty response to confirm it will loop

* formatting correction

* converted to pointer to allow for default allowance settings to be checked

* added readme for NTP server

* corrected some formatting

* updated configtest negativedifference value

* gofmt config_test.go for correct import order

* corrected typo value in test

* bugfix for windows newline and changes based on PR feedback

* added minus sign to output

* fixed negative number input

* Fixed spelling mistakes and removed redundant test

* reverted back to a positive number in the config instead of negative for allowednegativedifference

* restructured code for cleaner output
2019-04-18 10:08:19 +10:00
Adam
400c1cc84d Improved test coverage and functions in common.go (#266)
Improved directory creation, including permissions and expanded test coverage
2019-04-17 17:33:40 +10:00
Andrew
ca55f2f965 Optimisation: Large struct pointer conversion (final part) (#265)
Completes large struct pointer optomisations over the entire codebase and enables hugeParams linter by default
2019-04-04 15:31:49 +11:00
Andrew
dc236c251e Optimisation: large structs/huge param fixes (part 2) (#262)
* updated golangci config to enable hugeparam linter

* ModifyOrder struct usage converted to a pointer

* OrderBook conversion to struct

* More conversion of large structs to pointers

* updated golangci config to enable hugeparam linter

* ModifyOrder struct usage converted to a pointer

* OrderBook conversion to struct

* More conversion of large structs to pointers

* disabled hugeParam check for golang again

* changed based on suggested feedback and fix for no default provider

* fixed typing
2019-03-26 15:40:46 +11:00
Ryan O'Hara-Reid
0990f9d118 Currency package update (#247)
* Initial currency overhaul before service system implementation

* Remove redundant currency string in orderbook.Base
Unexport lastupdated field in orderbook.Base as it was being instantiated multiple times
Add error handling for process orderbook

*  Remove redundant currency string in ticker.Price
 Unexport lastupdated field in ticker.Price
 Add error handling for process ticker function and fix tests

* Phase Two Update

* Update translations to use map type - thankyou to kempeng for spotting this

* Change pair method name from Display -> Format for better readability

* Fixes misspelling and tests

* Implement requested changes from GloriousCode

* Remove reduntant function and streamlined return in currency_translation.go

* Revert pair method naming conventions

* Change currency naming conventions

* Changed code type to exported Item type with underlying string to reduce complexity

* Added interim orderbook process method to orderbook.Base type

* Changed feebuilder struct field to currency.Pair

* Adds fall over system for backup fx providers

* deprecate function and children and fix linter issue with btcmarkets

* Fixed requested changes

* Fix bug and move mtx for rates

* Fixed after rebase oopsies

* Fix linter issues

* Fixes race conditions in testing functions

* Final phase coinmarketcap update

* fix linter issues

* Implement requested changes

* Adds configuration variables to increase/decrease time durations between updating currency file and fetching new currency rates

* Add a collection of tests to improve codecov

* After rebase oopsy fixes for btse

* Fix requested changes

* fix after rebase oopsies and add more efficient comparison checks within currency pair

* Fix linter issues
2019-03-19 11:49:05 +11:00
Adrian Gallagher
7dcb1ab553 Migrate from gometalinter.v2 to golangci-lint (#249)
* Migrate from gometalinter.v2 to golangci-lint
2019-03-01 16:10:29 +11:00
Ryan O'Hara-Reid
82a622294c Coinmarketcap implementation (#243)
* Updates requester package to allow unpacking of zipped files and defaults to warn if no JSON is present

* Initial addition of coinmarketcap functionality

* fix requested changes

* Fix issue with displaying false positive in request.go && reorder plan list

* Rename CurrencyProvider -> CryptocurrencyProvider
Skip seeding currency data if not enabled
Rm line in main.go

* Update test procedures and relevant json files

* Fix const issue within config.go
2019-01-31 16:11:42 +11:00
Adrian Gallagher
291e404a4a Makefile: add new recipes and linter features (#244)
* Makefile: add new recipes and linter features

* expand linter coverage and fix issues

* Update makefile

* address PR nitterinos
2019-01-31 14:53:24 +11:00
Andrew
d01e7bad72 Implement Logger (#228)
* Added new base logger

* updated example and test configs

* updated exchange helpers restful router & server

* logPath is now passed to the logger to remove dependency on common package

* updated everything besides exchanges to use new logger

* alphapoint to bitmex done

* updated bitmex bitstamp bittrex btcc and also performance changes to logger

* btcmarkets coinbase coinut exmo gateio wrappers updated

* gateio and gemini logger updated

* hitbtc huobi itbit & kraken updated

* All exchanges updatd

* return correct error for disabled websocket

* don't disconnect client on invalid json

* updated router internal logging

* log.Fatal to t.Error for tests

* Changed from fatal to error failure to set maxprocs

* output ANSI codes for everything but windows for now due to lack of windows support

* added error handling to logger and unit tests

* clear wording on print -> log.print

* added benchmark test

* cleaned up import sections

* Updated logger based on PR requests (added default config options on failure/setting errors)

* ah this should fix travici enc config issue

* Load entire config and clear out logging to hopefully fix travisci issue

* wording & test error handling

* fixed formatting issues based on feedback

* fixed formatting issues based on feedback

* changed CheckDir to use mkdirall instead of mkdir and other changes based on feedback
2019-01-08 21:56:22 +11:00
Adrian Gallagher
e6027cfede Preserve ticker/orderbook output 2018-10-24 15:07:19 +11:00
Ryan O'Hara-Reid
d3c2800fe0 Initial overhaul of websocket connection and feeds (#189)
* Initial overhaul of websocket connection and feeds
* Added proxy support
* Piped to routines.go

* Added new websocket file in exchanges
Refactored orderbook handling into exchange_websocket.go
Added better error responses for binance_websocket.go
General clean for binance_websocket.go

* General fixes - bitfinex_websocket.go
Refactored orderbook cache code - bitfinex_websocket.go
Removed fatal error with unhandled type - routines.go

* Added general improvements to bitmex_websocket.go
Refactored orderbook handling to exchange_websocket.go
Added variable in Item struct in orderbook.go for looking up orders by ID

* Fix issue when routines are blocked due to Data Handler not started
Updated traffic handler
General fixes for bitstamp_websocket.go

* General fixes for coinbasepro_websocket.go

* General fixes for coinut_websocket.go
Fixed error return in exchange_websocket.go

* Removed comments in coinut_wrapper.go
Refactor orderbook logic from hitbtc_websocket.go to exchange_websocket.go

* General fixes

* Removed comments
General fixes

* Updated routines.go

* After rebase fix

* Fixed update config pairs in okcoin.go

* fixed config currency issue in okcoin.go for okcoin China

* exchange_websocket.go
*Removed unused const dec
*Removed state change routine
*Improved trafficMonitor routine
*Increased verbosity for error returns
*Removed uneeded mutex locks

exchange_websocket_test.go
*Added new tests for websocket and orderbook updating

routines.go
*Removed string cased

* Fixed race conditions on sync.waitgroup in exchanges_websocket.go

* Changes variable name in config.go

* Removes unnecessary comment

* Removes indefinite lock on error return

* Removes unnecessary comment

* Adds support for BTCC websocket
Drops support for BTCC REST

* Rewords comment in exchange_websocket.go
Moves types to poloniex_types.go

* Moves types to coinut_types.go

* Removes uneeded range for accessing array variables for coinbase_websocket.go
Removes comments in coinut_types.go

* Adds verbosity flag to GCT
Suppresses verbose output from routines.go

* Fixes setting proxy for REST and Websocket per exchange
Upgrades error handling
Drops unused *url.Url variable in exchange type

* Adds test for setting proxy

* Fixes bug that closes connection due to incorrect timeout time through a proxy connection

* Clarify verbose flag message
2018-10-24 14:22:40 +11:00
Adrian Gallagher
c2c7032858 Add debug logger and default data directory support 2018-10-19 17:26:00 +11:00
Adrian Gallagher
42d458d5ea Rename GDAX -> CoinbasePro 2018-07-11 10:42:37 +10:00
Ryan O'Hara-Reid
9d0616d8cf New communications package
Support for Slack, SMSGlobal, SMTP and Telegram

Supersedes: https://github.com/thrasher-/gocryptotrader/pull/126
2018-06-08 14:09:36 +10:00
Ryan O'Hara-Reid
58051b89c7 New package provider for foreign exchange rates.
Fixes issue: https://github.com/thrasher-/gocryptotrader/issues/131
Supersedes: https://github.com/thrasher-/gocryptotrader/pull/123
2018-06-06 21:05:17 +10:00
Adrian Gallagher
4903c788b1 Use key derivitive function for encryption/decryption of config data
Fixes https://github.com/thrasher-/gocryptotrader/issues/115
2018-06-04 18:43:13 +10:00
Adrian Gallagher
d3edd2845f Reorganise portfolio and prevent saving nil portfolio if not seeded 2018-05-31 14:07:23 +10:00
Adrian Gallagher
7b60bdd59f Prevent concurrent write for websocket data stream
Ref: https://github.com/thrasher-/gocryptotrader/issues/119
2018-05-29 16:27:03 +10:00
Adrian Gallagher
ac41a7cfad New features and bug fixes
- Modifications made to the request package. Planned improvements will be
sending requests on intervals, rate limiter back off support, dynamic tuning
and requests packaged into a request job group.
- Can modify each exchanges individual HTTP client (e.g timeout and
transport settings).
- Bot now uses an exchange config HTTP timeout value.
- Bot now uses a global HTTP timeout (configurable).
- Batched ticker request support for exchanges.
- Ticker and Orderbook fetching now are spanned accross multiple
go routines and regulated by a sync wait group.
- Fixes hack used to load exchanges, now uses a sync wait group.
- Ticker and Orderbook storage and fetching now uses mutex locks.
- New pair function for finding different pairs between two supplied
 pair arrays. This is used for currency pair updates for exchange which
support dynamic updating.
- Shows removal/additions of dynamic updates currencies.
2018-05-04 13:20:19 +10:00
Adrian Gallagher
ce3d2953f8 Various improvements, update config 2018-02-05 12:30:04 +11:00
Adrian Gallagher
5ca9db8ad4 Add banner 2018-01-19 17:30:08 +11:00
Adrian Gallagher
5a2c81346c Added helper functions, exchange method to set currencies and bug fixes 2018-01-18 12:45:19 +11:00
Adrian Gallagher
bb74b22ef7 Implement dry run mode, won't save config (to be used in future modes) 2018-01-17 21:24:24 +11:00