Commit Graph

25 Commits

Author SHA1 Message Date
Adrian Gallagher
a79e0d2b3e qa/spelling: Add Codespell support (#1121)
* Add codespell support

* Fix paths

* Add HTML files to exclusion list
2023-01-30 12:36:56 +11:00
Adrian Gallagher
9a4eb9de84 CI: Fix golangci-lint linter issues, add prealloc linter and bump version depends for Go 1.18 (#915)
* Bump CI versions

* Specifically set go version as 1.17.x bumps it to 1.18

* Another

* Adjust AppVeyor

* Part 1 of linter issues

* Part 2

* Fix various linters and improvements

* Part 3

* Finishing touches

* Tests and EqualFold

* Fix nitterinos plus bonus requester jobs bump for exchanges with large number of tests

* Fix nitterinos and bump golangci-lint timeout for AppVeyor

* Address nits, ensure all books are returned on err due to syncer regression

* Fix the wiggins

* Fix duplication

* Fix nitterinos
2022-04-20 13:45:15 +10:00
Adrian Gallagher
f0d45aa1d2 golangci-lint/CI: Bump versions and introduce new linters (#798)
* golangci-lint/CI: Bump versions

Fix remaining linter issues

* Specifically set AppVeyor version

* Fix the infamous typos 👀

* Add go env cmd to AppVeyor

* Add go version cmd to AppVeyor

* Specify AppVeyor image, adjust linters

* Update go get to go install due to deprecation

* Bump golangci-lint timeout time for AppVeyor

* Change NW contract to NQ

* Address nitters

* GetRandomPair -> Pair{}

* Address nits

* Address time nitterinos plus additional tweaks

* More time inception upgrades!

* Bending time and space
2021-10-14 16:38:53 +11:00
Ryan O'Hara-Reid
d636049fb2 exchanges: Initial context propagation (#744)
* gct: phase one context awareness pass

* exchanges: context propagation pass

* common/requester: force context requirement

* gctcli/exchanges: linter fix

* rpcserver: fix test using dummy rpc server

* backtester: fix comments

* grpc: add correct cancel and timeout for commands

* rpcserver_test: add comment on dummy server

* common: deprecated SendHTTPGetRequest

* linter: fix

* linter: turn on no context check

* apichecker: fix context linter issue

* binance: use param context

* common: remove checks as this gets executed before main

* common: change mutex to RW as clients can be used by multiple go routines.

* common: remove init and JIT default client. Unexport global variables and add protection.

* common: Add comments

* bithumb: after dinner mints fix
2021-09-11 13:52:07 +10:00
Scott
5ea5245afb Improvement: Subsystem separation (#664)
* Initial codes for a trade tracker

* Moving everything in a broken fashion

* Removes tradetracker. Removes some errors for subsystems

* Cleans up some subsystems, renames stuttering types. Removes some global Bot usage

* More basic subsystem renaming and file moving

* Removes engine dependency from events,ntpserver,ordermanager,comms manager

* Exports eventManager, fixes rpcserver. puts rpcserver back for now

* Removes redundant error message, further removes engine dependencies

* experimental end of day interface usage

* adds ability to build the application

* Withdraw and event manager handling

* cleans up apiserver and communications manager

* Cleans up some start/setup processes. Though should separate

* More consistency with Setup Start Stop IsRunning funcs

* Final consistency pass before testing phase

* Fixes engine tests. Fixes stop nil issue

* api server tests

* Communications manager testing

* Connection manager tests and nilsubsystem error

* End of day currencypairsyncer tests

* Adds databaseconnection/databaseconnection_test.go

* Adds withdrawal manager tests

* Deposit address testing. Moved orderbook sync first as its more important

* Adds test for event manager

* More full eventmanager testing

* Adds testfile. Enables skipped test.

* ntp manager tests

* Adds ordermanager tests, Extracts a whole new subsystem from engine and fanangles import cycles

* Adds websocket routine manager tests

* Basic portfolio manager testing

* Fixes issue with currency pair sync startup

* Fixes issue with event manager startup

* Starts the order manager before backtester starts

* Fixes fee tests. Expands testing. Doesnt fix races

* Fixes most test races

* Resolves data races

* Fixes subsystem test issues

* currency pair syncer coverage tests

* Refactors portfolio. Fixes tests. Withdraw validation

Portfolio didn't need to exist with a portfolio manager. Now the porfolio manager
is in charge how the portfolio is handled and all portfolio functions are attached
to the base instead of just exported at the package level

Withdrawal validation occurred at the exchange level when it can just be run at the
withdrawal manager level. All withdrawal requests go through that endpoint

* lint -fix

* golang lint fixes

* lints and comments everything

* Updates GCT logo, adds documentation for some subsystems

* More documentation and more logo updates

* Fixes backtesting and apiserver errors encountered

* Fixes errors and typos from reviewing

* More minor fixes

* Changes %h verb to %w

* reverbs to %s

* Humbly begins reverting to more flat engine package

The main reasoning for this is that the subsystem split doesn't make sense
in a golang environment. The subsystems are only meant to be used with engine
and so by placing them in a non-engine area, it does not work and is
inconsistent with the rest of the application's package layout.

This will begin salvaging the changes made by reverting to a flat
engine package, but maintaining the consistent designs introduced.
Further, I will look to remove any TestMains and decrease the scope
of testing to be more local and decrease the issues that have been
caused from our style of testing.

* Manages to re-flatten things. Everything is within its own file

* mini fixes

* Fixes tests and data races and lints

* Updates docs tool for engine to create filename readmes

* os -> ioutil

* remove err

* Appveyor version increase test

* Removes tCleanup as its unsupported on appveyor

* Adds stuff that I thought was in previous merge master commit

* Removes cancel from test

* Fixes really fun test-exclusive data race

* minor nit fixes

* niterinos

* docs gen

* rm;rf test

* Remove typoline. expands startstop helper. Splits apiserver

* Removes accidental folder

* Uses update instead of replace for order upsert

* addresses nits. Renames files. Regenerates documentation.

* lint and removal of comments

* Add new test for default scenario

* Fixes typo

* regen docs
2021-05-31 10:17:12 +10:00
Adrian Gallagher
0d787bc259 Bump golangci-lint to v1.24.0, linter fixes and general code improvements (#478)
* Bump golangci-lint version, update Go version deps and generic code improvements

* Fix wesbocket resp nil check and zip closures

* Update pprof path
2020-04-09 13:07:32 +10:00
Andrew
7fccc03164 (Logger) Rename package to log (#444)
* renamed package to log to stop side import requirement

* reverted comment changes

* reverted comment changes

* one more reverting wording back to logger

* wording changes on comments
2020-02-12 18:09:56 +11:00
Ryan O'Hara-Reid
0c5d75b22c (Engine) Variety of engine updates (#390)
* drop common uuid v4 func and imported package as needed

* removed common functions regarding json marshal and unmarshal and used the json package directly. WRT unmarshal it was calling reflect and converted to string which is also checked in the JSON package so it was doing a double up, this will be a tiny gain as it was directly used in the requester package for all our outbound requests.

* add in string

* explicitly throw away return error value

* atleast return the error that websocket initialise returns

* return error when not connected

* fix comment

* Adds comments

* move package declarations

* drop append whenever we call supported

* remove unused import

* Change incorrect spelling

* fix tests

* fix go import issue
2019-12-03 10:06:08 +11:00
Adrian Gallagher
da81ea6c08 Merge branch 'master' into engine 2019-11-11 17:11:11 +11:00
Adrian Gallagher
e9b3e3d328 Bump golangci-lint and AppVeyor versions now that 1.13.x is supported (#375) 2019-11-11 15:15:29 +11:00
Adrian Gallagher
8ecd8a0c16 Merge branch 'master' into engine 2019-08-09 15:46:24 +10:00
Adrian Gallagher
2078ba907f Update URLs for transfer into org repo (#338)
* Update URLs for transfer into org repo

* Update codecov, travis and othe remaining links

* Update appveyor paths
2019-08-09 11:50:39 +10:00
Scott
e209d85d0d Websocket request-response correlation (#328)
* Establishes new websocket functionality. Begins the creation of the websocket request

* Adding a wrapper over gorilla websocket connect,send,receive to handle ID messages. Doesn't work

* Successfully moved exchange_websocket into its own wshandler namespace. oof

* Sets up ZB to use a round trip WS request system

* Adds Kraken ID support to subscriptions. Renames duplicate func name UnsubscribeToChannels to RemoveSubscribedChannels. Adds some helper methods in the WebsocketConn to reduce duplicate code. Cleans up ZB implementation

* Fixes double locking which caused no websocket data to be read. Fixes requestid for kraken subscriptions

* Completes Huobi and Hadax implementation. Extends ZB error handling. Adds GZip support for reading messages

* Adds HitBTC support. Adds GetCurrencies, GetSymbols, GetTrades WS funcs. Adds super fun new parameter to GenerateMessageID for Unix and UnixNano

* Adds GateIO id support

* Adds Coinut support. Prevents nil reference error in constatus when there isnt one

* Standardises all Exchange websockets to use the wshandler websocket. Removes the wsRequestMtx as wshandler handles that now. Makes the Dialer a dialer, its not externally referenced that I can see.

* Fixes issue with coinut implementation. Updates bitmex currencies. Removes redundant log messages which are used to log messages

* Starts testing. Renames files

* Adds tests for websocket connection

* Reverts request.go change

* Linting everything

* Fixes rebase issue

* Final changes. Fixes variable names, removes log.Debug, removes lines, rearranges test types, removes order correlation websocket type

* Final final commit, fixing ZB issues.

* Adds traffic alerts where missed. Changes empty struct pointer addresses to nil instead. Removes empty lines

* Fixed string conversion

* Fixes issue with ZB not sending success codes

* Fixes issue with coinut processing due to nonce handling with subscriptions

* Fixes issue where ZB test failure was not caught. Removes unnecessary error handling from other ZB tests

* Removes unused interface

* Renames wshandler.Init() to wshandler.Run()

* Updates template file

* Capitalises cryptocurrencies in struct. Moves websocketResponseCheckTimeout and websocketResponseMaxLimit into config options. Moves connection configuration to main exchange Setup (where appropriate). Reverts currencylastupdated ticks. Improves reader close error checking

* Fixes two inconsistent websocket delay times

* Creates a default variable for websocket ResponseMaxLimit and ResponseCheckTimeout, then applies it to setdefaults and all tests

* Updates exchange template to set and use default websocket response limits
2019-08-07 15:15:01 +10:00
Andrew
3de1d94e5f New logging system (#319)
* First pass at adding new logging system

* NewLogger

* NewLogger

* WIP

* silly bug fix

* :D removed files

* removed old logging interface

* added tests

* added tests

* Started to add new lines to all f calls

* Added subsystem log types

* Logger improvements

* Further performance improvements

* changes to logger and sublogger creation

* Renamed Logging types

* removed old print statement

* changes based on feedback

* moved sublogger types to own file

* :)

* added console as output type

* added get level command

* added get/set log level via grpc command

* added check for output being empty for migration support

* first pass at log rotation

* added log rotation

* :D derp fixed

* added tests

* changes based on feedback

* changed log type

* comments

* renamed file -> fileSettings

* typo fix

* changes based on feedback

* gofmt ran on additional files

* gofmt ran on additional files
2019-07-07 05:20:31 +10:00
Adrian Gallagher
f777e68716 Engine improvements 2019-06-10 20:02:09 +10:00
Adrian Gallagher
e965e54e09 Split up common.go, file path fixes and much more 2019-06-04 17:04:32 +10:00
leilaes
46267e5975 Add missing close file and missing stop ticker (#300)
* Add missing close file and missing stop ticker

* Defer close file after checking error

* Update exchange_template.go
2019-05-17 11:17:35 +10:00
Adrian Gallagher
7dcb1ab553 Migrate from gometalinter.v2 to golangci-lint (#249)
* Migrate from gometalinter.v2 to golangci-lint
2019-03-01 16:10:29 +11:00
Adrian Gallagher
291e404a4a Makefile: add new recipes and linter features (#244)
* Makefile: add new recipes and linter features

* expand linter coverage and fix issues

* Update makefile

* address PR nitterinos
2019-01-31 14:53:24 +11:00
Andrew
d01e7bad72 Implement Logger (#228)
* Added new base logger

* updated example and test configs

* updated exchange helpers restful router & server

* logPath is now passed to the logger to remove dependency on common package

* updated everything besides exchanges to use new logger

* alphapoint to bitmex done

* updated bitmex bitstamp bittrex btcc and also performance changes to logger

* btcmarkets coinbase coinut exmo gateio wrappers updated

* gateio and gemini logger updated

* hitbtc huobi itbit & kraken updated

* All exchanges updatd

* return correct error for disabled websocket

* don't disconnect client on invalid json

* updated router internal logging

* log.Fatal to t.Error for tests

* Changed from fatal to error failure to set maxprocs

* output ANSI codes for everything but windows for now due to lack of windows support

* added error handling to logger and unit tests

* clear wording on print -> log.print

* added benchmark test

* cleaned up import sections

* Updated logger based on PR requests (added default config options on failure/setting errors)

* ah this should fix travici enc config issue

* Load entire config and clear out logging to hopefully fix travisci issue

* wording & test error handling

* fixed formatting issues based on feedback

* fixed formatting issues based on feedback

* changed CheckDir to use mkdirall instead of mkdir and other changes based on feedback
2019-01-08 21:56:22 +11:00
Adrian Gallagher
046e15a67e Periodic GoReportCard fixes 2018-12-27 15:10:41 +11:00
Ryan O'Hara-Reid
92534249bf Remove unwanted wrapper name stuttering and standardise common error returns (#213) 2018-11-23 12:55:00 +11:00
cranktakular
415332b446 Slack.go improvements and increasing coverage (#193)
* Slack.go improvements and increasing converage

Slack.go Adding handling for trying to send responses when the websocket isn't connected
Slack.go Adding handling for an error response other than "Socket URL has expired"
Slack.go Making handleErrorResponse return an error
Slack.go Making use of s.Connected
Slack.go Making HandleMessage return an error
Slack.go Removing SendHTTPGetRequestSlack, code now calls SendHTTPGetRequest from common.go instead

More coverage for slack GetChannelsString test
More coverage for slack GetUsernameByID test
Improving slack GetIDByName test
Adding slack GetGroupIDByName test
Improving slack GetChannelIDByName test
More coverage for slack GetUsersInGroup test
Adding slack WebSocketConnect test
Adding slack handlePresenceChange test
Adding slack handleMessageResponse test
Adding slack handleErrorResponse test
Adding slack handleHelloResponse test
Adding slack handleReconnectResponse test
Adding slack WebsocketSend test
Adding slack HandleMessage test
Adding slack SendHTTPGetRequestSlack test

* Fixing race conditions and missing exclamation mark.
2018-10-18 17:33:24 +11:00
Marco Franssen
0f209165d5 Improved code quality (#154)
* Removed package-lock.json form gitignore as it ensures specific package versions

* Updated all @angular web dependencies

* Resolved tslint errors using autofix option

* Resolved some more tslint issues

* Added lint scripts to package.json to easy lint the ts files

* Updated codelyzer and tslint

* Run web on travis using node 10 and run the lint task

* Resolved some more tslint issues after upgrading tslint and codelyzer

* Resolved golint issues with regards to exchange comments

* Resolved spelling errors shown by goreportcard.com

* Resolved gofmt warnings using goreportcard.com

* Resolved golint issue by removing unrequired else statement

* Refactored slack.go to reduce cyclomatic complexity

* Fixed govet issue where Slack was passed as value instead of reference
2018-07-18 15:46:47 +10:00
Ryan O'Hara-Reid
9d0616d8cf New communications package
Support for Slack, SMSGlobal, SMTP and Telegram

Supersedes: https://github.com/thrasher-/gocryptotrader/pull/126
2018-06-08 14:09:36 +10:00