Commit Graph

30 Commits

Author SHA1 Message Date
Adrian Gallagher
69bb645fc2 docs: Migrate development tracking from Trello to GitHub Projects (#1696)
* docs: Migrate development tracking from Trello to GitHub Projects

* docs: Replace non-template based references and correct templates
2024-11-08 10:10:49 +11:00
Adrian Gallagher
400bcb6b56 telegram: Fix pushing events to an authorised client list (#1208)
* fix_communications_authorised_clients

* Telegram: Link config to authorised clients list

* Telegram: Prevent multiple spam messages from unauthed user

* Telegram: Improve command handling for authenticated users

Telegram doesn't allow you to easily fetch the user ID of a user unless they have previously sent you a message and is currently waiting to be processed, or if they message you on the fly once the bot is connected. This ensures that the user ID is stored for future usage upon a single successful auth command.

It also fixes the offset as the previous code wouldn't be able to process incoming messages once connected and instead only relay them.

* Bump docs

* default to UTC time in case bot is run on a server with diff time zones

* Enhance config for already upgraded configs

---------

Co-authored-by: shanhuhai5739 <shanhu5739@gmail.com>
2023-06-05 10:37:13 +10:00
Samuael A
969669cb0d linters: Bump golangci-lint version to v1.52.2 and fix issues (#1169)
* fix linter issues

* linter issue suppressing and fixing

* change linter version and linter issues fix

* Bump version
2023-04-27 10:13:46 +10:00
Adrian Gallagher
7e08e483fb CI: Bump go version, linters and fix minor issues (#1130)
* CI: Bump go version, linters and fix minor issues

* Bump version, fix loop variables

* Revert

* Rid TODOs now that 1.51 has been released
2023-02-03 15:56:59 +11:00
Adrian Gallagher
9a4eb9de84 CI: Fix golangci-lint linter issues, add prealloc linter and bump version depends for Go 1.18 (#915)
* Bump CI versions

* Specifically set go version as 1.17.x bumps it to 1.18

* Another

* Adjust AppVeyor

* Part 1 of linter issues

* Part 2

* Fix various linters and improvements

* Part 3

* Finishing touches

* Tests and EqualFold

* Fix nitterinos plus bonus requester jobs bump for exchanges with large number of tests

* Fix nitterinos and bump golangci-lint timeout for AppVeyor

* Address nits, ensure all books are returned on err due to syncer regression

* Fix the wiggins

* Fix duplication

* Fix nitterinos
2022-04-20 13:45:15 +10:00
Scott
5ea5245afb Improvement: Subsystem separation (#664)
* Initial codes for a trade tracker

* Moving everything in a broken fashion

* Removes tradetracker. Removes some errors for subsystems

* Cleans up some subsystems, renames stuttering types. Removes some global Bot usage

* More basic subsystem renaming and file moving

* Removes engine dependency from events,ntpserver,ordermanager,comms manager

* Exports eventManager, fixes rpcserver. puts rpcserver back for now

* Removes redundant error message, further removes engine dependencies

* experimental end of day interface usage

* adds ability to build the application

* Withdraw and event manager handling

* cleans up apiserver and communications manager

* Cleans up some start/setup processes. Though should separate

* More consistency with Setup Start Stop IsRunning funcs

* Final consistency pass before testing phase

* Fixes engine tests. Fixes stop nil issue

* api server tests

* Communications manager testing

* Connection manager tests and nilsubsystem error

* End of day currencypairsyncer tests

* Adds databaseconnection/databaseconnection_test.go

* Adds withdrawal manager tests

* Deposit address testing. Moved orderbook sync first as its more important

* Adds test for event manager

* More full eventmanager testing

* Adds testfile. Enables skipped test.

* ntp manager tests

* Adds ordermanager tests, Extracts a whole new subsystem from engine and fanangles import cycles

* Adds websocket routine manager tests

* Basic portfolio manager testing

* Fixes issue with currency pair sync startup

* Fixes issue with event manager startup

* Starts the order manager before backtester starts

* Fixes fee tests. Expands testing. Doesnt fix races

* Fixes most test races

* Resolves data races

* Fixes subsystem test issues

* currency pair syncer coverage tests

* Refactors portfolio. Fixes tests. Withdraw validation

Portfolio didn't need to exist with a portfolio manager. Now the porfolio manager
is in charge how the portfolio is handled and all portfolio functions are attached
to the base instead of just exported at the package level

Withdrawal validation occurred at the exchange level when it can just be run at the
withdrawal manager level. All withdrawal requests go through that endpoint

* lint -fix

* golang lint fixes

* lints and comments everything

* Updates GCT logo, adds documentation for some subsystems

* More documentation and more logo updates

* Fixes backtesting and apiserver errors encountered

* Fixes errors and typos from reviewing

* More minor fixes

* Changes %h verb to %w

* reverbs to %s

* Humbly begins reverting to more flat engine package

The main reasoning for this is that the subsystem split doesn't make sense
in a golang environment. The subsystems are only meant to be used with engine
and so by placing them in a non-engine area, it does not work and is
inconsistent with the rest of the application's package layout.

This will begin salvaging the changes made by reverting to a flat
engine package, but maintaining the consistent designs introduced.
Further, I will look to remove any TestMains and decrease the scope
of testing to be more local and decrease the issues that have been
caused from our style of testing.

* Manages to re-flatten things. Everything is within its own file

* mini fixes

* Fixes tests and data races and lints

* Updates docs tool for engine to create filename readmes

* os -> ioutil

* remove err

* Appveyor version increase test

* Removes tCleanup as its unsupported on appveyor

* Adds stuff that I thought was in previous merge master commit

* Removes cancel from test

* Fixes really fun test-exclusive data race

* minor nit fixes

* niterinos

* docs gen

* rm;rf test

* Remove typoline. expands startstop helper. Splits apiserver

* Removes accidental folder

* Uses update instead of replace for order upsert

* addresses nits. Renames files. Regenerates documentation.

* lint and removal of comments

* Add new test for default scenario

* Fixes typo

* regen docs
2021-05-31 10:17:12 +10:00
Adrian Gallagher
051675255d CI: Add support for GitHub actions (#680)
* Add initial workflows config

* Add PSQL service

* Add 32bit and frontend jobs

* Add gcc-multilib and fix typo

* Fix backtester test for linux/unix filesystem errors and npm commands

* 32-bit Add -race to purposefully throw an error to ensure envs are set correctly (will revert)

* Revert "32-bit Add -race to purposefully throw an error to ensure envs are set correctly (will revert)"

This reverts commit b97e66c49fca859f8738e01107f96f9ca4040b2e.

* Bump workers count for orderbook processing

* Remove comments to trigger 32-bit cache hit after job success

* Cancel addition plus version bump, spawn more overlords

* Update build badge to Github Actions

* Brach -> Branch
2021-05-06 14:35:05 +10:00
Adam
7dd3f13c8d Exchange API modification monitor (#404)
* WIP

* Variable Name Changes and other PR Changes

* Name Changes and Error Checks

* WIP

* Some new functionality added and nits fixed

* WIP on Nits

* More trello functionality added and some PR changes

* Minor changes and linter fixes

* Errors Fixed and some PR Changes

* PR Changes

* WIP

* WIP

* Addition of Config

* WIP of Readme file

* Readme WIP

* Addition of ReadMe

* Linter Issues fixed

* Minor Errors fixed

* fixed test timeout issues and added api keys check for tests

* Errors fixed

* Nits WIP

* Nits fixed

* More efficient mutex usage

* Consts cleaned up

* Readme file fixed

* linter issues fixed

* format issues:

* readme updates and nits

* remove backup file and address nits

* backup file removed for real

* OkexOkcoin updates and linter fixes

* errors fixed

* Nits

* errors fixed

* formatting issues

* race conditions fixed

* files fixed

* go mod n go sum

* WIP adding rate limits

* Added rate limits

* Remove test vars

* go mod tidy

* improving readme file and 2digits for trello auto update

* Added space

* removed space

* addressing nits

* documentation update

* more nits

* board id check

* linter fixes

* test race fixed

* test file match

* errors fixed

* nitz

* LOWERCASE flags

* more nitz

* broken wip

* flag addition fixed

* error fixed

* documentation updated for adding exchange using flags

* something

* add comment

* nitz fixed

* pr changes

* added gemini exchange

* testfile saving changes

* nitz

* fixing done

* pr changes

* nits

* WIP

* minor error fix

* wip

* wip

* WIP

* pr changes

* pr changes

* pr changes

* nits

* nits fixed

* t.Fail removed

* nitz

* disable lakeBTC

* contributors fixed

contributors fixed

ALL THE NITZ

OMG

* OMGGG

ALL THE NITZ

OMG

* name change errors fixed

* linter issues fixed

* linterzz

* Added disabling feature for exchanges

* bug fix

* small feature added

* fixed

bug fix:

no keys

* nits

* test

* test2

* test 3

* linter fixed

* testing

* testing

* remove logs

* OMG

* bugfix

* cyam88888

* nits

* error handled

* minor nits

fixed createnewcheck issue

fixed createnewcheck issue

* functionality change

some functionality changes

no api keys 4u :D

* trello names thro config

* err check
2020-04-01 11:15:10 +11:00
Andrew
7fccc03164 (Logger) Rename package to log (#444)
* renamed package to log to stop side import requirement

* reverted comment changes

* reverted comment changes

* one more reverting wording back to logger

* wording changes on comments
2020-02-12 18:09:56 +11:00
Adrian Gallagher
b949388994 General engine improvements (#437)
* Add exchange manager to engine

* Several improvements for engine and friends

1) New file.Exists func
2) gRPC TLS cert expiration date check and regeneration
3) New donation var for use across the codebase
4) Use Go log package until the logger is initialised

* Add cert tests and create dir tree if it doesn't exist for file.Write

* Link up donation address to documentation tool plus minor adjustments

* Fix remaining donation addrs

* Move non-needed reload exchange funcs

* Revert accidental config_example.json changes 🕯️

* Use go logger for logging until the logger has initiliased, otherwise no output will be seen

* Link up portfolio delay val and other fixes

* Run go mod tidy after dependabot PR

* Address nitterinos
2020-02-06 12:32:01 +11:00
Ryan O'Hara-Reid
24bddcc090 Add directories to exclusion list && change default repo string (#387)
* Add directories to exclusion list && change default repo string

* Add in .idea folder to directory exclusion list

* cleaned

* Added support for calling the tool outside of its file

* fix formatting

* changed strings
2019-11-29 11:14:19 +11:00
Scott
ccfcdf26aa Engine: Protocol Features, coverage, types, BTC markets websocket (#368)
* Attempts to update orderbook so it doesn't need to sort

* Reverts the ws ob stuff. Gets rid of sorting because it happens later. Adds some exchange features

* update existing feature lists. Expands list definition to match my emotions

* Adds bithumb bitmex and bitstamp. adds a couple more types

* Features for you, features for me, features for bittrex, btcmarkets, btse, coinbasepro, coinut, exmo, gateio and gemini

* Features for hitbtc, huobi, itbit, kraken, lakebtc, lbank, localbitcoins, okcoin, okex, poloniex, yobit, zb

* Who can forget good old alphapoint?

* Adds btcmarksets websocket :glitch_crab: fixes alphapoint features

* Adds extra data not in the documentation :/

* Replaces websocket features by using protocol features. However, it breaks it due to import cycles. I'm not sure what I'll do just yet

* Removes import cycle via duplicate structs.

* Increases coverage of config with `TestCheckCurrencyConfigValues`. Moves all currency pair package types into their own files or places it at the bottom of files if necessary

* Increase coverage in code.go

* One way of determining a test has failed, is when to it fails. Removed redundant explanation

* Increases code coverage of conversion

* Lint fixes

* Fixes orderbook tests

* Re-adds sorting because its important to still have the internal pre-processed orderbook to be representative of a real orderbook

* Secret lints that did not show up via Windows linting

* Adds protocol package to contain exchange features

* Fixes protocol implementation

* Fixes ws tests

* Addresses the following: Removes st-st-stutters in config types, changes GetAvailableForexProviders -> GetSupportedForexProviders, removes errors from tests where error is nil, removes orderbook setup when not necessary, removes import newlines, removes false bools from declaration, changes should of to should have

* imports and casing

* Fixes two more nil error checks
2019-10-22 10:56:20 +11:00
Adrian Gallagher
132d2d03c2 Merge branch 'master' into engine 2019-09-24 17:05:58 +10:00
Adrian Gallagher
bbdf532b16 Update Slack shared invite link 2019-09-24 17:01:32 +10:00
Adrian Gallagher
8ecd8a0c16 Merge branch 'master' into engine 2019-08-09 15:46:24 +10:00
Adrian Gallagher
2078ba907f Update URLs for transfer into org repo (#338)
* Update URLs for transfer into org repo

* Update codecov, travis and othe remaining links

* Update appveyor paths
2019-08-09 11:50:39 +10:00
Andrew
3de1d94e5f New logging system (#319)
* First pass at adding new logging system

* NewLogger

* NewLogger

* WIP

* silly bug fix

* :D removed files

* removed old logging interface

* added tests

* added tests

* Started to add new lines to all f calls

* Added subsystem log types

* Logger improvements

* Further performance improvements

* changes to logger and sublogger creation

* Renamed Logging types

* removed old print statement

* changes based on feedback

* moved sublogger types to own file

* :)

* added console as output type

* added get level command

* added get/set log level via grpc command

* added check for output being empty for migration support

* first pass at log rotation

* added log rotation

* :D derp fixed

* added tests

* changes based on feedback

* changed log type

* comments

* renamed file -> fileSettings

* typo fix

* changes based on feedback

* gofmt ran on additional files

* gofmt ran on additional files
2019-07-07 05:20:31 +10:00
Adrian Gallagher
b901c4b670 Engine improvements
Add back events tests
Fill out SMTP comms handler
Add getcommunicationrelayers gRPC command
2019-06-14 18:00:42 +10:00
Adrian Gallagher
6b2cfe7905 Daily engine improvements
Link up various subsystems to be managed atomically with the ability to start/stop them
New subsystem APIs
Comms changes
2019-06-13 17:30:50 +10:00
Adrian Gallagher
f777e68716 Engine improvements 2019-06-10 20:02:09 +10:00
Adrian Gallagher
e965e54e09 Split up common.go, file path fixes and much more 2019-06-04 17:04:32 +10:00
Adrian Gallagher
6ebafb0f33 Merge branch 'master' into engine 2019-06-04 12:25:40 +10:00
Shogin Michael
416fbbd5ae Unit tests for communication/base package (#312)
* Added tests

* Added tests communications base

* Removed unnecessary field

* Review corrections: linter

* Review corrections: typo
2019-05-30 13:35:22 +10:00
Adrian Gallagher
f5914e8c10 Engine changes 2019-05-22 17:06:38 +10:00
Ryan O'Hara-Reid
0990f9d118 Currency package update (#247)
* Initial currency overhaul before service system implementation

* Remove redundant currency string in orderbook.Base
Unexport lastupdated field in orderbook.Base as it was being instantiated multiple times
Add error handling for process orderbook

*  Remove redundant currency string in ticker.Price
 Unexport lastupdated field in ticker.Price
 Add error handling for process ticker function and fix tests

* Phase Two Update

* Update translations to use map type - thankyou to kempeng for spotting this

* Change pair method name from Display -> Format for better readability

* Fixes misspelling and tests

* Implement requested changes from GloriousCode

* Remove reduntant function and streamlined return in currency_translation.go

* Revert pair method naming conventions

* Change currency naming conventions

* Changed code type to exported Item type with underlying string to reduce complexity

* Added interim orderbook process method to orderbook.Base type

* Changed feebuilder struct field to currency.Pair

* Adds fall over system for backup fx providers

* deprecate function and children and fix linter issue with btcmarkets

* Fixed requested changes

* Fix bug and move mtx for rates

* Fixed after rebase oopsies

* Fix linter issues

* Fixes race conditions in testing functions

* Final phase coinmarketcap update

* fix linter issues

* Implement requested changes

* Adds configuration variables to increase/decrease time durations between updating currency file and fetching new currency rates

* Add a collection of tests to improve codecov

* After rebase oopsy fixes for btse

* Fix requested changes

* fix after rebase oopsies and add more efficient comparison checks within currency pair

* Fix linter issues
2019-03-19 11:49:05 +11:00
Adrian Gallagher
7dcb1ab553 Migrate from gometalinter.v2 to golangci-lint (#249)
* Migrate from gometalinter.v2 to golangci-lint
2019-03-01 16:10:29 +11:00
Adrian Gallagher
928a332220 Use official Slack join link instead of herokuapp 2019-02-12 13:45:43 +11:00
Andrew
d01e7bad72 Implement Logger (#228)
* Added new base logger

* updated example and test configs

* updated exchange helpers restful router & server

* logPath is now passed to the logger to remove dependency on common package

* updated everything besides exchanges to use new logger

* alphapoint to bitmex done

* updated bitmex bitstamp bittrex btcc and also performance changes to logger

* btcmarkets coinbase coinut exmo gateio wrappers updated

* gateio and gemini logger updated

* hitbtc huobi itbit & kraken updated

* All exchanges updatd

* return correct error for disabled websocket

* don't disconnect client on invalid json

* updated router internal logging

* log.Fatal to t.Error for tests

* Changed from fatal to error failure to set maxprocs

* output ANSI codes for everything but windows for now due to lack of windows support

* added error handling to logger and unit tests

* clear wording on print -> log.print

* added benchmark test

* cleaned up import sections

* Updated logger based on PR requests (added default config options on failure/setting errors)

* ah this should fix travici enc config issue

* Load entire config and clear out logging to hopefully fix travisci issue

* wording & test error handling

* fixed formatting issues based on feedback

* fixed formatting issues based on feedback

* changed CheckDir to use mkdirall instead of mkdir and other changes based on feedback
2019-01-08 21:56:22 +11:00
Ryan O'Hara-Reid
3b8591bcc8 Updated documentation tool (#155)
* Updated documentation tool
Added templates
Updated documentation using tool

* Fixed incorrect version in web README

* Added new templates to tool.
Updated documents in templates across the code base.
Used tool to regenerate documentation.
2018-07-19 16:02:24 +10:00
Ryan O'Hara-Reid
9d0616d8cf New communications package
Support for Slack, SMSGlobal, SMTP and Telegram

Supersedes: https://github.com/thrasher-/gocryptotrader/pull/126
2018-06-08 14:09:36 +10:00