Commit Graph

55 Commits

Author SHA1 Message Date
Adrian Gallagher
bb122dcafa codebase: Remove web frontend and related services (#2067)
* codebase: Remove web frontend and related services

* refactor: Update StartPPROF to accept context and adjust related tests

* refactor: Simplify SetIfZero functions and update related tests

* config: Clarify DowngradeConfig method documentation regarding permanent removal of deprecated fields

* refactor: Rename setIfZeroAndWarn to setDefaultIfZeroWarn for clarity and update related calls

* refactor: Update error handling in DataHistoryManager and remove redundant error variable
2025-09-30 13:32:09 +10:00
Samuael A.
fc0f262c42 exchanges: Limit mock test JSON data size by truncating slices and maps (#1968)
* set limiter to first level mock data list and updated unit tests

* address nested slices length limit

* minor fix recording file and update unit tests

* minor updates on unit tests

* re-record mock files and minor fix on the unit tests ti adapt the mock data change

* improve http recording limit value and fix issues with mock data in binance

* added MockDataSliceLimit in request items and resolve minor unit test issues

* resolve missed conflict

* rename mock variables, resolve unit test issues, and other updates

* minor fix to CheckJSON and update unit tests

* minor unit test fix

* further optimization on mock CheckJSON method, unit tests, and re-record poloniex

* common and recording unit tests fix

* minor linter issues fix

* unit tests format fix

* fix miscellaneous error

* unit tests fix and minor docs update

* re-record and reduce mock file size

* indentation fix

* minor assertion test fix

* reverted log.Printf line in live testing

* rename variables

* update NewVCRServer unit test

* replace string comparison with *net.OpError check

* restructur net error test

* exchanges/mock: Remove redundant error assertion message in TestNewVCRServer

---------

Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2025-08-26 10:27:07 +10:00
Adrian Gallagher
ba92ba3254 docs: Add CODING_GUIDELINES.md and other adjustments (#1988)
* docs: Create new CODING_GUIDELINES doc

Rids excess contribution instructions from other packages
Adds AGENTS.md file for the AI overlords
Rids unused templates
Updates ADD_NEW_EXCHANGE.md with minor fixes

* docs: Fix linter issues and minor adjustments based on Copilot feedback

* docs: Update coding guidelines for API parameters and testing practices

* docs: Remove redundant GoDoc references

Adds copilot-instructions.md

* docs: Update CODING_GUIDELINES with export recommendations and test commentary

* docs: Fix formatting inconsistencies in ADD_NEW_EXCHANGE.md links

* docs: Update struct naming conventions for request and response types

* docs: Improve clarity and consistency in ADD_NEW_EXCHANGE.md and CODING_GUIDELINES.md

* refactor: Simplify error handling in QueryOrder method
2025-08-08 12:10:53 +10:00
Adrian Gallagher
16c1ff3bdf Slack: Update shared invite link (#1967)
* Slack: Update shared invite link

* fix: add rel="noopener noreferrer" to Slack invite links for security
2025-07-08 13:19:46 +10:00
Adrian Gallagher
a5b638bfb7 GHA: Add additional checks for common issues (#1922)
* GHA, tests: Add additional checks for common issues

These checks include:
- Ensuring that all testify funcs use their formatted variants (e.g., `assert.Equalf(t, expected, actual)` instead of `assert.Equal(t, expected, actual)`).
- Replacing `%s` with %q
- Enforcing consistent usage of should/must wording for testify assert/require messages

* Add support for checking backticked string format specifiers and fix issues

* tests: Fix error comparisons

* tests: Replace errors.Is(err, nil) usage with testify and automate check

* refactor: Rename ExtractPort to ExtractPortOrDefault

* tests: Replace assert with require for error handling in multiple test files

* tests: Replace assert with require for error handling and improve assertions in data tests

* tests: Fix typo in assertion message for StreamVol test

* OKX: Fix GetOpenInterestAndVolumeStrike test with instrument selection and improved assertions

* OKX: Revert intentional error check

* Improve error message for expiry time check in GetOpenInterestAndVolumeStrike test
2025-05-28 12:26:51 +10:00
Gareth Kirwan
b4e45e9a1b Websocket: Restructure files and types (#1859)
* Websocket: Rename stream package

* Websocket: Rename Websocket to Manager

* Websocket: Replace explicit errs with common.NilGuard

* Websocket: Move websocket_types.go to types.go

* Websocket: Minor field comment and alignment in types

* Webosocket: Rename WebsocketConnection to Connection

* Alphapoint: Make gorilla ws import explicit

Just to avoid confusion with our own packages.

* Websocket: Move stream_match to match

* Websocket: Move websocket_connection to connection

* Websocket: Move websocket.go to manager.go

* Websocket: Break out all subscription methods into subscriptions.go

* Websocket: Move connection type into its file

* Websocket: Remove PositionUpdated

Type is not used anywhere

* Kraken: Use local constant for pong

Was the only use of websocket.Pong and doesn't really feel right to
represent kraken's api resp in one of our packages

* Websocket: Move connection sub-types to connection package

* Websocket: Move manager types into manager

* Websocket: Move ConnectionWrapper into manager

* Websocket: Move websocket_test to manager_test

* Websocket: Privatise connectionWrapper

* Websocket: Remaining types into types.go

These really belong somewhere else mostly, but this will do for now

* Websocket: Tidy up connection method vars

* Gofumpt: Moving package imports around

* Websocket: Rename errDuplicateConnectionSetup

* Websocket: Fix duplicate import of gws

* Websocket: Fix gofumpt -extra

* Websocket: Standardise import of gws across other pkgs

* Kraken: Remove unused sub conf consts

These were replaced by the generic Levels and Depth fields on all subs

* Websocket: Privitise ConnectioWrapper fields

* Websocket: inline single use var WebsocketNotAuthenticatedUsingRest

* Websocket: Move documentation to template

* Bithumb: Assertify TestWsHandleData
2025-04-10 16:25:02 +10:00
Adrian Gallagher
f5faca2eb2 linter: Enable gofumpt and run against codebase (#1848)
* linter: Enable gofumpt and run against codebase

* Address shazbert's nits

* gofumpt: Fix issues after rebase
2025-03-18 10:23:16 +11:00
Ryan O'Hara-Reid
e99adca86f encoding/json: Add custom JSON package with build tag support for Sonic (#1623)
* tag optional sonic and allow full library conversion

* Add workflow and disallow arm and darwin usage

* Add basic hotswap benchmark

* linter: fix

* use bash

* linter: fix?

* Fix whoopsie, add to make file, also add mention in features list.

* test enforcement

* actually read documentation see if this works

* linter: fix

* linter: fix

* sonic: bump tagged version

* encoding/json: drop build tag arch and os filters

* encoding/json: consolidate tests

* encoding/json: log build tag usage

* rm superfluous builds

* glorious/nits: add template change and regen docs

* glorious/nits: update commentary on nolint directive

* glorious/nits: rm init func and log results in main.go

* Test to actually pull flag in

* linter: fix

* thrasher: nits

* gk: nits 4 goflags goooooooooo!

* gk: nits rn

* make sonic default json implementation

* screen 386

* linter: fix

* Add commentary

* glorious: nits Makefile not working

* gk: nits

* gk: nits whoops

* whoopsirino

* mention 32bit systems won't be sonic

* gk: super-duper nit of extremes

---------

Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
2025-02-20 16:05:55 +11:00
Adrian Gallagher
69bb645fc2 docs: Migrate development tracking from Trello to GitHub Projects (#1696)
* docs: Migrate development tracking from Trello to GitHub Projects

* docs: Replace non-template based references and correct templates
2024-11-08 10:10:49 +11:00
Adrian Gallagher
a79e0d2b3e qa/spelling: Add Codespell support (#1121)
* Add codespell support

* Fix paths

* Add HTML files to exclusion list
2023-01-30 12:36:56 +11:00
Adrian Gallagher
9a4eb9de84 CI: Fix golangci-lint linter issues, add prealloc linter and bump version depends for Go 1.18 (#915)
* Bump CI versions

* Specifically set go version as 1.17.x bumps it to 1.18

* Another

* Adjust AppVeyor

* Part 1 of linter issues

* Part 2

* Fix various linters and improvements

* Part 3

* Finishing touches

* Tests and EqualFold

* Fix nitterinos plus bonus requester jobs bump for exchanges with large number of tests

* Fix nitterinos and bump golangci-lint timeout for AppVeyor

* Address nits, ensure all books are returned on err due to syncer regression

* Fix the wiggins

* Fix duplication

* Fix nitterinos
2022-04-20 13:45:15 +10:00
Adrian Gallagher
f0d45aa1d2 golangci-lint/CI: Bump versions and introduce new linters (#798)
* golangci-lint/CI: Bump versions

Fix remaining linter issues

* Specifically set AppVeyor version

* Fix the infamous typos 👀

* Add go env cmd to AppVeyor

* Add go version cmd to AppVeyor

* Specify AppVeyor image, adjust linters

* Update go get to go install due to deprecation

* Bump golangci-lint timeout time for AppVeyor

* Change NW contract to NQ

* Address nitters

* GetRandomPair -> Pair{}

* Address nits

* Address time nitterinos plus additional tweaks

* More time inception upgrades!

* Bending time and space
2021-10-14 16:38:53 +11:00
Ryan O'Hara-Reid
d636049fb2 exchanges: Initial context propagation (#744)
* gct: phase one context awareness pass

* exchanges: context propagation pass

* common/requester: force context requirement

* gctcli/exchanges: linter fix

* rpcserver: fix test using dummy rpc server

* backtester: fix comments

* grpc: add correct cancel and timeout for commands

* rpcserver_test: add comment on dummy server

* common: deprecated SendHTTPGetRequest

* linter: fix

* linter: turn on no context check

* apichecker: fix context linter issue

* binance: use param context

* common: remove checks as this gets executed before main

* common: change mutex to RW as clients can be used by multiple go routines.

* common: remove init and JIT default client. Unexport global variables and add protection.

* common: Add comments

* bithumb: after dinner mints fix
2021-09-11 13:52:07 +10:00
Ryan O'Hara-Reid
a2381310da GCT: general updates across codebase (#699)
* orderbook: export orderbook nodes for external strategy inspection

* orderbook: Add in methods for locking and unlocking multiple books at the same time e.g. book1.LockWith(book2); defer book1.UnlockWith(book2)

* include waiting functionality for depth change alert

* backtester: add word.

* log: include logger changes to impl with downstream integration

* engine: reduce params for loading exchange

* assort: rm verbose in tests, change wording in ob, expose sync.waitgroup for ext. sync options

* ticker: reduce map look ups and contention when using RW mutex when there are over 80% writes adds find last function to get the latest rate

* engine/syncmanager: add in waitgroup for step over for external package calls

* cleaup

* engine: linter fix

* currency/fx: include all references to fiat currencies to default

* orderbook: Add in fields to Unsafe type for strategies to detect potential out of sync book operations

* syncmanager: changed config variable to display correct time

* ordermanager: Add time when none provided

* currency/manager: update getasset param to get enabled assets for minor optimizations

* ftx: use get all wallet balances for a better accounts breakdown

* orderbook: unlock in reverse order

* bithumb: fixes bug on market buy and sell orders

* bithumb: fix bug for nonce is also time window sensitive

* bithumb: get orders add required parameter

* bithumb: Add asset type to account struct

* currency: improve log output when checking currency and it fails

* bithumb: Add error return on incomplete pair

* ticker:unexport all service related methods

* ticker/currency: fixes

* orderbook: fix comment

* engine: revert variable name in LoadExchange method

* sync_manager: fix panic when enabling disabling manager

* engine: fix naming convention of exported function and comments

* engine: update comment

* orderbook: fix comment for unsafe type
2021-07-29 14:42:28 +10:00
Scott
63257ce4ca Improvement: Speeding up slow tests (#707)
* Speeds up tests

* Reduces time.Sleeps, lowers CreateTestBot complexity. Breaks things

* Removal of unecessary config reads. Parallel tests. Lower times

* Speeds up recent trades results

* mini update

* zoooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom

* Removes the dupes

* Lint

* post cherrypick

* Fix rare kraken data race

* Fixes banking global issues. Fixes postgres trades

* rmline for appveyor test

* Expands timeout in event that channel is closed before send

* Fix data race

* No rows, no bows and definitely no shows

* Removes parallel from createsnapshot tests

* Extends timedmutext test a smidge. Exchange fatality

* Shorter end timeframe and bigger candle
2021-07-07 12:42:03 +10:00
Scott
5ea5245afb Improvement: Subsystem separation (#664)
* Initial codes for a trade tracker

* Moving everything in a broken fashion

* Removes tradetracker. Removes some errors for subsystems

* Cleans up some subsystems, renames stuttering types. Removes some global Bot usage

* More basic subsystem renaming and file moving

* Removes engine dependency from events,ntpserver,ordermanager,comms manager

* Exports eventManager, fixes rpcserver. puts rpcserver back for now

* Removes redundant error message, further removes engine dependencies

* experimental end of day interface usage

* adds ability to build the application

* Withdraw and event manager handling

* cleans up apiserver and communications manager

* Cleans up some start/setup processes. Though should separate

* More consistency with Setup Start Stop IsRunning funcs

* Final consistency pass before testing phase

* Fixes engine tests. Fixes stop nil issue

* api server tests

* Communications manager testing

* Connection manager tests and nilsubsystem error

* End of day currencypairsyncer tests

* Adds databaseconnection/databaseconnection_test.go

* Adds withdrawal manager tests

* Deposit address testing. Moved orderbook sync first as its more important

* Adds test for event manager

* More full eventmanager testing

* Adds testfile. Enables skipped test.

* ntp manager tests

* Adds ordermanager tests, Extracts a whole new subsystem from engine and fanangles import cycles

* Adds websocket routine manager tests

* Basic portfolio manager testing

* Fixes issue with currency pair sync startup

* Fixes issue with event manager startup

* Starts the order manager before backtester starts

* Fixes fee tests. Expands testing. Doesnt fix races

* Fixes most test races

* Resolves data races

* Fixes subsystem test issues

* currency pair syncer coverage tests

* Refactors portfolio. Fixes tests. Withdraw validation

Portfolio didn't need to exist with a portfolio manager. Now the porfolio manager
is in charge how the portfolio is handled and all portfolio functions are attached
to the base instead of just exported at the package level

Withdrawal validation occurred at the exchange level when it can just be run at the
withdrawal manager level. All withdrawal requests go through that endpoint

* lint -fix

* golang lint fixes

* lints and comments everything

* Updates GCT logo, adds documentation for some subsystems

* More documentation and more logo updates

* Fixes backtesting and apiserver errors encountered

* Fixes errors and typos from reviewing

* More minor fixes

* Changes %h verb to %w

* reverbs to %s

* Humbly begins reverting to more flat engine package

The main reasoning for this is that the subsystem split doesn't make sense
in a golang environment. The subsystems are only meant to be used with engine
and so by placing them in a non-engine area, it does not work and is
inconsistent with the rest of the application's package layout.

This will begin salvaging the changes made by reverting to a flat
engine package, but maintaining the consistent designs introduced.
Further, I will look to remove any TestMains and decrease the scope
of testing to be more local and decrease the issues that have been
caused from our style of testing.

* Manages to re-flatten things. Everything is within its own file

* mini fixes

* Fixes tests and data races and lints

* Updates docs tool for engine to create filename readmes

* os -> ioutil

* remove err

* Appveyor version increase test

* Removes tCleanup as its unsupported on appveyor

* Adds stuff that I thought was in previous merge master commit

* Removes cancel from test

* Fixes really fun test-exclusive data race

* minor nit fixes

* niterinos

* docs gen

* rm;rf test

* Remove typoline. expands startstop helper. Splits apiserver

* Removes accidental folder

* Uses update instead of replace for order upsert

* addresses nits. Renames files. Regenerates documentation.

* lint and removal of comments

* Add new test for default scenario

* Fixes typo

* regen docs
2021-05-31 10:17:12 +10:00
Adrian Gallagher
051675255d CI: Add support for GitHub actions (#680)
* Add initial workflows config

* Add PSQL service

* Add 32bit and frontend jobs

* Add gcc-multilib and fix typo

* Fix backtester test for linux/unix filesystem errors and npm commands

* 32-bit Add -race to purposefully throw an error to ensure envs are set correctly (will revert)

* Revert "32-bit Add -race to purposefully throw an error to ensure envs are set correctly (will revert)"

This reverts commit b97e66c49fca859f8738e01107f96f9ca4040b2e.

* Bump workers count for orderbook processing

* Remove comments to trigger 32-bit cache hit after job success

* Cancel addition plus version bump, spawn more overlords

* Update build badge to Github Actions

* Brach -> Branch
2021-05-06 14:35:05 +10:00
Scott
0da62b7fbf Feature: Add mock testing to ZB (#569)
* Adds mock testing to ZB

* STEALS improved time validation code from the original STOLEN validation code :D

* Mini fixes from review

* happy fun comment stealing

* Moves the loop checker earlier to ensure no double appendages

* Fixes sneaky test

* Fixes the important part where mock tests work instead of live tests

* Skips authenticated endpoints for mock testing.

* lint

* Updates candle wrapper functions to respect design

* basic linting fix

* Reverts configtest.json, updates readme to be way better, adds coverage to validateCandlesRequest

* Tiniest grammatical fix

* Fixes more outdated code references

* Closing out a high

* Fixes spacing

* Replaces all instances of 4 spaces in tmpl files with a tab

* fixes spacing and tab related readme issues once and for all 🤞

* tidy

* indentation violation identification situation
2020-10-07 11:59:08 +11:00
Adrian Gallagher
0d787bc259 Bump golangci-lint to v1.24.0, linter fixes and general code improvements (#478)
* Bump golangci-lint version, update Go version deps and generic code improvements

* Fix wesbocket resp nil check and zip closures

* Update pprof path
2020-04-09 13:07:32 +10:00
Adam
7dd3f13c8d Exchange API modification monitor (#404)
* WIP

* Variable Name Changes and other PR Changes

* Name Changes and Error Checks

* WIP

* Some new functionality added and nits fixed

* WIP on Nits

* More trello functionality added and some PR changes

* Minor changes and linter fixes

* Errors Fixed and some PR Changes

* PR Changes

* WIP

* WIP

* Addition of Config

* WIP of Readme file

* Readme WIP

* Addition of ReadMe

* Linter Issues fixed

* Minor Errors fixed

* fixed test timeout issues and added api keys check for tests

* Errors fixed

* Nits WIP

* Nits fixed

* More efficient mutex usage

* Consts cleaned up

* Readme file fixed

* linter issues fixed

* format issues:

* readme updates and nits

* remove backup file and address nits

* backup file removed for real

* OkexOkcoin updates and linter fixes

* errors fixed

* Nits

* errors fixed

* formatting issues

* race conditions fixed

* files fixed

* go mod n go sum

* WIP adding rate limits

* Added rate limits

* Remove test vars

* go mod tidy

* improving readme file and 2digits for trello auto update

* Added space

* removed space

* addressing nits

* documentation update

* more nits

* board id check

* linter fixes

* test race fixed

* test file match

* errors fixed

* nitz

* LOWERCASE flags

* more nitz

* broken wip

* flag addition fixed

* error fixed

* documentation updated for adding exchange using flags

* something

* add comment

* nitz fixed

* pr changes

* added gemini exchange

* testfile saving changes

* nitz

* fixing done

* pr changes

* nits

* WIP

* minor error fix

* wip

* wip

* WIP

* pr changes

* pr changes

* pr changes

* nits

* nits fixed

* t.Fail removed

* nitz

* disable lakeBTC

* contributors fixed

contributors fixed

ALL THE NITZ

OMG

* OMGGG

ALL THE NITZ

OMG

* name change errors fixed

* linter issues fixed

* linterzz

* Added disabling feature for exchanges

* bug fix

* small feature added

* fixed

bug fix:

no keys

* nits

* test

* test2

* test 3

* linter fixed

* testing

* testing

* remove logs

* OMG

* bugfix

* cyam88888

* nits

* error handled

* minor nits

fixed createnewcheck issue

fixed createnewcheck issue

* functionality change

some functionality changes

no api keys 4u :D

* trello names thro config

* err check
2020-04-01 11:15:10 +11:00
Andrew
7fccc03164 (Logger) Rename package to log (#444)
* renamed package to log to stop side import requirement

* reverted comment changes

* reverted comment changes

* one more reverting wording back to logger

* wording changes on comments
2020-02-12 18:09:56 +11:00
Adrian Gallagher
b949388994 General engine improvements (#437)
* Add exchange manager to engine

* Several improvements for engine and friends

1) New file.Exists func
2) gRPC TLS cert expiration date check and regeneration
3) New donation var for use across the codebase
4) Use Go log package until the logger is initialised

* Add cert tests and create dir tree if it doesn't exist for file.Write

* Link up donation address to documentation tool plus minor adjustments

* Fix remaining donation addrs

* Move non-needed reload exchange funcs

* Revert accidental config_example.json changes 🕯️

* Use go logger for logging until the logger has initiliased, otherwise no output will be seen

* Link up portfolio delay val and other fixes

* Run go mod tidy after dependabot PR

* Address nitterinos
2020-02-06 12:32:01 +11:00
Adrian Gallagher
4e05ad41e3 Exchanges: Remove ANX from codebase (#408)
* Remove ANX from codebase

* Doc changes
2020-01-03 14:12:52 +11:00
Ryan O'Hara-Reid
0c5d75b22c (Engine) Variety of engine updates (#390)
* drop common uuid v4 func and imported package as needed

* removed common functions regarding json marshal and unmarshal and used the json package directly. WRT unmarshal it was calling reflect and converted to string which is also checked in the JSON package so it was doing a double up, this will be a tiny gain as it was directly used in the requester package for all our outbound requests.

* add in string

* explicitly throw away return error value

* atleast return the error that websocket initialise returns

* return error when not connected

* fix comment

* Adds comments

* move package declarations

* drop append whenever we call supported

* remove unused import

* Change incorrect spelling

* fix tests

* fix go import issue
2019-12-03 10:06:08 +11:00
Ryan O'Hara-Reid
24bddcc090 Add directories to exclusion list && change default repo string (#387)
* Add directories to exclusion list && change default repo string

* Add in .idea folder to directory exclusion list

* cleaned

* Added support for calling the tool outside of its file

* fix formatting

* changed strings
2019-11-29 11:14:19 +11:00
Adrian Gallagher
63191ce3ec Engine QA (#381)
* 1) Update Dockerfile/docker-compose.yml
2) Remove inline strings for buy/sell/test pairs
3) Remove dangerous order submission values
4) Fix consistency with audit_events (all other spec files use
CamelCase)
5) Update web websocket endpoint
6) Fix main param set (and induce dryrun mode on specific command line
params)

* Engine QA

Link up exchange syncer to cmd params, disarm market selling bombs and fix OKEX endpoints

* Fix linter issue after merge

* Engine QA changes

Template updates
Wrapper code cleanup
Disarmed order bombs
Documentation updates

* Daily engine QA

Bitstamp improvements
Spelling mistakes
Add Coinbene exchange to support list
Protect API authenticated calls for Coinbene/LBank

* Engine QA changes

Fix exchange_wrapper_coverage tool
Add SupportsAsset to exchange interface
Fix inline string usage and add BCH withdrawal support

* Engine QA

Fix Bitstamp types
Inform user of errors when parsing time accross the codebase
Change time parsing warnings to errors (as they are)
Update markdown docs [with linter fixes]

* Engine QA changes

1) Add test for dryrunParamInteraction
2) Disarm OKCoin/OKEX bombs if someone accidently sets canManipulateRealOrders to true and runs all package tests
3) Actually check exchange setup errors for BTSE and Coinbene, plus address this in the wrapper template
4) Hardcode missing/non-retrievable contributors and bump the contributors
5) Convert numbers/strings to meaningful types in Bitstamp and OKEX
6) If WS is supported for the exchange wrapper template, preset authWebsocketSupport var

* Fix the shadow people

* Link the SyncContinuously paramerino

* Also show SyncContinuously in engine.PrintSettings

* Address nitterinos and use correct filepath for logs

* Bitstamp: Extract ALL THE APM

* Fix additional nitterinos

* Fix time parsing error for Bittrex
2019-11-22 16:07:30 +11:00
Adrian Gallagher
da81ea6c08 Merge branch 'master' into engine 2019-11-11 17:11:11 +11:00
Adrian Gallagher
e9b3e3d328 Bump golangci-lint and AppVeyor versions now that 1.13.x is supported (#375) 2019-11-11 15:15:29 +11:00
Scott
ccfcdf26aa Engine: Protocol Features, coverage, types, BTC markets websocket (#368)
* Attempts to update orderbook so it doesn't need to sort

* Reverts the ws ob stuff. Gets rid of sorting because it happens later. Adds some exchange features

* update existing feature lists. Expands list definition to match my emotions

* Adds bithumb bitmex and bitstamp. adds a couple more types

* Features for you, features for me, features for bittrex, btcmarkets, btse, coinbasepro, coinut, exmo, gateio and gemini

* Features for hitbtc, huobi, itbit, kraken, lakebtc, lbank, localbitcoins, okcoin, okex, poloniex, yobit, zb

* Who can forget good old alphapoint?

* Adds btcmarksets websocket :glitch_crab: fixes alphapoint features

* Adds extra data not in the documentation :/

* Replaces websocket features by using protocol features. However, it breaks it due to import cycles. I'm not sure what I'll do just yet

* Removes import cycle via duplicate structs.

* Increases coverage of config with `TestCheckCurrencyConfigValues`. Moves all currency pair package types into their own files or places it at the bottom of files if necessary

* Increase coverage in code.go

* One way of determining a test has failed, is when to it fails. Removed redundant explanation

* Increases code coverage of conversion

* Lint fixes

* Fixes orderbook tests

* Re-adds sorting because its important to still have the internal pre-processed orderbook to be representative of a real orderbook

* Secret lints that did not show up via Windows linting

* Adds protocol package to contain exchange features

* Fixes protocol implementation

* Fixes ws tests

* Addresses the following: Removes st-st-stutters in config types, changes GetAvailableForexProviders -> GetSupportedForexProviders, removes errors from tests where error is nil, removes orderbook setup when not necessary, removes import newlines, removes false bools from declaration, changes should of to should have

* imports and casing

* Fixes two more nil error checks
2019-10-22 10:56:20 +11:00
Ryan O'Hara-Reid
e2d57540a6 Config overwrite bugfix (#363)
* Fix bug where on parsing an alternate new config it will overwrite main config.json in gct dir

* Stop movement of config.json file from root dir when a new config is parsed in

* Stop overiding config.json at gct dir with new config.json from root directory

* RM LN :D

* Fix bug where promptforconfig in config_encryption.go overwrites default config
Ensure periphery command packages do not interact or save over configuration
Ensure tests to not save over or change current testdata/config
2019-09-27 16:03:41 +10:00
Adrian Gallagher
132d2d03c2 Merge branch 'master' into engine 2019-09-24 17:05:58 +10:00
Adrian Gallagher
bbdf532b16 Update Slack shared invite link 2019-09-24 17:01:32 +10:00
Adrian Gallagher
8ecd8a0c16 Merge branch 'master' into engine 2019-08-09 15:46:24 +10:00
Adrian Gallagher
2078ba907f Update URLs for transfer into org repo (#338)
* Update URLs for transfer into org repo

* Update codecov, travis and othe remaining links

* Update appveyor paths
2019-08-09 11:50:39 +10:00
Scott
e209d85d0d Websocket request-response correlation (#328)
* Establishes new websocket functionality. Begins the creation of the websocket request

* Adding a wrapper over gorilla websocket connect,send,receive to handle ID messages. Doesn't work

* Successfully moved exchange_websocket into its own wshandler namespace. oof

* Sets up ZB to use a round trip WS request system

* Adds Kraken ID support to subscriptions. Renames duplicate func name UnsubscribeToChannels to RemoveSubscribedChannels. Adds some helper methods in the WebsocketConn to reduce duplicate code. Cleans up ZB implementation

* Fixes double locking which caused no websocket data to be read. Fixes requestid for kraken subscriptions

* Completes Huobi and Hadax implementation. Extends ZB error handling. Adds GZip support for reading messages

* Adds HitBTC support. Adds GetCurrencies, GetSymbols, GetTrades WS funcs. Adds super fun new parameter to GenerateMessageID for Unix and UnixNano

* Adds GateIO id support

* Adds Coinut support. Prevents nil reference error in constatus when there isnt one

* Standardises all Exchange websockets to use the wshandler websocket. Removes the wsRequestMtx as wshandler handles that now. Makes the Dialer a dialer, its not externally referenced that I can see.

* Fixes issue with coinut implementation. Updates bitmex currencies. Removes redundant log messages which are used to log messages

* Starts testing. Renames files

* Adds tests for websocket connection

* Reverts request.go change

* Linting everything

* Fixes rebase issue

* Final changes. Fixes variable names, removes log.Debug, removes lines, rearranges test types, removes order correlation websocket type

* Final final commit, fixing ZB issues.

* Adds traffic alerts where missed. Changes empty struct pointer addresses to nil instead. Removes empty lines

* Fixed string conversion

* Fixes issue with ZB not sending success codes

* Fixes issue with coinut processing due to nonce handling with subscriptions

* Fixes issue where ZB test failure was not caught. Removes unnecessary error handling from other ZB tests

* Removes unused interface

* Renames wshandler.Init() to wshandler.Run()

* Updates template file

* Capitalises cryptocurrencies in struct. Moves websocketResponseCheckTimeout and websocketResponseMaxLimit into config options. Moves connection configuration to main exchange Setup (where appropriate). Reverts currencylastupdated ticks. Improves reader close error checking

* Fixes two inconsistent websocket delay times

* Creates a default variable for websocket ResponseMaxLimit and ResponseCheckTimeout, then applies it to setdefaults and all tests

* Updates exchange template to set and use default websocket response limits
2019-08-07 15:15:01 +10:00
Andrew
3de1d94e5f New logging system (#319)
* First pass at adding new logging system

* NewLogger

* NewLogger

* WIP

* silly bug fix

* :D removed files

* removed old logging interface

* added tests

* added tests

* Started to add new lines to all f calls

* Added subsystem log types

* Logger improvements

* Further performance improvements

* changes to logger and sublogger creation

* Renamed Logging types

* removed old print statement

* changes based on feedback

* moved sublogger types to own file

* :)

* added console as output type

* added get level command

* added get/set log level via grpc command

* added check for output being empty for migration support

* first pass at log rotation

* added log rotation

* :D derp fixed

* added tests

* changes based on feedback

* changed log type

* comments

* renamed file -> fileSettings

* typo fix

* changes based on feedback

* gofmt ran on additional files

* gofmt ran on additional files
2019-07-07 05:20:31 +10:00
Ryan O'Hara-Reid
897bcfd9a4 Documentation Update (#318)
* Initial update

* update

* Fix linter issues

* Add new documentation template and fix
2019-06-21 12:38:35 +10:00
Adrian Gallagher
f777e68716 Engine improvements 2019-06-10 20:02:09 +10:00
Adrian Gallagher
e965e54e09 Split up common.go, file path fixes and much more 2019-06-04 17:04:32 +10:00
Adrian Gallagher
f5914e8c10 Engine changes 2019-05-22 17:06:38 +10:00
leilaes
46267e5975 Add missing close file and missing stop ticker (#300)
* Add missing close file and missing stop ticker

* Defer close file after checking error

* Update exchange_template.go
2019-05-17 11:17:35 +10:00
Adrian Gallagher
7dcb1ab553 Migrate from gometalinter.v2 to golangci-lint (#249)
* Migrate from gometalinter.v2 to golangci-lint
2019-03-01 16:10:29 +11:00
Adrian Gallagher
928a332220 Use official Slack join link instead of herokuapp 2019-02-12 13:45:43 +11:00
Adrian Gallagher
5e5ca8a887 Linter fixes (#246)
Linter fixes
2019-02-05 16:26:04 +11:00
Scott
978b91a692 GetActiveOrders/GetOrderHistory wrapper implementation (#239)
* Adds signature to all exchange wrappers

* Adds funky new OrderHistoryRequest type. Updates signature for GetOrderHistory to use funky new type. Adds tests for GetOrderHistory on all exchanges. Implements GetOrderHistory for ANX

* Fixes alphapoint, bitstamp, itbit, zb tests. Adds exchange functions FilterOrdersByStatusAndType, FilterOrdersByTickRange, FilterOrdersByCurrencies to easily filter returned orders. Adds tests for filters. Implements GetOrderHistory wrapper for Binance, bitfinex, bithumb, bitmex. Adds new filter funcs to implementations.

* Adds bitstamp wrapper support

* Splits up GetOrderHistory into GetOpenOrders and GetOrderHistory wraapper functions to distinguish between active and past. Renames exchange.GetOrderHistoryRequest to exchange.GetOrdersRequest. Renames any API exchange method named GetOpenOrders to GetActiveOrders. Adds test function TestGetOpenOrders for each exchange

* Reimplements the split GetOrders and GetOrderHistory for alphapoint, anx, binance, bitfinex, bithumb, bitmex, bitstamp and bittrex. Renames orderType, orderStatus constants. Adds new exchange.FilterOrdersBySide and exchange.FilterOrdersByType and removes old exchange.FilterOrdersByStatusAndType.

* Changes orderHistoryRequest to use currencypair array instead of strings, also adds fees and trade breakdown. Removes if statement preventing ANX/BTCMarkets testing. Implements Active order + Order history retrieval for Bittrex and BTCMarkets.

* Adds support for coinut and coinbasepro

* Adds Exmo support

* Adds GateIO support

* Adds Gemini support

* Adds hitbtc, huobi, hadax,  itbit, kraken support for open orders & order history. Fixes switch case break and fallthroughs. Adds filtering to gateio and gemini results

* Adds support for LakeBTC, Liqui, Localbitcoin, OKCoin, OKEX

* Adds poloniex support

* Adds Wex support

* Adds Yobit support. Updates Wex support

* Adds ZB support. Removes ArrangeActAssert from tests

* Changes baseCurrency + quoteCurrency exchange.OrderDetail properties to a pair.CurrencyPair. Adds exchange name to all implementations. Fixes EXMO TestSetup

* Removes verbose setting from tests as verbosity increases the amount of noise return when testing. Noise is only helpful when debugging tests to get more helpful information to resolve the issue and so it is unnecessary to have such lengthy output when testing in bulk or via Travis CI. This commit therefore improves readability when there are no issues

* Fixes issue where gemini test sandbox api url was overridden. Handles blank response from Gemini

* Fixes verbose typo

* Removes spacing for old act assert test comments. Limits previous infinite loop to 10

* Fixes issue with filtering where orderside is never specified

* Uses proper capitalisation for ServerOrderID and OpenOrders. Reverts commenting out orde_id param for bithumb.GetOrderDetails. Removes unnecessary int logic

* Removes JSON ID fields. Uses map where appropriate for exchange order side/type. Updates OrderDetail/GetOrdersRequest type to use time fields. Remvoes comments. Removes inappropriate variable name. Adds AccountID field for alphapoint. Fixes log message formatting. Lowers errorfs to warnfs for time conversion

* Adds missed files

* Removes blank line

* Adds sorting options for orders. Adds concurrency warnings in comments. Adds test for NewCurrencyPairWithDelimiter. Removes (e *Base) from filter funcs. Updates references to filter funcs

* Fixes rebase issues. Condenses append loops.

* Fixes more receive typos. Removes some inline strings. Adds AskOrderSide and BidOrderSide. Removes hypothetical infinite loop

* Fixes issue where allTrades wasn't used in loop. Fixes assignment/typing issues

* Fixes formatting
2019-02-05 10:44:05 +11:00
Adrian Gallagher
291e404a4a Makefile: add new recipes and linter features (#244)
* Makefile: add new recipes and linter features

* expand linter coverage and fix issues

* Update makefile

* address PR nitterinos
2019-01-31 14:53:24 +11:00
Andrew
d01e7bad72 Implement Logger (#228)
* Added new base logger

* updated example and test configs

* updated exchange helpers restful router & server

* logPath is now passed to the logger to remove dependency on common package

* updated everything besides exchanges to use new logger

* alphapoint to bitmex done

* updated bitmex bitstamp bittrex btcc and also performance changes to logger

* btcmarkets coinbase coinut exmo gateio wrappers updated

* gateio and gemini logger updated

* hitbtc huobi itbit & kraken updated

* All exchanges updatd

* return correct error for disabled websocket

* don't disconnect client on invalid json

* updated router internal logging

* log.Fatal to t.Error for tests

* Changed from fatal to error failure to set maxprocs

* output ANSI codes for everything but windows for now due to lack of windows support

* added error handling to logger and unit tests

* clear wording on print -> log.print

* added benchmark test

* cleaned up import sections

* Updated logger based on PR requests (added default config options on failure/setting errors)

* ah this should fix travici enc config issue

* Load entire config and clear out logging to hopefully fix travisci issue

* wording & test error handling

* fixed formatting issues based on feedback

* fixed formatting issues based on feedback

* changed CheckDir to use mkdirall instead of mkdir and other changes based on feedback
2019-01-08 21:56:22 +11:00
Adrian Gallagher
046e15a67e Periodic GoReportCard fixes 2018-12-27 15:10:41 +11:00
Ryan O'Hara-Reid
92534249bf Remove unwanted wrapper name stuttering and standardise common error returns (#213) 2018-11-23 12:55:00 +11:00
cranktakular
8cbe99cf2f Telegram testing and race condition fix. (#195)
* Telegram testing and race condition fix.

Improving telegram Setup test
More coverage for telegram PushEvent test
Adding telegram HandleMessages test
Adding telegram GetUpdates test
Adding telegram TestConnection test
Adding telegram SendMessage test
(Hopefully) completely fixed race conditions with slack testing

* Adding testing to communications.go
2018-10-23 15:40:25 +11:00