* adds funding rate implementations and improvements
* merge fixes x1
* lint
* kucoin funding rates func make
* migrate sync-manager to keys
* some kucoin work
* adds some kucoin wrapper funcs
* ehhh, todo
* kucoin position
* start of orders
* adds the kucoin tests yay
* multiplier
* nits, EWS includes order limits
* NotYetImplemented, IsPerp improvements, cleaning
* lint, test fix, huobi time
* fixes issues, improves testing
* fixes linters I WRECKED
* local lint but remote lint, lint, lint, lint
* fixes err
* skip CI
* lint
* Supported rates, binance endpoints
* fixes weird mocktest problems
* no, CZ is invalid
* fixes some new EWS test errors
* sync manager: don't fall over to rest when websocket connection active
* glorious: nits and fix bug
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* allows sync manager customisation for values and logs
* config-example add
* who doesnt like more coverage?
* ensures you can actually disable it via config el oh el
* less ifs, better control
* fix verbose
* sync trades default false
* fix summary being printed when not enabled
* fixes config checker and output
* nits
* I can put this behind me now
* Fixed logCaSiNg
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* combines if statements
---------
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* SyncManager: Optimise and fixes
This is a fairly invasive change which addresses the amount of work the
sync manager does each cycle and the cycle intervals.
We switch to using discrete locks for each type of work on each pair,
so each worker can take a discrete chunk of work safely.
For performance and simplicity we now use a map for the currencyPairs.
* fix reporting when a websocket is reconnected
* fix not switching REST off after websocket available again
* fix race condition in isProcessing flag
This PR still could go further by avoiding cycling through everything
each time, and by pushing some one-time work for adding enabled pairs
down to a later stage off the hot path.
This was the smallest chunk of refactoring I felt could address
everything without changing too much.
Significant manual testing done with a variety of Timeouts to test for
edgecases and handling.
* SyncManager: Fix ticker/orderbook tracker linked
* SyncManager: Fix sync complete logging in update
* SyncManager: Fix pair format breaking sync key
Kraken seems to always switch to XBT_USDT format, but websockets still
pass around XBTUSDT format. Just to be safe this just removes the
delimiter to avoid any such issues
* SyncManager: Remove unused error
* SyncManager: Remove unused IsProcessing flag
* SyncManager: Fix Update test add() pair format
We had to unify pair format inside sync manager, so test needs to do the
same
* engine: make sure upper and lower case names standardize from config to default and return error if sync item never matches
* nits: Add tests for @gloriouscode
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* orderbook/buffer: data integrity and resubscription pass
* btcmarkets: REMOVE THAT LIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIINE!!!!!!!!!!!!!!!!!
* buffer: reinstate publish, refaactor, invalidate more and comments
* buffer/orderbook: improve update and snapshot performance. Move Update type to orderbook package to util. pointer through entire function calls. (cleanup). Change action string to uint8 for easier comparison. Add parsing helper. Update current test benchmark comments.
* dispatch: change publish func to variadic id param
* dispatch: remove sender receiver wait time as this adds overhead and complexity. update tests.
* dispatch: don't create pointers for every job container
* rpcserver: fix assertion issues with data publishing change
* linter: fixes
* glorious: nits addr
* depth: change validation handling to incorporate and store err
* linter: fix more issues
* dispatch: fix race
* travis: update before fetching
* depth: wrap and return wrapped error in invalidate call and fix tests
* btcmarkets: fix commenting
* workflow: check
* workflow: check
* orderbook: check error
* buffer/depth: return invalidation error and fix tests
* gctcli: display errors on orderbook streams
* buffer: remove unused types
* orderbook/bitmex: shift function to bitmex
* orderbook: Add specific comments to unexported functions that don't have locking require locking.
* orderbook: restrict published data functionality to orderbook.Outbound interface
* common: add assertion failure helper for error
* dispatch: remove atomics, add mutex protection, remove add/remove worker, redo main tests
* dispatch: export function
* engine: revert and change sub logger to manager
* engine: remove old test
* dispatch: add common variable ;)
* btcmarket: don't overflow int in tests on 32bit systems
* ci: force 1.17.7 usage for go
* Revert "ci: force 1.17.7 usage for go"
This reverts commit af2f95563bf218cf2b9f36a9fcf3258e2c6a2d91.
* golangci: bump version add and remove linter items
* Revert "golangci: bump version add and remove linter items"
This reverts commit 3c98bffc9d030e39faca0387ea40c151df2ab06b.
* dispatch: remove unsused mutex from mux
* order: slight optimizations
* nits: glorious
* dispatch: fix regression on uuid generation and input inline with master
* linter: fix
* linter: fix
* glorious: nit - rm slice segration
* account: fix test after merge
* coinbasepro: revert change
* account: close channel instead of needing a receiver, push alert in routine to prepare for waiter.
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* currency: Add method to derive pair
* currency: Add method to lower entire charset but used the slice copy and returned that. This will change the original, just gotta see if this is an issue, but the slice usually goes out of scope anyway.
* currency/pairs: add filter method
* currency: add function to derive select currencies from currency pairs
* currency/engine: slight adjustments
* currency: fix linter issue also shift burden of proof to caller instead of repair, more performant.
* currency: more linter
* pairs: optimize; reduce allocs/op and B/op
* currency: Add in function 'NewPairsFromString' for testing purposes
* currency: don't suppress error
* currency: stop panic on empty currency code
* currency: Add helper method to match currencies between exchanges
* currency: fixed my bad spelling
* currency: Implement stable coin checks, refactored base code methods, optimized upper and lower case strings for currency code/pairs
* currency: add pairs method to derive stable coins from internal list.
* Currency: Cleanup, fix tests.
* engine/exchanges/currency: fix whoops
* Currency: force govet no copy on Item datatype
* Currency: fix naughty linter issues
* exchange: revert change
* currency/config: fix config upgrade mistake
* currency: re-implement currency sub-systems
* *RetrieveConfigCurrencyPairs removed
*CheckCurrencyConfigValues to only provide warnings, add additional support when, disable when support is lost or not available and set default values.
*Drop Cryptocurrencies from configuration as this is not needed.
*Drop REST Poll delay field as this was unused.
*Update default values for currencyFileUpdateDuration & foreignExchangeUpdateDuration.
*Allow Role to be marshalled for file type.
*Refactor RunUpdater to verify and check config values and set default running foreign exchange provider.
* currency: cleanup
* currency: change match -> equal for comparison which is more of a standard and little easier to find
* currency: address nits
* currency: fix whoops
* currency: Add some more pairs methods
* currency: linter issues
* currency: RM unused field
* currency: rm verbose
* currency: fix word
* currency: gocritic
* currency: fix another whoopsie
* example_config: default to show log system name
* Currency: Force all support packages to use Equal method for comparison as there is a small comparison bug when checking upper and lower casing, this has a more of a pronounced impact between exchanges and client instances of currency generation
* currency: fix log name
* ordermanager: fix potential panic
* currency: small optim.
* engine: display correct bool and force shutdown
* currency: add function and fix regression
* Change ConvertCurrency -> ConvertFiat to be more precise
* ADD GetForeignExchangeRate to get specific exchange rate for fiat pair
* Fix currency display and formatting regression and tied in with config.Currency fields
* engine: fix tests
* currency: return the amount when no conversion needs to take place
* currency: reduce method name
* currency: Address nits glorious nits
* currency: fix linter
* currency: addr nits
* currency: check underlying role in test
* gct: change to EMPTYCODE and EMPTYPAIR across codebase
* currency: fix nits
* currency: this fixes test race but this issue has not been resolved. Please see: https://trello.com/c/54eizOIo/143-currency-package-upgrades
* currency: Add temp dir for testing
* Update engine/engine.go
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* documentation: update and regen
* currency: Address niterinos
* currency: Add test case for config upgrade when falling over to exchange rate host as default from exchangeRates provider
* currency: addr nits
* currency: fix whoops
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Initial REST managed order updating
* Apply gloriousCode's changes.go patch
* Update internal order ID handling
* Check error
* Replace string with string pointer
* Avoid nil pointers in upsert
* Update test for UpdateOrderFromDetail()
* Add tests for orders.go
* Remove unnecessary newline
* Address comments
* Add missing nil check
* Add tests for new functions in order_manager.go
* Remove empty line
* Change log level for updates from Info to Debug (keep added orders at Info)
* Initialize orders before running the timer
* [TEMP] Add verbosity for debugging
* Nil checking on exchangeManager in GetExchanges()
- exchangeManager.GetExchanges() and iExchangeManager.GetExchanges() return an error on nil
- bot.GetExchanges() wraps exchangeManager.GetExchanges() and returns an empty slice
* Revert b5afe1a46b
* Do not start the order manager runner thread
Instead, mark the order manager as running
* Remove redundant error.Is() and remove print wrapper on msg
* Add atomic blocker and waitgroup on processOrders()
* Disable unnecessary orderManager runner thread for rpcserver_test
* Remove redundant err from orderStore.getActiveOrders()
* [FIX] Populate requiresProcessing using UpsertResponse data instead of REST return data
.. because the data returned by the REST calls do not include the internal user ID's
* [TEST] Verify that processOrders() actually processes queried order data
* Remove leftover warning and add nil check on wg.Done()
* Apply suggestions from code review
Log category changes - as suggested
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* Return when no exchanges available
Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
* gct: phase one context awareness pass
* exchanges: context propagation pass
* common/requester: force context requirement
* gctcli/exchanges: linter fix
* rpcserver: fix test using dummy rpc server
* backtester: fix comments
* grpc: add correct cancel and timeout for commands
* rpcserver_test: add comment on dummy server
* common: deprecated SendHTTPGetRequest
* linter: fix
* linter: turn on no context check
* apichecker: fix context linter issue
* binance: use param context
* common: remove checks as this gets executed before main
* common: change mutex to RW as clients can be used by multiple go routines.
* common: remove init and JIT default client. Unexport global variables and add protection.
* common: Add comments
* bithumb: after dinner mints fix
* exchanges: add an UpdateTickers method to the main exchange interface
This method will fetch all currency pair tickers of a given asset type
and update them internally, does nothing for now.
* exchanges: refactor UpdateTicker on all exchanges
Keep the exact previous behaviour but implement the UpdateTickers
method and refactor UpdateTicker by using it where applicable.
* sync_manager: update all tickers when batching is enabled
* binance: UpdateTicker to fetch single ticker symbol
* ftx: UpdateTicker to fetch single ticker symbol
* orderbook: export orderbook nodes for external strategy inspection
* orderbook: Add in methods for locking and unlocking multiple books at the same time e.g. book1.LockWith(book2); defer book1.UnlockWith(book2)
* include waiting functionality for depth change alert
* backtester: add word.
* log: include logger changes to impl with downstream integration
* engine: reduce params for loading exchange
* assort: rm verbose in tests, change wording in ob, expose sync.waitgroup for ext. sync options
* ticker: reduce map look ups and contention when using RW mutex when there are over 80% writes adds find last function to get the latest rate
* engine/syncmanager: add in waitgroup for step over for external package calls
* cleaup
* engine: linter fix
* currency/fx: include all references to fiat currencies to default
* orderbook: Add in fields to Unsafe type for strategies to detect potential out of sync book operations
* syncmanager: changed config variable to display correct time
* ordermanager: Add time when none provided
* currency/manager: update getasset param to get enabled assets for minor optimizations
* ftx: use get all wallet balances for a better accounts breakdown
* orderbook: unlock in reverse order
* bithumb: fixes bug on market buy and sell orders
* bithumb: fix bug for nonce is also time window sensitive
* bithumb: get orders add required parameter
* bithumb: Add asset type to account struct
* currency: improve log output when checking currency and it fails
* bithumb: Add error return on incomplete pair
* ticker:unexport all service related methods
* ticker/currency: fixes
* orderbook: fix comment
* engine: revert variable name in LoadExchange method
* sync_manager: fix panic when enabling disabling manager
* engine: fix naming convention of exported function and comments
* engine: update comment
* orderbook: fix comment for unsafe type
* Adds lovely initial concept for historical data doer
* Adds ability to save tasks. Adds config. Adds startStop to engine
* Has a database microservice without use of globals! Further infrastructure design. Adds readme
* Commentary to help design
* Adds migrations for database
* readme and adds database models
* Some modelling that doesn't work end of day
* Completes datahistoryjob sql.Begins datahistoryjobresult
* Adds datahistoryjob functions to retreive job results. Adapts subsystem
* Adds process for upserting jobs and job results to the database
* Broken end of day weird sqlboiler crap
* Fixes issue with SQL generation.
* RPC generation and addition of basic upsert command
* Renames types
* Adds rpc functions
* quick commit before context swithc. Exchanges aren't being populated
* Begin the tests!
* complete sql tests. stop failed jobs. CLI command creation
* Defines rpc commands
* Fleshes out RPC implementation
* Expands testing
* Expands testing, removes double remove
* Adds coverage of data history subsystem, expands errors and nil checks
* Minor logic improvement
* streamlines datahistory test setup
* End of day minor linting
* Lint, convert simplify, rpc expansion, type expansion, readme expansion
* Documentation update
* Renames for consistency
* Completes RPC server commands
* Fixes tests
* Speeds up testing by reducing unnecessary actions. Adds maxjobspercycle config
* Comments for everything
* Adds missing result string. checks interval supported. default start end cli
* Fixes ID problem. Improves binance trade fetch. job ranges are processed
* adds dbservice coverage. adds rpcserver coverage
* docs regen, uses dbcon interface, reverts binance, fixes races, toggle manager
* Speed up tests, remove bad global usage, fix uuid check
* Adds verbose. Updates docs. Fixes postgres
* Minor changes to logging and start stop
* Fixes postgres db tests, fixes postgres column typo
* Fixes old string typo,removes constraint,error parsing for nonreaders
* prevents dhm running when table doesn't exist. Adds prereq documentation
* Adds parallel, rmlines, err fix, comment fix, minor param fixes
* doc regen, common time range check and test updating
* Fixes job validation issues. Updates candle range checker.
* Ensures test cannot fail due to time.Now() shenanigans
* Fixes oopsie, adds documentation and a warn
* Fixes another time test, adjusts copy
* Drastically speeds up data history manager tests via function overrides
* Fixes summary bug and better logs
* Fixes local time test, fixes websocket tests
* removes defaults and comment,updates error messages,sets cli command args
* Fixes FTX trade processing
* Fixes issue where jobs got stuck if data wasn't returned but retrieval was successful
* Improves test speed. Simplifies trade verification SQL. Adds command help
* Fixes the oopsies
* Fixes use of query within transaction. Fixes trade err
* oopsie, not needed
* Adds missing data status. Properly ends job even when data is missing
* errors are more verbose and so have more words to describe them
* Doc regen for new status
* tiny test tinkering
* str := string("Removes .String()").String()
* Merge fixups
* Fixes a data race discovered during github actions
* Allows websocket test to pass consistently
* Fixes merge issue preventing datahistorymanager from starting via config
* Niterinos cmd defaults and explanations
* fixes default oopsie
* Fixes lack of nil protection
* Additional oopsie
* More detailed error for validating job exchange
* Initial codes for a trade tracker
* Moving everything in a broken fashion
* Removes tradetracker. Removes some errors for subsystems
* Cleans up some subsystems, renames stuttering types. Removes some global Bot usage
* More basic subsystem renaming and file moving
* Removes engine dependency from events,ntpserver,ordermanager,comms manager
* Exports eventManager, fixes rpcserver. puts rpcserver back for now
* Removes redundant error message, further removes engine dependencies
* experimental end of day interface usage
* adds ability to build the application
* Withdraw and event manager handling
* cleans up apiserver and communications manager
* Cleans up some start/setup processes. Though should separate
* More consistency with Setup Start Stop IsRunning funcs
* Final consistency pass before testing phase
* Fixes engine tests. Fixes stop nil issue
* api server tests
* Communications manager testing
* Connection manager tests and nilsubsystem error
* End of day currencypairsyncer tests
* Adds databaseconnection/databaseconnection_test.go
* Adds withdrawal manager tests
* Deposit address testing. Moved orderbook sync first as its more important
* Adds test for event manager
* More full eventmanager testing
* Adds testfile. Enables skipped test.
* ntp manager tests
* Adds ordermanager tests, Extracts a whole new subsystem from engine and fanangles import cycles
* Adds websocket routine manager tests
* Basic portfolio manager testing
* Fixes issue with currency pair sync startup
* Fixes issue with event manager startup
* Starts the order manager before backtester starts
* Fixes fee tests. Expands testing. Doesnt fix races
* Fixes most test races
* Resolves data races
* Fixes subsystem test issues
* currency pair syncer coverage tests
* Refactors portfolio. Fixes tests. Withdraw validation
Portfolio didn't need to exist with a portfolio manager. Now the porfolio manager
is in charge how the portfolio is handled and all portfolio functions are attached
to the base instead of just exported at the package level
Withdrawal validation occurred at the exchange level when it can just be run at the
withdrawal manager level. All withdrawal requests go through that endpoint
* lint -fix
* golang lint fixes
* lints and comments everything
* Updates GCT logo, adds documentation for some subsystems
* More documentation and more logo updates
* Fixes backtesting and apiserver errors encountered
* Fixes errors and typos from reviewing
* More minor fixes
* Changes %h verb to %w
* reverbs to %s
* Humbly begins reverting to more flat engine package
The main reasoning for this is that the subsystem split doesn't make sense
in a golang environment. The subsystems are only meant to be used with engine
and so by placing them in a non-engine area, it does not work and is
inconsistent with the rest of the application's package layout.
This will begin salvaging the changes made by reverting to a flat
engine package, but maintaining the consistent designs introduced.
Further, I will look to remove any TestMains and decrease the scope
of testing to be more local and decrease the issues that have been
caused from our style of testing.
* Manages to re-flatten things. Everything is within its own file
* mini fixes
* Fixes tests and data races and lints
* Updates docs tool for engine to create filename readmes
* os -> ioutil
* remove err
* Appveyor version increase test
* Removes tCleanup as its unsupported on appveyor
* Adds stuff that I thought was in previous merge master commit
* Removes cancel from test
* Fixes really fun test-exclusive data race
* minor nit fixes
* niterinos
* docs gen
* rm;rf test
* Remove typoline. expands startstop helper. Splits apiserver
* Removes accidental folder
* Uses update instead of replace for order upsert
* addresses nits. Renames files. Regenerates documentation.
* lint and removal of comments
* Add new test for default scenario
* Fixes typo
* regen docs
