* orderbook: export orderbook nodes for external strategy inspection
* orderbook: Add in methods for locking and unlocking multiple books at the same time e.g. book1.LockWith(book2); defer book1.UnlockWith(book2)
* include waiting functionality for depth change alert
* backtester: add word.
* log: include logger changes to impl with downstream integration
* engine: reduce params for loading exchange
* assort: rm verbose in tests, change wording in ob, expose sync.waitgroup for ext. sync options
* ticker: reduce map look ups and contention when using RW mutex when there are over 80% writes adds find last function to get the latest rate
* engine/syncmanager: add in waitgroup for step over for external package calls
* cleaup
* engine: linter fix
* currency/fx: include all references to fiat currencies to default
* orderbook: Add in fields to Unsafe type for strategies to detect potential out of sync book operations
* syncmanager: changed config variable to display correct time
* ordermanager: Add time when none provided
* currency/manager: update getasset param to get enabled assets for minor optimizations
* ftx: use get all wallet balances for a better accounts breakdown
* orderbook: unlock in reverse order
* bithumb: fixes bug on market buy and sell orders
* bithumb: fix bug for nonce is also time window sensitive
* bithumb: get orders add required parameter
* bithumb: Add asset type to account struct
* currency: improve log output when checking currency and it fails
* bithumb: Add error return on incomplete pair
* ticker:unexport all service related methods
* ticker/currency: fixes
* orderbook: fix comment
* engine: revert variable name in LoadExchange method
* sync_manager: fix panic when enabling disabling manager
* engine: fix naming convention of exported function and comments
* engine: update comment
* orderbook: fix comment for unsafe type
Current generation of protobuf stubs is hard and there's open issues
around it:
https://github.com/grpc-ecosystem/grpc-gateway/issues/1065
grpc-gateway author and README suggest to use `buf` instead which
intends to be a simpler and more reliable method of protobuf stub
generation.
* config: add remoteControl/gRPC/timeInNanoSeconds
* grpc: consult with remoteControl/gRPC/timeInNanoSeconds whether timestamps should be in seconds or nanos
* engine: test if RPCServer.unixTimestamp() respects config/remoteControl/gRPC/timeInNanoSeconds
* engine: implement TestRPCServer_GetTicker_LastUpdatedNanos that makes sure TickerResponse.LastUpdated is returned in nanoseconds if configured
* config_example.json: add remoteControl/gRPC/timeInNanoSeconds
* engine: (1) test RPCServer.unixTimestamp() in parallel, (2) increase time tolerance of TestRPCServer_GetTicker_LastUpdatedNanos()
* engine: TestRPCServer_GetTicker_LastUpdatedNanos() now fetches a mock-up ticker to check timestamps
* gctrpc: generate protobuf files with latest versions of tools
* gctrpc: generation scripts now use protoc-gen-openapiv2 (instead of protoc-gen-swagger)
* gtcrpc: update readme
* first draft of getmanaged orders RPC call
* - ClientIDs for binance, especially spot asset
- applied old ClientOrderId for cancelled orders
- added clientOrderId to GCTRPC
* added tests for Matchfilter and GetManagedOrders
* smaller fixes
* comment fix
added getFilteredOrders to store
changed store mutex to RWMutex
smaller fixes
* fixed bug in Detail Copy and added test
* fixes for Scotts review
* processSubmittedOrder was missing clientOrderId
* changed: TestGetOrdersFiltered expanded
fixed: warning, where variable name collided with package name
fixed: used req.AssetType in binance_wrapper.go
Co-authored-by: Mark Dzulko <81071907+Mark-numus@users.noreply.github.com>
* Speeds up tests
* Reduces time.Sleeps, lowers CreateTestBot complexity. Breaks things
* Removal of unecessary config reads. Parallel tests. Lower times
* Speeds up recent trades results
* mini update
* zoooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom
* Removes the dupes
* Lint
* post cherrypick
* Fix rare kraken data race
* Fixes banking global issues. Fixes postgres trades
* rmline for appveyor test
* Expands timeout in event that channel is closed before send
* Fix data race
* No rows, no bows and definitely no shows
* Removes parallel from createsnapshot tests
* Extends timedmutext test a smidge. Exchange fatality
* Shorter end timeframe and bigger candle
* Adds lovely initial concept for historical data doer
* Adds ability to save tasks. Adds config. Adds startStop to engine
* Has a database microservice without use of globals! Further infrastructure design. Adds readme
* Commentary to help design
* Adds migrations for database
* readme and adds database models
* Some modelling that doesn't work end of day
* Completes datahistoryjob sql.Begins datahistoryjobresult
* Adds datahistoryjob functions to retreive job results. Adapts subsystem
* Adds process for upserting jobs and job results to the database
* Broken end of day weird sqlboiler crap
* Fixes issue with SQL generation.
* RPC generation and addition of basic upsert command
* Renames types
* Adds rpc functions
* quick commit before context swithc. Exchanges aren't being populated
* Begin the tests!
* complete sql tests. stop failed jobs. CLI command creation
* Defines rpc commands
* Fleshes out RPC implementation
* Expands testing
* Expands testing, removes double remove
* Adds coverage of data history subsystem, expands errors and nil checks
* Minor logic improvement
* streamlines datahistory test setup
* End of day minor linting
* Lint, convert simplify, rpc expansion, type expansion, readme expansion
* Documentation update
* Renames for consistency
* Completes RPC server commands
* Fixes tests
* Speeds up testing by reducing unnecessary actions. Adds maxjobspercycle config
* Comments for everything
* Adds missing result string. checks interval supported. default start end cli
* Fixes ID problem. Improves binance trade fetch. job ranges are processed
* adds dbservice coverage. adds rpcserver coverage
* docs regen, uses dbcon interface, reverts binance, fixes races, toggle manager
* Speed up tests, remove bad global usage, fix uuid check
* Adds verbose. Updates docs. Fixes postgres
* Minor changes to logging and start stop
* Fixes postgres db tests, fixes postgres column typo
* Fixes old string typo,removes constraint,error parsing for nonreaders
* prevents dhm running when table doesn't exist. Adds prereq documentation
* Adds parallel, rmlines, err fix, comment fix, minor param fixes
* doc regen, common time range check and test updating
* Fixes job validation issues. Updates candle range checker.
* Ensures test cannot fail due to time.Now() shenanigans
* Fixes oopsie, adds documentation and a warn
* Fixes another time test, adjusts copy
* Drastically speeds up data history manager tests via function overrides
* Fixes summary bug and better logs
* Fixes local time test, fixes websocket tests
* removes defaults and comment,updates error messages,sets cli command args
* Fixes FTX trade processing
* Fixes issue where jobs got stuck if data wasn't returned but retrieval was successful
* Improves test speed. Simplifies trade verification SQL. Adds command help
* Fixes the oopsies
* Fixes use of query within transaction. Fixes trade err
* oopsie, not needed
* Adds missing data status. Properly ends job even when data is missing
* errors are more verbose and so have more words to describe them
* Doc regen for new status
* tiny test tinkering
* str := string("Removes .String()").String()
* Merge fixups
* Fixes a data race discovered during github actions
* Allows websocket test to pass consistently
* Fixes merge issue preventing datahistorymanager from starting via config
* Niterinos cmd defaults and explanations
* fixes default oopsie
* Fixes lack of nil protection
* Additional oopsie
* More detailed error for validating job exchange
* improved functions and new helper functions
* bitfinex margin info func
* small rate change
* rate changes
* adding some currencies for margin funding translation
* adding index candles
* added test
* slight improvement in params
* time func
* orderbook helper avgprice func
* broken test + removing some tlogs and prints
* adding test cases
* error fix
* remove unused
* another unused
* shazbert changes
* wip
* bitfinex func and more nits
* final shazzy nits
* most shazzy nits
* few prior requested changes
* shazbert nits final WIP
* shazbert changes
* minor linter issue
* unused val
* glorious changes
* more verbositiy improvements
* quick changes
* unused remaining amount oops
* thrasher changes
* reverting changes that were only for testing purposes and bymistake pushed up
* bfx shadow dec + huobi fetch tradable pairs formatted so as to return config format for ease of comparison and requests
* more linters
* glorious final nits wip
* formatting tradable pairs for different asset types + remove println
* glorious changes
* added portfolio support for bech32 segwit addresses
* segwit addresses may have l values so including this
* removed debug line
* Expands bech32 address validation support
* lint
Co-authored-by: gloriousCode <scott.grant@thrasher.io>
Co-authored-by: Scott <gloriousCode@users.noreply.github.com>
* engine: Set full state on settings validation before flag check to minimise future work when settings are added.
* engine: purge sillyness comment
* engine: address logic issues with change
* engine: invert boolean check
* Initial codes for a trade tracker
* Moving everything in a broken fashion
* Removes tradetracker. Removes some errors for subsystems
* Cleans up some subsystems, renames stuttering types. Removes some global Bot usage
* More basic subsystem renaming and file moving
* Removes engine dependency from events,ntpserver,ordermanager,comms manager
* Exports eventManager, fixes rpcserver. puts rpcserver back for now
* Removes redundant error message, further removes engine dependencies
* experimental end of day interface usage
* adds ability to build the application
* Withdraw and event manager handling
* cleans up apiserver and communications manager
* Cleans up some start/setup processes. Though should separate
* More consistency with Setup Start Stop IsRunning funcs
* Final consistency pass before testing phase
* Fixes engine tests. Fixes stop nil issue
* api server tests
* Communications manager testing
* Connection manager tests and nilsubsystem error
* End of day currencypairsyncer tests
* Adds databaseconnection/databaseconnection_test.go
* Adds withdrawal manager tests
* Deposit address testing. Moved orderbook sync first as its more important
* Adds test for event manager
* More full eventmanager testing
* Adds testfile. Enables skipped test.
* ntp manager tests
* Adds ordermanager tests, Extracts a whole new subsystem from engine and fanangles import cycles
* Adds websocket routine manager tests
* Basic portfolio manager testing
* Fixes issue with currency pair sync startup
* Fixes issue with event manager startup
* Starts the order manager before backtester starts
* Fixes fee tests. Expands testing. Doesnt fix races
* Fixes most test races
* Resolves data races
* Fixes subsystem test issues
* currency pair syncer coverage tests
* Refactors portfolio. Fixes tests. Withdraw validation
Portfolio didn't need to exist with a portfolio manager. Now the porfolio manager
is in charge how the portfolio is handled and all portfolio functions are attached
to the base instead of just exported at the package level
Withdrawal validation occurred at the exchange level when it can just be run at the
withdrawal manager level. All withdrawal requests go through that endpoint
* lint -fix
* golang lint fixes
* lints and comments everything
* Updates GCT logo, adds documentation for some subsystems
* More documentation and more logo updates
* Fixes backtesting and apiserver errors encountered
* Fixes errors and typos from reviewing
* More minor fixes
* Changes %h verb to %w
* reverbs to %s
* Humbly begins reverting to more flat engine package
The main reasoning for this is that the subsystem split doesn't make sense
in a golang environment. The subsystems are only meant to be used with engine
and so by placing them in a non-engine area, it does not work and is
inconsistent with the rest of the application's package layout.
This will begin salvaging the changes made by reverting to a flat
engine package, but maintaining the consistent designs introduced.
Further, I will look to remove any TestMains and decrease the scope
of testing to be more local and decrease the issues that have been
caused from our style of testing.
* Manages to re-flatten things. Everything is within its own file
* mini fixes
* Fixes tests and data races and lints
* Updates docs tool for engine to create filename readmes
* os -> ioutil
* remove err
* Appveyor version increase test
* Removes tCleanup as its unsupported on appveyor
* Adds stuff that I thought was in previous merge master commit
* Removes cancel from test
* Fixes really fun test-exclusive data race
* minor nit fixes
* niterinos
* docs gen
* rm;rf test
* Remove typoline. expands startstop helper. Splits apiserver
* Removes accidental folder
* Uses update instead of replace for order upsert
* addresses nits. Renames files. Regenerates documentation.
* lint and removal of comments
* Add new test for default scenario
* Fixes typo
* regen docs
* Binance: REST respect proxy variable
* Binance: add rest API functionality
* margin account
* move accountInfo to authenticated endpoints
* myTrades endpoint (not yet implemented)
* add BUSD (available in Binance) to currencies enumeration
* Binance: websocket fix
* like REST, websocket dialer respects HTTP(S)_PROXY env vars
* handle situation when orderbook buffers websocket depth updates, the check on FastUpdateID and FirstUpdateID is done right before WebsocketDepthStream gets staged in orderbook manager's buffer channel. The assertion is this depth's FirstUpdateID should equal (last depth's LastUpdateID + 1)
* Binance: add Margin account test case
* Binance: fix typo in MarginAccount, add more fields
* Binance: margin account holdings bookkeeping
* Binance: add rest API functionality
* spot historical trades (public), needs API key in header
* change how margin account holdings are accounted in accordance with the PR
* Binance: use websocket message timestamp as orderbook update time
* Binance:
* fix mock test TestGetHistoricalTrades
* comment exported types
* Binance: fix linter issue
* Binance: add a lock to prevent orderbook test race
* Update Bittrex API from v1.1 to v3
V1.1 has been retired as of 9/30/2020
- Update REST to V3
- Add initial websocket support
* Bittrex update - enable websockets in testdata config file
* Update Bittrex - add Websocket capability to docs
* Update Bittrex connector
- AppVeyor warnings
- Update tests
- Generate documentation
- Fix nits
* Update Bittrex - add websocket order processing
* Update Bittrex connector
* Bittrex connector - fix ineffectual err assignment
* Fix nits
* Orderbook synchronization
* Remove redundant nil
* Log WS fetch orderbook message as debug message instead of as warning
* Update after rebase
* Add tests
* Add allowed candle interval values
* Replace literals with declared constants
* Replace variable name 'request' with 'req'
* Add check and update for deprecated REST URL
* Nits and some cleaning up
* Change ParseInt bit size to 64
* [FIX] Remove several shadow declarations
* Do not export constructTicker
* Remove parseTime()
* Update GetHistoricCandles()
* [FIX] Address gocritic nits
* [FIX] Address gocritic nits
* Use SendMessageReturnResponse() instead of local map
* Rate limit subscribing and unsubscribing
* [FIX] use go routine for subscribing and unsubscribing
* [FIX] Set correct index for map
* [FIX] Address unused vars, literals, time format
* Adjusted timing when subscribing to many order books
* Cache partial updates to tickers instead of calling REST function
* [FIX] Update sequence nr when multiple updates are queued
* Address golint issues
* Fix nits
* Add initial workflows config
* Add PSQL service
* Add 32bit and frontend jobs
* Add gcc-multilib and fix typo
* Fix backtester test for linux/unix filesystem errors and npm commands
* 32-bit Add -race to purposefully throw an error to ensure envs are set correctly (will revert)
* Revert "32-bit Add -race to purposefully throw an error to ensure envs are set correctly (will revert)"
This reverts commit b97e66c49fca859f8738e01107f96f9ca4040b2e.
* Bump workers count for orderbook processing
* Remove comments to trigger 32-bit cache hit after job success
* Cancel addition plus version bump, spawn more overlords
* Update build badge to Github Actions
* Brach -> Branch
* Add new forex provider ExchangeRateHost.io
* Fix linter paramTypeComine
* Add templates and README files
* Convert all times to UTC
* Fix cosmetic issue and address nits
* Add support for fx exchangerate.host engine override
* Address nit plus use remove plural
* add buy and sell limit to signal event
* add buy limit and sell limit
* add test case
* add verify limit before order
* fix sell max && min bugs
* add equal when sell & buy limit comparison && add received to buy & sell limit testcase
* fix bugs in description of SetSellLimit
* remote backtester\eventhandlers\exchange\exchange.go:115: unnecessary trailing newline (whitespace)
* add timeout=10m to golangci-lint
* add timeout=10m to .golangci.yml
* Revert "remote backtester\eventhandlers\exchange\exchange.go:115: unnecessary trailing newline (whitespace)"
This reverts commit 5f7f34903eb9d11a83d3643141a26388c8364a67.
* Revert "add timeout=10m to .golangci.yml"
This reverts commit c83fa972b58327b8de7af3c8fc1d7c19f537838f.
* Revert "add timeout=10m to golangci-lint"
This reverts commit a9da40e91af05d4bb3eee52a61106686c03f9ff4.
* trailing whitespace && revert timeout for linter ci
* add check when buy & sell limit is 0 && passed test cases in size_test
* fix bugs when buy & sell min & max limit is zero && pass testcase TestExecuteOrder
* check MaximumSize if zero or not && add test cases TestExecuteOrderBuySellSizeLimit
* clean logs
* add update buy sell limit in exchange && update testcase
* fix bugs when max is zero calculateBuySize && add testcase TestMaximumBuySizeEqualZero
* fix bugs when max is zero calculateSellSize && add testcase TestMaximumSellSizeEqualZero
Co-authored-by: Tony Wang <tonywang.data@gmail.com>
