2 Commits

Author	SHA1	Message	Date
Scott	197ef2df21	Feature: Data history manager engine subsystem (#693) ... * Adds lovely initial concept for historical data doer * Adds ability to save tasks. Adds config. Adds startStop to engine * Has a database microservice without use of globals! Further infrastructure design. Adds readme * Commentary to help design * Adds migrations for database * readme and adds database models * Some modelling that doesn't work end of day * Completes datahistoryjob sql.Begins datahistoryjobresult * Adds datahistoryjob functions to retreive job results. Adapts subsystem * Adds process for upserting jobs and job results to the database * Broken end of day weird sqlboiler crap * Fixes issue with SQL generation. * RPC generation and addition of basic upsert command * Renames types * Adds rpc functions * quick commit before context swithc. Exchanges aren't being populated * Begin the tests! * complete sql tests. stop failed jobs. CLI command creation * Defines rpc commands * Fleshes out RPC implementation * Expands testing * Expands testing, removes double remove * Adds coverage of data history subsystem, expands errors and nil checks * Minor logic improvement * streamlines datahistory test setup * End of day minor linting * Lint, convert simplify, rpc expansion, type expansion, readme expansion * Documentation update * Renames for consistency * Completes RPC server commands * Fixes tests * Speeds up testing by reducing unnecessary actions. Adds maxjobspercycle config * Comments for everything * Adds missing result string. checks interval supported. default start end cli * Fixes ID problem. Improves binance trade fetch. job ranges are processed * adds dbservice coverage. adds rpcserver coverage * docs regen, uses dbcon interface, reverts binance, fixes races, toggle manager * Speed up tests, remove bad global usage, fix uuid check * Adds verbose. Updates docs. Fixes postgres * Minor changes to logging and start stop * Fixes postgres db tests, fixes postgres column typo * Fixes old string typo,removes constraint,error parsing for nonreaders * prevents dhm running when table doesn't exist. Adds prereq documentation * Adds parallel, rmlines, err fix, comment fix, minor param fixes * doc regen, common time range check and test updating * Fixes job validation issues. Updates candle range checker. * Ensures test cannot fail due to time.Now() shenanigans * Fixes oopsie, adds documentation and a warn * Fixes another time test, adjusts copy * Drastically speeds up data history manager tests via function overrides * Fixes summary bug and better logs * Fixes local time test, fixes websocket tests * removes defaults and comment,updates error messages,sets cli command args * Fixes FTX trade processing * Fixes issue where jobs got stuck if data wasn't returned but retrieval was successful * Improves test speed. Simplifies trade verification SQL. Adds command help * Fixes the oopsies * Fixes use of query within transaction. Fixes trade err * oopsie, not needed * Adds missing data status. Properly ends job even when data is missing * errors are more verbose and so have more words to describe them * Doc regen for new status * tiny test tinkering * str := string("Removes .String()").String() * Merge fixups * Fixes a data race discovered during github actions * Allows websocket test to pass consistently * Fixes merge issue preventing datahistorymanager from starting via config * Niterinos cmd defaults and explanations * fixes default oopsie * Fixes lack of nil protection * Additional oopsie * More detailed error for validating job exchange	2021-07-01 16:21:48 +10:00
Andrew	f6fd94ea69	Engine: Scripting support (#383) ... * WIP * updated appveyor and increased deadline 5 seconds due to increased linters being added * revert files to upstream/engine * WIP * WIP * mod file changes * added script manager * Added manager/and cli interfaces to scripting * Added script task handler * WIP - Added timer/repeat support and fleshed out wrapper further * autoload support added + WIP * WIP commit * added account balance info * btc markets temp work around * WIP - merged with upstream for new order package BTC Markets responses broken * Cancel order wrapper WIP * order wrapper update * Added test coverage for VM * moved to map for VM List shutdown of all VM now handled added gctcli commands for list and stop of running scripts * added override to load/execute for path * fixed incorrect channel shutdown added further test coverage and restructured gctcli commands into sub commands * increased test coverage for packages * Added docs cleaned up tests and example scripts * Test coverage increased for module/gct/exchange package * windows fixes * merged upstream/engine * WIP * logger fixes - removed pointer to bool check removed duplicate test check for logger * remove unused mutex * added inital upload support * fix linter issues for go-fmt * added zip support for uploading and added base for fund withdrawing * changed error return types and also log errors, fix zip path issue * improved error outputs and code flow * pairs response fix added protobuf defs for stop all and list all * added stop all running scripts general clean up and moved across to OrderManager * linter fixes (gofmt) * added list all command * rewrote zip handler to be cleaner also fixed file overwrite on upload * added query command reworked tests * added further error checking to compileandrun corrected use of pointers for accountinfo * bumped tengo version * Removed named returns reworded log messages removed unused falseptr * WIP * Added virutal machine limit improved config options * added model for script event added upload validation * script_event table has been completed, tests for wrapper functions implemented * README updates * reverted changes opened new PR to move withdraw struct outs * intial work on adding withdraw support after merger of withdraw package * started work on examples * Added crypto withdraw support * fix switch case assignment and gofmt project * Reworking Fiat withdraw request pending #402 * removed double pointer call * added withdraw support for fiat currencies * added tests for withdraw methods increased readme * removed local tengo require and also fix linter issues * Added default log size const added basic test for invalid script execution * First pass at moving wrapper to validator package to allow proper validation of uploaded scripts * Added script details to README added config test added test for no file extension * moved tests to const and fixed incorrect pathing * added test coverage to withdraw package * corrected file close handling * point to included configtest.json * extended validator support when a script is uploaded * Bug fix on bool logic * Added mutex * Don't create autit events on test execution * reverted common to master * moved file rename to unix timestamp format * converted logger enabled back to pointer as i need nilness check also moved scriptid to text over blob * started work on autoload add/remove support * First round of PR fixes (mostly commented exports) * Moved GCTScript load to last, removed unneeded error from cleanup() * Comment clairty for AuitEventID * added autoload add/remove command to cli * added tests for autoload * Test updates for Exchanges * linter fixes (gofmt) * Removed double check of engine pointer * remove possible nil pointer on GetSpecificTicker * Fixed not closing file handler on write that causes archive removal to fail * file handler Close clean ups * corrected spelling on error return and return invalid name n autoload * moved strings to cosnt moved bool pointer creation to convert package * new zip extractor added * Validation has been added to archive uploads * removed shadow var on err * added ok check to conversion * converted condition check * basic test for zip extract added * new zip handler * reverted back to old atomic loading system * removed shadow err * lets add a new line * added space to error return * command line toggle for script now works properly * readme updated * set configLoaded to true * check for configLoaded condition * added mutex to allow for multiple access on virtual machine increased test coverage disable script manager if scripting is disabled * linked up to enable/disablesubsystem commands * added start/stop example to readme * reworked logic on test as check should be done on Load() * updated to tengo v2 * linters * lower time on ntp client to stop slippage * remove all fails if any fail validtion from an archive * remove vm from list if timer is invalid * removed shadow on err * remove config creation from NTPCheck test * WIP testing DB changes * add unique constraint * WIP: created has many model * linters run * basic sqlite3 support added for new database format * linters run * Added test coverage for script repo * removed unused print * updated env vars for CI instances * updated env vars for CI instances * Updated test packages * Test updates for postgresql * removed invalid tests from postgres * remove duplication of struct and improved code flow * general cleanup * wording changes on log output * use databasemgr logger and add support for autoload without file extension * corrected test naming * return correct error * return correct error again version 82 * store scriptdata on creation * Hello * Errorln -> Errorf * Removed unused vars * Read me updates * testing without parallel * comment on exported type * added nil check against VM for test * add debugging information * gofmt * remove verbose and data sent to channel * Added debug information * linter fixes (gofmt) * remove unused CompileAndRun() call * test sleep to see if issue is timing related * semi-concurrent map fixes * one day i will run gofmt or setup precommit hooks * new line :D * increased test coverage * added correct sleep time * Moved over to sync map * linter fixes (gofmt) * goimports * moved VM related methods to vm.go * new line at end of file * trying increased timeout on golangci-lint for appveyor * add debugging information * removed timeout * reworked timeout logic * linter fixes (gofmt) * increased test coverage * increased test coverage * one day i will run gofmt or setup precommit hooks * removed unused exchange test * increased golangci-lint timeout * Added nil check on shutdown and test coverage for it lowered timeout back to 1:30 * reworked ID system * removed script hash as it was unused * added comments on exported methods and read me update * reorder code * removed to atomic.value for test execution flag * increased test coverage * move add further up execution * point to correct script file	2020-01-23 13:54:04 +11:00