2 Commits

Author	SHA1	Message	Date
Scott	48434dfd46	Feature: Candle conversion & Candle validation (#716) ... * Remove old concept. Introduce new job types and candle scaling * Adds extra processing, commands * new concept for queued jobs. Jobs can pause. New commands to manage status * =End of day commit designing tables and implementing prerequisites further. * Adds postgres data history relations * Fixes table design for sqlite. Fixes all issues from merge * Fixes craziness of database design. Adds some functions to get related jobs * Fixes errors * Updates some documentation, manages prerequisite jobs a little better, adds rpc funcs * Fixes database design and adjust repo functions * Tests database relationship * Test coverage of new job functions * Finishes coverage of new functions * Commands and RPC coverage * New database modifications for new job types * Adds db support of new columns. Adds conversion validation. lint * command blurb changes * Allows websocket test to pass consistently * Fixes merge issue preventing datahistorymanager from starting via config * Minor fixes for different job type processing * Fixes rangeholder issue, fixes validation, does not address jobs not starting or wrong status * Fixes database tests, but at what cost. Fixes dhm tests * Fixes dhj completion issue. Adds prerequisite by nickname * Fixes validation processing. Adds db tests and validation * Fixes validation job processing range * Fixes trade sql. Reduces defaults. Validation processing and errors * Updates cli job commands. adds validation decimal. fix job validation * Expands run job handling and tests * Validation work * Fixes validation processing * candle relations. new job type. updating database design * Adds secondary exchange support. Sets stage for candle override * Re adds accidentally deleted relationship * Updates loading and saving candles to have relationship data when relevant * Now validates and replaces candle data appropriately * Fixes getting and setting datahistory data. Neatens DHM * Test coverage * Updates proto for new db types. New test coverage. Secondary exchange work * Investigation into never-ending validation jobs. Now that intervals are ruled out, now need to complete the job.... * Fixes issues with validation job completion. Fixes validation volume issue for secondary exchange * Adds candle warning support to the backtester * Fixes warnings * lint and begin docs * Documentation updates. Final testing changes * Minor fixes * docs, prerequisite checks, more testing * Fixes binance trade test. Rename err * Documentation fixes. Figure fixes * documentation update * Fixes remote PSQL tests * Fix binance mock test * Remove unnecessary JSON * regen proto * Some minor nit fixes * Var usage, query sorting, log improving, sql mirroring * Extra coverage * Experimental removal of m.jobs and mutex. Fix messaging * Fixes error * Lint fixes, command description improvements. More isRunning gates * description improvements * Lint * BUFF regenerate * Rough concept to fix insertions taking up long periods of time * New calculation for trade data. Adds batch saving This also adds an experimental request feature to shut down lingering requests. However, its uncertain whether or not this is having any impact. Initially thought it was the trades that was taking time and not SQL. Will investigate further * Removes experimental requester. Adds documentation. Fixes typo * rm unused error * re-adds more forgotten contributors * Now with proper commit count	2021-08-05 10:27:27 +10:00
Scott	197ef2df21	Feature: Data history manager engine subsystem (#693) ... * Adds lovely initial concept for historical data doer * Adds ability to save tasks. Adds config. Adds startStop to engine * Has a database microservice without use of globals! Further infrastructure design. Adds readme * Commentary to help design * Adds migrations for database * readme and adds database models * Some modelling that doesn't work end of day * Completes datahistoryjob sql.Begins datahistoryjobresult * Adds datahistoryjob functions to retreive job results. Adapts subsystem * Adds process for upserting jobs and job results to the database * Broken end of day weird sqlboiler crap * Fixes issue with SQL generation. * RPC generation and addition of basic upsert command * Renames types * Adds rpc functions * quick commit before context swithc. Exchanges aren't being populated * Begin the tests! * complete sql tests. stop failed jobs. CLI command creation * Defines rpc commands * Fleshes out RPC implementation * Expands testing * Expands testing, removes double remove * Adds coverage of data history subsystem, expands errors and nil checks * Minor logic improvement * streamlines datahistory test setup * End of day minor linting * Lint, convert simplify, rpc expansion, type expansion, readme expansion * Documentation update * Renames for consistency * Completes RPC server commands * Fixes tests * Speeds up testing by reducing unnecessary actions. Adds maxjobspercycle config * Comments for everything * Adds missing result string. checks interval supported. default start end cli * Fixes ID problem. Improves binance trade fetch. job ranges are processed * adds dbservice coverage. adds rpcserver coverage * docs regen, uses dbcon interface, reverts binance, fixes races, toggle manager * Speed up tests, remove bad global usage, fix uuid check * Adds verbose. Updates docs. Fixes postgres * Minor changes to logging and start stop * Fixes postgres db tests, fixes postgres column typo * Fixes old string typo,removes constraint,error parsing for nonreaders * prevents dhm running when table doesn't exist. Adds prereq documentation * Adds parallel, rmlines, err fix, comment fix, minor param fixes * doc regen, common time range check and test updating * Fixes job validation issues. Updates candle range checker. * Ensures test cannot fail due to time.Now() shenanigans * Fixes oopsie, adds documentation and a warn * Fixes another time test, adjusts copy * Drastically speeds up data history manager tests via function overrides * Fixes summary bug and better logs * Fixes local time test, fixes websocket tests * removes defaults and comment,updates error messages,sets cli command args * Fixes FTX trade processing * Fixes issue where jobs got stuck if data wasn't returned but retrieval was successful * Improves test speed. Simplifies trade verification SQL. Adds command help * Fixes the oopsies * Fixes use of query within transaction. Fixes trade err * oopsie, not needed * Adds missing data status. Properly ends job even when data is missing * errors are more verbose and so have more words to describe them * Doc regen for new status * tiny test tinkering * str := string("Removes .String()").String() * Merge fixups * Fixes a data race discovered during github actions * Allows websocket test to pass consistently * Fixes merge issue preventing datahistorymanager from starting via config * Niterinos cmd defaults and explanations * fixes default oopsie * Fixes lack of nil protection * Additional oopsie * More detailed error for validating job exchange	2021-07-01 16:21:48 +10:00