4 Commits

Author	SHA1	Message	Date
Scott	197ef2df21	Feature: Data history manager engine subsystem (#693) ... * Adds lovely initial concept for historical data doer * Adds ability to save tasks. Adds config. Adds startStop to engine * Has a database microservice without use of globals! Further infrastructure design. Adds readme * Commentary to help design * Adds migrations for database * readme and adds database models * Some modelling that doesn't work end of day * Completes datahistoryjob sql.Begins datahistoryjobresult * Adds datahistoryjob functions to retreive job results. Adapts subsystem * Adds process for upserting jobs and job results to the database * Broken end of day weird sqlboiler crap * Fixes issue with SQL generation. * RPC generation and addition of basic upsert command * Renames types * Adds rpc functions * quick commit before context swithc. Exchanges aren't being populated * Begin the tests! * complete sql tests. stop failed jobs. CLI command creation * Defines rpc commands * Fleshes out RPC implementation * Expands testing * Expands testing, removes double remove * Adds coverage of data history subsystem, expands errors and nil checks * Minor logic improvement * streamlines datahistory test setup * End of day minor linting * Lint, convert simplify, rpc expansion, type expansion, readme expansion * Documentation update * Renames for consistency * Completes RPC server commands * Fixes tests * Speeds up testing by reducing unnecessary actions. Adds maxjobspercycle config * Comments for everything * Adds missing result string. checks interval supported. default start end cli * Fixes ID problem. Improves binance trade fetch. job ranges are processed * adds dbservice coverage. adds rpcserver coverage * docs regen, uses dbcon interface, reverts binance, fixes races, toggle manager * Speed up tests, remove bad global usage, fix uuid check * Adds verbose. Updates docs. Fixes postgres * Minor changes to logging and start stop * Fixes postgres db tests, fixes postgres column typo * Fixes old string typo,removes constraint,error parsing for nonreaders * prevents dhm running when table doesn't exist. Adds prereq documentation * Adds parallel, rmlines, err fix, comment fix, minor param fixes * doc regen, common time range check and test updating * Fixes job validation issues. Updates candle range checker. * Ensures test cannot fail due to time.Now() shenanigans * Fixes oopsie, adds documentation and a warn * Fixes another time test, adjusts copy * Drastically speeds up data history manager tests via function overrides * Fixes summary bug and better logs * Fixes local time test, fixes websocket tests * removes defaults and comment,updates error messages,sets cli command args * Fixes FTX trade processing * Fixes issue where jobs got stuck if data wasn't returned but retrieval was successful * Improves test speed. Simplifies trade verification SQL. Adds command help * Fixes the oopsies * Fixes use of query within transaction. Fixes trade err * oopsie, not needed * Adds missing data status. Properly ends job even when data is missing * errors are more verbose and so have more words to describe them * Doc regen for new status * tiny test tinkering * str := string("Removes .String()").String() * Merge fixups * Fixes a data race discovered during github actions * Allows websocket test to pass consistently * Fixes merge issue preventing datahistorymanager from starting via config * Niterinos cmd defaults and explanations * fixes default oopsie * Fixes lack of nil protection * Additional oopsie * More detailed error for validating job exchange	2021-07-01 16:21:48 +10:00
Scott	5ea5245afb	Improvement: Subsystem separation (#664) ... * Initial codes for a trade tracker * Moving everything in a broken fashion * Removes tradetracker. Removes some errors for subsystems * Cleans up some subsystems, renames stuttering types. Removes some global Bot usage * More basic subsystem renaming and file moving * Removes engine dependency from events,ntpserver,ordermanager,comms manager * Exports eventManager, fixes rpcserver. puts rpcserver back for now * Removes redundant error message, further removes engine dependencies * experimental end of day interface usage * adds ability to build the application * Withdraw and event manager handling * cleans up apiserver and communications manager * Cleans up some start/setup processes. Though should separate * More consistency with Setup Start Stop IsRunning funcs * Final consistency pass before testing phase * Fixes engine tests. Fixes stop nil issue * api server tests * Communications manager testing * Connection manager tests and nilsubsystem error * End of day currencypairsyncer tests * Adds databaseconnection/databaseconnection_test.go * Adds withdrawal manager tests * Deposit address testing. Moved orderbook sync first as its more important * Adds test for event manager * More full eventmanager testing * Adds testfile. Enables skipped test. * ntp manager tests * Adds ordermanager tests, Extracts a whole new subsystem from engine and fanangles import cycles * Adds websocket routine manager tests * Basic portfolio manager testing * Fixes issue with currency pair sync startup * Fixes issue with event manager startup * Starts the order manager before backtester starts * Fixes fee tests. Expands testing. Doesnt fix races * Fixes most test races * Resolves data races * Fixes subsystem test issues * currency pair syncer coverage tests * Refactors portfolio. Fixes tests. Withdraw validation Portfolio didn't need to exist with a portfolio manager. Now the porfolio manager is in charge how the portfolio is handled and all portfolio functions are attached to the base instead of just exported at the package level Withdrawal validation occurred at the exchange level when it can just be run at the withdrawal manager level. All withdrawal requests go through that endpoint * lint -fix * golang lint fixes * lints and comments everything * Updates GCT logo, adds documentation for some subsystems * More documentation and more logo updates * Fixes backtesting and apiserver errors encountered * Fixes errors and typos from reviewing * More minor fixes * Changes %h verb to %w * reverbs to %s * Humbly begins reverting to more flat engine package The main reasoning for this is that the subsystem split doesn't make sense in a golang environment. The subsystems are only meant to be used with engine and so by placing them in a non-engine area, it does not work and is inconsistent with the rest of the application's package layout. This will begin salvaging the changes made by reverting to a flat engine package, but maintaining the consistent designs introduced. Further, I will look to remove any TestMains and decrease the scope of testing to be more local and decrease the issues that have been caused from our style of testing. * Manages to re-flatten things. Everything is within its own file * mini fixes * Fixes tests and data races and lints * Updates docs tool for engine to create filename readmes * os -> ioutil * remove err * Appveyor version increase test * Removes tCleanup as its unsupported on appveyor * Adds stuff that I thought was in previous merge master commit * Removes cancel from test * Fixes really fun test-exclusive data race * minor nit fixes * niterinos * docs gen * rm;rf test * Remove typoline. expands startstop helper. Splits apiserver * Removes accidental folder * Uses update instead of replace for order upsert * addresses nits. Renames files. Regenerates documentation. * lint and removal of comments * Add new test for default scenario * Fixes typo * regen docs	2021-05-31 10:17:12 +10:00
Adrian Gallagher	0d787bc259	Bump golangci-lint to v1.24.0, linter fixes and general code improvements (#478) ... * Bump golangci-lint version, update Go version deps and generic code improvements * Fix wesbocket resp nil check and zip closures * Update pprof path	2020-04-09 13:07:32 +10:00
Andrew	92147cdc5f	(Engine): Database system improvements (#358) ... * Migrated to goose & sqlboiler * create tests with sqlboiler * code clean up * Added gct -> sqlboiler config gen * dropped pgx support * dropped pgx support because who needs connection pools * reenable sqlite audit tests * first pass of migration changes * stuff is broken :D * sqlboiler :D * end of date commit * Added comments code clean up * revert go module files back to upstream * bug fix * pushed go.mod update to use correc goose version * renamed sqlite to sqlite3 for consistency across codebase and PR feedback changes * makefile updates * things are broken end of day commit * added postgresql test * use correct database name * travis fixes for env vars * travis fixes for env vars * test fixes * run migration on test setup * test adding postgres support to appveyor * Skip tests on appveyor due to issues with missing binaries * oh yeah i have to support windows don't i * bumped goose version up * add postgres to osx * fix travis config as osx does not support services move spin up to before_script * added PGDATA path fix * pass PG_DATA to pg_ctl * added initdb to before install * fixes to wording and bumps up goose version * who needs ssl anyway * moved ssl to correct section :D * bumped goose version up * unbreak travis * unbreak travis * fix if database is disabled in config * move strings to consts * converted more strings to const * improvements to sqlboiler mmodel gen * Added contrib\sqlboiler file * sqlboiler windows contrib fixes * bumped goose version up * :D whoops * further fixes to sql models * further fixes to sql models * database type fix for config gen * README update * go.mod clean up * added config details for appveyor * appveyor ordering fix * force psql9.6 * appveyor config changes * all the environmen vars * model changes for psql * model changes for psql * sqlite model fixes * attempt at osx fix * added error check for migration * typos and check against goose error instead of string :D * updated sqlboiler commit id * bump sqlboiler version again * set decimal package to @0bb1631 * readme and makefile updates * bump goose version update readme and add override flag to config gen * README typo fix and lowered inserts in test down to 20 as we are only testing that inserts work running 200 was unnecessary * added gctcli command for audit event * Added debug output toggle to config added both postgres & sqlite support to gctcli command * Wording changes on errors * set sqlite to 1 connection to stop locke database issues * Usage update for order * README updates with config examples * go.mod/sum tidy * removed lines in import second * removed lines in imports * convert local time to utc for database and display output * go mod clean up and error checking to time * renamed all packages to sqlite3 * added windows command output for sql model gen * time conversion fix * time conversion on gctcli	2019-10-08 15:28:31 +11:00