Feature: Candle conversion & Candle validation (#716)

* Remove old concept. Introduce new job types and candle scaling

* Adds extra processing, commands

* new concept for queued jobs. Jobs can pause. New commands to manage status

* =End of day commit designing tables and implementing prerequisites further.

* Adds postgres data history relations

* Fixes table design for sqlite. Fixes all issues from merge

* Fixes craziness of database design. Adds some functions to get related jobs

* Fixes errors

* Updates some documentation, manages prerequisite jobs a little better, adds rpc funcs

* Fixes database design and adjust repo functions

* Tests database relationship

* Test coverage of new job functions

* Finishes coverage of new functions

* Commands and RPC coverage

* New database modifications for new job types

* Adds db support of new columns. Adds conversion validation. lint

* command blurb changes

* Allows websocket test to pass consistently

* Fixes merge issue preventing datahistorymanager from starting via config

* Minor fixes for different job type processing

* Fixes rangeholder issue, fixes validation, does not address jobs not starting or wrong status

* Fixes database tests, but at what cost. Fixes dhm tests

* Fixes dhj completion issue. Adds prerequisite by nickname

* Fixes validation processing. Adds db tests and validation

* Fixes validation job processing range

* Fixes trade sql. Reduces defaults. Validation processing and errors

* Updates cli job commands. adds validation decimal. fix job validation

* Expands run job handling and tests

* Validation work

* Fixes validation processing

* candle relations. new job type. updating database design

* Adds secondary exchange support. Sets stage for candle override

* Re adds accidentally deleted relationship

* Updates loading and saving candles to have relationship data when relevant

* Now validates and replaces candle data appropriately

* Fixes getting and setting datahistory data. Neatens DHM

* Test coverage

* Updates proto for new db types. New test coverage. Secondary exchange work

* Investigation into never-ending validation jobs. Now that intervals are ruled out, now need to complete the job....

* Fixes issues with validation job completion. Fixes validation volume issue for secondary exchange

* Adds candle warning support to the backtester

* Fixes warnings

* lint and begin docs

* Documentation updates. Final testing changes

* Minor fixes

* docs, prerequisite checks, more testing

* Fixes binance trade test. Rename err

* Documentation fixes. Figure fixes

* documentation update

* Fixes remote PSQL tests

* Fix binance mock test

* Remove unnecessary JSON

* regen proto

* Some minor nit fixes

* Var usage, query sorting, log improving, sql mirroring

* Extra coverage

* Experimental removal of m.jobs and mutex. Fix messaging

* Fixes error

* Lint fixes, command description improvements. More isRunning gates

* description improvements

* Lint

* BUFF regenerate

* Rough concept to fix insertions taking up long periods of time

* New calculation for trade data. Adds batch saving

This also adds an experimental request feature to shut down lingering requests. However, its uncertain whether or not this is having any impact. Initially thought it was the trades that was taking time and not SQL. Will investigate further

* Removes experimental requester. Adds documentation. Fixes typo

* rm unused error

* re-adds more forgotten contributors

* Now with proper commit count

database/migrations/20210617165159_datahistoryjob_relations/postgres.sql

@@ -0,0 +1,9 @@
+-- +goose Up
+CREATE TABLE datahistoryjobrelations
+(
+    prerequisite_job_id uuid not null REFERENCES datahistoryjob(id),
+    job_id uuid not null REFERENCES datahistoryjob(id),
+    PRIMARY KEY (prerequisite_job_id, job_id)
+);
+-- +goose Down
+DROP TABLE datahistoryjobrelations;

database/migrations/20210617165159_datahistoryjob_relations/sqlite3.sql

@@ -0,0 +1,11 @@
+-- +goose Up
+CREATE TABLE datahistoryjobrelations
+(
+    prerequisite_job_id text not null,
+    job_id text not null,
+    PRIMARY KEY (prerequisite_job_id, job_id),
+    FOREIGN KEY (prerequisite_job_id) REFERENCES datahistoryjob(id) ON DELETE RESTRICT,
+    FOREIGN KEY (job_id) REFERENCES datahistoryjob(id)  ON DELETE RESTRICT
+);
+-- +goose Down
+DROP TABLE datahistoryjobrelations;

database/migrations/20210630125121_datahistoryjob_conversions/postgres.sql

@@ -0,0 +1,19 @@
+-- +goose Up
+ALTER TABLE datahistoryjob
+    ADD conversion_interval DOUBLE PRECISION,
+    ADD overwrite_data boolean,
+    ADD decimal_place_comparison INTEGER,
+    ADD secondary_exchange_id uuid REFERENCES exchange(id),
+    ADD issue_tolerance_percentage DOUBLE PRECISION,
+    ADD replace_on_issue boolean;
+
+-- +goose Down
+ALTER TABLE datahistoryjob
+    DROP replace_on_issue,
+    DROP issue_tolerance_percentage,
+    DROP CONSTRAINT datahistoryjob_secondary_exchange_id_fkey,
+    DROP secondary_exchange_id,
+    DROP decimal_place_comparison,
+    DROP overwrite_data,
+    DROP conversion_interval;
+

database/migrations/20210630125121_datahistoryjob_conversions/sqlite3.sql

@@ -0,0 +1,27 @@
+-- +goose Up
+ALTER TABLE datahistoryjob
+    ADD conversion_interval real;
+ALTER TABLE datahistoryjob
+    ADD overwrite_data integer;
+ALTER TABLE datahistoryjob
+    ADD decimal_place_comparison integer;
+ALTER TABLE datahistoryjob
+    ADD secondary_exchange_id text REFERENCES exchange(id) ON DELETE RESTRICT;
+ALTER TABLE datahistoryjob
+    ADD issue_tolerance_percentage real;
+ALTER TABLE datahistoryjob
+    ADD replace_on_issue integer;
+
+-- +goose Down
+ALTER TABLE datahistoryjob
+    DROP replace_on_issue;
+ALTER TABLE datahistoryjob
+    DROP issue_tolerance_percentage;
+ALTER TABLE datahistoryjob
+    DROP secondary_exchange_id;
+ALTER TABLE datahistoryjob
+    DROP decimal_place_comparison;
+ALTER TABLE datahistoryjob
+    DROP overwrite_data;
+ALTER TABLE datahistoryjob
+    DROP conversion_interval;

database/migrations/20210709133512_candle_details/postgres.sql

@@ -0,0 +1,12 @@
+-- +goose Up
+ALTER TABLE candle
+    ADD source_job_id uuid references datahistoryjob(id),
+    ADD validation_job_id uuid REFERENCES datahistoryjob(id),
+    ADD validation_issues TEXT;
+-- +goose Down
+ALTER TABLE candle
+    DROP validation_issues,
+    DROP CONSTRAINT candle_validation_job_id_fkey,
+    DROP validation_job_id,
+    DROP CONSTRAINT candle_source_job_id_fkey,
+    DROP source_job_id;

database/migrations/20210709133512_candle_details/sqlite3.sql

@@ -0,0 +1,15 @@
+-- +goose Up
+ALTER TABLE candle
+    ADD source_job_id TEXT REFERENCES datahistoryjob(id);
+ALTER TABLE candle
+    ADD validation_job_id TEXT REFERENCES datahistoryjob(id);
+ALTER TABLE candle
+    ADD validation_issues TEXT;
+-- +goose Down
+ALTER TABLE candle
+    DROP validation_issues;
+ALTER TABLE candle
+    DROP validation_job_id;
+ALTER TABLE candle
+    DROP source_job_id;
+