(Engine): Database system improvements (#358)

* Migrated to goose & sqlboiler * create tests with sqlboiler * code clean up * Added gct -> sqlboiler config gen * dropped pgx support * dropped pgx support because who needs connection pools * reenable sqlite audit tests * first pass of migration changes * stuff is broken :D * sqlboiler :D * end of date commit * Added comments code clean up * revert go module files back to upstream * bug fix * pushed go.mod update to use correc goose version * renamed sqlite to sqlite3 for consistency across codebase and PR feedback changes * makefile updates * things are broken end of day commit * added postgresql test * use correct database name * travis fixes for env vars * travis fixes for env vars * test fixes * run migration on test setup * test adding postgres support to appveyor * Skip tests on appveyor due to issues with missing binaries * oh yeah i have to support windows don't i * bumped goose version up * add postgres to osx * fix travis config as osx does not support services move spin up to before_script * added PGDATA path fix * pass PG_DATA to pg_ctl * added initdb to before install * fixes to wording and bumps up goose version * who needs ssl anyway * moved ssl to correct section :D * bumped goose version up * unbreak travis * unbreak travis * fix if database is disabled in config * move strings to consts * converted more strings to const * improvements to sqlboiler mmodel gen * Added contrib\sqlboiler file * sqlboiler windows contrib fixes * bumped goose version up * :D whoops * further fixes to sql models * further fixes to sql models * database type fix for config gen * README update * go.mod clean up * added config details for appveyor * appveyor ordering fix * force psql9.6 * appveyor config changes * all the environmen vars * model changes for psql * model changes for psql * sqlite model fixes * attempt at osx fix * added error check for migration * typos and check against goose error instead of string :D * updated sqlboiler commit id * bump sqlboiler version again * set decimal package to @0bb1631 * readme and makefile updates * bump goose version update readme and add override flag to config gen * README typo fix and lowered inserts in test down to 20 as we are only testing that inserts work running 200 was unnecessary * added gctcli command for audit event * Added debug output toggle to config added both postgres & sqlite support to gctcli command * Wording changes on errors * set sqlite to 1 connection to stop locke database issues * Usage update for order * README updates with config examples * go.mod/sum tidy * removed lines in import second * removed lines in imports * convert local time to utc for database and display output * go mod clean up and error checking to time * renamed all packages to sqlite3 * added windows command output for sql model gen * time conversion fix * time conversion on gctcli
2026-05-31 07:26:44 +00:00 · 2019-10-08 15:28:31 +11:00
parent 2a13551dd1
commit 92147cdc5f
66 changed files with 6018 additions and 2745 deletions
--- a/database/README.md
+++ b/database/README.md
@@ -14,46 +14,135 @@ This database package is part of the GoCryptoTrader codebase.

 ## This is still in active development

-You can track ideas, planned features and what's in progresss on this Trello board: [https://trello.com/b/ZAhMhpOy/gocryptotrader](https://trello.com/b/ZAhMhpOy/gocryptotrader).
+You can track ideas, planned features and what's in progress on this Trello board: [https://trello.com/b/ZAhMhpOy/gocryptotrader](https://trello.com/b/ZAhMhpOy/gocryptotrader).

 Join our slack to discuss all things related to GoCryptoTrader! [GoCryptoTrader Slack](https://join.slack.com/t/gocryptotrader/shared_invite/enQtNTQ5NDAxMjA2Mjc5LTc5ZDE1ZTNiOGM3ZGMyMmY1NTAxYWZhODE0MWM5N2JlZDk1NDU0YTViYzk4NTk3OTRiMDQzNGQ1YTc4YmRlMTk)

 ## Current Features for database package

 + Establishes & Maintains database connection across program life cycle
-+ Multiple database support via simple repository model
-+ Run migration on connection to assure database is at correct version
+ Migration handed by [Goose](https://github.com/thrasher-corp/goose) 
+ Model generation handled by [SQLBoiler](https://github.com/thrasher-corp/sqlboiler) 

 ## How to use

-##### To Manually migrate to the latest database you can run the "dbmigrate" helper in the cmd folder
+##### Prerequisites

-This will parse and run all migration files in your $GoCryptoTrader/database/migrations
-
-_This is also run from the bot when a connection is established to the database_
-
-```sh
-go run ./cmd/dbmigrate
-```
-A Makefile command has also been added for this
-```sh
-make db_migrate
+[SQLBoiler](https://github.com/thrasher-corp/sqlboiler)
+```shell script
+go get -u github.com/thrasher-corp/sqlboiler
 ```

-##### To create a new migrate file you can also run the same command with the -create "migration name" flag
+[Postgres Driver](https://github.com/thrasher-corp/sqlboiler/drivers/sqlboiler-psql)
+```shell script
+go get -u github.com/thrasher-corp/sqlboiler/drivers/sqlboiler-psql
+```
+
+[SQLite Driver](https://github.com/thrasher-corp/sqlboiler-sqlite3)
+```shell script
+go get -u github.com/thrasher-corp/sqlboiler-sqlite3
+```
+
+##### Configuration
+
+The database configuration struct is currently: 
+```shell script
+type Config struct {
+	Enabled                   bool   `json:"enabled"`
+	Verbose                   bool   `json:"verbose"`
+	Driver                    string `json:"driver"`
+	drivers.ConnectionDetails `json:"connectionDetails"`
+}
+```
+And Connection Details:
+```sh
+type ConnectionDetails struct {
+	Host     string `json:"host"`
+	Port     uint16 `json:"port"`
+	Username string `json:"username"`
+	Password string `json:"password"`
+	Database string `json:"database"`
+	SSLMode  string `json:"sslmode"`
+}
+```
+
+With an example configuration being:

 ```sh
-go run ./cmd/dbmigrate -create "alter some table"
+ "database": {
+  "enabled": true,
+  "verbose": true,
+  "driver": "postgres",
+  "connectionDetails": {
+   "host": "localhost",
+   "port": 5432,
+   "username": "gct-dev",
+   "password": "gct-dev",
+   "database": "gct-dev",
+   "sslmode": "disable"
+  }
+ },
 ```

+##### Create and Run migrations
+ Migrations are created using a modified version of [Goose](https://github.com/thrasher-corp/goose) 
+ 
+ A helper tool sits in the ./cmd/dbmigrate folder that includes the following features:
+ 
+ Check current database version with the "status" command
+```shell script
+dbmigrate -command status
+```
+
+ Create a new migration
+```sh
+dbmigrate -command "create" -args "model"
+```
+_This will create a folder in the ./database/migration folder that contains postgres.sql and sqlite.sql files_
+ + Run dbmigrate command with -command up 
+```shell script
+dbmigrate -command "up"
+```
+
+dbmigrate provides a -migrationdir flag override to tell it what path to look in for migrations
+
 ##### Adding a new model
+Model's are generated using [SQLBoiler](https://github.com/thrasher-corp/sqlboiler) 
+A helper tool has been made located in gen_sqlboiler_config that will parse your GoCryptoTrader config and output a SQLBoiler config

-+ Create Model in github.com/thrasher-corp/gocryptotrader/database/models directory
+```sh
+gen_sqlboiler_config
+```
+
+By default this will look in your gocryptotrader data folder and default config, these can be overwritten 
+along with the location of the sqlboiler generated config
+
+```shell script
+-config "configname.json"
+-datadir "~/.gocryptotrader/"
+-outdir "~/.gocryptotrader/"
+```
+
+
+Generate a new model that gets placed in ./database/models/<databasetype> folder
+
+Linux:
+```shell script
+sqlboiler -o database/models/postgres -p postgres --no-auto-timestamps --wipe psql 
+```
+Windows: 
+```sh
+sqlboiler -o database\\models\\postgres -p postgres --no-auto-timestamps --wipe psql
+```
+
+Helpers have been provided in the Makefile for linux users 
+```
+make gen_db_models
+```
+And in the contrib/sqlboiler.cmd for windows users

 ##### Adding a Repository
 + Create Repository directory in github.com/thrasher-corp/gocryptotrader/database/repository/
-+ Create a base Repository interface with any required Methods
-+ Create a per driver implementation of the Repository that implement all required methods to match the interface

 ## Contribution