Backtester: Live trading upgrades (#1023)

* Modifications for a smoother live run * Fixes data appending * Successfully allows multi-currency live trading. Adds multiple currencies to live DCA strategy * Attempting to get cash and carry working * Poor attempts at sorting out data and appending it properly with USD in mind * =designs new live data handler * Updates cash and carry strat to work * adds test coverage. begins closeallpositions function * Updates cash and carry to work live * New kline.Event type. Cancels orders on close. Rn types * =Fixes USD funding issue * =fixes tests * fixes tests AGAIN * adds coverage to close all orders * crummy tests, should override * more tests * more tests * more coverage * removes scourge of currency.Pair maps. More tests * missed currency stuff * Fixes USD data issue & collateral issue. Needs to close ALL orders * Now triggers updates on the very first data entry * All my problems are solved now???? * fixes tests, extends coverage * there is some really funky candle stuff going on * my brain is melting * better shutdown management, fixes freezing bug * fixes data duplication issues, adds retries to requests * reduces logging, adds verbose options * expands coverage over all new functionality * fixes fun bug from curr == curr to curr.Equal(curr) * fixes setup issues and tests * starts adding external wallet amounts for funding * more setup for assets * setup live fund calcs and placing orders * successfully performs automated cash and carry * merge fixes * funding properly set at all times * fixes some bugs, need to address currencystatistics still * adds 'appeneded' trait, attempts to fix some stats * fixes stat bugs, adds cool new fetchfees feature * fixes terrible processing bugs * tightens realorder stats, sadly loses some live stats * this actually sets everything correctly for bothcd ..cd ..cd ..cd ..cd ..! * fix tests * coverage * beautiful new test coverage * docs * adds new fee getter delayer * commits from the correct directory * Lint * adds verbose to fund manager * Fix bug in t2b2 strat. Update dca live config. Docs * go mod tidy * update buf * buf + test improvement * Post merge fixes * fixes surprise offset bug * fix sizing restrictions for cash and carry * fix server lints * merge fixes * test fixesss * lintle fixles * slowloris * rn run to task, bug fixes, close all on close * rpc lint and fixes * bugfix: order manager not processing orders properly * somewhat addresses nits * absolutely broken end of day commit * absolutely massive knockon effects from nits * massive knockon effects continue * fixes things * address remaining nits * jk now fixes things * addresses the easier nits * more nit fixers * more niterinos addressederinos * refactors holdings and does some nits * so buf * addresses some nits, fixes holdings bugs * cleanup * attempts to fix alert chans to prevent many chans waiting? * terrible code, will revert * to be reviewed in detail tomorrow * Fixes up channel system * smashes those nits * fixes extra candles, fixes collateral bug, tests * fixes data races, introduces reflection * more checks n tests * Fixes cash and carry issues. Fixes more cool bugs * fixes ~typer~ typo * replace spot strats from ftx to binance * fixes all the tests I just destroyed * removes example path, rm verbose * 1) what 2) removes FTX references from the Backtester * renamed, non-working strategies * Removes FTX references almost as fast as sbf removes funds * regen docs, add contrib names,sort contrib names * fixes merge renamings * Addresses nits. Fixes setting API credentials. Fixes Binance limit retrieval * Fixes live order bugs with real orders and without * Apply suggestions from code review Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io> * Update backtester/engine/live.go Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io> * Update backtester/engine/live.go Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io> * Update backtester/config/strategyconfigbuilder/main.go Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io> * updates docs * even better docs Co-authored-by: Adrian Gallagher <adrian.gallagher@thrasher.io>
2026-06-01 15:10:44 +00:00 · 2023-01-05 13:03:17 +11:00
parent d92ffe6e9e
commit 017cdf1384
195 changed files with 13783 additions and 8048 deletions
--- a/engine/currency_state_manager.md
+++ b/engine/currency_state_manager.md
@@ -1,22 +1,22 @@
-# GoCryptoTrader package Currency state manager
-
-<img src="/common/gctlogo.png?raw=true" width="350px" height="350px" hspace="70">
-
-
-[![Build Status](https://github.com/thrasher-corp/gocryptotrader/actions/workflows/tests.yml/badge.svg?branch=master)](https://github.com/thrasher-corp/gocryptotrader/actions/workflows/tests.yml)
-[![Software License](https://img.shields.io/badge/License-MIT-orange.svg?style=flat-square)](https://github.com/thrasher-corp/gocryptotrader/blob/master/LICENSE)
-[![GoDoc](https://godoc.org/github.com/thrasher-corp/gocryptotrader?status.svg)](https://godoc.org/github.com/thrasher-corp/gocryptotrader/engine/currency_state_manager)
-[![Coverage Status](http://codecov.io/github/thrasher-corp/gocryptotrader/coverage.svg?branch=master)](http://codecov.io/github/thrasher-corp/gocryptotrader?branch=master)
-[![Go Report Card](https://goreportcard.com/badge/github.com/thrasher-corp/gocryptotrader)](https://goreportcard.com/report/github.com/thrasher-corp/gocryptotrader)
-
-
-This currency_state_manager package is part of the GoCryptoTrader codebase.
-
-## This is still in active development
-
-You can track ideas, planned features and what's in progress on this Trello board: [https://trello.com/b/ZAhMhpOy/gocryptotrader](https://trello.com/b/ZAhMhpOy/gocryptotrader).
-
-Join our slack to discuss all things related to GoCryptoTrader! [GoCryptoTrader Slack](https://join.slack.com/t/gocryptotrader/shared_invite/enQtNTQ5NDAxMjA2Mjc5LTc5ZDE1ZTNiOGM3ZGMyMmY1NTAxYWZhODE0MWM5N2JlZDk1NDU0YTViYzk4NTk3OTRiMDQzNGQ1YTc4YmRlMTk)
+# GoCryptoTrader package Currency state manager
+
+<img src="/common/gctlogo.png?raw=true" width="350px" height="350px" hspace="70">
+
+
+[![Build Status](https://github.com/thrasher-corp/gocryptotrader/actions/workflows/tests.yml/badge.svg?branch=master)](https://github.com/thrasher-corp/gocryptotrader/actions/workflows/tests.yml)
+[![Software License](https://img.shields.io/badge/License-MIT-orange.svg?style=flat-square)](https://github.com/thrasher-corp/gocryptotrader/blob/master/LICENSE)
+[![GoDoc](https://godoc.org/github.com/thrasher-corp/gocryptotrader?status.svg)](https://godoc.org/github.com/thrasher-corp/gocryptotrader/engine/currency_state_manager)
+[![Coverage Status](http://codecov.io/github/thrasher-corp/gocryptotrader/coverage.svg?branch=master)](http://codecov.io/github/thrasher-corp/gocryptotrader?branch=master)
+[![Go Report Card](https://goreportcard.com/badge/github.com/thrasher-corp/gocryptotrader)](https://goreportcard.com/report/github.com/thrasher-corp/gocryptotrader)
+
+
+This currency_state_manager package is part of the GoCryptoTrader codebase.
+
+## This is still in active development
+
+You can track ideas, planned features and what's in progress on this Trello board: [https://trello.com/b/ZAhMhpOy/gocryptotrader](https://trello.com/b/ZAhMhpOy/gocryptotrader).
+
+Join our slack to discuss all things related to GoCryptoTrader! [GoCryptoTrader Slack](https://join.slack.com/t/gocryptotrader/shared_invite/enQtNTQ5NDAxMjA2Mjc5LTc5ZDE1ZTNiOGM3ZGMyMmY1NTAxYWZhODE0MWM5N2JlZDk1NDU0YTViYzk4NTk3OTRiMDQzNGQ1YTc4YmRlMTk)

 ## Current Features for Currency state manager
 + The state manager keeps currency states up to date, which include:
@@ -27,22 +27,22 @@ Join our slack to discuss all things related to GoCryptoTrader! [GoCryptoTrader
 + This allows for an internal state check to compliment internal and external 
 strategies.

-
-## Contribution
-
-Please feel free to submit any pull requests or suggest any desired features to be added.
-
-When submitting a PR, please abide by our coding guidelines:
-
-+ Code must adhere to the official Go [formatting](https://golang.org/doc/effective_go.html#formatting) guidelines (i.e. uses [gofmt](https://golang.org/cmd/gofmt/)).
-+ Code must be documented adhering to the official Go [commentary](https://golang.org/doc/effective_go.html#commentary) guidelines.
-+ Code must adhere to our [coding style](https://github.com/thrasher-corp/gocryptotrader/blob/master/doc/coding_style.md).
-+ Pull requests need to be based on and opened against the `master` branch.

-## Donations
-
-<img src="https://github.com/thrasher-corp/gocryptotrader/blob/master/web/src/assets/donate.png?raw=true" hspace="70">
-
-If this framework helped you in any way, or you would like to support the developers working on it, please donate Bitcoin to:
-
+## Contribution
+
+Please feel free to submit any pull requests or suggest any desired features to be added.
+
+When submitting a PR, please abide by our coding guidelines:
+
+ Code must adhere to the official Go [formatting](https://golang.org/doc/effective_go.html#formatting) guidelines (i.e. uses [gofmt](https://golang.org/cmd/gofmt/)).
+ Code must be documented adhering to the official Go [commentary](https://golang.org/doc/effective_go.html#commentary) guidelines.
+ Code must adhere to our [coding style](https://github.com/thrasher-corp/gocryptotrader/blob/master/doc/coding_style.md).
+ Pull requests need to be based on and opened against the `master` branch.
+
+## Donations
+
+<img src="https://github.com/thrasher-corp/gocryptotrader/blob/master/web/src/assets/donate.png?raw=true" hspace="70">
+
+If this framework helped you in any way, or you would like to support the developers working on it, please donate Bitcoin to:
+
 ***bc1qk0jareu4jytc0cfrhr5wgshsq8282awpavfahc***
--- a/engine/helpers.go
+++ b/engine/helpers.go
@@ -576,7 +576,7 @@ func GetCollatedExchangeAccountInfoByCoin(accounts []account.Holdings) map[curre
 	for x := range accounts {
 		for y := range accounts[x].Accounts {
 			for z := range accounts[x].Accounts[y].Currencies {
-				currencyName := accounts[x].Accounts[y].Currencies[z].CurrencyName
+				currencyName := accounts[x].Accounts[y].Currencies[z].Currency
 				total := accounts[x].Accounts[y].Currencies[z].Total
 				onHold := accounts[x].Accounts[y].Currencies[z].Hold
 				avail := accounts[x].Accounts[y].Currencies[z].AvailableWithoutBorrow
@@ -586,7 +586,7 @@ func GetCollatedExchangeAccountInfoByCoin(accounts []account.Holdings) map[curre
 				info, ok := result[currencyName]
 				if !ok {
 					accountInfo := account.Balance{
-						CurrencyName:           currencyName,
+						Currency:               currencyName,
 						Total:                  total,
 						Hold:                   onHold,
 						Free:                   free,
--- a/engine/helpers_test.go
+++ b/engine/helpers_test.go
@@ -855,9 +855,9 @@ func TestGetCollatedExchangeAccountInfoByCoin(t *testing.T) {
 		account.SubAccount{
 			Currencies: []account.Balance{
 				{
-					CurrencyName: currency.BTC,
-					Total:        100,
-					Hold:         0,
+					Currency: currency.BTC,
+					Total:    100,
+					Hold:     0,
 				},
 			},
 		})
@@ -870,14 +870,14 @@ func TestGetCollatedExchangeAccountInfoByCoin(t *testing.T) {
 		account.SubAccount{
 			Currencies: []account.Balance{
 				{
-					CurrencyName: currency.LTC,
-					Total:        100,
-					Hold:         0,
+					Currency: currency.LTC,
+					Total:    100,
+					Hold:     0,
 				},
 				{
-					CurrencyName: currency.BTC,
-					Total:        100,
-					Hold:         0,
+					Currency: currency.BTC,
+					Total:    100,
+					Hold:     0,
 				},
 			},
 		})
--- a/engine/order_manager.go
+++ b/engine/order_manager.go
@@ -465,7 +465,6 @@ func (m *OrderManager) Submit(ctx context.Context, newOrder *order.Submit) (*Ord
 	if err != nil {
 		return nil, err
 	}
-
 	// Checks for exchange min max limits for order amounts before order
 	// execution can occur
 	err = exch.CheckOrderExecutionLimits(newOrder.AssetType,
@@ -478,7 +477,6 @@ func (m *OrderManager) Submit(ctx context.Context, newOrder *order.Submit) (*Ord
 			newOrder.Exchange,
 			err)
 	}
-
 	// Determines if current trading activity is turned off by the exchange for
 	// the currency pair
 	err = exch.CanTradePair(newOrder.Pair, newOrder.AssetType)
@@ -683,22 +681,20 @@ func (m *OrderManager) processOrders() {
 					err)
 				continue
 			}
-			if len(orders) > 0 && len(result) > 0 {
-				for z := range result {
-					var upsertResponse *OrderUpsertResponse
-					upsertResponse, err = m.UpsertOrder(&result[z])
-					if err != nil {
-						log.Error(log.OrderMgr, err)
-					} else {
-						for i := range orders {
-							if orders[i].InternalOrderID != upsertResponse.OrderDetails.InternalOrderID {
-								continue
-							}
-							orders[i] = orders[len(orders)-1]
-							orders = orders[:len(orders)-1]
-							break
-						}
+			for z := range result {
+				var upsertResponse *OrderUpsertResponse
+				upsertResponse, err = m.UpsertOrder(&result[z])
+				if err != nil {
+					log.Error(log.OrderMgr, err)
+					continue
+				}
+				for i := range orders {
+					if orders[i].InternalOrderID != upsertResponse.OrderDetails.InternalOrderID {
+						continue
 					}
+					orders[i] = orders[len(orders)-1]
+					orders = orders[:len(orders)-1]
+					break
 				}
 			}

--- a/engine/portfolio_manager.go
+++ b/engine/portfolio_manager.go
@@ -162,7 +162,7 @@ func (m *portfolioManager) seedExchangeAccountInfo(accounts []account.Holdings)
 		next:
 			for z := range accounts[x].Accounts[y].Currencies {
 				for i := range currencies {
-					if !accounts[x].Accounts[y].Currencies[z].CurrencyName.Equal(currencies[i].CurrencyName) {
+					if !accounts[x].Accounts[y].Currencies[z].Currency.Equal(currencies[i].Currency) {
 						continue
 					}
 					currencies[i].Hold += accounts[x].Accounts[y].Currencies[z].Hold
@@ -173,7 +173,7 @@ func (m *portfolioManager) seedExchangeAccountInfo(accounts []account.Holdings)
 					continue next
 				}
 				currencies = append(currencies, account.Balance{
-					CurrencyName:           accounts[x].Accounts[y].Currencies[z].CurrencyName,
+					Currency:               accounts[x].Accounts[y].Currencies[z].Currency,
 					Total:                  accounts[x].Accounts[y].Currencies[z].Total,
 					Hold:                   accounts[x].Accounts[y].Currencies[z].Hold,
 					Free:                   accounts[x].Accounts[y].Currencies[z].Free,
@@ -184,20 +184,20 @@ func (m *portfolioManager) seedExchangeAccountInfo(accounts []account.Holdings)
 		}

 		for j := range currencies {
-			if !m.base.ExchangeAddressExists(accounts[x].Exchange, currencies[j].CurrencyName) {
+			if !m.base.ExchangeAddressExists(accounts[x].Exchange, currencies[j].Currency) {
 				if currencies[j].Total <= 0 {
 					continue
 				}

 				log.Debugf(log.PortfolioMgr, "Portfolio: Adding new exchange address: %s, %s, %f, %s\n",
 					accounts[x].Exchange,
-					currencies[j].CurrencyName,
+					currencies[j].Currency,
 					currencies[j].Total,
 					portfolio.ExchangeAddress)

 				m.base.Addresses = append(m.base.Addresses, portfolio.Address{
 					Address:     accounts[x].Exchange,
-					CoinType:    currencies[j].CurrencyName,
+					CoinType:    currencies[j].Currency,
 					Balance:     currencies[j].Total,
 					Description: portfolio.ExchangeAddress,
 				})
@@ -207,14 +207,14 @@ func (m *portfolioManager) seedExchangeAccountInfo(accounts []account.Holdings)
 			if currencies[j].Total <= 0 {
 				log.Debugf(log.PortfolioMgr, "Portfolio: Removing %s %s entry.\n",
 					accounts[x].Exchange,
-					currencies[j].CurrencyName)
-				m.base.RemoveExchangeAddress(accounts[x].Exchange, currencies[j].CurrencyName)
+					currencies[j].Currency)
+				m.base.RemoveExchangeAddress(accounts[x].Exchange, currencies[j].Currency)
 				continue
 			}

 			balance, ok := m.base.GetAddressBalance(accounts[x].Exchange,
 				portfolio.ExchangeAddress,
-				currencies[j].CurrencyName)
+				currencies[j].Currency)
 			if !ok {
 				continue
 			}
@@ -222,10 +222,10 @@ func (m *portfolioManager) seedExchangeAccountInfo(accounts []account.Holdings)
 			if balance != currencies[j].Total {
 				log.Debugf(log.PortfolioMgr, "Portfolio: Updating %s %s entry with balance %f.\n",
 					accounts[x].Exchange,
-					currencies[j].CurrencyName,
+					currencies[j].Currency,
 					currencies[j].Total)
 				m.base.UpdateExchangeAddressBalance(accounts[x].Exchange,
-					currencies[j].CurrencyName,
+					currencies[j].Currency,
 					currencies[j].Total)
 			}
 		}
--- a/engine/rpcserver.go
+++ b/engine/rpcserver.go
@@ -192,9 +192,14 @@ func (s *RPCServer) StartRPCRESTProxy() {
 	}

 	go func() {
-		if err := http.ListenAndServe(s.Config.RemoteControl.GRPC.GRPCProxyListenAddress, mux); err != nil {
-			log.Errorf(log.GRPCSys, "gRPC proxy failed to server: %s\n", err)
-			return
+		server := &http.Server{
+			Addr:              s.Config.RemoteControl.GRPC.GRPCProxyListenAddress,
+			ReadHeaderTimeout: time.Minute,
+			ReadTimeout:       time.Minute,
+		}
+
+		if err = server.ListenAndServe(); err != nil {
+			log.Errorf(log.GRPCSys, "GRPC proxy failed to server: %s\n", err)
 		}
 	}()

@@ -636,7 +641,7 @@ func createAccountInfoRequest(h account.Holdings) (*gctrpc.GetAccountInfoRespons
 				continue
 			}
 			a.Currencies = append(a.Currencies, &gctrpc.AccountCurrencyInfo{
-				Currency:          y.CurrencyName.String(),
+				Currency:          y.Currency.String(),
 				TotalValue:        y.Total,
 				Hold:              y.Hold,
 				Free:              y.Free,
@@ -677,7 +682,7 @@ func (s *RPCServer) GetAccountInfoStream(r *gctrpc.GetAccountInfoRequest, stream
 		subAccounts := make([]*gctrpc.AccountCurrencyInfo, len(initAcc.Accounts[x].Currencies))
 		for y := range initAcc.Accounts[x].Currencies {
 			subAccounts[y] = &gctrpc.AccountCurrencyInfo{
-				Currency:   initAcc.Accounts[x].Currencies[y].CurrencyName.String(),
+				Currency:   initAcc.Accounts[x].Currencies[y].Currency.String(),
 				TotalValue: initAcc.Accounts[x].Currencies[y].Total,
 				Hold:       initAcc.Accounts[x].Currencies[y].Hold,
 			}
@@ -724,7 +729,7 @@ func (s *RPCServer) GetAccountInfoStream(r *gctrpc.GetAccountInfoRequest, stream
 			subAccounts := make([]*gctrpc.AccountCurrencyInfo, len(holdings.Accounts[x].Currencies))
 			for y := range holdings.Accounts[x].Currencies {
 				subAccounts[y] = &gctrpc.AccountCurrencyInfo{
-					Currency:   holdings.Accounts[x].Currencies[y].CurrencyName.String(),
+					Currency:   holdings.Accounts[x].Currencies[y].Currency.String(),
 					TotalValue: holdings.Accounts[x].Currencies[y].Total,
 					Hold:       holdings.Accounts[x].Currencies[y].Hold,
 				}
@@ -4824,15 +4829,15 @@ func (s *RPCServer) GetCollateral(ctx context.Context, r *gctrpc.GetCollateralRe
 		free := decimal.NewFromFloat(acc.Currencies[i].AvailableWithoutBorrow)
 		cal := order.CollateralCalculator{
 			CalculateOffline:   r.CalculateOffline,
-			CollateralCurrency: acc.Currencies[i].CurrencyName,
+			CollateralCurrency: acc.Currencies[i].Currency,
 			Asset:              a,
 			FreeCollateral:     free,
 			LockedCollateral:   total.Sub(free),
 		}
 		if r.CalculateOffline &&
-			!acc.Currencies[i].CurrencyName.Equal(currency.USD) {
+			!acc.Currencies[i].Currency.Equal(currency.USD) {
 			var tick *ticker.Price
-			tickerCurr := currency.NewPair(acc.Currencies[i].CurrencyName, currency.USD)
+			tickerCurr := currency.NewPair(acc.Currencies[i].Currency, currency.USD)
 			if !spotPairs.Contains(tickerCurr, true) {
 				// cannot price currency to calculate collateral
 				continue
--- a/engine/rpcserver_test.go
+++ b/engine/rpcserver_test.go
@@ -245,12 +245,12 @@ func (f fExchange) FetchAccountInfo(_ context.Context, a asset.Item) (account.Ho
 				AssetType: a,
 				Currencies: []account.Balance{
 					{
-						CurrencyName: currency.USD,
-						Total:        1337,
+						Currency: currency.USD,
+						Total:    1337,
 					},
 					{
-						CurrencyName: currency.BTC,
-						Total:        13337,
+						Currency: currency.BTC,
+						Total:    13337,
 					},
 				},
 			},