Files
gocryptotrader/backtester/config/config.go
Scott f929b4d51e backtester: Futures handling & FTX Cash and Carry example strategy (#930)
* implements futures functions and GRPC functions on new branch

* lint and test fixes

* Fix uneven split pnl. Adds collateral weight test. docs. New clear func

* Test protection if someone has zero collateral

* Uses string instead of double for accuracy

* Fixes old code panic

* context, match, docs

* Addresses Shazniterinos, var names, expanded tests

* Returns subaccount name, provides USD values when offlinecalc

* Fixes oopsie

* Fixes cool bug which allowed made up subaccount results

* Subaccount override on FTX, subaccount results for collateral

* Strenghten collateral account info checks. Improve FTX test

* English is my first language

* Fixes oopsies

* Adds some conceptual futures order details to track PNL

* Initial design of future order processing in the backtester

* Introduces futures concept for collateral and spot/futures config diffs

* Fixes most tests

* Simple designs for collateral funding pair concept

* Expands interface use so much it hurts

* Implements more collateral interfaces

* Adds liquidation, adds strategy, struggles with Binance

* Attempts at getting FTX to work

* Adds calculatePNL as a wrapper function and adds an `IsFutures` asset check

* Successfully loads backtester with collateral currency

* Fails to really get much going for supporting futures

* Merges master changes

* Fleshes out how FTX processes collateral

* Further FTX collateral workings

* hooks up more ftx collateral and pnl calculations

* more funcs to flesh out handling

* Adds more links, just can't fit the pieces together :(

* Greatly expands futures order processing

* Fleshes out position tracker to also handle asset and exchange +testing

* RM linkedOrderID. rn positioncontroller, unexport

* Successfully tracks futures order positions

* Fails to calculate PNL

* Calculates pnl from orders accurately with exception to flipping orders

* Calculates PNL from orders

* Adds another controller layer to make it ez from orderstore

* Backtester now compiles. Adds test coverage

* labels things add scaling collateral test

* Calculates pnl in line with fees

* Mostly accurate PNL, with exception to appending with diff prices

* Adds locks, adds rpc function

* grpc implementations

* Gracefully handles rpc function

* beautiful tests!

* rejiggles tests to polish

* Finishes FTX testing, adds comments

* Exposes collateral calculations to rpc

* Adds commands and testing for rpcserver.go functions

* Increase testing and fix up backtester code

* Returns cool changes to original branch

* end of day fixes

* Fixing some tests

* Fixing tests 🎉

* Fixes all the tests

* Splits the backtester setup and running into different files

* Merge, minor fixes

* Messing with some strategy updates

* Failed understanding at collateral usage

* Begins the creation of cash and carry strategy

* Adds underlying pair, adds filldependentevent for futures

* Completes fill prerequsite event implementation. Can't short though

* Some bug fixes

* investigating funds

* CAN NOW CREATE A SHORT ORDER

* Minor change in short size

* Fixes for unrealised PNL & collateral rendering

* Fixes lint and tests

* Adds some verbosity

* Updates to pnl calc

* Tracks pnl for short orders, minor update to strategy

* Close and open event based on conditions

* Adds pnl data for currency statistics

* Working through PNL calculation automatically. Now panics

* Adds tracking, is blocked from design

* Work to flesh out closing a position

* vain attempts at tracking zeroing out bugs

* woww, super fun new subloggers 🎉

* Begins attempt at automatically handling contracts and collateral based on direction

* Merge master + fixes

* Investigating issues with pnl and holdings

* Minor pnl fixes

* Fixes future position sizing, needs contract sizing

* Can render pnl results, focussing on funding statistics

* tracking candles for futures, but why not btc

* Improves funding statistics

* Colours and stats

* Fixes collateral and snapshot bugs

* Completes test

* Fixes totals bug

* Fix double buy, expand stats, fixes usd totals, introduce interface

* Begins report formatting and calculations

* Appends pnl to receiving curr. Fixes map[time]. accurate USD

* Improves report output rendering

* PNL stats in report. New tests for futures

* Fixes existing tests before adding new coverage

* Test coverage

* Completes portfolio coverage

* Increase coverage exchange, portfolio. fix size bug. NEW CHART

* WHAT IS GOING ON WITH PNL

* Fixes PNL calculation. Adds ability to skip om futures tracking

* minor commit before merge

* Adds basic liquidation to backtester

* Changes liquidation to order based

* Liquidationnnnnn

* Further fleshes out liquidations

* Completes liquidations in a honorable manner. Adds AppendReasonf

* Beginnings of spot futures gap chart. Needs to link currencies to render difference

* Removes fake liquidation. Adds cool new chart

* Fixes somet tests,allows for zero fee value v nil distinction,New tests

* Some annoying test fixes that took too long

* portfolio coverage

* holding coverage, privatisation funding

* Testwork

* boring tests

* engine coverage

* More backtesting coverage

* Funding, strategy, report test coverage

* Completes coverage of report package

* Documentation, fixes some assumptions on asset errors

* Changes before master merge

* Lint and Tests

* defaults to non-coloured rendering

* Chart rendering

* Fixes surprise non-local-lints

* Niterinos to the extremeos

* Fixes merge problems

* The linter splintered across the glinting plinths

* Many nits addressed. Now sells spot position on final candle

* Adds forgotten coverage

* Adds ability to size futures contracts to match spot positions.

* fixes order sell sizing

* Adds tests to sizing. Fixes charting issue

* clint splintered the linters with flint

* Improves stats, stat rendering

* minifix

* Fixes tests and fee bug

* Merge fixeroos

* Microfixes

* Updates orderPNL on first Correctly utilises fees. Adds committed funds

* New base funcs. New order summary

* Fun test updates

* Fix logo colouring

* Fixes niteroonies

* Fix report

* BAD COMMIT

* Fixes funding issues.Updates default fee rates.Combines cashcarry case

* doc regen

* Now returns err

* Fixes sizing bug issue introduced in PR

* Fixes fun fee/total US value bug

* Fix chart bug. Show log charts with disclaimer

* sellside fee

* fixes fee and slippage view

* Fixed slippage price issue

* Fixes calculation and removes rendering

* Fixes stats and some rendering

* Merge fix

* Fixes merge issues

* go mod tidy, lint updates

* New linter attempt

* Version bump in appveyor and makefile

* Regex filename, config fixes, template h2 fixes

* Removes bad stats.

* neatens config builder. Moves filename generator

* Fixes issue where linter wants to fix my spelling

* Fixes pointers and starts
2022-06-30 15:43:41 +10:00

338 lines
14 KiB
Go

package config
import (
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"github.com/shopspring/decimal"
"github.com/thrasher-corp/gocryptotrader/backtester/common"
"github.com/thrasher-corp/gocryptotrader/backtester/eventhandlers/strategies"
"github.com/thrasher-corp/gocryptotrader/backtester/eventhandlers/strategies/base"
gctcommon "github.com/thrasher-corp/gocryptotrader/common"
"github.com/thrasher-corp/gocryptotrader/common/file"
"github.com/thrasher-corp/gocryptotrader/exchanges/asset"
"github.com/thrasher-corp/gocryptotrader/log"
)
// ReadConfigFromFile will take a config from a path
func ReadConfigFromFile(path string) (*Config, error) {
if !file.Exists(path) {
return nil, errors.New("file not found")
}
fileData, err := os.ReadFile(path)
if err != nil {
return nil, err
}
return LoadConfig(fileData)
}
// LoadConfig unmarshalls byte data into a config struct
func LoadConfig(data []byte) (resp *Config, err error) {
err = json.Unmarshal(data, &resp)
return resp, err
}
// Validate checks all config settings
func (c *Config) Validate() error {
err := c.validateDate()
if err != nil {
return err
}
err = c.validateStrategySettings()
if err != nil {
return err
}
err = c.validateCurrencySettings()
if err != nil {
return err
}
return c.validateMinMaxes()
}
// validate ensures no one sets bad config values on purpose
func (m *MinMax) validate() error {
if m.MaximumSize.IsNegative() {
return fmt.Errorf("invalid maximum size %w", errSizeLessThanZero)
}
if m.MinimumSize.IsNegative() {
return fmt.Errorf("invalid minimum size %w", errSizeLessThanZero)
}
if m.MaximumTotal.IsNegative() {
return fmt.Errorf("invalid maximum total set to %w", errSizeLessThanZero)
}
if m.MaximumSize.LessThan(m.MinimumSize) && !m.MinimumSize.IsZero() && !m.MaximumSize.IsZero() {
return fmt.Errorf("%w maximum size %v vs minimum size %v",
errMaxSizeMinSizeMismatch,
m.MaximumSize,
m.MinimumSize)
}
if m.MaximumSize.Equal(m.MinimumSize) && !m.MinimumSize.IsZero() && !m.MaximumSize.IsZero() {
return fmt.Errorf("%w %v",
errMinMaxEqual,
m.MinimumSize)
}
return nil
}
func (c *Config) validateMinMaxes() (err error) {
for i := range c.CurrencySettings {
err = c.CurrencySettings[i].BuySide.validate()
if err != nil {
return err
}
err = c.CurrencySettings[i].SellSide.validate()
if err != nil {
return err
}
}
err = c.PortfolioSettings.BuySide.validate()
if err != nil {
return err
}
err = c.PortfolioSettings.SellSide.validate()
if err != nil {
return err
}
return nil
}
func (c *Config) validateStrategySettings() error {
if c.FundingSettings.UseExchangeLevelFunding && !c.StrategySettings.SimultaneousSignalProcessing {
return errSimultaneousProcessingRequired
}
if len(c.FundingSettings.ExchangeLevelFunding) > 0 && !c.FundingSettings.UseExchangeLevelFunding {
return errExchangeLevelFundingRequired
}
if c.FundingSettings.UseExchangeLevelFunding && len(c.FundingSettings.ExchangeLevelFunding) == 0 {
return errExchangeLevelFundingDataRequired
}
if c.FundingSettings.UseExchangeLevelFunding {
for i := range c.FundingSettings.ExchangeLevelFunding {
if c.FundingSettings.ExchangeLevelFunding[i].InitialFunds.IsNegative() {
return fmt.Errorf("%w for %v %v %v",
errBadInitialFunds,
c.FundingSettings.ExchangeLevelFunding[i].ExchangeName,
c.FundingSettings.ExchangeLevelFunding[i].Asset,
c.FundingSettings.ExchangeLevelFunding[i].Currency,
)
}
}
}
strats := strategies.GetStrategies()
for i := range strats {
if strings.EqualFold(strats[i].Name(), c.StrategySettings.Name) {
return nil
}
}
return fmt.Errorf("strategty %v %w", c.StrategySettings.Name, base.ErrStrategyNotFound)
}
// validateDate checks whether someone has set a date poorly in their config
func (c *Config) validateDate() error {
if c.DataSettings.DatabaseData != nil {
if c.DataSettings.DatabaseData.StartDate.IsZero() ||
c.DataSettings.DatabaseData.EndDate.IsZero() {
return errStartEndUnset
}
if c.DataSettings.DatabaseData.StartDate.After(c.DataSettings.DatabaseData.EndDate) ||
c.DataSettings.DatabaseData.StartDate.Equal(c.DataSettings.DatabaseData.EndDate) {
return errBadDate
}
}
if c.DataSettings.APIData != nil {
if c.DataSettings.APIData.StartDate.IsZero() ||
c.DataSettings.APIData.EndDate.IsZero() {
return errStartEndUnset
}
if c.DataSettings.APIData.StartDate.After(c.DataSettings.APIData.EndDate) ||
c.DataSettings.APIData.StartDate.Equal(c.DataSettings.APIData.EndDate) {
return errBadDate
}
}
return nil
}
// validateCurrencySettings checks whether someone has set invalid currency setting data in their config
func (c *Config) validateCurrencySettings() error {
if len(c.CurrencySettings) == 0 {
return errNoCurrencySettings
}
var hasFutures, hasSlippage bool
for i := range c.CurrencySettings {
if c.CurrencySettings[i].Asset == asset.PerpetualSwap ||
c.CurrencySettings[i].Asset == asset.PerpetualContract {
return errPerpetualsUnsupported
}
if c.CurrencySettings[i].Asset == asset.Futures &&
(c.CurrencySettings[i].Quote.String() == "PERP" || c.CurrencySettings[i].Base.String() == "PI") {
return errPerpetualsUnsupported
}
if c.CurrencySettings[i].Asset.IsFutures() {
hasFutures = true
}
if c.CurrencySettings[i].SpotDetails != nil {
if c.FundingSettings.UseExchangeLevelFunding {
if c.CurrencySettings[i].SpotDetails.InitialQuoteFunds != nil &&
c.CurrencySettings[i].SpotDetails.InitialQuoteFunds.GreaterThan(decimal.Zero) {
return fmt.Errorf("non-nil quote %w", errBadInitialFunds)
}
if c.CurrencySettings[i].SpotDetails.InitialBaseFunds != nil &&
c.CurrencySettings[i].SpotDetails.InitialBaseFunds.GreaterThan(decimal.Zero) {
return fmt.Errorf("non-nil base %w", errBadInitialFunds)
}
} else {
if c.CurrencySettings[i].SpotDetails.InitialQuoteFunds == nil &&
c.CurrencySettings[i].SpotDetails.InitialBaseFunds == nil {
return fmt.Errorf("nil base and quote %w", errBadInitialFunds)
}
if c.CurrencySettings[i].SpotDetails.InitialQuoteFunds != nil &&
c.CurrencySettings[i].SpotDetails.InitialBaseFunds != nil &&
c.CurrencySettings[i].SpotDetails.InitialBaseFunds.IsZero() &&
c.CurrencySettings[i].SpotDetails.InitialQuoteFunds.IsZero() {
return fmt.Errorf("base or quote funds set to zero %w", errBadInitialFunds)
}
if c.CurrencySettings[i].SpotDetails.InitialQuoteFunds == nil {
c.CurrencySettings[i].SpotDetails.InitialQuoteFunds = &decimal.Zero
}
if c.CurrencySettings[i].SpotDetails.InitialBaseFunds == nil {
c.CurrencySettings[i].SpotDetails.InitialBaseFunds = &decimal.Zero
}
}
}
if c.CurrencySettings[i].Base.IsEmpty() {
return errUnsetCurrency
}
if !c.CurrencySettings[i].Asset.IsValid() {
return fmt.Errorf("%v %w", c.CurrencySettings[i].Asset, asset.ErrNotSupported)
}
if c.CurrencySettings[i].ExchangeName == "" {
return errUnsetExchange
}
if !c.CurrencySettings[i].MinimumSlippagePercent.IsZero() ||
!c.CurrencySettings[i].MaximumSlippagePercent.IsZero() {
hasSlippage = true
}
if c.CurrencySettings[i].MinimumSlippagePercent.LessThan(decimal.Zero) ||
c.CurrencySettings[i].MaximumSlippagePercent.LessThan(decimal.Zero) ||
c.CurrencySettings[i].MinimumSlippagePercent.GreaterThan(c.CurrencySettings[i].MaximumSlippagePercent) {
return errBadSlippageRates
}
c.CurrencySettings[i].ExchangeName = strings.ToLower(c.CurrencySettings[i].ExchangeName)
}
if hasSlippage && hasFutures {
return fmt.Errorf("%w futures sizing currently incompatible with slippage", errFeatureIncompatible)
}
return nil
}
// PrintSetting prints relevant settings to the console for easy reading
func (c *Config) PrintSetting() {
log.Info(common.Config, common.ColourH1+"------------------Backtester Settings------------------------"+common.ColourDefault)
log.Info(common.Config, common.ColourH2+"------------------Strategy Settings--------------------------"+common.ColourDefault)
log.Infof(common.Config, "Strategy: %s", c.StrategySettings.Name)
if len(c.StrategySettings.CustomSettings) > 0 {
log.Info(common.Config, "Custom strategy variables:")
for k, v := range c.StrategySettings.CustomSettings {
log.Infof(common.Config, "%s: %v", k, v)
}
} else {
log.Info(common.Config, "Custom strategy variables: unset")
}
log.Infof(common.Config, "Simultaneous Signal Processing: %v", c.StrategySettings.SimultaneousSignalProcessing)
log.Infof(common.Config, "USD value tracking: %v", !c.StrategySettings.DisableUSDTracking)
if c.FundingSettings.UseExchangeLevelFunding && c.StrategySettings.SimultaneousSignalProcessing {
log.Info(common.Config, common.ColourH2+"------------------Funding Settings---------------------------"+common.ColourDefault)
log.Infof(common.Config, "Use Exchange Level Funding: %v", c.FundingSettings.UseExchangeLevelFunding)
for i := range c.FundingSettings.ExchangeLevelFunding {
log.Infof(common.Config, "Initial funds for %v %v %v: %v",
c.FundingSettings.ExchangeLevelFunding[i].ExchangeName,
c.FundingSettings.ExchangeLevelFunding[i].Asset,
c.FundingSettings.ExchangeLevelFunding[i].Currency,
c.FundingSettings.ExchangeLevelFunding[i].InitialFunds.Round(8))
}
}
for i := range c.CurrencySettings {
currStr := fmt.Sprintf(common.ColourH2+"------------------%v %v-%v Currency Settings---------------------------------------------------------"+common.ColourDefault,
c.CurrencySettings[i].Asset,
c.CurrencySettings[i].Base,
c.CurrencySettings[i].Quote)
log.Infof(common.Config, currStr[:61])
log.Infof(common.Config, "Exchange: %v", c.CurrencySettings[i].ExchangeName)
if !c.FundingSettings.UseExchangeLevelFunding && c.CurrencySettings[i].SpotDetails != nil {
if c.CurrencySettings[i].SpotDetails.InitialBaseFunds != nil {
log.Infof(common.Config, "Initial base funds: %v %v",
c.CurrencySettings[i].SpotDetails.InitialBaseFunds.Round(8),
c.CurrencySettings[i].Base)
}
if c.CurrencySettings[i].SpotDetails.InitialQuoteFunds != nil {
log.Infof(common.Config, "Initial quote funds: %v %v",
c.CurrencySettings[i].SpotDetails.InitialQuoteFunds.Round(8),
c.CurrencySettings[i].Quote)
}
}
if c.CurrencySettings[i].TakerFee != nil {
if c.CurrencySettings[i].UsingExchangeTakerFee {
log.Infof(common.Config, "Taker fee: Using Exchange's API default taker rate: %v", c.CurrencySettings[i].TakerFee.Round(8))
} else {
log.Infof(common.Config, "Taker fee: %v", c.CurrencySettings[i].TakerFee.Round(8))
}
}
if c.CurrencySettings[i].MakerFee != nil {
if c.CurrencySettings[i].UsingExchangeMakerFee {
log.Infof(common.Config, "Maker fee: Using Exchange's API default maker rate: %v", c.CurrencySettings[i].MakerFee.Round(8))
} else {
log.Infof(common.Config, "Maker fee: %v", c.CurrencySettings[i].MakerFee.Round(8))
}
}
log.Infof(common.Config, "Minimum slippage percent: %v", c.CurrencySettings[i].MinimumSlippagePercent.Round(8))
log.Infof(common.Config, "Maximum slippage percent: %v", c.CurrencySettings[i].MaximumSlippagePercent.Round(8))
log.Infof(common.Config, "Buy rules: %+v", c.CurrencySettings[i].BuySide)
log.Infof(common.Config, "Sell rules: %+v", c.CurrencySettings[i].SellSide)
if c.CurrencySettings[i].FuturesDetails != nil && c.CurrencySettings[i].Asset == asset.Futures {
log.Infof(common.Config, "Leverage rules: %+v", c.CurrencySettings[i].FuturesDetails.Leverage)
}
log.Infof(common.Config, "Can use exchange defined order execution limits: %+v", c.CurrencySettings[i].CanUseExchangeLimits)
}
log.Info(common.Config, common.ColourH2+"------------------Portfolio Settings-------------------------"+common.ColourDefault)
log.Infof(common.Config, "Buy rules: %+v", c.PortfolioSettings.BuySide)
log.Infof(common.Config, "Sell rules: %+v", c.PortfolioSettings.SellSide)
log.Infof(common.Config, "Leverage rules: %+v", c.PortfolioSettings.Leverage)
if c.DataSettings.LiveData != nil {
log.Info(common.Config, common.ColourH2+"------------------Live Settings------------------------------"+common.ColourDefault)
log.Infof(common.Config, "Data type: %v", c.DataSettings.DataType)
log.Infof(common.Config, "Interval: %v", c.DataSettings.Interval)
log.Infof(common.Config, "REAL ORDERS: %v", c.DataSettings.LiveData.RealOrders)
log.Infof(common.Config, "Overriding GCT API settings: %v", c.DataSettings.LiveData.APIClientIDOverride != "")
}
if c.DataSettings.APIData != nil {
log.Info(common.Config, common.ColourH2+"------------------API Settings-------------------------------"+common.ColourDefault)
log.Infof(common.Config, "Data type: %v", c.DataSettings.DataType)
log.Infof(common.Config, "Interval: %v", c.DataSettings.Interval)
log.Infof(common.Config, "Start date: %v", c.DataSettings.APIData.StartDate.Format(gctcommon.SimpleTimeFormat))
log.Infof(common.Config, "End date: %v", c.DataSettings.APIData.EndDate.Format(gctcommon.SimpleTimeFormat))
}
if c.DataSettings.CSVData != nil {
log.Info(common.Config, common.ColourH2+"------------------CSV Settings-------------------------------"+common.ColourDefault)
log.Infof(common.Config, "Data type: %v", c.DataSettings.DataType)
log.Infof(common.Config, "Interval: %v", c.DataSettings.Interval)
log.Infof(common.Config, "CSV file: %v", c.DataSettings.CSVData.FullPath)
}
if c.DataSettings.DatabaseData != nil {
log.Info(common.Config, common.ColourH2+"------------------Database Settings--------------------------"+common.ColourDefault)
log.Infof(common.Config, "Data type: %v", c.DataSettings.DataType)
log.Infof(common.Config, "Interval: %v", c.DataSettings.Interval)
log.Infof(common.Config, "Start date: %v", c.DataSettings.DatabaseData.StartDate.Format(gctcommon.SimpleTimeFormat))
log.Infof(common.Config, "End date: %v", c.DataSettings.DatabaseData.EndDate.Format(gctcommon.SimpleTimeFormat))
}
}