backtester: Futures handling & FTX Cash and Carry example strategy (#930)

* implements futures functions and GRPC functions on new branch

* lint and test fixes

* Fix uneven split pnl. Adds collateral weight test. docs. New clear func

* Test protection if someone has zero collateral

* Uses string instead of double for accuracy

* Fixes old code panic

* context, match, docs

* Addresses Shazniterinos, var names, expanded tests

* Returns subaccount name, provides USD values when offlinecalc

* Fixes oopsie

* Fixes cool bug which allowed made up subaccount results

* Subaccount override on FTX, subaccount results for collateral

* Strenghten collateral account info checks. Improve FTX test

* English is my first language

* Fixes oopsies

* Adds some conceptual futures order details to track PNL

* Initial design of future order processing in the backtester

* Introduces futures concept for collateral and spot/futures config diffs

* Fixes most tests

* Simple designs for collateral funding pair concept

* Expands interface use so much it hurts

* Implements more collateral interfaces

* Adds liquidation, adds strategy, struggles with Binance

* Attempts at getting FTX to work

* Adds calculatePNL as a wrapper function and adds an `IsFutures` asset check

* Successfully loads backtester with collateral currency

* Fails to really get much going for supporting futures

* Merges master changes

* Fleshes out how FTX processes collateral

* Further FTX collateral workings

* hooks up more ftx collateral and pnl calculations

* more funcs to flesh out handling

* Adds more links, just can't fit the pieces together :(

* Greatly expands futures order processing

* Fleshes out position tracker to also handle asset and exchange +testing

* RM linkedOrderID. rn positioncontroller, unexport

* Successfully tracks futures order positions

* Fails to calculate PNL

* Calculates pnl from orders accurately with exception to flipping orders

* Calculates PNL from orders

* Adds another controller layer to make it ez from orderstore

* Backtester now compiles. Adds test coverage

* labels things add scaling collateral test

* Calculates pnl in line with fees

* Mostly accurate PNL, with exception to appending with diff prices

* Adds locks, adds rpc function

* grpc implementations

* Gracefully handles rpc function

* beautiful tests!

* rejiggles tests to polish

* Finishes FTX testing, adds comments

* Exposes collateral calculations to rpc

* Adds commands and testing for rpcserver.go functions

* Increase testing and fix up backtester code

* Returns cool changes to original branch

* end of day fixes

* Fixing some tests

* Fixing tests 🎉

* Fixes all the tests

* Splits the backtester setup and running into different files

* Merge, minor fixes

* Messing with some strategy updates

* Failed understanding at collateral usage

* Begins the creation of cash and carry strategy

* Adds underlying pair, adds filldependentevent for futures

* Completes fill prerequsite event implementation. Can't short though

* Some bug fixes

* investigating funds

* CAN NOW CREATE A SHORT ORDER

* Minor change in short size

* Fixes for unrealised PNL & collateral rendering

* Fixes lint and tests

* Adds some verbosity

* Updates to pnl calc

* Tracks pnl for short orders, minor update to strategy

* Close and open event based on conditions

* Adds pnl data for currency statistics

* Working through PNL calculation automatically. Now panics

* Adds tracking, is blocked from design

* Work to flesh out closing a position

* vain attempts at tracking zeroing out bugs

* woww, super fun new subloggers 🎉

* Begins attempt at automatically handling contracts and collateral based on direction

* Merge master + fixes

* Investigating issues with pnl and holdings

* Minor pnl fixes

* Fixes future position sizing, needs contract sizing

* Can render pnl results, focussing on funding statistics

* tracking candles for futures, but why not btc

* Improves funding statistics

* Colours and stats

* Fixes collateral and snapshot bugs

* Completes test

* Fixes totals bug

* Fix double buy, expand stats, fixes usd totals, introduce interface

* Begins report formatting and calculations

* Appends pnl to receiving curr. Fixes map[time]. accurate USD

* Improves report output rendering

* PNL stats in report. New tests for futures

* Fixes existing tests before adding new coverage

* Test coverage

* Completes portfolio coverage

* Increase coverage exchange, portfolio. fix size bug. NEW CHART

* WHAT IS GOING ON WITH PNL

* Fixes PNL calculation. Adds ability to skip om futures tracking

* minor commit before merge

* Adds basic liquidation to backtester

* Changes liquidation to order based

* Liquidationnnnnn

* Further fleshes out liquidations

* Completes liquidations in a honorable manner. Adds AppendReasonf

* Beginnings of spot futures gap chart. Needs to link currencies to render difference

* Removes fake liquidation. Adds cool new chart

* Fixes somet tests,allows for zero fee value v nil distinction,New tests

* Some annoying test fixes that took too long

* portfolio coverage

* holding coverage, privatisation funding

* Testwork

* boring tests

* engine coverage

* More backtesting coverage

* Funding, strategy, report test coverage

* Completes coverage of report package

* Documentation, fixes some assumptions on asset errors

* Changes before master merge

* Lint and Tests

* defaults to non-coloured rendering

* Chart rendering

* Fixes surprise non-local-lints

* Niterinos to the extremeos

* Fixes merge problems

* The linter splintered across the glinting plinths

* Many nits addressed. Now sells spot position on final candle

* Adds forgotten coverage

* Adds ability to size futures contracts to match spot positions.

* fixes order sell sizing

* Adds tests to sizing. Fixes charting issue

* clint splintered the linters with flint

* Improves stats, stat rendering

* minifix

* Fixes tests and fee bug

* Merge fixeroos

* Microfixes

* Updates orderPNL on first Correctly utilises fees. Adds committed funds

* New base funcs. New order summary

* Fun test updates

* Fix logo colouring

* Fixes niteroonies

* Fix report

* BAD COMMIT

* Fixes funding issues.Updates default fee rates.Combines cashcarry case

* doc regen

* Now returns err

* Fixes sizing bug issue introduced in PR

* Fixes fun fee/total US value bug

* Fix chart bug. Show log charts with disclaimer

* sellside fee

* fixes fee and slippage view

* Fixed slippage price issue

* Fixes calculation and removes rendering

* Fixes stats and some rendering

* Merge fix

* Fixes merge issues

* go mod tidy, lint updates

* New linter attempt

* Version bump in appveyor and makefile

* Regex filename, config fixes, template h2 fixes

* Removes bad stats.

* neatens config builder. Moves filename generator

* Fixes issue where linter wants to fix my spelling

* Fixes pointers and starts

cmd/documentation/backtester_templates/backtester_config_examples_readme.tmpl

@@ -16,6 +16,7 @@
 | dca-database-candles.strat | The same DCA strategy, but uses a database to retrieve candle data |
 | rsi-api-candles.strat | Runs a strategy using rsi figures to make buy or sell orders based on market figures |
 | t2b2-api-candles-exchange-funding.strat | Runs a more complex strategy using simultaneous signal processing, exchange level funding and MFI values to make buy or sell signals based on the two strongest and weakest MFI values |
+| ftx-cash-carry.strat | Executes a cash and carry trade on FTX, buying BTC-USD while shorting the long dated futures contract BTC-20210924 |
 
 ### Want to make your own configs?
 Use the provided config builder under `/backtester/config/configbuilder` or modify tests under `/backtester/config/config_test.go` to generates strategy files quickly

cmd/documentation/backtester_templates/backtester_config_readme.tmpl

@@ -26,9 +26,9 @@ See below for a set of tables and fields, expected values and what they can do
 | Goal | A description of what you would hope the outcome to be. When verifying output, you can review and confirm whether the strategy met that goal  |
 | CurrencySettings | Currency settings is an array of settings for each individual currency you wish to run the strategy against |
 | StrategySettings | Select which strategy to run, what custom settings to load and whether the strategy can assess multiple currencies at once to make more in-depth decisions |
+| FundingSettings | Defines whether individual funding settings can be used. Defines the funding exchange, asset, currencies at an individual level |
 | PortfolioSettings | Contains a list of global rules for the portfolio manager. CurrencySettings contain their own rules on things like how big a position is allowable, the portfolio manager rules are the same, but override any individual currency's settings |
 | StatisticSettings | Contains settings that impact statistics calculation. Such as the risk-free rate for the sharpe ratio |
-| GoCryptoTraderConfigPath | The filepath for the location of GoCryptoTrader's config path. The Backtester utilises settings from GoCryptoTrader. If unset, will utilise the default filepath via `config.DefaultFilePath`, implemented [here](/config/config.go#L1460) |
 
 
 #### Strategy Settings
@@ -38,11 +38,18 @@ See below for a set of tables and fields, expected values and what they can do
 | Name | The strategy to use | `rsi` |
 | UsesSimultaneousProcessing | This denotes whether multiple currencies are processed simultaneously with the strategy function `OnSimultaneousSignals`. Eg If you have multiple CurrencySettings and only wish to purchase BTC-USDT when XRP-DOGE is 1337, this setting is useful as you can analyse both signal events to output a purchase call for BTC | `true` |
 | CustomSettings | This is a map where you can enter custom settings for a strategy. The RSI strategy allows for customisation of the upper, lower and length variables to allow you to change them from 70, 30 and 14 respectively to 69, 36, 12 | `"custom-settings": { "rsi-high": 70, "rsi-low": 30, "rsi-period": 14 } ` |
-| UseExchangeLevelFunding | Allows shared funding at an exchange asset level. You can set funding for `USDT` and all pairs that feature `USDT` will have access to those funds when making orders. See [this](/backtester/funding/README.md) for more information | `false` |
-| ExchangeLevelFunding | An array of exchange level funding settings.  See below, or [this](/backtester/funding/README.md) for more information | `[]` |
 | DisableUSDTracking | If `false`, will track all currencies used in your strategy against USD equivalent candles. For example, if you are running a strategy for BTC/XRP, then the GoCryptoTrader Backtester will also retreive candles data for BTC/USD and XRP/USD to then track strategy performance against a single currency. This also tracks against USDT and other USD tracked stablecoins, so one exchange supporting USDT and another BUSD will still allow unified strategy performance analysis. If disabled, will not track against USD, this can be especially helpful when running strategies under live, database and CSV based data  | `false` |
 
-##### Funding Config Settings
+
+#### Funding Config Settings
+
+| Key | Description | Example |
+| --- | ------- | --- |
+| UseExchangeLevelFunding | Allows shared funding at an exchange asset level. You can set funding for `USDT` and all pairs that feature `USDT` will have access to those funds when making orders. See [this](/backtester/funding/README.md) for more information | `false` |
+| ExchangeLevelFunding | An array of exchange level funding settings.  See below, or [this](/backtester/funding/README.md) for more information | `[]` |
+
+
+##### Funding Item Config Settings
 
 | Key | Description | Example |
 | --- | ------- | ----- |
@@ -62,18 +69,30 @@ See below for a set of tables and fields, expected values and what they can do
 | Base | The base of a currency | `BTC` |
 | Quote | The quote of a currency | `USDT` |
 | InitialFunds | A legacy field, will be temporarily migrated to `InitialQuoteFunds` if present in your strat config | `` |
-| InitialBaseFunds | The funds that the GoCryptoTraderBacktester has for the base currency. This is only required if the strategy setting `UseExchangeLevelFunding` is `false` | `2` |
-| InitialQuoteFunds | The funds that the GoCryptoTraderBacktester has for the quote currency. This is only required if the strategy setting `UseExchangeLevelFunding` is `false` | `10000` |
-| Leverage | This struct defines the leverage rules that this specific currency setting must abide by | `1` |
 | BuySide | This struct defines the buying side rules this specific currency setting must abide by such as maximum purchase amount | - |
 | SellSide | This struct defines the selling side rules this specific currency setting must abide by such as maximum selling amount | - |
 | MinimumSlippagePercent | Is the lower bounds in a random number generated that make purchases more expensive, or sell events less valuable. If this value is 90, then the most a price can be affected is 10% | `90` |
 | MaximumSlippagePercent | Is the upper bounds in a random number generated that make purchases more expensive, or sell events less valuable. If this value is 99, then the least a price can be affected is 1%. Set both upper and lower to 100 to have no randomness applied to purchase events | `100` |
-| MakerFee | The fee to use when sizing and purchasing currency | `0.001` |
-| TakerFee | Unused fee for when an order is placed in the orderbook, rather than taken from the orderbook | `0.002` |
+| MakerFee | The fee to use when sizing and purchasing currency. If `nil`, will lookup an exchange's fee details | `0.001` |
+| TakerFee | Unused fee for when an order is placed in the orderbook, rather than taken from the orderbook. If `nil`, will lookup an exchange's fee details | `0.002` |
 | MaximumHoldingsRatio | When multiple currency settings are used, you may set a maximum holdings ratio to prevent having too large a stake in a single currency | `0.5` |
 | CanUseExchangeLimits | Will lookup exchange rules around purchase sizing eg minimum order increments of 0.0005. Note: Will retrieve up-to-date rules which may not have existed for the data you are using. Best to use this when considering to use this strategy live | `false` |
 | SkipCandleVolumeFitting | When placing orders, by default the BackTester will shrink an order's size to fit the candle data's volume so as to not rewrite history. Set this to `true` to ignore this and to set order size at what the portfolio manager prescribes | `false` |
+| SpotSettings | An optional field which contains initial funding data for SPOT currency pairs | See SpotSettings table below |
+| FuturesSettings | An optional field which contains leverage data for FUTURES currency pairs | See FuturesSettings table below |
+
+##### SpotSettings
+
+| Key | Description | Example |
+| --- | ------- | ----- |
+| InitialBaseFunds | The funds that the GoCryptoTraderBacktester has for the base currency. This is only required if the strategy setting `UseExchangeLevelFunding` is `false` | `2` |
+| InitialQuoteFunds | The funds that the GoCryptoTraderBacktester has for the quote currency. This is only required if the strategy setting `UseExchangeLevelFunding` is `false` | `10000` |
+
+##### FuturesSettings
+
+| Key | Description | Example |
+| --- | ------- | ----- |
+| Leverage | This struct defines the leverage rules that this specific currency setting must abide by | `1` |
 
 #### PortfolioSettings
 

cmd/documentation/backtester_templates/backtester_engine_readme.tmpl

@@ -1,8 +1,8 @@
-{{define "backtester backtest" -}}
+{{define "backtester engine" -}}
 {{template "backtester-header" .}}
 ## {{.CapitalName}} package overview
 
-The backtest package is the most important package of the GoCryptoTrader backtester. It is the engine which combines all elements.
+The engine package is the most important package of the GoCryptoTrader backtester. It is the engine which combines all elements.
 It is responsible for the following functionality
 - Loading settings from a provided config file
 - Retrieving data

cmd/documentation/backtester_templates/backtester_eventhandlers_strategies_ftxcashandcarry_readme.tmpl

@@ -0,0 +1,34 @@
+{{define "backtester eventhandlers strategies ftxcashandcarry" -}}
+{{template "backtester-header" .}}
+## FTX Cash and carry strategy overview
+
+### Description
+Cash and carry is a strategy which takes advantage of the difference in pricing between a long-dated futures contract and a SPOT asset.
+By default, this cash and carry strategy will, upon the first data event, purchase BTC-USD SPOT asset from FTX exchange and then, once filled, raise a SHORT for BTC-20210924 FUTURES contract.
+On the last event, the strategy will close the SHORT position by raising a LONG of the same contract amount, thereby netting the difference in prices
+
+### Requirements
+- At this time of writing, this strategy is only compatible with FTX
+- This strategy *requires* `Simultaneous Signal Processing` aka [use-simultaneous-signal-processing](/backtester/config/README.md).
+- This strategy *requires* `Exchange Level Funding` aka [use-exchange-level-funding](/backtester/config/README.md).
+
+### Creating a strategy config
+- The long-dated futures contract will need to be part of the `currency-settings` of the contract
+- Funding for purchasing SPOT assets will need to be part of `funding-settings`
+- See the [example config](./config/examples/ftx-cash-carry.strat)
+
+### Customisation
+This strategy does support strategy customisation in the following ways:
+
+| Field | Description |  Example |
+| --- | ------- | --- |
+| openShortDistancePercentage | If there is no short position open, and the difference between FUTURES and SPOT pricing goes above this this percentage threshold, raise a SHORT order of the FUTURES contract | 10 |
+| closeShortDistancePercentage | If there is an open SHORT position on a FUTURES contract, and the difference in FUTURES and SPOT pricing goes below this percentage threshold, close the SHORT position | 1 |
+
+### External Resources
+- [This](https://ftxcashandcarry.com/) is a very informative site on describing what a cash and carry trade will look like
+
+### Please click GoDocs chevron above to view current GoDoc information for this package
+{{template "contributions"}}
+{{template "donations" .}}
+{{end}}

cmd/documentation/backtester_templates/backtester_funding_readme.tmpl

@@ -18,6 +18,9 @@ A funding item holds the initial funding, current funding, reserved funding and
 ### What is a funding Pair?
 A funding Pair consists of two funding Items, the Base and Quote. If Exchange Level Funding is disabled, the Base and Quote are linked to each other and the funds cannot be shared with other Pairs or Items. If Exchange Level Funding is enabled, the pair can access the same funds as every other currency that shares the exchange and asset type.
 
+### What is a collateral Pair?
+A collateral Pair consists of two funding Items, the Contract and Collateral. These are exclusive to FUTURES asset type and help track how much money there is, along with how many contract holdings there are
+
 ### What does Exchange Level Funding mean?
 Exchange level funding allows funds to be shared during a backtesting run. If the strategy contains the two pairs BTC-USDT and BNB-USDT and the strategy sells 3 BTC for $100,000 USDT, then BNB-USDT can use that $100,000 USDT to make a purchase of $20,000 BNB.
 It is restricted to an exchange and asset type, so BTC used in spot, cannot be used in a futures contract (futures backtesting is not currently supported). However, the funding manager can transfer funds between exchange and asset types.
@@ -49,6 +52,11 @@ No. The already existing `CurrencySettings` will populate the funding manager wi
 | Name | The strategy to use | `rsi` |
 | UsesSimultaneousProcessing | This denotes whether multiple currencies are processed simultaneously with the strategy function `OnSimultaneousSignals`. Eg If you have multiple CurrencySettings and only wish to purchase BTC-USDT when XRP-DOGE is 1337, this setting is useful as you can analyse both signal events to output a purchase call for BTC | `true` |
 | CustomSettings | This is a map where you can enter custom settings for a strategy. The RSI strategy allows for customisation of the upper, lower and length variables to allow you to change them from 70, 30 and 14 respectively to 69, 36, 12 | `"custom-settings": { "rsi-high": 70, "rsi-low": 30, "rsi-period": 14 } ` |
+
+#### Funding Settings
+
+| Key | Description | Example |
+| --- | ------- | --- |
 | UseExchangeLevelFunding | This allows shared exchange funds to be used in your strategy. Requires `UsesSimultaneousProcessing` to be set to `true` to use  | `false` |
 | ExchangeLevelFunding | This is a list of funding definitions if `UseExchangeLevelFunding` is set to true  | See below table |
 

cmd/documentation/backtester_templates/backtester_readme.tmpl

@@ -25,14 +25,17 @@ An event-driven backtesting tool to test and iterate trading strategies using hi
 - Compliance manager to keep snapshots of every transaction and their changes at every interval
 - Exchange level funding allows funding to be shared across multiple currency pairs and to allow for complex strategy design
 - Fund transfer. At a strategy level, transfer funds between exchanges to allow for complex strategy design
+- Backtesting support for futures asset types
+- Example cash and carry spot futures strategy
 
 ## Planned Features
 We welcome pull requests on any feature for the Backtester! We will be especially appreciative of any contribution towards the following planned features:
 
 | Feature | Description |
 |---------|-------------|
-| Add backtesting support for futures asset types | Spot trading is currently the only supported asset type. Futures trading greatly expands the Backtester's potential |
-| Example futures pairs trading strategy | Providing a basic example will allow for esteemed traders to build and customise their own |
+| Long-running application | Transform the Backtester to run a GRPC server, where commands can be sent to run Backtesting operations. Allowing for many strategies to be run, analysed and tweaked in a more efficient manner |
+| Leverage support | Leverage is a good way to enhance profit and loss and is important to include in strategies |
+| Enhance config-builder | Create an application that can create strategy configs in a more visual manner and execute them via GRPC to allow for faster customisation of strategies |
 | Save Backtester results to database | This will allow for easier comparison of results over time |
 | Backtester result comparison report | Providing an executive summary of Backtester database results |
 | Currency correlation | Compare multiple exchange, asset, currencies for a candle interval against indicators to highlight correlated pairs for use in pairs trading |