Adding additional indicator changes outcome of Strategy (without being used in next())

[mseidlx]

First of all, awesome job with this library! I really like backtesting.py and it's great to see that you added multi-asset strategies.

I'm trying to create a strategy that relies on a couple of data points. I'm still trying to wrap my head around how multiple conditions can be applied and while doing some tests I noticed that just adding an additional indicator, even without referencing it in the next() method, can change the strategy outcome.

here is an example:
`class FJScore(Strategy):
df = get_score_df()
start_date = '2022-01-01'
end_date = '2023-09-30'

def init(self):
    self.metric1= self.I(prep_df, self.df, 'Metric1', self.start_date, self.end_date, name='Metric1')
    self.metric2 = self.I(prep_df, self.df, 'Metric2', self.start_date, self.end_date, name='Metric2')
    #self.metric3 = self.I(prep_df, self.df, 'Metric3', self.start_date, self.end_date, name='Metric3')

def next(self):
    metric1= self.metric1.df.iloc[-1]
    #metric2= self.metric2.df.iloc[-1]
    #metric3= self.metric3.df.iloc[-1]

    self.alloc.add(
        metric1 == 1
        #,metric2 >= 7
        #,metric2 > 1.1
    ).equal_weight()

    self.rebalance()`

even if I keep the line "metric2 >= 7" commented out but add the indicator for Metric2 (in the init) it will change my final results. I don't really understand why though?

[dodid]

I suspect the results returned from two prep_df() calls somehow share the same underlying memory and get messed up when indicators are prepared for next(). To help debugging the issue, can you please try adding .copy() in prep_df()'s return and see if the issue still shows up? Another experiment is to swap the order of the two lines and see if it makes any difference:

self.metric1= self.I(prep_df, self.df, 'Metric1', self.start_date, self.end_date, name='Metric1')
self.metric2 = self.I(prep_df, self.df, 'Metric2', self.start_date, self.end_date, name='Metric2')

[mseidlx]

Thank you for getting back so quickly!

I tried the .copy() idea but same output (indicators are also correctly plotted).
I also tried to swap the order of the indicators and it did make a difference - the output changed. Why is that the case?

[dodid]

Likely to do with how we identify a variable as indicator. Indicators are added to a list and a class variable is an indicator if it's in the list. I can't easily replicate the issue though. Will you be able to provide a minimal runnable example?

[mseidlx]

sure, here is an example. Just uncommenting one of the lines that adds a new indicator will change the backtesting output:

import requests
import pandas as pd
import numpy as np
from minitrade.backtest import Strategy
from minitrade.backtest import Backtest


def get_binance_ohlc(symbol:str):
    # Set API endpoint and parameters
    endpoint = 'https://api.binance.com/api/v3/klines'
    params = {
        'symbol': symbol.upper()+'USDT',
        'interval': '1d',
        'limit': 1000
    }

    # Make GET request to API
    response = requests.get(endpoint, params=params)

    # Load data into pandas DataFrame
    data = response.json()
    df = pd.DataFrame(data, columns=['time', 'open', 'high', 'low', 'close', 'volume', 'close_time', 'quote_asset_volume', 'num_trades', 'taker_buy_base_asset_volume', 'taker_buy_quote_asset_volume', 'ignore'])
    df['Date'] = pd.to_datetime(df['time'], unit='ms')
    df.set_index('Date', inplace=True)
    df = df[['open', 'high', 'low', 'close', 'volume']]
    df.rename(columns={'open': 'Open', 'high': 'High', 'low': 'Low', 'close': 'Close', 'volume': 'Volume'}, inplace=True)
    df = df.astype(float)
    return df

def get_binanace_multi_ohlc(symbols:list):
    ## call get_binance_ohlc for each symbol in symbols
    ## return a multi-index dataframe
    df = pd.DataFrame()
    for symbol in symbols:
        df1 = get_binance_ohlc(symbol)
        df1.columns = pd.MultiIndex.from_product([[symbol], df1.columns])
        df = pd.concat([df, df1], axis=1)
    return df

def create_dummy_metrics():
    start_date = '2022-01-01'
    end_date = '2023-09-30'
    np.random.seed(123)

    df_multi = pd.DataFrame()
    for symbol in ['SOL', 'AVAX', 'BNB']:
        dates = pd.date_range(start_date, end_date)
        df = pd.DataFrame(index=dates)
        df['Metric1'] = np.random.randint(1, 100, size=len(dates))
        df['Metric1'] = df['Metric1'].rolling(10).mean()
        df['Metric2'] = np.random.randint(1, 100, size=len(dates))
        df['Metric2'] = df['Metric2'].rolling(20).mean()
        df['Metric3'] = np.random.randint(1, 100, size=len(dates))
        df['Metric3'] = df['Metric3'].rolling(30).mean()
        df.columns = pd.MultiIndex.from_product([[symbol], df.columns])
        df_multi = pd.concat([df_multi, df], axis=1)
    return df_multi

def prep_df(df, column:str, start_date:str, end_date:str):
    df = df.loc[:, df.columns.get_level_values(1) == column]
    df = df.loc[start_date:end_date]
    ## delete the 2nd level column header
    df.columns = df.columns.droplevel(1)
    return df.copy()

class TestStrategy(Strategy):
    df = create_dummy_metrics()
    start_date = '2022-01-01'
    end_date = '2023-09-30'

    def init(self):
        self.metric1 = self.I(prep_df, self.df, 'Metric1', self.start_date, self.end_date, name='Metric 1')
        #self.metric2 = self.I(prep_df, self.df, 'Metric2', self.start_date, self.end_date, name='Metric 2')
        #self.metric3 = self.I(prep_df, self.df, 'Metric3', self.start_date, self.end_date, name='Metric 3')

    def next(self):
        metric1= self.metric1.df.iloc[-1]

        self.alloc.add(
            metric1 > 50
        ).equal_weight()

        self.rebalance()


asset_ohlcvs = get_binanace_multi_ohlc(['SOL', 'BNB', 'AVAX'])
asset_ohlcvs = asset_ohlcvs[(asset_ohlcvs.index >= '2022-01-01') & (asset_ohlcvs.index <= '2023-09-30')]

bt = Backtest(asset_ohlcvs, TestStrategy, cash=100000, commission=0.002)
stats = bt.run()
print(stats)

[dodid]

dce738f should fix the issue. Thanks for reporting!

Please note backtest starts from when all indicators have non-NA values. Adding metric2, even if not used, can still change the outcome since it has a different warmup period, which effectively changes when backtest starts. You can capture and inspect metric1 in next() by:

self.record(metric1_seen=metric1)

[mseidlx]

Thank you for looking into it and also thank you for the tip around the non-NA values @dodid!
With you latest commit everything works smoothly now