Hidden side-effects of make_future_dataframe function

[sivikt]

Hi All!

Great library - loving it so far.

I observe a strange thing. Starting from the 0.4.2 the function mode.make_future_dataframe(...) silently changes the model state: removing predefined regressors.

Consider the code. It creates a model with 1 future regressor r_lagged and then trying to make a future using make_future_dataframe function. Result contains columns r_lagged but with NaNs for the last 5 past data points which is strange since the whole past is provided.

## pip install neuralprophet==0.4.2

import numpy as np
import pandas as pd
from neuralprophet import NeuralProphet
from neuralprophet import set_random_seed

set_random_seed(0)

data_location = "https://raw.githubusercontent.com/ourownstory/neuralprophet-data/main/datasets/"

df = pd.read_csv(data_location + 'wp_log_peyton_manning.csv')
df['r_lagged'] = np.random.choice(5, len(df))
df.loc[df[-20:].index, 'r_lagged'] = 5

m = NeuralProphet(n_forecasts=10, n_lags=5, epochs=2)
m = m.add_future_regressor(name='r_lagged')
metrics = m.fit(df, freq='D')

last_date = pd.to_datetime(df["ds"].copy(deep=True).dropna()).sort_values().max()
future_dates = pd.date_range(start=last_date, periods=12, freq='D')
future_dates = future_dates[future_dates > last_date]
future_df = pd.DataFrame({"ds": future_dates})
future_df['r_lagged'] = np.random.choice(5, len(future_df))
future = m.make_future_dataframe(df=df, periods=10, regressors_df=future_df, n_historic_predictions=False)
print(future_df)
print(future)

idx	ds	y	r_lagged
0	2016-01-16	7.8172	NaN
1	2016-01-17	9.2739	NaN
2	2016-01-18	10.3338	NaN
3	2016-01-19	9.1259	NaN
4	2016-01-20	8.8914	NaN
5	2016-01-21	NaN	0.0
6	2016-01-22	NaN	0.0
7	2016-01-23	NaN	1.0
8	2016-01-24	NaN	0.0
9	2016-01-25	NaN	4.0
10	2016-01-26	NaN	3.0
11	2016-01-27	NaN	3.0
12	2016-01-28	NaN	4.0
13	2016-01-29	NaN	2.0
14	2016-01-30	NaN	3.0

It is because of check_single_dataframe gets the last past historic data points where r_lagged has the only value 5.

check_single_dataframe, df_utils.py:455
check_dataframe, df_utils.py:519
_check_dataframe, forecaster.py:2155
_make_future_dataframe, forecaster.py:2765
make_future_dataframe, forecaster.py:1246

I do not quite understand why it so having trained model with multiple past regressor unique. Why simple utility function changes the model state, it was not the case in the previous versions. One can see that check_single_dataframe marks a regressor as "to be removed" if the regressor has only 1 unique value but it considers only N past points (N lags) putting aside the future_df.

What do you think? Should we consider opening an issue/PR to adjust the behavior or there some hidden algorithm intrinsics which require such model state manipulation?

[judussoari]

Hey @sivikt
thanks for raising this! It looks like you are absolutely right - the future regressors, because they have the same value (5) when passed to check_single_dataframe. you can set 'n_historic_predictions=True' which gives a workaround problem, because with all the historic predictions used in the future df, the future regressors does no longer have only unique values (5).
I will create an Issue to address this, thanks!
Best
Julio