Regressor setup with third-party forecasts

[yanirs]

Hi! I have a problem where I have a third-party forecast that I'm trying to improve on. The forecast values are given as a black box – I don't have access to the underlying model. I figured it makes sense to use the third-party forecasts as input to the NeuralProphet model, but I'm not sure what's the best way of doing this.

To make this more concrete, here's a quick example. Let's say I'm dealing with hourly data where the third-party forecast is produced every hour with a three-hour horizon. My dataset may look like this, with fcst_<i>hr_ago representing the third-party forecast for ds, i hours ago:

ds	y	fcst_3hr_ago	fcst_2hr_ago	fcst_1hr_ago
00:00	1.5	1.3	1.6	1.45
01:00	0.5	10.0	5.0	0.4
02:00	1.0	-4.0	3.2	0.9

fcst_<i>hr_ago is pretty much the equivalent of yhat<i> produced by NeuralProphet. As the example shows, the forecast is likely to improve as we get closer to the target ds (i.e., for a smaller i).

Taking the simpler scenario first, let's say I only want to forecast one hour into the future (n_forecasts=1). I think this means I can use all the fcst_<i>hr_ago values without worrying about leakage, but would they be considered future regressors or lagged regressors? I was thinking future because they are for a future time point (the one we're forecasting for), but I may be wrong...

The more complicated scenario is when I want to produce a forecast that's equivalent or longer than the third-party forecast, e.g., with n_forecasts=3. What's the best way of going about it without introducing data leakage? For example, when forecasting two steps into the future, I can't assume I'll have the latest third-party forecast, so it shouldn't be included as a regressor.

Any help would be appreciated! 🙂

[yanirs]

OK, I thought about it more and I think I figured it out. Sticking with the same example and moving things around so that each row only contains what we know at ds, we get:

ds	y	fcst_for_1hr_from_now	fcst_for_2hr_from_now	fcst_for_3hr_from_now
21:00				1.3
22:00			1.6	10.0
23:00		1.45	5.0	-4.0
00:00	1.5	0.4	3.2
01:00	0.5	0.9
02:00	1.0

This table can be used as input to the model, with fcst_for_<i>hr_from_now as lagged regressors.

One thing to note is that performance can be pretty poor when using the default normalisation settings, as the regressors and y may end up on different scales.

Any other thoughts and feedback are still welcome. 🙂

[bhausleitner]

Hi @yanirs,

sorry for the slow response here, we are experiencing a massive rise in users these days and are working through the comments and suggestions of our amazing open source community :)

Thank you providing such a detailed description to your problem, that is super helpful. It seems like that this ended up working for you. We will consider to potentially adjust our input_format of the lagged regressors. Thanks!

[ourownstory]

hi @yanirs
Your thinking seems sound to me - you can use all three as lagged regressors.
The normalization scales should not matter, as each will have it's own regression weight set, too.

Further, you could also use the forecasts as future regressor - if you do a 1step forecast, use the other 1ahead forecast. if you predict 3 steps, the use the other 3ahead forecast. Note, that in the second case, when producing real forecasts with make_future, you could replace the first two regressor values with the 1. and 2. step ahead forecasts. Also note, if you produce more than 3 step forecast with NP, you can't use any for a future regressor.

An idea: if your other provider's forecast is not bad, and you plan to not forecast more than 3 steps, you can predict it's forecast error (making y-yhat your series). This may possibly be able to find more remaining signal. However, you may need to fit one model for each of the forecast steps.

Curious to hear more about your findings.
Happy Forecasting!

[yanirs]

Great, thanks @ourownstory! That's a good point about the normalisation scales not mattering. I didn't do much tuning in my preliminary, which may be why turning off normalisation helped.

Trying to predict the forecast error is a good idea. I'll give it a go. 🙂