ELEX-2418-find-optimal-lambda

[rishasurana]

Description

We can pass a regularization constant to our model lambda_, but on election night we don’t know what the optimal lambda_ is. Add a feature to the client that take options for lambda_, the features and fixed effects. It should read in the data and use k-fold cross validation to find the optimal lambda as the lowest average MAPE on the k hold out sets. The function should probably return the lambda_ and the average MAPE.

AC:

1. A function that takes a list of possible lambda_ values, the features and fixed effects we are using, any other inputs that are necessary and run k-fold cross validation to pick the best lambda_ value to use
2. Unit tests

Jira Ticket

https://arcpublishing.atlassian.net/browse/ELEX-2418

[rishasurana]

If I run the code with postal code fixed effects I get a nan error:

elexmodel 2020-11-03_USA_G --office_id=P --estimands=dem --geographic_unit_type=county --pi_method=nonparametric --percent_reporting=30 --aggregates=postal_code --aggregates=unit --fixed_effects=postal_code --lambda='[1, 10, 100, 1000]'

When getting unit predictions for each fold nonreporting_units_features = self.featurizer.featurize_heldout_data(nonreporting_units), there is number of NAN values (1-2% of the total df) in nonreporting_units_features post calling featurize_heldout_data on the test set of each fold. I can't seem to replicate this issue for other fixed-effect/aggregate combinations and it works fine when not splitting the data into test/train sets. Each time I run this command, certain states have NAN values and cause errors more often than others (HI, DE, NJ). Which states cause errors varies each time because the fold data is now randomly selected.

Example runs:
Columns that have NAN values, total NAN values in dataframe

1. ['postal_code_HI'], 303
2. ['postal_code_HI', 'postal_code_NJ', 'postal_code_NV'], 927
3. ['postal_code_DE', 'postal_code_NJ', 'postal_code_WY'], 930
4. ['postal_code_HI', 'postal_code_WA'], 602
5. ['postal_code_DE', 'postal_code_MD'], 632
6. ['postal_code_DE'], 281

Using fillna(0) here in the Featurizer class: new_heldout_data = new_heldout_data.join(missing_expanded_fixed_effects_df) resolves this issue because it seems like certain states may be inconsistent in certain folds for the smaller test sets and so when we use .join it fills them with NAN.

Switching to use .merge doesn't work here because there are no common columns to merge on.