Use of hyperthreading in hyperparameter sweeps

[pomorigio]

Hello back,

I aimed to run your code using Python hyperthreading module (I don't know if you are familiar with it) to speed up hyperparameter sweeps, as you may find in the file attached (it is in a .txt extension as I could not directly update a .py file).

However, most cases show how results hardly improve and strongly fluctuate even when using very low learning rates, opposite to when you do so sequentially without hyperthreading (where both loss and val_loss keep gradually decreasing).

Do you know if this is just an incompatibility issue or am I doing something wrong? How do you actually deal with hyperparameter sweeps? Are you using any scikit_tools or you do it manually such as I?

Thank you in advance, and best regards! May you have a nice weekend!

# x_data, y_data generated from a .csv file
SAMPLE = x_data.shape[0]
N_OUTPUTS = x_data.shape[1]
N_INPUTS = y_data.shape[1]

N_EPOCHS = [6000]
N_LAYERS = [1]
N_HIDDEN = [100]
N_MIXES = [8, 12]
DROPOUT = [0]
ACT_FUNCTION = 'tanh'
LR = [0.00005, 0.00001]
BATCH_SIZE = [NSAMPLE]
PTEST = [0.3]
beta1 = [0.9]
beta2 = [0.999]

def MDN(N_MIXES, LR, BATCH_SIZE, N_LAYERS, N_HIDDEN, DROPOUT, PTEST, N_EPOCHS, beta1, beta2):
	model = keras.Sequential()
	model.add(Dense(N_HIDDEN, batch_input_shape = (None, N_INPUTS), activation = ACT_FUNCTION))
	model.add(Dropout(DROPOUT))
	for layer in range(N_LAYERS - 1):
		model.add(Dense(N_HIDDEN, activation = ACT_FUNCTION))
		model.add(Dropout(DROPOUT))
	model.add(mdn.MDN(N_OUTPUTS, N_MIXES))
	return model	
	adam = keras.optimizers.Adam(lr=LR, beta_1 = beta1, beta_2 = beta2)
	model.compile(loss=mdn.get_mixture_loss_func(N_OUTPUTS,N_MIXES), optimizer=adam)	
	
	H = model.fit(x=x_data, y=y_data, verbose=0, batch_size=BATCH_SIZE, epochs=N_EPOCHS, validation_split=PTEST)
		
	return N_MIXES, LR, BATCH_SIZE, N_LAYERS, N_HIDDEN, DROPOUT, beta1, beta2, H.history['loss'], H.history['val_loss']
	
params = list(itertools.product(*[N_MIXES, LR, BATCH_SIZE, N_LAYERS, N_HIDDEN, DROPOUT, PTEST, N_EPOCHS, beta1, beta2]))

pool = ThreadPool()
results = pool.starmap(MDN, params)
pool.close()
pool.join()

[cpmpercussion]

Hi @MisterTellini - I'm not sure what's going wrong in that instance, but I would suggest using of the tools in sklearn.model_selection, for instance, GridSearchCV seems to take care of threading for you.

Maybe you could follow the instructions in https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/ and see how that goes?