Skip to content

SMOTE-BD: A distributed Synthetic Minority Oversampling Technique (SMOTE) for Big Data.

License

Notifications You must be signed in to change notification settings

majobasgall/smote-bd

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 

Repository files navigation

SMOTE-BD

It is a fully scalable preprocessing approach for imbalanced classification in Big Data. It is based on one of the most widespread preprocessing solutions for imbalanced classification, namely the SMOTE [1] algorithm, which creates new synthetic instances according to the neighborhood of each example of the minority class.

How to run it?

A generic example to run it could be:

spark-submit --master "URL" --executor-memory "XG" "path-to-jar".jar --class "path-to-main" --datasetName="aName" --headerFile="path-to-header" --inputFile="path-to-input" --delimiter=", " --outputPah="path-to-output" --seed="aSeed" --K="number-of-neighbours" --numPartitions="number-of-parts" --nReducers="number-of-reducers" --numIterations="number-of-iterations" --minClassName="min-class-name" -overPercentage=100

  • Parameters of spark: --master "URL" | --executor-memory "XG" . They can be usefull for launch with diferent settings and datasets.
  • --class path.to.the.main aJarFile.jar Determine the jar file to be run.
  • datasetName The name of the current dataset.
  • headerFile Full path to header file.
  • inputFile Full path to input file.
  • delimiter Delimiter of each attribute value.
  • outputPah Full path to output directory.
  • seed A seed to generate random numbers.
  • K Number of nearest neighbours.
  • numPartitions Number of partitions to split data.
  • nReducers Number of reducers (required by the K-NN stage).
  • numIterations Number of iterations (required by the K-NN stage).
  • minClassName Name of the minority class (according to the header file).
  • overPercentage Percentage of balancing between classes.

Please, cite this software as:

Basgall, M. J., Hasperué, W., Naiouf, M., Fernández, A., & Herrera, F. (2018). SMOTE-BD: An Exact and Scalable Oversampling Method for Imbalanced Classification in Big Data. Journal of Computer Science and Technology, 18(03), e23. https://doi.org/10.24215/16666038.18.e23

Bibtex:

@article{Basgall_Hasperué_Naiouf_Fernández_Herrera_2018,
title={SMOTE-BD: An Exact and Scalable Oversampling Method for Imbalanced Classification in Big Data},
volume={18},
url={http://journal.info.unlp.edu.ar/JCST/article/view/1122},
DOI={10.24215/16666038.18.e23},
number={03},
journal={Journal of Computer Science and Technology},
author={Basgall, María José and Hasperué, Waldo and Naiouf, Marcelo and Fernández, Alberto and Herrera, Francisco},
year={2018},
month={Dec.},
pages={e23}
}

References

[1] Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. J. Artif. Int. Res., 16(1), 321–357.

About

SMOTE-BD: A distributed Synthetic Minority Oversampling Technique (SMOTE) for Big Data.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages