Model retrieval

Some ML Model retrieval

• VSM(updated in 2018/11/13)
• SMN(updated in 2018/12/04)
• Bert(updated in 2019/02/02)
• LightGBM(updated in 2019/03/24)
• DMN(updated in 2019/02/02)
• flask(updated in 2019/02/02)
• bert_embedding(updated in 2019/03/24)
• word2vec(updated in 2019/03/24)
• wordNet(updated in 2019/03/24)

VSM

VSM = Vector Space Model

This a hand write VSM retrieval

├── utils
│   └── utils.py         // public function
└── vsm
    ├── pre.sh           // data preprocessing shell
    └── vsm.py           // vsm py

VSM process:

1. word alignment
2. TF - IDF (smooth, similarity)
3. one by one calaulate

• VSM.vsmCalaulate()
  • Consider about bias by smooth
  • Choose one tuple(artile1, artile2) have specific (tf-idf1, tf-idf2)
  • In this way, we have low performance, even we have two class Threadings
• VSM.vsmTest()
  • Ignore bias by smooth
  • Calculate tf-idf in the pre processing which decided by artile instead of tuple(artile1, artile2)
  • In this way, we have fantastic performance
  • We calculate dataset of 3100✖️3100 in 215s

SMN

SMN = Sequential Matching Network

some change from MarkWuNLP/MultiTurnResponseSelection

.
├── NN
│   ├── CNN.py              // CNN function
│   ├── Classifier.py       // classifier function
│   ├── Optimization.py     // NN optimization function
│   ├── RNN.py              // RNN function
│   └── logistic_sgd.py     // sgd function
├── SMN
│   ├── PreProcess.py       // pre deal function
│   ├── SMN_Last.py         // model function
│   ├── SimAsImage.py       // cnn pool & conv
│   └── sampleConduct.py    // got negative and true sample
└── utils
    ├── constant.py         // constant parameter
    └── utils.py            // public function

SMN process:

1. word embemdding
2. GRU
3. CNN
4. GRU
5. score

• SMN.PreProcess.ParseMultiTurn(input_file)
  • prepare deal sample to matrix
• SMN.PreProcess..ParseMultiTurnTest(input_file)
  • prepare deal test sample to matrix
• SMN.sampleConduct.preWord2vec(input_file, out_file)
  • embedding sample
• SMN.sampleConduct.SampleConduct()
  • got negative & true sample
• SMN.SMN_Last.run_model()
  • run SMN model

Bert

LightGBM

DMN

bert_embedding

word2vec

wordNet

flask