-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
305 lines (296 loc) · 245 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
<!doctype html>
<html>
<head>
<meta charset='UTF-8'><meta name='viewport' content='width=device-width initial-scale=1'>
<title>Twitter Tweet Classification Using BERT</title><link href='https://fonts.loli.net/css?family=Open+Sans:400italic,700italic,700,400&subset=latin,latin-ext' rel='stylesheet' type='text/css' /><style type='text/css'>html {overflow-x: initial !important;}:root { --bg-color:#ffffff; --text-color:#333333; --select-text-bg-color:#B5D6FC; --select-text-font-color:auto; --monospace:"Lucida Console",Consolas,"Courier",monospace; }
html { font-size: 14px; background-color: var(--bg-color); color: var(--text-color); font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; -webkit-font-smoothing: antialiased; }
body { margin: 0px; padding: 0px; height: auto; bottom: 0px; top: 0px; left: 0px; right: 0px; font-size: 1rem; line-height: 1.42857; overflow-x: hidden; background: inherit; }
iframe { margin: auto; }
a.url { word-break: break-all; }
a:active, a:hover { outline: 0px; }
.in-text-selection, ::selection { text-shadow: none; background: var(--select-text-bg-color); color: var(--select-text-font-color); }
#write { margin: 0px auto; height: auto; width: inherit; word-break: normal; word-wrap: break-word; position: relative; white-space: normal; overflow-x: visible; padding-top: 40px; }
#write.first-line-indent p { text-indent: 2em; }
#write.first-line-indent li p, #write.first-line-indent p * { text-indent: 0px; }
#write.first-line-indent li { margin-left: 2em; }
.for-image #write { padding-left: 8px; padding-right: 8px; }
body.typora-export { padding-left: 30px; padding-right: 30px; }
.typora-export .footnote-line, .typora-export li, .typora-export p { white-space: pre-wrap; }
@media screen and (max-width: 500px) {
body.typora-export { padding-left: 0px; padding-right: 0px; }
#write { padding-left: 20px; padding-right: 20px; }
.CodeMirror-sizer { margin-left: 0px !important; }
.CodeMirror-gutters { display: none !important; }
}
#write li > figure:first-child { margin-top: -20px; }
#write ol, #write ul { position: relative; }
img { max-width: 100%; vertical-align: middle; }
button, input, select, textarea { color: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; }
input[type="checkbox"], input[type="radio"] { line-height: normal; padding: 0px; }
*, ::after, ::before { box-sizing: border-box; }
#write h1, #write h2, #write h3, #write h4, #write h5, #write h6, #write p, #write pre { width: inherit; }
#write h1, #write h2, #write h3, #write h4, #write h5, #write h6, #write p { position: relative; }
h1, h2, h3, h4, h5, h6 { break-after: avoid-page; break-inside: avoid; orphans: 2; }
p { orphans: 4; }
h1 { font-size: 2rem; }
h2 { font-size: 1.8rem; }
h3 { font-size: 1.6rem; }
h4 { font-size: 1.4rem; }
h5 { font-size: 1.2rem; }
h6 { font-size: 1rem; }
.md-math-block, .md-rawblock, h1, h2, h3, h4, h5, h6, p { margin-top: 1rem; margin-bottom: 1rem; }
.hidden { display: none; }
.md-blockmeta { color: rgb(204, 204, 204); font-weight: 700; font-style: italic; }
a { cursor: pointer; }
sup.md-footnote { padding: 2px 4px; background-color: rgba(238, 238, 238, 0.7); color: rgb(85, 85, 85); border-radius: 4px; cursor: pointer; }
sup.md-footnote a, sup.md-footnote a:hover { color: inherit; text-transform: inherit; text-decoration: inherit; }
#write input[type="checkbox"] { cursor: pointer; width: inherit; height: inherit; }
figure { overflow-x: auto; margin: 1.2em 0px; max-width: calc(100% + 16px); padding: 0px; }
figure > table { margin: 0px !important; }
tr { break-inside: avoid; break-after: auto; }
thead { display: table-header-group; }
table { border-collapse: collapse; border-spacing: 0px; width: 100%; overflow: auto; break-inside: auto; text-align: left; }
table.md-table td { min-width: 32px; }
.CodeMirror-gutters { border-right: 0px; background-color: inherit; }
.CodeMirror-linenumber { user-select: none; }
.CodeMirror { text-align: left; }
.CodeMirror-placeholder { opacity: 0.3; }
.CodeMirror pre { padding: 0px 4px; }
.CodeMirror-lines { padding: 0px; }
div.hr:focus { cursor: none; }
#write pre { white-space: pre-wrap; }
#write.fences-no-line-wrapping pre { white-space: pre; }
#write pre.ty-contain-cm { white-space: normal; }
.CodeMirror-gutters { margin-right: 4px; }
.md-fences { font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; overflow: visible; white-space: pre; background: inherit; position: relative !important; }
.md-diagram-panel { width: 100%; margin-top: 10px; text-align: center; padding-top: 0px; padding-bottom: 8px; overflow-x: auto; }
#write .md-fences.mock-cm { white-space: pre-wrap; }
.md-fences.md-fences-with-lineno { padding-left: 0px; }
#write.fences-no-line-wrapping .md-fences.mock-cm { white-space: pre; overflow-x: auto; }
.md-fences.mock-cm.md-fences-with-lineno { padding-left: 8px; }
.CodeMirror-line, twitterwidget { break-inside: avoid; }
.footnotes { opacity: 0.8; font-size: 0.9rem; margin-top: 1em; margin-bottom: 1em; }
.footnotes + .footnotes { margin-top: 0px; }
.md-reset { margin: 0px; padding: 0px; border: 0px; outline: 0px; vertical-align: top; background: 0px 0px; text-decoration: none; text-shadow: none; float: none; position: static; width: auto; height: auto; white-space: nowrap; cursor: inherit; -webkit-tap-highlight-color: transparent; line-height: normal; font-weight: 400; text-align: left; box-sizing: content-box; direction: ltr; }
li div { padding-top: 0px; }
blockquote { margin: 1rem 0px; }
li .mathjax-block, li p { margin: 0.5rem 0px; }
li { margin: 0px; position: relative; }
blockquote > :last-child { margin-bottom: 0px; }
blockquote > :first-child, li > :first-child { margin-top: 0px; }
.footnotes-area { color: rgb(136, 136, 136); margin-top: 0.714rem; padding-bottom: 0.143rem; white-space: normal; }
#write .footnote-line { white-space: pre-wrap; }
@media print {
body, html { border: 1px solid transparent; height: 99%; break-after: avoid; break-before: avoid; }
#write { margin-top: 0px; padding-top: 0px; border-color: transparent !important; }
.typora-export * { -webkit-print-color-adjust: exact; }
html.blink-to-pdf { font-size: 13px; }
.typora-export #write { padding-left: 32px; padding-right: 32px; padding-bottom: 0px; break-after: avoid; }
.typora-export #write::after { height: 0px; }
@page { margin: 20mm 0px; }
}
.footnote-line { margin-top: 0.714em; font-size: 0.7em; }
a img, img a { cursor: pointer; }
pre.md-meta-block { font-size: 0.8rem; min-height: 0.8rem; white-space: pre-wrap; background: rgb(204, 204, 204); display: block; overflow-x: hidden; }
p > .md-image:only-child:not(.md-img-error) img, p > img:only-child { display: block; margin: auto; }
p > .md-image:only-child { display: inline-block; width: 100%; }
#write .MathJax_Display { margin: 0.8em 0px 0px; }
.md-math-block { width: 100%; }
.md-math-block:not(:empty)::after { display: none; }
[contenteditable="true"]:active, [contenteditable="true"]:focus { outline: 0px; box-shadow: none; }
.md-task-list-item { position: relative; list-style-type: none; }
.task-list-item.md-task-list-item { padding-left: 0px; }
.md-task-list-item > input { position: absolute; top: 0px; left: 0px; margin-left: -1.2em; margin-top: calc(1em - 10px); border: none; }
.math { font-size: 1rem; }
.md-toc { min-height: 3.58rem; position: relative; font-size: 0.9rem; border-radius: 10px; }
.md-toc-content { position: relative; margin-left: 0px; }
.md-toc-content::after, .md-toc::after { display: none; }
.md-toc-item { display: block; color: rgb(65, 131, 196); }
.md-toc-item a { text-decoration: none; }
.md-toc-inner:hover { text-decoration: underline; }
.md-toc-inner { display: inline-block; cursor: pointer; }
.md-toc-h1 .md-toc-inner { margin-left: 0px; font-weight: 700; }
.md-toc-h2 .md-toc-inner { margin-left: 2em; }
.md-toc-h3 .md-toc-inner { margin-left: 4em; }
.md-toc-h4 .md-toc-inner { margin-left: 6em; }
.md-toc-h5 .md-toc-inner { margin-left: 8em; }
.md-toc-h6 .md-toc-inner { margin-left: 10em; }
@media screen and (max-width: 48em) {
.md-toc-h3 .md-toc-inner { margin-left: 3.5em; }
.md-toc-h4 .md-toc-inner { margin-left: 5em; }
.md-toc-h5 .md-toc-inner { margin-left: 6.5em; }
.md-toc-h6 .md-toc-inner { margin-left: 8em; }
}
a.md-toc-inner { font-size: inherit; font-style: inherit; font-weight: inherit; line-height: inherit; }
.footnote-line a:not(.reversefootnote) { color: inherit; }
.md-attr { display: none; }
.md-fn-count::after { content: "."; }
code, pre, samp, tt { font-family: var(--monospace); }
kbd { margin: 0px 0.1em; padding: 0.1em 0.6em; font-size: 0.8em; color: rgb(36, 39, 41); background: rgb(255, 255, 255); border: 1px solid rgb(173, 179, 185); border-radius: 3px; box-shadow: rgba(12, 13, 14, 0.2) 0px 1px 0px, rgb(255, 255, 255) 0px 0px 0px 2px inset; white-space: nowrap; vertical-align: middle; }
.md-comment { color: rgb(162, 127, 3); opacity: 0.8; font-family: var(--monospace); }
code { text-align: left; vertical-align: initial; }
a.md-print-anchor { white-space: pre !important; border-width: initial !important; border-style: none !important; border-color: initial !important; display: inline-block !important; position: absolute !important; width: 1px !important; right: 0px !important; outline: 0px !important; background: 0px 0px !important; text-decoration: initial !important; text-shadow: initial !important; }
.md-inline-math .MathJax_SVG .noError { display: none !important; }
.html-for-mac .inline-math-svg .MathJax_SVG { vertical-align: 0.2px; }
.md-math-block .MathJax_SVG_Display { text-align: center; margin: 0px; position: relative; text-indent: 0px; max-width: none; max-height: none; min-height: 0px; min-width: 100%; width: auto; overflow-y: hidden; display: block !important; }
.MathJax_SVG_Display, .md-inline-math .MathJax_SVG_Display { width: auto; margin: inherit; display: inline-block !important; }
.MathJax_SVG .MJX-monospace { font-family: var(--monospace); }
.MathJax_SVG .MJX-sans-serif { font-family: sans-serif; }
.MathJax_SVG { display: inline; font-style: normal; font-weight: 400; line-height: normal; zoom: 90%; text-indent: 0px; text-align: left; text-transform: none; letter-spacing: normal; word-spacing: normal; word-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border: 0px; padding: 0px; margin: 0px; }
.MathJax_SVG * { transition: none; }
.MathJax_SVG_Display svg { vertical-align: middle !important; margin-bottom: 0px !important; }
.os-windows.monocolor-emoji .md-emoji { font-family: "Segoe UI Symbol", sans-serif; }
.md-diagram-panel > svg { max-width: 100%; }
[lang="mermaid"] svg, [lang="flow"] svg { max-width: 100%; }
[lang="mermaid"] .node text { font-size: 1rem; }
table tr th { border-bottom: 0px; }
video { max-width: 100%; display: block; margin: 0px auto; }
iframe { max-width: 100%; width: 100%; border: none; }
.highlight td, .highlight tr { border: 0px; }
.CodeMirror { height: auto; }
.CodeMirror.cm-s-inner { background: inherit; }
.CodeMirror-scroll { overflow-y: hidden; overflow-x: auto; z-index: 3; }
.CodeMirror-gutter-filler, .CodeMirror-scrollbar-filler { background-color: rgb(255, 255, 255); }
.CodeMirror-gutters { border-right: 1px solid rgb(221, 221, 221); background: inherit; white-space: nowrap; }
.CodeMirror-linenumber { padding: 0px 3px 0px 5px; text-align: right; color: rgb(153, 153, 153); }
.cm-s-inner .cm-keyword { color: rgb(119, 0, 136); }
.cm-s-inner .cm-atom, .cm-s-inner.cm-atom { color: rgb(34, 17, 153); }
.cm-s-inner .cm-number { color: rgb(17, 102, 68); }
.cm-s-inner .cm-def { color: rgb(0, 0, 255); }
.cm-s-inner .cm-variable { color: rgb(0, 0, 0); }
.cm-s-inner .cm-variable-2 { color: rgb(0, 85, 170); }
.cm-s-inner .cm-variable-3 { color: rgb(0, 136, 85); }
.cm-s-inner .cm-string { color: rgb(170, 17, 17); }
.cm-s-inner .cm-property { color: rgb(0, 0, 0); }
.cm-s-inner .cm-operator { color: rgb(152, 26, 26); }
.cm-s-inner .cm-comment, .cm-s-inner.cm-comment { color: rgb(170, 85, 0); }
.cm-s-inner .cm-string-2 { color: rgb(255, 85, 0); }
.cm-s-inner .cm-meta { color: rgb(85, 85, 85); }
.cm-s-inner .cm-qualifier { color: rgb(85, 85, 85); }
.cm-s-inner .cm-builtin { color: rgb(51, 0, 170); }
.cm-s-inner .cm-bracket { color: rgb(153, 153, 119); }
.cm-s-inner .cm-tag { color: rgb(17, 119, 0); }
.cm-s-inner .cm-attribute { color: rgb(0, 0, 204); }
.cm-s-inner .cm-header, .cm-s-inner.cm-header { color: rgb(0, 0, 255); }
.cm-s-inner .cm-quote, .cm-s-inner.cm-quote { color: rgb(0, 153, 0); }
.cm-s-inner .cm-hr, .cm-s-inner.cm-hr { color: rgb(153, 153, 153); }
.cm-s-inner .cm-link, .cm-s-inner.cm-link { color: rgb(0, 0, 204); }
.cm-negative { color: rgb(221, 68, 68); }
.cm-positive { color: rgb(34, 153, 34); }
.cm-header, .cm-strong { font-weight: 700; }
.cm-del { text-decoration: line-through; }
.cm-em { font-style: italic; }
.cm-link { text-decoration: underline; }
.cm-error { color: red; }
.cm-invalidchar { color: red; }
.cm-constant { color: rgb(38, 139, 210); }
.cm-defined { color: rgb(181, 137, 0); }
div.CodeMirror span.CodeMirror-matchingbracket { color: rgb(0, 255, 0); }
div.CodeMirror span.CodeMirror-nonmatchingbracket { color: rgb(255, 34, 34); }
.cm-s-inner .CodeMirror-activeline-background { background: inherit; }
.CodeMirror { position: relative; overflow: hidden; }
.CodeMirror-scroll { height: 100%; outline: 0px; position: relative; box-sizing: content-box; background: inherit; }
.CodeMirror-sizer { position: relative; }
.CodeMirror-gutter-filler, .CodeMirror-hscrollbar, .CodeMirror-scrollbar-filler, .CodeMirror-vscrollbar { position: absolute; z-index: 6; display: none; }
.CodeMirror-vscrollbar { right: 0px; top: 0px; overflow: hidden; }
.CodeMirror-hscrollbar { bottom: 0px; left: 0px; overflow: hidden; }
.CodeMirror-scrollbar-filler { right: 0px; bottom: 0px; }
.CodeMirror-gutter-filler { left: 0px; bottom: 0px; }
.CodeMirror-gutters { position: absolute; left: 0px; top: 0px; padding-bottom: 30px; z-index: 3; }
.CodeMirror-gutter { white-space: normal; height: 100%; box-sizing: content-box; padding-bottom: 30px; margin-bottom: -32px; display: inline-block; }
.CodeMirror-gutter-wrapper { position: absolute; z-index: 4; background: 0px 0px !important; border: none !important; }
.CodeMirror-gutter-background { position: absolute; top: 0px; bottom: 0px; z-index: 4; }
.CodeMirror-gutter-elt { position: absolute; cursor: default; z-index: 4; }
.CodeMirror-lines { cursor: text; }
.CodeMirror pre { border-radius: 0px; border-width: 0px; background: 0px 0px; font-family: inherit; font-size: inherit; margin: 0px; white-space: pre; word-wrap: normal; color: inherit; z-index: 2; position: relative; overflow: visible; }
.CodeMirror-wrap pre { word-wrap: break-word; white-space: pre-wrap; word-break: normal; }
.CodeMirror-code pre { border-right: 30px solid transparent; width: fit-content; }
.CodeMirror-wrap .CodeMirror-code pre { border-right: none; width: auto; }
.CodeMirror-linebackground { position: absolute; left: 0px; right: 0px; top: 0px; bottom: 0px; z-index: 0; }
.CodeMirror-linewidget { position: relative; z-index: 2; overflow: auto; }
.CodeMirror-wrap .CodeMirror-scroll { overflow-x: hidden; }
.CodeMirror-measure { position: absolute; width: 100%; height: 0px; overflow: hidden; visibility: hidden; }
.CodeMirror-measure pre { position: static; }
.CodeMirror div.CodeMirror-cursor { position: absolute; visibility: hidden; border-right: none; width: 0px; }
.CodeMirror div.CodeMirror-cursor { visibility: hidden; }
.CodeMirror-focused div.CodeMirror-cursor { visibility: inherit; }
.cm-searching { background: rgba(255, 255, 0, 0.4); }
@media print {
.CodeMirror div.CodeMirror-cursor { visibility: hidden; }
}
:root { --side-bar-bg-color: #fafafa; --control-text-color: #777; }
html { font-size: 16px; }
body { font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; color: rgb(51, 51, 51); line-height: 1.6; }
#write { max-width: 860px; margin: 0px auto; padding: 30px 30px 100px; }
#write > ul:first-child, #write > ol:first-child { margin-top: 30px; }
a { color: rgb(65, 131, 196); }
h1, h2, h3, h4, h5, h6 { position: relative; margin-top: 1rem; margin-bottom: 1rem; font-weight: bold; line-height: 1.4; cursor: text; }
h1:hover a.anchor, h2:hover a.anchor, h3:hover a.anchor, h4:hover a.anchor, h5:hover a.anchor, h6:hover a.anchor { text-decoration: none; }
h1 tt, h1 code { font-size: inherit; }
h2 tt, h2 code { font-size: inherit; }
h3 tt, h3 code { font-size: inherit; }
h4 tt, h4 code { font-size: inherit; }
h5 tt, h5 code { font-size: inherit; }
h6 tt, h6 code { font-size: inherit; }
h1 { padding-bottom: 0.3em; font-size: 2.25em; line-height: 1.2; border-bottom: 1px solid rgb(238, 238, 238); }
h2 { padding-bottom: 0.3em; font-size: 1.75em; line-height: 1.225; border-bottom: 1px solid rgb(238, 238, 238); }
h3 { font-size: 1.5em; line-height: 1.43; }
h4 { font-size: 1.25em; }
h5 { font-size: 1em; }
h6 { font-size: 1em; color: rgb(119, 119, 119); }
p, blockquote, ul, ol, dl, table { margin: 0.8em 0px; }
li > ol, li > ul { margin: 0px; }
hr { height: 2px; padding: 0px; margin: 16px 0px; background-color: rgb(231, 231, 231); border: 0px none; overflow: hidden; box-sizing: content-box; }
li p.first { display: inline-block; }
ul, ol { padding-left: 30px; }
ul:first-child, ol:first-child { margin-top: 0px; }
ul:last-child, ol:last-child { margin-bottom: 0px; }
blockquote { border-left: 4px solid rgb(223, 226, 229); padding: 0px 15px; color: rgb(119, 119, 119); }
blockquote blockquote { padding-right: 0px; }
table { padding: 0px; word-break: initial; }
table tr { border-top: 1px solid rgb(223, 226, 229); margin: 0px; padding: 0px; }
table tr:nth-child(2n), thead { background-color: rgb(248, 248, 248); }
table tr th { font-weight: bold; border-width: 1px 1px 0px; border-top-style: solid; border-right-style: solid; border-left-style: solid; border-top-color: rgb(223, 226, 229); border-right-color: rgb(223, 226, 229); border-left-color: rgb(223, 226, 229); border-image: initial; border-bottom-style: initial; border-bottom-color: initial; text-align: left; margin: 0px; padding: 6px 13px; }
table tr td { border: 1px solid rgb(223, 226, 229); text-align: left; margin: 0px; padding: 6px 13px; }
table tr th:first-child, table tr td:first-child { margin-top: 0px; }
table tr th:last-child, table tr td:last-child { margin-bottom: 0px; }
.CodeMirror-lines { padding-left: 4px; }
.code-tooltip { box-shadow: rgba(0, 28, 36, 0.3) 0px 1px 1px 0px; border-top: 1px solid rgb(238, 242, 242); }
.md-fences, code, tt { border: 1px solid rgb(231, 234, 237); background-color: rgb(248, 248, 248); border-radius: 3px; padding: 2px 4px 0px; font-size: 0.9em; }
code { background-color: rgb(243, 244, 244); padding: 0px 2px; }
.md-fences { margin-bottom: 15px; margin-top: 15px; padding-top: 8px; padding-bottom: 6px; }
.md-task-list-item > input { margin-left: -1.3em; }
@media print {
html { font-size: 13px; }
table, pre { break-inside: avoid; }
pre { word-wrap: break-word; }
}
.md-fences { background-color: rgb(248, 248, 248); }
#write pre.md-meta-block { padding: 1rem; font-size: 85%; line-height: 1.45; background-color: rgb(247, 247, 247); border: 0px; border-radius: 3px; color: rgb(119, 119, 119); margin-top: 0px !important; }
.mathjax-block > .code-tooltip { bottom: 0.375rem; }
.md-mathjax-midline { background: rgb(250, 250, 250); }
#write > h3.md-focus::before { left: -1.5625rem; top: 0.375rem; }
#write > h4.md-focus::before { left: -1.5625rem; top: 0.285714rem; }
#write > h5.md-focus::before { left: -1.5625rem; top: 0.285714rem; }
#write > h6.md-focus::before { left: -1.5625rem; top: 0.285714rem; }
.md-image > .md-meta { border-radius: 3px; padding: 2px 0px 0px 4px; font-size: 0.9em; color: inherit; }
.md-tag { color: rgb(167, 167, 167); opacity: 1; }
.md-toc { margin-top: 20px; padding-bottom: 20px; }
.sidebar-tabs { border-bottom: none; }
#typora-quick-open { border: 1px solid rgb(221, 221, 221); background-color: rgb(248, 248, 248); }
#typora-quick-open-item { background-color: rgb(250, 250, 250); border-color: rgb(254, 254, 254) rgb(229, 229, 229) rgb(229, 229, 229) rgb(238, 238, 238); border-style: solid; border-width: 1px; }
.on-focus-mode blockquote { border-left-color: rgba(85, 85, 85, 0.12); }
header, .context-menu, .megamenu-content, footer { font-family: "Segoe UI", Arial, sans-serif; }
.file-node-content:hover .file-node-icon, .file-node-content:hover .file-node-open-state { visibility: visible; }
.mac-seamless-mode #typora-sidebar { background-color: var(--side-bar-bg-color); }
.md-lang { color: rgb(180, 101, 77); }
.html-for-mac .context-menu { --item-hover-bg-color: #E6F0FE; }
#md-notification .btn { border: 0px; }
.dropdown-menu .divider { border-color: rgb(229, 229, 229); }
.typora-export li, .typora-export p, .typora-export, .footnote-line {white-space: normal;}
</style>
</head>
<body class='typora-export os-windows' >
<div id='write' class = 'is-node'><h1><a name='header-n21' class='md-header-anchor '></a>Twitter Tweet Classification Using BERT</h1><div class='md-toc' mdtype='toc'><p class="md-toc-content"><span class="md-toc-item md-toc-h1" data-ref="n21"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n21">Twitter Tweet Classification Using BERT</a></span><span class="md-toc-item md-toc-h2" data-ref="n45"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n45">1. Introduction and Background</a></span><span class="md-toc-item md-toc-h2" data-ref="n46"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n46">2. BERT</a></span><span class="md-toc-item md-toc-h2" data-ref="n68"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n68">3. Dataset</a></span><span class="md-toc-item md-toc-h2" data-ref="n48"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n48">4. Tweet Preprocessing</a></span><span class="md-toc-item md-toc-h2" data-ref="n352"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n352">5. Classification Model</a></span><span class="md-toc-item md-toc-h3" data-ref="n359"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n359">5.1 Baseline Model with LSTM and GloVE Embedding</a></span><span class="md-toc-item md-toc-h3" data-ref="n363"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n363">5.2 BERT Sequence Classification Model</a></span><span class="md-toc-item md-toc-h3" data-ref="n367"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n367">5.3 BERT with LSTM Classification Model</a></span><span class="md-toc-item md-toc-h3" data-ref="n365"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n365">5.4 Helper Functions</a></span><span class="md-toc-item md-toc-h2" data-ref="n392"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n392">6. Model Training</a></span><span class="md-toc-item md-toc-h2" data-ref="n396"><a class="md-toc-inner" style="cursor: pointer;" href="#header-n396">7. Results</a></span></p></div><h2><a name='header-n45' class='md-header-anchor '></a>1. Introduction and Background</h2><p>During a mass casualty event such as a natural disaster or a mass shooting, social networks such as Twitter or Facebook act as a conduit of information. These information include location and type of personal injury, infrastructure damage, donations, advice, and emotional support. Useful information can be harnessed by first responders and agencies to assess damage and coordinate rescue operations. However, the speed and the mass at which the information come in presents a challenge to rescue personnel to discern the relevant ones from extraneous ones. In this light, we want to create a machine learning model that can automatically classify social media posts into different categories. The model aims to extract useful information from a sea of posts, and sort the useful information into several classes.</p><p>In this project, we will classify Twitter tweets during natural disasters. We attempt to classify tweets into 7 categories:</p><ol><li>not informative </li><li>caution and advice</li><li>affected individuals</li><li>infrastructure and utilities damage</li><li>donations and volunteering</li><li>sympathy and support</li><li>other useful information</li></ol><p>Our baseline model will be a LSTM model using the Stanford GloVE Twitter word embedding. We will compare the base model with a Google BERT base classifier model and BERT model modified with an LSTM. The models will be written in Pytorch.</p><h2><a name='header-n46' class='md-header-anchor '></a>2. BERT</h2><p>In natural language processing, a word is represented by a vector of numbers before input into a machine learning model for processing. These word vectors are commonly referred to as <strong>word embeddings</strong>. A word embedding should not be a random vector, but rather be able to express the meaning of the word or relations between different words. For example, the embedding for "queen" subtract the embedding for "woman" and then add the embedding for "man" should result in the embedding for "king".</p><div mdtype="math_block" cid="n72" id="mathjax-n72" class="mathjax-block md-end-block md-math-block md-rawblock" spellcheck="false" contenteditable="false"><div class="md-rawblock-container md-math-container" contenteditable="false" tabindex="-1"><div class="MathJax_SVG_Display" style="text-align: center;"><span class="MathJax_SVG" id="MathJax-Element-55-Frame" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="471.47px" height="22.0631px" viewBox="0 -806.1 23707.4 1109.7" role="img" focusable="false" style="vertical-align: -0.705ex; max-width: 100%;"><defs><path stroke-width="0" id="E58-MJMATHI-65" d="M39 168Q39 225 58 272T107 350T174 402T244 433T307 442H310Q355 442 388 420T421 355Q421 265 310 237Q261 224 176 223Q139 223 138 221Q138 219 132 186T125 128Q125 81 146 54T209 26T302 45T394 111Q403 121 406 121Q410 121 419 112T429 98T420 82T390 55T344 24T281 -1T205 -11Q126 -11 83 42T39 168ZM373 353Q367 405 305 405Q272 405 244 391T199 357T170 316T154 280T149 261Q149 260 169 260Q282 260 327 284T373 353Z"></path><path stroke-width="0" id="E58-MJMATHI-6D" d="M21 287Q22 293 24 303T36 341T56 388T88 425T132 442T175 435T205 417T221 395T229 376L231 369Q231 367 232 367L243 378Q303 442 384 442Q401 442 415 440T441 433T460 423T475 411T485 398T493 385T497 373T500 364T502 357L510 367Q573 442 659 442Q713 442 746 415T780 336Q780 285 742 178T704 50Q705 36 709 31T724 26Q752 26 776 56T815 138Q818 149 821 151T837 153Q857 153 857 145Q857 144 853 130Q845 101 831 73T785 17T716 -10Q669 -10 648 17T627 73Q627 92 663 193T700 345Q700 404 656 404H651Q565 404 506 303L499 291L466 157Q433 26 428 16Q415 -11 385 -11Q372 -11 364 -4T353 8T350 18Q350 29 384 161L420 307Q423 322 423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 181Q151 335 151 342Q154 357 154 369Q154 405 129 405Q107 405 92 377T69 316T57 280Q55 278 41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E58-MJMATHI-62" d="M73 647Q73 657 77 670T89 683Q90 683 161 688T234 694Q246 694 246 685T212 542Q204 508 195 472T180 418L176 399Q176 396 182 402Q231 442 283 442Q345 442 383 396T422 280Q422 169 343 79T173 -11Q123 -11 82 27T40 150V159Q40 180 48 217T97 414Q147 611 147 623T109 637Q104 637 101 637H96Q86 637 83 637T76 640T73 647ZM336 325V331Q336 405 275 405Q258 405 240 397T207 376T181 352T163 330L157 322L136 236Q114 150 114 114Q114 66 138 42Q154 26 178 26Q211 26 245 58Q270 81 285 114T318 219Q336 291 336 325Z"></path><path stroke-width="0" id="E58-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E58-MJMATHI-71" d="M33 157Q33 258 109 349T280 441Q340 441 372 389Q373 390 377 395T388 406T404 418Q438 442 450 442Q454 442 457 439T460 434Q460 425 391 149Q320 -135 320 -139Q320 -147 365 -148H390Q396 -156 396 -157T393 -175Q389 -188 383 -194H370Q339 -192 262 -192Q234 -192 211 -192T174 -192T157 -193Q143 -193 143 -185Q143 -182 145 -170Q149 -154 152 -151T172 -148Q220 -148 230 -141Q238 -136 258 -53T279 32Q279 33 272 29Q224 -10 172 -10Q117 -10 75 30T33 157ZM352 326Q329 405 277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q233 26 290 98L298 109L352 326Z"></path><path stroke-width="0" id="E58-MJMATHI-75" d="M21 287Q21 295 30 318T55 370T99 420T158 442Q204 442 227 417T250 358Q250 340 216 246T182 105Q182 62 196 45T238 27T291 44T328 78L339 95Q341 99 377 247Q407 367 413 387T427 416Q444 431 463 431Q480 431 488 421T496 402L420 84Q419 79 419 68Q419 43 426 35T447 26Q469 29 482 57T512 145Q514 153 532 153Q551 153 551 144Q550 139 549 130T540 98T523 55T498 17T462 -8Q454 -10 438 -10Q372 -10 347 46Q345 45 336 36T318 21T296 6T267 -6T233 -11Q189 -11 155 7Q103 38 103 113Q103 170 138 262T173 379Q173 380 173 381Q173 390 173 393T169 400T158 404H154Q131 404 112 385T82 344T65 302T57 280Q55 278 41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E58-MJMATHI-6E" d="M21 287Q22 293 24 303T36 341T56 388T89 425T135 442Q171 442 195 424T225 390T231 369Q231 367 232 367L243 378Q304 442 382 442Q436 442 469 415T503 336T465 179T427 52Q427 26 444 26Q450 26 453 27Q482 32 505 65T540 145Q542 153 560 153Q580 153 580 145Q580 144 576 130Q568 101 554 73T508 17T439 -10Q392 -10 371 17T350 73Q350 92 386 193T423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 180T152 343Q153 348 153 366Q153 405 129 405Q91 405 66 305Q60 285 60 284Q58 278 41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E58-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E58-MJMAIN-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path stroke-width="0" id="E58-MJMATHI-77" d="M580 385Q580 406 599 424T641 443Q659 443 674 425T690 368Q690 339 671 253Q656 197 644 161T609 80T554 12T482 -11Q438 -11 404 5T355 48Q354 47 352 44Q311 -11 252 -11Q226 -11 202 -5T155 14T118 53T104 116Q104 170 138 262T173 379Q173 380 173 381Q173 390 173 393T169 400T158 404H154Q131 404 112 385T82 344T65 302T57 280Q55 278 41 278H27Q21 284 21 287Q21 293 29 315T52 366T96 418T161 441Q204 441 227 416T250 358Q250 340 217 250T184 111Q184 65 205 46T258 26Q301 26 334 87L339 96V119Q339 122 339 128T340 136T341 143T342 152T345 165T348 182T354 206T362 238T373 281Q402 395 406 404Q419 431 449 431Q468 431 475 421T483 402Q483 389 454 274T422 142Q420 131 420 107V100Q420 85 423 71T442 42T487 26Q558 26 600 148Q609 171 620 213T632 273Q632 306 619 325T593 357T580 385Z"></path><path stroke-width="0" id="E58-MJMATHI-6F" d="M201 -11Q126 -11 80 38T34 156Q34 221 64 279T146 380Q222 441 301 441Q333 441 341 440Q354 437 367 433T402 417T438 387T464 338T476 268Q476 161 390 75T201 -11ZM121 120Q121 70 147 48T206 26Q250 26 289 58T351 142Q360 163 374 216T388 308Q388 352 370 375Q346 405 306 405Q243 405 195 347Q158 303 140 230T121 120Z"></path><path stroke-width="0" id="E58-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E58-MJMAIN-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path stroke-width="0" id="E58-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="0" id="E58-MJMATHI-6B" d="M121 647Q121 657 125 670T137 683Q138 683 209 688T282 694Q294 694 294 686Q294 679 244 477Q194 279 194 272Q213 282 223 291Q247 309 292 354T362 415Q402 442 438 442Q468 442 485 423T503 369Q503 344 496 327T477 302T456 291T438 288Q418 288 406 299T394 328Q394 353 410 369T442 390L458 393Q446 405 434 405H430Q398 402 367 380T294 316T228 255Q230 254 243 252T267 246T293 238T320 224T342 206T359 180T365 147Q365 130 360 106T354 66Q354 26 381 26Q429 26 459 145Q461 153 479 153H483Q499 153 499 144Q499 139 496 130Q455 -11 378 -11Q333 -11 305 15T277 90Q277 108 280 121T283 145Q283 167 269 183T234 206T200 217T182 220H180Q168 178 159 139T145 81T136 44T129 20T122 7T111 -2Q98 -11 83 -11Q66 -11 57 -1T48 16Q48 26 85 176T158 471L195 616Q196 629 188 632T149 637H144Q134 637 131 637T124 640T121 647Z"></path><path stroke-width="0" id="E58-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E58-MJMATHI-67" d="M311 43Q296 30 267 15T206 0Q143 0 105 45T66 160Q66 265 143 353T314 442Q361 442 401 394L404 398Q406 401 409 404T418 412T431 419T447 422Q461 422 470 413T480 394Q480 379 423 152T363 -80Q345 -134 286 -169T151 -205Q10 -205 10 -137Q10 -111 28 -91T74 -71Q89 -71 102 -80T116 -111Q116 -121 114 -130T107 -144T99 -154T92 -162L90 -164H91Q101 -167 151 -167Q189 -167 211 -155Q234 -144 254 -122T282 -75Q288 -56 298 -13Q311 35 311 43ZM384 328L380 339Q377 350 375 354T369 368T359 382T346 393T328 402T306 405Q262 405 221 352Q191 313 171 233T151 117Q151 38 213 38Q269 38 323 108L331 118L384 328Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E58-MJMATHI-65" x="0" y="0"></use><use xlink:href="#E58-MJMATHI-6D" x="466" y="0"></use><use xlink:href="#E58-MJMATHI-62" x="1344" y="0"></use><use xlink:href="#E58-MJMAIN-28" x="1773" y="0"></use><use xlink:href="#E58-MJMATHI-71" x="2162" y="0"></use><use xlink:href="#E58-MJMATHI-75" x="2622" y="0"></use><use xlink:href="#E58-MJMATHI-65" x="3194" y="0"></use><use xlink:href="#E58-MJMATHI-65" x="3660" y="0"></use><use xlink:href="#E58-MJMATHI-6E" x="4126" y="0"></use><use xlink:href="#E58-MJMAIN-29" x="4726" y="0"></use><use xlink:href="#E58-MJMAIN-2212" x="5337" y="0"></use><use xlink:href="#E58-MJMATHI-65" x="6337" y="0"></use><use xlink:href="#E58-MJMATHI-6D" x="6803" y="0"></use><use xlink:href="#E58-MJMATHI-62" x="7681" y="0"></use><use xlink:href="#E58-MJMAIN-28" x="8110" y="0"></use><use xlink:href="#E58-MJMATHI-77" x="8499" y="0"></use><use xlink:href="#E58-MJMATHI-6F" x="9215" y="0"></use><use xlink:href="#E58-MJMATHI-6D" x="9700" y="0"></use><use xlink:href="#E58-MJMATHI-61" x="10578" y="0"></use><use xlink:href="#E58-MJMATHI-6E" x="11107" y="0"></use><use xlink:href="#E58-MJMAIN-29" x="11707" y="0"></use><use xlink:href="#E58-MJMAIN-2B" x="12318" y="0"></use><use xlink:href="#E58-MJMATHI-65" x="13318" y="0"></use><use xlink:href="#E58-MJMATHI-6D" x="13784" y="0"></use><use xlink:href="#E58-MJMATHI-62" x="14662" y="0"></use><use xlink:href="#E58-MJMAIN-28" x="15091" y="0"></use><use xlink:href="#E58-MJMATHI-6D" x="15480" y="0"></use><use xlink:href="#E58-MJMATHI-61" x="16358" y="0"></use><use xlink:href="#E58-MJMATHI-6E" x="16887" y="0"></use><use xlink:href="#E58-MJMAIN-29" x="17487" y="0"></use><use xlink:href="#E58-MJMAIN-3D" x="18154" y="0"></use><use xlink:href="#E58-MJMATHI-65" x="19210" y="0"></use><use xlink:href="#E58-MJMATHI-6D" x="19676" y="0"></use><use xlink:href="#E58-MJMATHI-62" x="20554" y="0"></use><use xlink:href="#E58-MJMAIN-28" x="20983" y="0"></use><use xlink:href="#E58-MJMATHI-6B" x="21372" y="0"></use><use xlink:href="#E58-MJMATHI-69" x="21893" y="0"></use><use xlink:href="#E58-MJMATHI-6E" x="22238" y="0"></use><use xlink:href="#E58-MJMATHI-67" x="22838" y="0"></use><use xlink:href="#E58-MJMAIN-29" x="23318" y="0"></use></g></svg></span></div><script type="math/tex; mode=display" id="MathJax-Element-55">emb(queen)-emb(woman)+emb(man)=emb(king)</script></div></div><p>Thus these word representations allow mathematical operations to be conducted on the "meaning" of the words in the machine learning model, and allows the model to discern the overall meaning of an utterance. </p><p>Word representations are often trained on large text corpus based on co-occurrence statistics, which are the number of times words appear close to each other in a sentence. Pre-trained presentations can be broadly into two classes, <em>contextual</em> or <em>non-contextual</em>. Contextual representations can be further divided into <em>unidirectional</em> or <em>bidirectional</em> . </p><p><em>Noncontextual</em> representations use the same vector to represent words that have multiple meanings, regardless of which context the words appears in. For example, in both of the following sentences, "He flipped the light switch", and "His steps were light ", the same vector is used for the word "light". This is despite the fact that based on context, the same word have completely different meanings. Commonly used <a href='https://www.tensorflow.org/tutorials/representation/word2vec'>word2vec</a> and <a href='https://nlp.stanford.edu/projects/glove/'>GloVE</a> embeddings belong to this class of representation.</p><p><em>Contextual</em> representations would use different vectors to represent a word based on the context in which the word appears in. This type of representation is more accurate and performs better This class of word representation can be divided into <em>unidirectional</em> or <em>bidirectional</em>. In unidirectional contextual representations, only the context on one side of the word, either left or right, is used to generate the representation. The <a href='https://allennlp.org/elmo'>ELMo</a> representation belongs to the class of unidirectional contextual word representations.</p><p><a href='https://github.com/google-research/bert'>BERT</a>, or Bidirectional Encoder Representations from Transformers, is a bidirectional contextual word representation. Bidirectional representations are better because the meaning of a words depends on both the words before and after it. Unidirectional contextual embeddings are commonly generated by scanning forward or backwards from the word of interest. Scanning both sides of the word of interest together would cause the word to "see itself" in the context. BERT solves this problem by first masking a certain percentage of the words in the sequence, then use an bidirectional encoder called <a href='https://arxiv.org/abs/1706.03762'>transformers</a> to scan the entire sequence, and finally predict the masked words. For example,</p><blockquote><p>Input: He took his children to the [mask1] to read some books after [mask2] ended.</p><p>Labels: mask1 = library, mask2 = school </p></blockquote><p> </p><h2><a name='header-n68' class='md-header-anchor '></a>3. Dataset</h2><p>Labelled tweets are gathered from 2 online sources, <a href='https://crisisnlp.qcri.org/'>CrisisNLP</a>, and <a href='https://crisislex.org/'>CrissisLex</a>. Each website contains several repositories. Each repository contains tweets gathered during various natural disasters in a csv file. We will only choose English tweets. The labels from each repository are not identical but largely similar. We will map the labels to a set of common labels. </p><h2><a name='header-n48' class='md-header-anchor '></a>4. Tweet Preprocessing</h2><p>Lets preprocess the tweets into the appropriate format before feeding them into the network. We download the English tweets from the CrisisNLP and CrisisLex online repositories and story them in separate file directories. </p><p>First we list all the csv files we have saved in the crisisNLP directory.</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre><span>xxxxxxxxxx</span></pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">os</span></span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">pandas</span> <span class="cm-keyword">as</span> <span class="cm-variable">pd</span></span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre></div><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">data_dir</span> = <span class="cm-string">'data/crisisNLP'</span></span></pre></div><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">files_csv</span> = []</span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">for</span> <span class="cm-variable">file</span> <span class="cm-keyword">in</span> <span class="cm-variable">os</span>.<span class="cm-property">listdir</span>(<span class="cm-variable">data_dir</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">if</span> <span class="cm-variable">file</span>.<span class="cm-property">endswith</span>(<span class="cm-string">".csv"</span>):</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">files_csv</span>.<span class="cm-property">append</span>(<span class="cm-variable">data_dir</span><span class="cm-operator">+</span><span class="cm-variable">file</span>)</span></pre></div></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 180px;"></div><div class="CodeMirror-gutters" style="display: none; height: 180px;"></div></div></div></pre><p> </p><p>All the csv files are loaded and appended into a single Pandas data frame .</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre>x</pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation"><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">crisisNLP_df</span> = <span class="cm-variable">pd</span>.<span class="cm-property">DataFrame</span>() </span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">for</span> <span class="cm-variable">file</span> <span class="cm-keyword">in</span> <span class="cm-variable">files_csv</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">temp_df</span> = <span class="cm-variable">pd</span>.<span class="cm-property">read_csv</span>(<span class="cm-variable">file</span>, <span class="cm-variable">encoding</span>=<span class="cm-string">'utf-8'</span>)</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">crisisNLP_df</span> = <span class="cm-variable">pd</span>.<span class="cm-property">concat</span>([<span class="cm-variable">crisisNLP_df</span>, <span class="cm-variable">temp_df</span>])</span></pre></div></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 90px;"></div><div class="CodeMirror-gutters" style="display: none; height: 90px;"></div></div></div></pre><p> </p><p>Lets examine the csv file.</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre><span>xxxxxxxxxx</span></pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation"><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">crisisNLP_df</span>.<span class="cm-property">head</span>(<span class="cm-number">10</span>)</span></pre></div></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 23px;"></div><div class="CodeMirror-gutters" style="display: none; height: 23px;"></div></div></div></pre><figure><table><thead><tr><th>_unit_id</th><th>_golden</th><th>_unit_state</th><th>_trusted_judgments</th><th>_last_judgment_at</th><th>choose_one_category</th><th>choose_one_category:confidence</th><th>choose_one_category_gold</th><th>tweet_id</th><th>tweet_text</th></tr></thead><tbody><tr><td>879819338</td><td>False</td><td>finalized</td><td>3</td><td>2/11/2016 11:41:40</td><td>other_useful_information</td><td>0.3429</td><td>NaN</td><td>'382813388503412736'</td><td>USGS reports a M1.7 #earthquake 70km ESE of He...</td></tr><tr><td>879819339</td><td>False</td><td>finalized</td><td>3</td><td>2/11/2016 09:52:13</td><td>not_related_or_irrelevant</td><td>0.6345</td><td>NaN</td><td>382813391435206659</td><td>QuakeFactor M 3.0, Southern Alaska: Sunday, Se...</td></tr></tbody></table></figure><p>As on can see there are 10 columns, the relevant ones are the "choose_one_category", "choose_one_category:confidence", and "tweet_text" columns. In particular, the "choose_one_category" is the tweet labels, and the "choose_one_category:confidence" is the confidence of the labels.</p><p>We will drop the columns that are not relevant, and also only keep the rows that have confidence level greater than 0.6.</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre><span>xxxxxxxxxx</span></pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">drop_cols</span> = [<span class="cm-string">'_unit_id'</span>, <span class="cm-string">'_golden'</span>, <span class="cm-string">'_unit_state'</span>, <span class="cm-string">'_trusted_judgments'</span>,</span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'_last_judgment_at'</span>, <span class="cm-string">'choose_one_category_gold'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'tweet_id'</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">crisisNLP_df</span>.<span class="cm-property">drop</span>(<span class="cm-variable">columns</span>=<span class="cm-variable">drop_cols</span>, <span class="cm-variable">inplace</span>=<span class="cm-keyword">True</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">crisisNLP_df</span>[<span class="cm-variable">crisisNLP_df</span>[<span class="cm-string">'choose_one_category:confidence'</span>] <span class="cm-operator">></span>=<span class="cm-number">0.6</span>]</span></pre></div></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 180px;"></div><div class="CodeMirror-gutters" style="display: none; height: 180px;"></div></div></div></pre><p>We do the same for CrisisLex data. We drop the columns not needed by us.</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" style="break-inside: unset;"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre>x</pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">data_dir</span> = <span class="cm-string">'data/CrisisLexT26/'</span></span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">files_csv</span> = []</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">for</span> (<span class="cm-variable">dirpath</span>, <span class="cm-variable">dirnames</span>, <span class="cm-variable">filenames</span>) <span class="cm-keyword">in</span> <span class="cm-variable">os</span>.<span class="cm-property">walk</span>(<span class="cm-variable">data_dir</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">for</span> <span class="cm-variable">file</span> <span class="cm-keyword">in</span> <span class="cm-variable">filenames</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">if</span> <span class="cm-variable">file</span>.<span class="cm-property">endswith</span>(<span class="cm-string">"labeled.csv"</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">files_csv</span>.<span class="cm-property">append</span>(<span class="cm-variable">dirpath</span><span class="cm-operator">+</span><span class="cm-string">'/'</span><span class="cm-operator">+</span><span class="cm-variable">file</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">crisisLex_df</span> = <span class="cm-variable">pd</span>.<span class="cm-property">DataFrame</span>() </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">for</span> <span class="cm-variable">file</span> <span class="cm-keyword">in</span> <span class="cm-variable">files_csv</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">temp_df</span> = <span class="cm-variable">pd</span>.<span class="cm-property">read_csv</span>(<span class="cm-variable">file</span>, <span class="cm-variable">encoding</span>=<span class="cm-string">'utf-8'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">crisisLex_df</span> = <span class="cm-variable">pd</span>.<span class="cm-property">concat</span>([<span class="cm-variable">crisisLex_df</span>, <span class="cm-variable">temp_df</span>])</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">drop_cols</span> = [<span class="cm-string">'Tweet ID'</span>,<span class="cm-string">' Information Source'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">' Informativeness'</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">crisisLex_df</span>.<span class="cm-property">drop</span>(<span class="cm-variable">columns</span>=<span class="cm-variable">drop_cols</span>, <span class="cm-variable">inplace</span>=<span class="cm-keyword">True</span>)</span></pre></div></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 360px;"></div><div class="CodeMirror-gutters" style="display: none; height: 360px;"></div></div></div></pre><p>We combine the two data frames into a common data frame. But before we do that, we need to give the same name to the corresponding columns in both data frames. We will give the tweet column the name "tweet" and category column the name "type". We will drop the confidence column in CrisisNLP and finall concatenate the two data frames.</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre>x</pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">crisisLex_df</span>.<span class="cm-property">rename</span>(<span class="cm-variable">index</span>=<span class="cm-builtin">str</span>, <span class="cm-variable">columns</span>={<span class="cm-string">" Information Type"</span>: <span class="cm-string">"type"</span>, <span class="cm-string">" Tweet Text"</span>: <span class="cm-string">"tweet"</span>}, <span class="cm-variable">inplace</span>=<span class="cm-keyword">True</span>)</span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">crisisNLP_df</span>.<span class="cm-property">rename</span>(<span class="cm-variable">index</span>=<span class="cm-builtin">str</span>, <span class="cm-variable">columns</span>={<span class="cm-string">"choose_one_category"</span>: <span class="cm-string">"type"</span>, <span class="cm-string">"tweet_text"</span>: <span class="cm-string">"tweet"</span>}, <span class="cm-variable">inplace</span>=<span class="cm-keyword">True</span>)</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre></div><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">crisisNLP_df</span>.<span class="cm-property">drop</span>(<span class="cm-variable">columns</span>=<span class="cm-string">'choose_one_category:confidence'</span>, <span class="cm-variable">inplace</span>=<span class="cm-keyword">True</span>)</span></pre></div><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>=<span class="cm-variable">pd</span>.<span class="cm-property">concat</span>([<span class="cm-variable">crisisLex_df</span>,<span class="cm-variable">crisisNLP_df</span>])</span></pre></div></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 158px;"></div><div class="CodeMirror-gutters" style="display: none; height: 158px;"></div></div></div></pre><p>Now that we have stripped the necessary data from the csv file, we need to clean up the tweets. We will</p><ol><li>Remove the URL linkes</li><li>Remove the "RT", "&", "<", ">", and "@" symbles</li><li>Remove non-ascii words</li><li>Remove extra spaces</li><li>Insert spaces between punctuation marks</li><li>Strip leading and ending spaces</li><li>Convert all words to lower-case</li></ol><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" style="break-inside: unset;"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre><span>xxxxxxxxxx</span></pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-comment">#remove URL</span></span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'http(\S)+'</span>, <span class="cm-string">r''</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'http ...'</span>, <span class="cm-string">r''</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'http'</span>, <span class="cm-string">r''</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">contains</span>(<span class="cm-string">r'http'</span>)]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-comment"># remove RT, @</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'(RT|rt)[ ]*@[ ]*[\S]+'</span>,<span class="cm-string">r''</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">contains</span>(<span class="cm-string">r'RT[ ]?@'</span>)]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'@[\S]+'</span>,<span class="cm-string">r''</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-comment">#remove non-ascii words and characters</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = [<span class="cm-string">''</span>.<span class="cm-property">join</span>([<span class="cm-variable">i</span> <span class="cm-keyword">if</span> <span class="cm-builtin">ord</span>(<span class="cm-variable">i</span>) <span class="cm-operator"><</span> <span class="cm-number">128</span> <span class="cm-keyword">else</span> <span class="cm-string">''</span> <span class="cm-keyword">for</span> <span class="cm-variable">i</span> <span class="cm-keyword">in</span> <span class="cm-variable">text</span>]) <span class="cm-keyword">for</span> <span class="cm-variable">text</span> <span class="cm-keyword">in</span> <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>]]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'_[\S]?'</span>,<span class="cm-string">r''</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-comment">#remove &, < and ></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'&amp;?'</span>,<span class="cm-string">r'and'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'&lt;'</span>,<span class="cm-string">r'<'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'&gt;'</span>,<span class="cm-string">r'>'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-comment"># remove extra space</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'[ ]{2, }'</span>,<span class="cm-string">r' '</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-comment"># insert space between punctuation marks</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'([\w\d]+)([^\w\d ]+)'</span>, <span class="cm-string">r'\1 \2'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">replace</span>(<span class="cm-string">r'([^\w\d ]+)([\w\d]+)'</span>, <span class="cm-string">r'\1 \2'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-comment"># lower case and strip white spaces at both ends</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">lower</span>()</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>].<span class="cm-property">str</span>.<span class="cm-property">strip</span>()</span></pre></div></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 698px;"></div><div class="CodeMirror-gutters" style="display: none; height: 698px;"></div></div></div></pre><p>We will calculate the length of each tweet and only keep unique tweets that are 3 words or longer.</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre>x</pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc_length'</span>] = [<span class="cm-builtin">len</span>(<span class="cm-variable">text</span>.<span class="cm-property">split</span>(<span class="cm-string">' '</span>)) <span class="cm-keyword">for</span> <span class="cm-variable">text</span> <span class="cm-keyword">in</span> <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>]]</span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc_length'</span>].<span class="cm-property">value_counts</span>()</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span> = <span class="cm-variable">df</span>[<span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc_length'</span>]<span class="cm-operator">></span><span class="cm-number">3</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span> = <span class="cm-variable">df</span>.<span class="cm-property">drop_duplicates</span>(<span class="cm-variable">subset</span>=[<span class="cm-string">'tweet_proc'</span>])</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre></div></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 135px;"></div><div class="CodeMirror-gutters" style="display: none; height: 135px;"></div></div></div></pre><p>Now we have our set of working data. We will consolidate all the labels into the following 7 labels.</p><ol><li>Not informative</li><li>Other useful information</li><li>Caution and advice</li><li>Infrastructure and Utilities damage</li><li>donations and volunteering</li><li>sympathy and support</li><li>affected individuals</li></ol><p>Afterwards, we will assign a numeric type to each of the labels,</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" style="break-inside: unset;"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre><span>xxxxxxxxxx</span></pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">type_map</span>={<span class="cm-string">'Not labeled'</span>: <span class="cm-string">'not informative'</span>, </span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'Other Useful Information'</span>: <span class="cm-string">'other useful information'</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'Caution and advice'</span>: <span class="cm-string">'caution and advice'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'Affected individuals'</span>: <span class="cm-string">'affected individuals'</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'Infrastructure and utilities'</span>: <span class="cm-string">'infrastructure and utilities damage'</span>,</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'Donations and volunteering'</span>: <span class="cm-string">'donations and volunteering'</span>,</span></pre></div><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'Sympathy and support'</span>:<span class="cm-string">'sympathy and support'</span>,</span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'Not applicable'</span>: <span class="cm-string">'not informative'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'other_useful_information'</span>: <span class="cm-string">'other useful information'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'not_related_or_irrelevant'</span>: <span class="cm-string">'not informative'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'donation_needs_or_offers_or_volunteering_services'</span>: <span class="cm-string">'donations and volunteering'</span>,</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'injured_or_dead_people'</span>:<span class="cm-string">'affected individuals'</span>, </span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'missing_trapped_or_found_people'</span>:<span class="cm-string">'affected individuals'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'caution_and_advice'</span>:<span class="cm-string">'caution and advice'</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'infrastructure_and_utilities_damage'</span>:<span class="cm-string">'infrastructure and utilities damage'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'sympathy_and_emotional_support'</span>:<span class="cm-string">'sympathy and support'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'displaced_people_and_evacuations'</span>:<span class="cm-string">'affected individuals'</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> }</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'type'</span>] = <span class="cm-variable">df</span>[<span class="cm-string">'type'</span>].<span class="cm-property">map</span>(<span class="cm-variable">type_map</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'type'</span>].<span class="cm-property">value_counts</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">label_dict</span> = <span class="cm-builtin">dict</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">for</span> <span class="cm-variable">i</span>, <span class="cm-variable">l</span> <span class="cm-keyword">in</span> <span class="cm-builtin">enumerate</span>(<span class="cm-builtin">list</span>(<span class="cm-variable">df</span>.<span class="cm-property">type</span>.<span class="cm-property">value_counts</span>().<span class="cm-property">keys</span>())):</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">label_dict</span>.<span class="cm-property">update</span>({<span class="cm-variable">l</span>: <span class="cm-variable">i</span>})</span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-comment"># for each unique label, assign a numeric identiifer</span></span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre></div><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'type_label'</span>] = [<span class="cm-variable">label_dict</span>[<span class="cm-variable">label</span>] <span class="cm-keyword">for</span> <span class="cm-variable">label</span> <span class="cm-keyword">in</span> <span class="cm-variable">df</span>.<span class="cm-property">type</span>] <span class="cm-comment">#create a column in df to store the numeric ids</span></span></pre></div></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 675px;"></div><div class="CodeMirror-gutters" style="display: none; height: 675px;"></div></div></div></pre><p>The final step is to preprocess the tweet texts for input into the BERT classifier. BER classifier requires the input be prefixed by the "[CLS]" token. We will also tokenize the tweet text with the BERT Tokenizer and calculate the length of the tokenized text.</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre>x</pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc_bert'</span>] = <span class="cm-string">'[CLS] '</span><span class="cm-operator">+</span><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc'</span>]</span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">from</span> <span class="cm-variable">pytorch_pretrained_bert</span> <span class="cm-keyword">import</span> <span class="cm-variable">BertTokenizer</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">tokenizer</span> = <span class="cm-variable">BertTokenizer</span>.<span class="cm-property">from_pretrained</span>(<span class="cm-string">'bert-base-uncased'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc_BERTbase_length'</span>] = [<span class="cm-builtin">len</span>(<span class="cm-variable">tokenizer</span>.<span class="cm-property">tokenize</span>(<span class="cm-variable">sent</span>)) <span class="cm-keyword">for</span> <span class="cm-variable">sent</span> <span class="cm-keyword">in</span> <span class="cm-variable">df</span>[<span class="cm-string">'tweet_proc_bert'</span>]]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 180px;"></div><div class="CodeMirror-gutters" style="display: none; height: 180px;"></div></div></div></pre><h2><a name='header-n352' class='md-header-anchor '></a>5. Classification Model</h2><p>The models will be programmed using Pytorch. We will compare 3 different classification models. The baseline model is a LSTM network using the GloVE twitter word embedding. It will be compared with two BERT based model. The basic BERT model is the pretrained BertForSequenceClassification model. We will be finetuning it on the twitter dataset. The second BERT based model stacks a LSTM on top of BERT.</p><h3><a name='header-n359' class='md-header-anchor '></a>5.1 Baseline Model with LSTM and GloVE Embedding</h3><p>We use a single layer bi-directional LSTM neural network model as our baseline. The hidden size of the LSTM cell is 256. Tweets are first embedded using the GloVE Twitter embedding with 50 dimensions. The stacked final state of the LSTM cell is linked to a softmax classifier through a fully connected layer.</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" style="break-inside: unset;"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 922.5px; left: 250.664px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre>x</pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">torch</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">torch</span>.<span class="cm-property">nn</span> <span class="cm-keyword">as</span> <span class="cm-variable">nn</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">torch</span>.<span class="cm-property">nn</span>.<span class="cm-property">utils</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">from</span> <span class="cm-variable">torch</span>.<span class="cm-property">nn</span>.<span class="cm-property">utils</span>.<span class="cm-property">rnn</span> <span class="cm-keyword">import</span> <span class="cm-variable">pack_padded_sequence</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">sys</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">pickle</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">from</span> <span class="cm-variable">vocab</span> <span class="cm-keyword">import</span> <span class="cm-variable">VocabEntry</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">numpy</span> <span class="cm-keyword">as</span> <span class="cm-variable">np</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">class</span> <span class="cm-def">BaselineModel</span>(<span class="cm-variable">nn</span>.<span class="cm-property">Module</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">__init__</span>(<span class="cm-variable-2">self</span>, <span class="cm-variable">rnn_state_size</span>, <span class="cm-variable">embedding</span>, <span class="cm-variable">vocab</span>, <span class="cm-variable">num_tweet_class</span>, <span class="cm-variable">dropout_rate</span>=<span class="cm-number">0</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">"""</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param hidden_size (int): size of lstm hidden layer</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param embedding (torch.Tensor): shape (num_of_words_in_dict, embed_dim), glove embedding matrix</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param vocab (VocabEntry): GloVe word dictionary/index</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param num_tweet_class (int): number of labels / classes</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param dropout_rate (float): dropout rate for training</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">super</span>(<span class="cm-variable">BaselineModel</span>, <span class="cm-variable-2">self</span>).<span class="cm-property">__init__</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">rnn_state_size</span> = <span class="cm-variable">rnn_state_size</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">embed_dim</span> = <span class="cm-variable">embedding</span>.<span class="cm-property">size</span>(<span class="cm-number">1</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">vocab</span> = <span class="cm-variable">vocab</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">num_tweet_class</span> = <span class="cm-variable">num_tweet_class</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">padding_idx</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">vocab</span>[<span class="cm-string">'<pad>'</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">dropout_rate</span> = <span class="cm-variable">dropout_rate</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">embedding_layer</span> = <span class="cm-variable">nn</span>.<span class="cm-property">Embedding</span>.<span class="cm-property">from_pretrained</span>(<span class="cm-variable">embeddings</span> = <span class="cm-variable">embedding</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">freeze</span>=<span class="cm-keyword">True</span>, <span class="cm-comment"># freeze weights from training</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">padding_idx</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">padding_idx</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">max_norm</span>=<span class="cm-keyword">None</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">norm_type</span>=<span class="cm-number">2.0</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">scale_grad_by_freq</span>=<span class="cm-keyword">False</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">sparse</span>=<span class="cm-keyword">False</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># Create a embedding using GloVE pretrained weights</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">lstm</span> = <span class="cm-variable">nn</span>.<span class="cm-property">LSTM</span>(<span class="cm-variable">input_size</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">embed_dim</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">hidden_size</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">rnn_state_size</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">num_layers</span> = <span class="cm-number">1</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">bias</span> = <span class="cm-keyword">True</span>, </span></pre><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">batch_first</span> = <span class="cm-keyword">False</span>, </span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">dropout</span> = <span class="cm-number">0</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">bidirectional</span> = <span class="cm-keyword">True</span> <span class="cm-comment">#use a bi-directional LSTM</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> )</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">affine</span> = <span class="cm-variable">nn</span>.<span class="cm-property">Linear</span>(<span class="cm-variable">in_features</span> = <span class="cm-number">2</span><span class="cm-operator">*</span><span class="cm-variable-2">self</span>.<span class="cm-property">rnn_state_size</span>, <span class="cm-comment">#the hidden stats of the biLSTM is stacked</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">out_features</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">num_tweet_class</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">bias</span>=<span class="cm-keyword">True</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># fully connected layer before softmax</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">dropout</span> = <span class="cm-variable">nn</span>.<span class="cm-property">Dropout</span>(<span class="cm-variable">p</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">dropout_rate</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">forward</span>(<span class="cm-variable-2">self</span>, <span class="cm-variable">sents</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">"""</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param sents (list[list[str]]): a list of a list of words, sorted in descending length</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @return output (torch.Tensor): logits to put into softmax function to calculate prob</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">text_lengths</span> = <span class="cm-variable">torch</span>.<span class="cm-property">tensor</span>([<span class="cm-builtin">len</span>(<span class="cm-variable">sent</span>) <span class="cm-keyword">for</span> <span class="cm-variable">sent</span> <span class="cm-keyword">in</span> <span class="cm-variable">sents</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">sents_tensor</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">vocab</span>.<span class="cm-property">to_input_tensor</span>(<span class="cm-variable">sents</span>) <span class="cm-comment"># Convert from list to tensor (max_sent_length, batch_size)</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">x_embed</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">embedding_layer</span>(<span class="cm-variable">sents_tensor</span>) <span class="cm-comment"># create embedding for words (max_sent_length, batch_size, embed_size)</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">seq</span> = <span class="cm-variable">pack_padded_sequence</span>(<span class="cm-variable">x_embed</span>.<span class="cm-property">float</span>(), <span class="cm-variable">text_lengths</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">enc_hiddens</span>, (<span class="cm-variable">last_hidden</span>, <span class="cm-variable">last_cell</span>) = <span class="cm-variable-2">self</span>.<span class="cm-property">lstm</span>(<span class="cm-variable">seq</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">output_hidden</span> = <span class="cm-variable">torch</span>.<span class="cm-property">cat</span>((<span class="cm-variable">last_hidden</span>[<span class="cm-number">0</span>], <span class="cm-variable">last_hidden</span>[<span class="cm-number">1</span>]), <span class="cm-variable">dim</span>=<span class="cm-number">1</span>) <span class="cm-comment"># (batch_size, 2*hidden_size)</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">output_hidden</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">dropout</span>(<span class="cm-variable">output_hidden</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">output</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">affine</span>(<span class="cm-variable">output_hidden</span>) <span class="cm-comment"># (batch_size, n_class)</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">return</span> <span class="cm-variable">output</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-meta">@staticmethod</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">load</span>(<span class="cm-variable">model_path</span>: <span class="cm-builtin">str</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">""" Load the model from a file.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param model_path (str): path to model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @return model (nn.Module): model with saved parameters</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">params</span> = <span class="cm-variable">torch</span>.<span class="cm-property">load</span>(<span class="cm-variable">model_path</span>, <span class="cm-variable">map_location</span>=<span class="cm-keyword">lambda</span> <span class="cm-variable">storage</span>, <span class="cm-variable">loc</span>: <span class="cm-variable">storage</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">args</span> = <span class="cm-variable">params</span>[<span class="cm-string">'args'</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span> = <span class="cm-variable">BaselineModel</span>(<span class="cm-variable">vocab</span>=<span class="cm-variable">params</span>[<span class="cm-string">'vocab'</span>], <span class="cm-variable">embedding</span>=<span class="cm-variable">params</span>[<span class="cm-string">'embedding'</span>], <span class="cm-operator">**</span><span class="cm-variable">args</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span>.<span class="cm-property">load_state_dict</span>(<span class="cm-variable">params</span>[<span class="cm-string">'state_dict'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">return</span> <span class="cm-variable">model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">save</span>(<span class="cm-variable-2">self</span>, <span class="cm-variable">path</span>: <span class="cm-builtin">str</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">""" Save the model to a file.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param path (str): path to the model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'save model parameters to [%s]'</span> <span class="cm-operator">%</span> <span class="cm-variable">path</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">params</span> = {</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" class="cm-tab-wrap-hack" style="padding-right: 0.1px;"> <span class="cm-string">'args'</span>: <span class="cm-builtin">dict</span>(<span class="cm-variable">rnn_state_size</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">rnn_state_size</span>, <span class="cm-tab" role="presentation" cm-text=" "> </span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">dropout_rate</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">dropout_rate</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">num_tweet_class</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">num_tweet_class</span>),</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'vocab'</span>: <span class="cm-variable-2">self</span>.<span class="cm-property">vocab</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'embedding'</span>: <span class="cm-variable-2">self</span>.<span class="cm-property">embedding_layer</span>.<span class="cm-property">weight</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'state_dict'</span>: <span class="cm-variable-2">self</span>.<span class="cm-property">state_dict</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> }</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">torch</span>.<span class="cm-property">save</span>(<span class="cm-variable">params</span>, <span class="cm-variable">path</span>)</span></pre></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 2318px;"></div><div class="CodeMirror-gutters" style="display: none; height: 2318px;"></div></div></div></pre><p> </p><h3><a name='header-n363' class='md-header-anchor '></a>5.2 BERT Sequence Classification Model</h3><p>We finetune the basic BERT sequence classification model, BertForSequenceClassification. We use the English BERT uncased base model, which has 12 transformer layers, 12 self-attention heads, and a hidden size of 768.</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" style="break-inside: unset;"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre>x</pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">from</span> <span class="cm-variable">pytorch_pretrained_bert</span> <span class="cm-keyword">import</span> <span class="cm-variable">BertTokenizer</span>, <span class="cm-variable">BertModel</span>, <span class="cm-variable">BertForSequenceClassification</span></span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">torch</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">from</span> <span class="cm-variable">torch</span>.<span class="cm-property">nn</span>.<span class="cm-property">utils</span>.<span class="cm-property">rnn</span> <span class="cm-keyword">import</span> <span class="cm-variable">pack_padded_sequence</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">sys</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">numpy</span> <span class="cm-keyword">as</span> <span class="cm-variable">np</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">class</span> <span class="cm-def">default_bert</span>(<span class="cm-variable">torch</span>.<span class="cm-property">nn</span>.<span class="cm-property">Module</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">__init__</span>(<span class="cm-variable-2">self</span>, <span class="cm-variable">num_class</span>, <span class="cm-variable">bert_config</span>=<span class="cm-string">'bert-base-uncased'</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">"""</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> :param num_class: number of classification categories</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> :param bert_config: str, BERT model used</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">super</span>(<span class="cm-variable">default_bert</span>, <span class="cm-variable-2">self</span>).<span class="cm-property">__init__</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">num_class</span> = <span class="cm-variable">num_class</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">bert_config</span> = <span class="cm-variable">bert_config</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">model</span> = <span class="cm-variable">BertForSequenceClassification</span>.<span class="cm-property">from_pretrained</span>(<span class="cm-variable-2">self</span>.<span class="cm-property">bert_config</span>, <span class="cm-variable">num_labels</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">num_class</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">tokenizer</span> = <span class="cm-variable">BertTokenizer</span>.<span class="cm-property">from_pretrained</span>(<span class="cm-variable-2">self</span>.<span class="cm-property">bert_config</span>) </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">forward</span>(<span class="cm-variable-2">self</span>, <span class="cm-variable">sents</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">"""</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> :param sents: list[str], list of untokenized sentences </span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">sents_tensor</span>, <span class="cm-variable">masks_tensor</span>, <span class="cm-variable">sents_lengths</span> = <span class="cm-variable">sents_to_tensor</span>(<span class="cm-variable-2">self</span>.<span class="cm-property">tokenizer</span>, <span class="cm-variable">sents</span>) <span class="cm-comment">#helpfer function to convert string to tensors, and create padding mask</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">output</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">model</span>(<span class="cm-variable">input_ids</span>=<span class="cm-variable">sents_tensor</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">attention_mask</span>=<span class="cm-variable">masks_tensor</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">return</span> <span class="cm-variable">output</span> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-meta">@staticmethod</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">load</span>(<span class="cm-variable">model_path</span>: <span class="cm-builtin">str</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">""" Load the model from a file.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param model_path (str): path to model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @return model (nn.Module): model with saved parameters</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">params</span> = <span class="cm-variable">torch</span>.<span class="cm-property">load</span>(<span class="cm-variable">model_path</span>, <span class="cm-variable">map_location</span>=<span class="cm-keyword">lambda</span> <span class="cm-variable">storage</span>, <span class="cm-variable">loc</span>: <span class="cm-variable">storage</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">args</span> = <span class="cm-variable">params</span>[<span class="cm-string">'args'</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span> = <span class="cm-variable">default_bert</span>(<span class="cm-operator">**</span><span class="cm-variable">args</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span>.<span class="cm-property">load_state_dict</span>(<span class="cm-variable">params</span>[<span class="cm-string">'state_dict'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">return</span> <span class="cm-variable">model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">save</span>(<span class="cm-variable-2">self</span>, <span class="cm-variable">path</span>: <span class="cm-builtin">str</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">""" Save the model to a file.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param path (str): path to the model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'save model parameters to [%s]'</span> <span class="cm-operator">%</span> <span class="cm-variable">path</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">params</span> = {</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'args'</span>: <span class="cm-builtin">dict</span>(<span class="cm-variable">bert_config</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">bert_config</span>, <span class="cm-variable">num_class</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">num_class</span>),</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'state_dict'</span>: <span class="cm-variable-2">self</span>.<span class="cm-property">state_dict</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> }</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">torch</span>.<span class="cm-property">save</span>(<span class="cm-variable">params</span>, <span class="cm-variable">path</span>)</span></pre></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 1238px;"></div><div class="CodeMirror-gutters" style="display: none; height: 1238px;"></div></div></div></pre><p> </p><h3><a name='header-n367' class='md-header-anchor '></a>5.3 BERT with LSTM Classification Model</h3><p>We stack a bidirectional LSTM on top of BERT. The input to the LSTM is the BERT final hidden states of the entire tweet. Then, as the baseline model, the stacked hidden states of the LSTM is connected to a softmax classifier through a affine layer.</p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" style="break-inside: unset;"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre>x</pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">class</span> <span class="cm-def">LSTM_bert</span>(<span class="cm-variable">torch</span>.<span class="cm-property">nn</span>.<span class="cm-property">Module</span>):</span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">__init__</span>(<span class="cm-variable-2">self</span>, <span class="cm-variable">num_class</span>, <span class="cm-variable">dropout_rate</span>, <span class="cm-variable">bert_config</span>=<span class="cm-string">'bert-base-uncased'</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">"""</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> :param num_class: int, number of categories</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> :param bert_config: str, BERT configuration description</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> :param dropout_rate: float</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">super</span>(<span class="cm-variable">LSTM_bert</span>, <span class="cm-variable-2">self</span>).<span class="cm-property">__init__</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">num_class</span> = <span class="cm-variable">num_class</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">bert_config</span> = <span class="cm-variable">bert_config</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">tokenizer</span> = <span class="cm-variable">BertTokenizer</span>.<span class="cm-property">from_pretrained</span>(<span class="cm-variable-2">self</span>.<span class="cm-property">bert_config</span>) </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">bert</span> = <span class="cm-variable">BertModel</span>.<span class="cm-property">from_pretrained</span>(<span class="cm-variable-2">self</span>.<span class="cm-property">bert_config</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">dropout_rate</span> = <span class="cm-variable">dropout_rate</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">lstm_input_size</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">bert</span>.<span class="cm-property">config</span>.<span class="cm-property">hidden_size</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">lstm_hidden_size</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">bert</span>.<span class="cm-property">config</span>.<span class="cm-property">hidden_size</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">lstm</span> = <span class="cm-variable">torch</span>.<span class="cm-property">nn</span>.<span class="cm-property">LSTM</span>(<span class="cm-variable">input_size</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">lstm_input_size</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">hidden_size</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">lstm_hidden_size</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">bidirectional</span> = <span class="cm-keyword">True</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">dropout</span> = <span class="cm-variable">torch</span>.<span class="cm-property">nn</span>.<span class="cm-property">Dropout</span>(<span class="cm-variable">p</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">dropout_rate</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable-2">self</span>.<span class="cm-property">fc</span> = <span class="cm-variable">torch</span>.<span class="cm-property">nn</span>.<span class="cm-property">Linear</span>(<span class="cm-variable">in_features</span> = <span class="cm-number">2</span><span class="cm-operator">*</span><span class="cm-variable-2">self</span>.<span class="cm-property">lstm_hidden_size</span>, <span class="cm-comment">#LSTM stacked hidden state</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">out_features</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">num_class</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">bias</span>=<span class="cm-keyword">True</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">forward</span>(<span class="cm-variable-2">self</span>, <span class="cm-variable">sents</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">"""</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> :param sents: list[str], list of untokenized sentences </span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> :return: torch.tensor of shape (batch_size, num_class)</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">sents_tensor</span>, <span class="cm-variable">masks_tensor</span>, <span class="cm-variable">sents_lengths</span> = <span class="cm-variable">sents_to_tensor</span>(<span class="cm-variable-2">self</span>.<span class="cm-property">tokenizer</span>, <span class="cm-variable">sents</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># 1. The tweet is first input to the model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">encoded_layers</span>, <span class="cm-variable">pooled_output</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">bert</span>(<span class="cm-variable">input_ids</span>=<span class="cm-variable">sents_tensor</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">attention_mask</span>=<span class="cm-variable">masks_tensor</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">output_all_encoded_layers</span>=<span class="cm-keyword">False</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-tab" role="presentation" cm-text=" "> </span><span class="cm-tab" role="presentation" cm-text=" "> </span><span class="cm-comment"># 2. The output is reshuffled to the correct format</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">encoded_layers</span> = <span class="cm-variable">encoded_layers</span>.<span class="cm-property">permute</span>(<span class="cm-number">1</span>, <span class="cm-number">0</span>, <span class="cm-number">2</span>) <span class="cm-comment">#permute dimensions to fit LSTM input</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># 3. The encoded layers are fed into the biLSTM</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">enc_hiddens</span>, (<span class="cm-variable">last_hidden</span>, <span class="cm-variable">last_cell</span>) = <span class="cm-variable-2">self</span>.<span class="cm-property">lstm</span>(<span class="cm-variable">pack_padded_sequence</span>(<span class="cm-variable">encoded_layers</span>, <span class="cm-variable">sents_lengths</span>))</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># 4. final hidden states of the biLSTM is concatenated together</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">output_hidden</span> = <span class="cm-variable">torch</span>.<span class="cm-property">cat</span>((<span class="cm-variable">last_hidden</span>[<span class="cm-number">0</span>,:,:], <span class="cm-variable">last_hidden</span>[<span class="cm-number">1</span>,:,:]), <span class="cm-variable">dim</span>=<span class="cm-number">1</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># h_n of shape (num_layers * num_directions, batch, hidden_size)</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># 5. Dropout applied</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">output_hidden</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">dropout</span>(<span class="cm-variable">output_hidden</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># 6. Affine layer before softmax</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">output</span> = <span class="cm-variable-2">self</span>.<span class="cm-property">fc</span>(<span class="cm-variable">output_hidden</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">return</span> <span class="cm-variable">output</span> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-meta">@staticmethod</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">load</span>(<span class="cm-variable">model_path</span>: <span class="cm-builtin">str</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">""" Load the model from a file.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param model_path (str): path to model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @return model (nn.Module): model with saved parameters</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">params</span> = <span class="cm-variable">torch</span>.<span class="cm-property">load</span>(<span class="cm-variable">model_path</span>, <span class="cm-variable">map_location</span>=<span class="cm-keyword">lambda</span> <span class="cm-variable">storage</span>, <span class="cm-variable">loc</span>: <span class="cm-variable">storage</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">args</span> = <span class="cm-variable">params</span>[<span class="cm-string">'args'</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span> = <span class="cm-variable">LSTM_bert</span>(<span class="cm-operator">**</span><span class="cm-variable">args</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span>.<span class="cm-property">load_state_dict</span>(<span class="cm-variable">params</span>[<span class="cm-string">'state_dict'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">return</span> <span class="cm-variable">model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">def</span> <span class="cm-def">save</span>(<span class="cm-variable-2">self</span>, <span class="cm-variable">path</span>: <span class="cm-builtin">str</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">""" Save the model to a file.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param path (str): path to the model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'save model parameters to [%s]'</span> <span class="cm-operator">%</span> <span class="cm-variable">path</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">params</span> = {</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'args'</span>: <span class="cm-builtin">dict</span>(<span class="cm-variable">bert_config</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">bert_config</span>, <span class="cm-variable">num_class</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">num_class</span>, <span class="cm-variable">dropout_rate</span>=<span class="cm-variable-2">self</span>.<span class="cm-property">dropout_rate</span>),</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'state_dict'</span>: <span class="cm-variable-2">self</span>.<span class="cm-property">state_dict</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> }</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">torch</span>.<span class="cm-property">save</span>(<span class="cm-variable">params</span>, <span class="cm-variable">path</span>)</span></pre></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 1845px;"></div><div class="CodeMirror-gutters" style="display: none; height: 1845px;"></div></div></div></pre><p> </p><h3><a name='header-n365' class='md-header-anchor '></a>5.4 Helper Functions</h3><p>These are the helper functions used to convert batches of tweets to tensors, and pad to a common length in the process.</p><pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="python" style="break-inside: unset;"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre>x</pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">def</span> <span class="cm-def">pad_sents</span>(<span class="cm-variable">sents</span>, <span class="cm-variable">pad_token</span>):</span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">""" Pad list of sentences to the longest length in the batch.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param sents (list[list[str]]): list of tokenized strings</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param pad_token (int): pad token</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @returns sents_padded (list[list[int]]): list of tokenized sentences with padding shape: (batch_size, max_sentence_length)</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">sents_padded</span> = []</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">max_len</span> = <span class="cm-builtin">max</span>(<span class="cm-builtin">len</span>(<span class="cm-variable">s</span>) <span class="cm-keyword">for</span> <span class="cm-variable">s</span> <span class="cm-keyword">in</span> <span class="cm-variable">sents</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">for</span> <span class="cm-variable">s</span> <span class="cm-keyword">in</span> <span class="cm-variable">sents</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">padded</span> = [<span class="cm-variable">pad_token</span>] <span class="cm-operator">*</span> <span class="cm-variable">max_len</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">padded</span>[:<span class="cm-builtin">len</span>(<span class="cm-variable">s</span>)] = <span class="cm-variable">s</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">sents_padded</span>.<span class="cm-property">append</span>(<span class="cm-variable">padded</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">return</span> <span class="cm-variable">sents_padded</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">def</span> <span class="cm-def">sents_to_tensor</span>(<span class="cm-variable">tokenizer</span>, <span class="cm-variable">sents</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">"""</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> :param tokenizer</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> :param sents: list[str], list of untokenized strings</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">tokens_list</span> = [<span class="cm-variable">tokenizer</span>.<span class="cm-property">tokenize</span>(<span class="cm-variable">sent</span>) <span class="cm-keyword">for</span> <span class="cm-variable">sent</span> <span class="cm-keyword">in</span> <span class="cm-variable">sents</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">sents_lengths</span> = [<span class="cm-builtin">len</span>(<span class="cm-variable">tokens</span>) <span class="cm-keyword">for</span> <span class="cm-variable">tokens</span> <span class="cm-keyword">in</span> <span class="cm-variable">tokens_list</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">sents_lengths</span> = <span class="cm-variable">torch</span>.<span class="cm-property">tensor</span>(<span class="cm-variable">sents_lengths</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">tokens_list_padded</span> = <span class="cm-variable">pad_sents</span>(<span class="cm-variable">tokens_list</span>, <span class="cm-string">'[PAD]'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">masks</span> = <span class="cm-variable">np</span>.<span class="cm-property">asarray</span>(<span class="cm-variable">tokens_list_padded</span>)<span class="cm-operator">!</span>=<span class="cm-string">'[PAD]'</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">masks_tensor</span> = <span class="cm-variable">torch</span>.<span class="cm-property">tensor</span>(<span class="cm-variable">masks</span>, <span class="cm-variable">dtype</span>=<span class="cm-variable">torch</span>.<span class="cm-property">long</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">tokens_id_list</span> = [<span class="cm-variable">tokenizer</span>.<span class="cm-property">convert_tokens_to_ids</span>(<span class="cm-variable">tokens</span>) <span class="cm-keyword">for</span> <span class="cm-variable">tokens</span> <span class="cm-keyword">in</span> <span class="cm-variable">tokens_list_padded</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">sents_tensor</span> = <span class="cm-variable">torch</span>.<span class="cm-property">tensor</span>(<span class="cm-variable">tokens_id_list</span>, <span class="cm-variable">dtype</span>=<span class="cm-variable">torch</span>.<span class="cm-property">long</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">return</span> <span class="cm-variable">sents_tensor</span>, <span class="cm-variable">masks_tensor</span>, <span class="cm-variable">sents_lengths</span></span></pre></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 743px;"></div><div class="CodeMirror-gutters" style="display: none; height: 743px;"></div></div></div></pre><p> </p><h2><a name='header-n392' class='md-header-anchor '></a>6. Model Training</h2><p>We use BertAdam optimizer for all BERT related models. The initial learning rate for the BERT model is set to 0.00002 and for non-BERT part of the model is set to 0.001. The batch-size is set to 64. Patients is set to 5, and the learning rate is reduced by half after reaching this number. The training is stopped early if 5 reduction in learning rate occurred. Cross entropy loss is used as the loss function. With each category weighted by <span class="MathJax_SVG" tabindex="-1" style="font-size: 100%; display: inline-block;"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="38.992ex" height="2.694ex" viewBox="0 -806.1 16788.3 1160" role="img" focusable="false" style="vertical-align: -0.822ex;"><defs><path stroke-width="0" id="E101-MJMATHI-6D" d="M21 287Q22 293 24 303T36 341T56 388T88 425T132 442T175 435T205 417T221 395T229 376L231 369Q231 367 232 367L243 378Q303 442 384 442Q401 442 415 440T441 433T460 423T475 411T485 398T493 385T497 373T500 364T502 357L510 367Q573 442 659 442Q713 442 746 415T780 336Q780 285 742 178T704 50Q705 36 709 31T724 26Q752 26 776 56T815 138Q818 149 821 151T837 153Q857 153 857 145Q857 144 853 130Q845 101 831 73T785 17T716 -10Q669 -10 648 17T627 73Q627 92 663 193T700 345Q700 404 656 404H651Q565 404 506 303L499 291L466 157Q433 26 428 16Q415 -11 385 -11Q372 -11 364 -4T353 8T350 18Q350 29 384 161L420 307Q423 322 423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 181Q151 335 151 342Q154 357 154 369Q154 405 129 405Q107 405 92 377T69 316T57 280Q55 278 41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E101-MJMATHI-61" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path stroke-width="0" id="E101-MJMATHI-78" d="M52 289Q59 331 106 386T222 442Q257 442 286 424T329 379Q371 442 430 442Q467 442 494 420T522 361Q522 332 508 314T481 292T458 288Q439 288 427 299T415 328Q415 374 465 391Q454 404 425 404Q412 404 406 402Q368 386 350 336Q290 115 290 78Q290 50 306 38T341 26Q378 26 414 59T463 140Q466 150 469 151T485 153H489Q504 153 504 145Q504 144 502 134Q486 77 440 33T333 -11Q263 -11 227 52Q186 -10 133 -10H127Q78 -10 57 16T35 71Q35 103 54 123T99 143Q142 143 142 101Q142 81 130 66T107 46T94 41L91 40Q91 39 97 36T113 29T132 26Q168 26 194 71Q203 87 217 139T245 247T261 313Q266 340 266 352Q266 380 251 392T217 404Q177 404 142 372T93 290Q91 281 88 280T72 278H58Q52 284 52 289Z"></path><path stroke-width="0" id="E101-MJMAIN-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path stroke-width="0" id="E101-MJMAIN-33" d="M127 463Q100 463 85 480T69 524Q69 579 117 622T233 665Q268 665 277 664Q351 652 390 611T430 522Q430 470 396 421T302 350L299 348Q299 347 308 345T337 336T375 315Q457 262 457 175Q457 96 395 37T238 -22Q158 -22 100 21T42 130Q42 158 60 175T105 193Q133 193 151 175T169 130Q169 119 166 110T159 94T148 82T136 74T126 70T118 67L114 66Q165 21 238 21Q293 21 321 74Q338 107 338 175V195Q338 290 274 322Q259 328 213 329L171 330L168 332Q166 335 166 348Q166 366 174 366Q202 366 232 371Q266 376 294 413T322 525V533Q322 590 287 612Q265 626 240 626Q208 626 181 615T143 592T132 580H135Q138 579 143 578T153 573T165 566T175 555T183 540T186 520Q186 498 172 481T127 463Z"></path><path stroke-width="0" id="E101-MJMAIN-2C" d="M78 35T78 60T94 103T137 121Q165 121 187 96T210 8Q210 -27 201 -60T180 -117T154 -158T130 -185T117 -194Q113 -194 104 -185T95 -172Q95 -168 106 -156T131 -126T157 -76T173 -3V9L172 8Q170 7 167 6T161 3T152 1T140 0Q113 0 96 17Z"></path><path stroke-width="0" id="E101-MJMATHI-6A" d="M297 596Q297 627 318 644T361 661Q378 661 389 651T403 623Q403 595 384 576T340 557Q322 557 310 567T297 596ZM288 376Q288 405 262 405Q240 405 220 393T185 362T161 325T144 293L137 279Q135 278 121 278H107Q101 284 101 286T105 299Q126 348 164 391T252 441Q253 441 260 441T272 442Q296 441 316 432Q341 418 354 401T367 348V332L318 133Q267 -67 264 -75Q246 -125 194 -164T75 -204Q25 -204 7 -183T-12 -137Q-12 -110 7 -91T53 -71Q70 -71 82 -81T95 -112Q95 -148 63 -167Q69 -168 77 -168Q111 -168 139 -140T182 -74L193 -32Q204 11 219 72T251 197T278 308T289 365Q289 372 288 376Z"></path><path stroke-width="0" id="E101-MJMATHI-73" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path stroke-width="0" id="E101-MJMATHI-69" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path><path stroke-width="0" id="E101-MJMATHI-7A" d="M347 338Q337 338 294 349T231 360Q211 360 197 356T174 346T162 335T155 324L153 320Q150 317 138 317Q117 317 117 325Q117 330 120 339Q133 378 163 406T229 440Q241 442 246 442Q271 442 291 425T329 392T367 375Q389 375 411 408T434 441Q435 442 449 442H462Q468 436 468 434Q468 430 463 420T449 399T432 377T418 358L411 349Q368 298 275 214T160 106L148 94L163 93Q185 93 227 82T290 71Q328 71 360 90T402 140Q406 149 409 151T424 153Q443 153 443 143Q443 138 442 134Q425 72 376 31T278 -11Q252 -11 232 6T193 40T155 57Q111 57 76 -3Q70 -11 59 -11H54H41Q35 -5 35 -2Q35 13 93 84Q132 129 225 214T340 322Q352 338 347 338Z"></path><path stroke-width="0" id="E101-MJMATHI-65" d="M39 168Q39 225 58 272T107 350T174 402T244 433T307 442H310Q355 442 388 420T421 355Q421 265 310 237Q261 224 176 223Q139 223 138 221Q138 219 132 186T125 128Q125 81 146 54T209 26T302 45T394 111Q403 121 406 121Q410 121 419 112T429 98T420 82T390 55T344 24T281 -1T205 -11Q126 -11 83 42T39 168ZM373 353Q367 405 305 405Q272 405 244 391T199 357T170 316T154 280T149 261Q149 260 169 260Q282 260 327 284T373 353Z"></path><path stroke-width="0" id="E101-MJMATHI-6C" d="M117 59Q117 26 142 26Q179 26 205 131Q211 151 215 152Q217 153 225 153H229Q238 153 241 153T246 151T248 144Q247 138 245 128T234 90T214 43T183 6T137 -11Q101 -11 70 11T38 85Q38 97 39 102L104 360Q167 615 167 623Q167 626 166 628T162 632T157 634T149 635T141 636T132 637T122 637Q112 637 109 637T101 638T95 641T94 647Q94 649 96 661Q101 680 107 682T179 688Q194 689 213 690T243 693T254 694Q266 694 266 686Q266 675 193 386T118 83Q118 81 118 75T117 65V59Z"></path><path stroke-width="0" id="E101-MJMATHI-62" d="M73 647Q73 657 77 670T89 683Q90 683 161 688T234 694Q246 694 246 685T212 542Q204 508 195 472T180 418L176 399Q176 396 182 402Q231 442 283 442Q345 442 383 396T422 280Q422 169 343 79T173 -11Q123 -11 82 27T40 150V159Q40 180 48 217T97 414Q147 611 147 623T109 637Q104 637 101 637H96Q86 637 83 637T76 640T73 647ZM336 325V331Q336 405 275 405Q258 405 240 397T207 376T181 352T163 330L157 322L136 236Q114 150 114 114Q114 66 138 42Q154 26 178 26Q211 26 245 58Q270 81 285 114T318 219Q336 291 336 325Z"></path><path stroke-width="0" id="E101-MJMAIN-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path stroke-width="0" id="E101-MJMAIN-2F" d="M423 750Q432 750 438 744T444 730Q444 725 271 248T92 -240Q85 -250 75 -250Q68 -250 62 -245T56 -231Q56 -221 230 257T407 740Q411 750 423 750Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#E101-MJMATHI-6D" x="0" y="0"></use><use xlink:href="#E101-MJMATHI-61" x="878" y="0"></use><use xlink:href="#E101-MJMATHI-78" x="1407" y="0"></use><use xlink:href="#E101-MJMAIN-28" x="1979" y="0"></use><use xlink:href="#E101-MJMAIN-33" x="2368" y="0"></use><use xlink:href="#E101-MJMAIN-2C" x="2868" y="0"></use><use xlink:href="#E101-MJMATHI-6D" x="3312" y="0"></use><use xlink:href="#E101-MJMATHI-61" x="4190" y="0"></use><g transform="translate(4719,0)"><use xlink:href="#E101-MJMATHI-78" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E101-MJMATHI-6A" x="808" y="-213"></use></g><use xlink:href="#E101-MJMAIN-28" x="5682" y="0"></use><use xlink:href="#E101-MJMATHI-73" x="6071" y="0"></use><use xlink:href="#E101-MJMATHI-69" x="6540" y="0"></use><use xlink:href="#E101-MJMATHI-7A" x="6885" y="0"></use><use xlink:href="#E101-MJMATHI-65" x="7353" y="0"></use><use xlink:href="#E101-MJMAIN-28" x="7819" y="0"></use><use xlink:href="#E101-MJMATHI-6C" x="8208" y="0"></use><use xlink:href="#E101-MJMATHI-61" x="8506" y="0"></use><use xlink:href="#E101-MJMATHI-62" x="9035" y="0"></use><use xlink:href="#E101-MJMATHI-65" x="9464" y="0"></use><g transform="translate(9930,0)"><use xlink:href="#E101-MJMATHI-6C" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E101-MJMATHI-6A" x="421" y="-213"></use></g><use xlink:href="#E101-MJMAIN-29" x="10620" y="0"></use><use xlink:href="#E101-MJMAIN-2F" x="11009" y="0"></use><use xlink:href="#E101-MJMATHI-73" x="11509" y="0"></use><use xlink:href="#E101-MJMATHI-69" x="11978" y="0"></use><use xlink:href="#E101-MJMATHI-7A" x="12323" y="0"></use><use xlink:href="#E101-MJMATHI-65" x="12791" y="0"></use><use xlink:href="#E101-MJMAIN-28" x="13257" y="0"></use><use xlink:href="#E101-MJMATHI-6C" x="13646" y="0"></use><use xlink:href="#E101-MJMATHI-61" x="13944" y="0"></use><use xlink:href="#E101-MJMATHI-62" x="14473" y="0"></use><use xlink:href="#E101-MJMATHI-65" x="14902" y="0"></use><g transform="translate(15368,0)"><use xlink:href="#E101-MJMATHI-6C" x="0" y="0"></use><use transform="scale(0.707)" xlink:href="#E101-MJMATHI-69" x="421" y="-213"></use></g><use xlink:href="#E101-MJMAIN-29" x="16010" y="0"></use><use xlink:href="#E101-MJMAIN-29" x="16399" y="0"></use></g></svg></span><script type="math/tex">max(3, max_j(size(label_j)/size(label_i))</script>. </p><pre lang="python" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" style="break-inside: unset;"><div class="CodeMirror cm-s-inner CodeMirror-wrap" lang="python"><div style="overflow: hidden; position: relative; width: 3px; height: 0px; top: 0px; left: 8.16406px;"><textarea autocorrect="off" autocapitalize="off" spellcheck="false" tabindex="0" style="position: absolute; bottom: -1em; padding: 0px; width: 1000px; height: 1em; outline: none;"></textarea></div><div class="CodeMirror-scrollbar-filler" cm-not-content="true"></div><div class="CodeMirror-gutter-filler" cm-not-content="true"></div><div class="CodeMirror-scroll" tabindex="-1"><div class="CodeMirror-sizer" style="margin-left: 0px; margin-bottom: 0px; border-right-width: 0px; padding-right: 0px; padding-bottom: 0px;"><div style="position: relative; top: 0px;"><div class="CodeMirror-lines" role="presentation"><div role="presentation" style="position: relative; outline: none;"><div class="CodeMirror-measure"><pre>x</pre></div><div class="CodeMirror-measure"></div><div style="position: relative; z-index: 1;"></div><div class="CodeMirror-code" role="presentation" style=""><div class="CodeMirror-activeline" style="position: relative;"><div class="CodeMirror-activeline-background CodeMirror-linebackground"></div><div class="CodeMirror-gutter-background CodeMirror-activeline-gutter" style="left: 0px; width: 0px;"></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">from</span> <span class="cm-variable">pytorch_pretrained_bert</span> <span class="cm-keyword">import</span> <span class="cm-variable">BertAdam</span></span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">from</span> <span class="cm-variable">bert</span> <span class="cm-keyword">import</span> <span class="cm-variable">default_bert</span>, <span class="cm-variable">LSTM_bert</span> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">pickle</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">numpy</span> <span class="cm-keyword">as</span> <span class="cm-variable">np</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">torch</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">pandas</span> <span class="cm-keyword">as</span> <span class="cm-variable">pd</span></span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">time</span></span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">import</span> <span class="cm-variable">sys</span></span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">from</span> <span class="cm-variable">utils</span> <span class="cm-keyword">import</span> <span class="cm-variable">batch_iter</span></span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">def</span> <span class="cm-def">validation</span>(<span class="cm-variable">model</span>, <span class="cm-variable">df_val</span>, <span class="cm-variable">loss_func</span>, <span class="cm-variable">bert_size</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">""" validation of model during training.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param model (nn.Module): the model being trained</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param df_val (dataframe): validation dataset, sorted in descending text length</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @param loss_func(nn.Module): loss function</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> @return avg loss value across validation dataset</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-string"> """</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">was_training</span> = <span class="cm-variable">model</span>.<span class="cm-property">training</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span>.<span class="cm-property">eval</span>() <span class="cm-comment">#model.eval() put all layers in model in eval mode, that way, batchnorm or dropout layers will work in eval mode instead of training mode.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">df_val</span> = <span class="cm-variable">df_val</span>.<span class="cm-property">sort_values</span>(<span class="cm-variable">by</span>=<span class="cm-string">'tweet_proc_BERT'</span><span class="cm-operator">+</span><span class="cm-variable">bert_size</span><span class="cm-operator">+</span><span class="cm-string">'_length'</span>, <span class="cm-variable">ascending</span>=<span class="cm-keyword">False</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">tweet_proc_bert</span> = <span class="cm-builtin">list</span>(<span class="cm-variable">df_val</span>[<span class="cm-string">'tweet_proc_bert'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">type_label</span> = <span class="cm-builtin">list</span>(<span class="cm-variable">df_val</span>[<span class="cm-string">'type_label'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">val_batch_size</span> = <span class="cm-number">32</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">num_val_samples</span> = <span class="cm-variable">df_val</span>.<span class="cm-property">shape</span>[<span class="cm-number">0</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">n_batch</span> = <span class="cm-builtin">int</span>(<span class="cm-variable">np</span>.<span class="cm-property">ceil</span>(<span class="cm-variable">num_val_samples</span><span class="cm-operator">/</span><span class="cm-variable">val_batch_size</span>))</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">total_loss</span> = <span class="cm-number">0.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">with</span> <span class="cm-variable">torch</span>.<span class="cm-property">no_grad</span>():</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">for</span> <span class="cm-variable">i</span> <span class="cm-keyword">in</span> <span class="cm-builtin">range</span>(<span class="cm-variable">n_batch</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">sents</span> = <span class="cm-variable">tweet_proc_bert</span>[<span class="cm-variable">i</span><span class="cm-operator">*</span><span class="cm-variable">val_batch_size</span>: (<span class="cm-variable">i</span><span class="cm-operator">+</span><span class="cm-number">1</span>)<span class="cm-operator">*</span><span class="cm-variable">val_batch_size</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">targets</span> = <span class="cm-variable">torch</span>.<span class="cm-property">tensor</span>(<span class="cm-variable">type_label</span>[<span class="cm-variable">i</span><span class="cm-operator">*</span><span class="cm-variable">val_batch_size</span>: (<span class="cm-variable">i</span><span class="cm-operator">+</span><span class="cm-number">1</span>)<span class="cm-operator">*</span><span class="cm-variable">val_batch_size</span>],</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">dtype</span>=<span class="cm-variable">torch</span>.<span class="cm-property">long</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">batch_size</span> = <span class="cm-builtin">len</span>(<span class="cm-variable">sents</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">output</span> = <span class="cm-variable">model</span>(<span class="cm-variable">sents</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">batch_loss</span> = <span class="cm-variable">loss_func</span>(<span class="cm-variable">output</span>, <span class="cm-variable">targets</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">total_loss</span> += <span class="cm-variable">batch_loss</span>.<span class="cm-property">item</span>()<span class="cm-operator">*</span><span class="cm-variable">batch_size</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">if</span> <span class="cm-variable">was_training</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span>.<span class="cm-property">train</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">return</span> <span class="cm-variable">total_loss</span><span class="cm-operator">/</span><span class="cm-variable">num_val_samples</span> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-keyword">def</span> <span class="cm-def">train</span>(<span class="cm-variable">args</span>):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">label_name</span> = [<span class="cm-string">'not informative'</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'other useful information'</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'caution and advice'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'affected individuals'</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'infrastructure and utilities damage'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'donations and volunteering'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'sympathy and support'</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> ]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-comment"># save_file_name = args['--model']+'_model.bin'</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">bert_size</span> = <span class="cm-variable">args</span>[<span class="cm-string">'--bert-config'</span>].<span class="cm-property">split</span>(<span class="cm-string">'-'</span>)[<span class="cm-number">1</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">start_time</span> = <span class="cm-variable">time</span>.<span class="cm-property">time</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'Importing data...'</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">df_train</span> = <span class="cm-variable">pd</span>.<span class="cm-property">read_csv</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--train'</span>], <span class="cm-variable">index_col</span>=<span class="cm-number">0</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">df_val</span> = <span class="cm-variable">pd</span>.<span class="cm-property">read_csv</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--dev'</span>], <span class="cm-variable">index_col</span>=<span class="cm-number">0</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">train_label</span> = <span class="cm-builtin">dict</span>(<span class="cm-variable">df_train</span>[<span class="cm-string">'type_label'</span>].<span class="cm-property">value_counts</span>())</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">label_max</span> = <span class="cm-builtin">float</span>(<span class="cm-builtin">max</span>(<span class="cm-variable">train_label</span>.<span class="cm-property">values</span>()))</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-variable">train_label</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">train_label_weight</span> = <span class="cm-variable">torch</span>.<span class="cm-property">tensor</span>([<span class="cm-variable">label_max</span><span class="cm-operator">/</span><span class="cm-variable">train_label</span>[<span class="cm-variable">i</span>] <span class="cm-keyword">for</span> <span class="cm-variable">i</span> <span class="cm-keyword">in</span> <span class="cm-builtin">range</span>(<span class="cm-builtin">len</span>(<span class="cm-variable">train_label</span>))])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'Done! time elapsed %.2f sec'</span> <span class="cm-operator">%</span> (<span class="cm-variable">time</span>.<span class="cm-property">time</span>() <span class="cm-operator">-</span> <span class="cm-variable">start_time</span>), <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'-'</span> <span class="cm-operator">*</span> <span class="cm-number">80</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">start_time</span> = <span class="cm-variable">time</span>.<span class="cm-property">time</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'Set up model...'</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><div class="" style="position: relative;"><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre></div><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">if</span> <span class="cm-variable">args</span>[<span class="cm-string">'--model'</span>] == <span class="cm-string">'default_bert'</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span> = <span class="cm-variable">default_bert</span>(<span class="cm-variable">num_class</span>=<span class="cm-builtin">len</span>(<span class="cm-variable">label_name</span>), <span class="cm-variable">bert_config</span>=<span class="cm-variable">args</span>[<span class="cm-string">'--bert-config'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">optimizer_grouped_parameters</span> = [</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> {<span class="cm-string">'params'</span>: <span class="cm-variable">model</span>.<span class="cm-property">model</span>.<span class="cm-property">bert</span>.<span class="cm-property">parameters</span>()},</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> {<span class="cm-string">'params'</span>: <span class="cm-variable">model</span>.<span class="cm-property">model</span>.<span class="cm-property">classifier</span>.<span class="cm-property">parameters</span>(), <span class="cm-string">'lr'</span>: <span class="cm-builtin">float</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--lr'</span>])}</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> ]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">optimizer</span> = <span class="cm-variable">BertAdam</span>(<span class="cm-variable">optimizer_grouped_parameters</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">lr</span>=<span class="cm-builtin">float</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--lr-bert'</span>]),</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">max_grad_norm</span>=<span class="cm-builtin">float</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--clip-grad'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> )</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">elif</span> <span class="cm-variable">args</span>[<span class="cm-string">'--model'</span>] == <span class="cm-string">'LSTM_bert'</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span> = <span class="cm-variable">LSTM_bert</span>(<span class="cm-variable">num_class</span>=<span class="cm-builtin">len</span>(<span class="cm-variable">label_name</span>), <span class="cm-variable">dropout_rate</span>=<span class="cm-builtin">float</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--dropout'</span>]), <span class="cm-variable">bert_config</span>=<span class="cm-variable">args</span>[<span class="cm-string">'--bert-config'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">optimizer_grouped_parameters</span> = [</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> {<span class="cm-string">'params'</span>: <span class="cm-variable">model</span>.<span class="cm-property">bert</span>.<span class="cm-property">parameters</span>()},</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> {<span class="cm-string">'params'</span>: <span class="cm-variable">model</span>.<span class="cm-property">lstm</span>.<span class="cm-property">parameters</span>(), <span class="cm-string">'lr'</span>: <span class="cm-builtin">float</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--lr'</span>])},</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> {<span class="cm-string">'params'</span>: <span class="cm-variable">model</span>.<span class="cm-property">fc</span>.<span class="cm-property">parameters</span>(), <span class="cm-string">'lr'</span>: <span class="cm-builtin">float</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--lr'</span>])}]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">optimizer</span> = <span class="cm-variable">BertAdam</span>(<span class="cm-variable">optimizer_grouped_parameters</span>, </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">lr</span>=<span class="cm-builtin">float</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--lr-bert'</span>]),</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">max_grad_norm</span>=<span class="cm-builtin">float</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--clip-grad'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> )</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">else</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'wrong model...'</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'Done! time elapsed %.2f sec'</span> <span class="cm-operator">%</span> (<span class="cm-variable">time</span>.<span class="cm-property">time</span>() <span class="cm-operator">-</span> <span class="cm-variable">start_time</span>), <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'-'</span> <span class="cm-operator">*</span> <span class="cm-number">80</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span>.<span class="cm-property">train</span>() <span class="cm-comment">#set model for training mode</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">criterion</span> = <span class="cm-variable">torch</span>.<span class="cm-property">nn</span>.<span class="cm-property">CrossEntropyLoss</span>(<span class="cm-variable">weight</span>=<span class="cm-variable">train_label_weight</span>, <span class="cm-variable">reduction</span>=<span class="cm-string">'mean'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">torch</span>.<span class="cm-property">save</span>(<span class="cm-variable">criterion</span>, <span class="cm-string">'loss_func'</span>) <span class="cm-comment"># for later testing</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">train_batch_size</span> = <span class="cm-builtin">int</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--batch-size'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">valid_niter</span> = <span class="cm-builtin">int</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--valid-niter'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">display_num</span> = <span class="cm-builtin">int</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--display_num'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model_save_path</span> = <span class="cm-variable">args</span>[<span class="cm-string">'--save-to'</span>]</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">num_restarts</span> = <span class="cm-number">0</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">train_iter</span> = <span class="cm-variable">patience</span> = <span class="cm-variable">cum_loss</span> = <span class="cm-variable">report_loss</span> = <span class="cm-number">0</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">total_samples</span> = <span class="cm-variable">display_samples</span> = <span class="cm-variable">epoch</span> = <span class="cm-number">0</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">valid_loss_hist</span> = []</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">train_time</span> = <span class="cm-variable">begin_time</span> = <span class="cm-variable">time</span>.<span class="cm-property">time</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'Begin training...'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">while</span> <span class="cm-keyword">True</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">epoch</span> += <span class="cm-number">1</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">for</span> <span class="cm-variable">sents</span>, <span class="cm-variable">targets</span> <span class="cm-keyword">in</span> <span class="cm-variable">batch_iter</span>(<span class="cm-variable">df_train</span>, <span class="cm-variable">batch_size</span>=<span class="cm-variable">train_batch_size</span>, <span class="cm-variable">shuffle</span>=<span class="cm-keyword">False</span>, <span class="cm-variable">bert</span>=(<span class="cm-variable">args</span>[<span class="cm-string">'--bert-config'</span>])): <span class="cm-comment"># for each epoch</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">train_iter</span> += <span class="cm-number">1</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">batch_size</span> = <span class="cm-builtin">len</span>(<span class="cm-variable">sents</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">labels</span> = <span class="cm-variable">torch</span>.<span class="cm-property">tensor</span>(<span class="cm-variable">targets</span>, <span class="cm-variable">dtype</span>=<span class="cm-variable">torch</span>.<span class="cm-property">long</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">optimizer</span>.<span class="cm-property">zero_grad</span>() <span class="cm-comment">#restarting the grad accumulations between mini-batches</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">output</span> = <span class="cm-variable">model</span>(<span class="cm-variable">sents</span>) <span class="cm-comment">#pass through model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">loss</span> = <span class="cm-variable">criterion</span>(<span class="cm-variable">output</span>, <span class="cm-variable">labels</span>) <span class="cm-comment">#calculate loss</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">loss</span>.<span class="cm-property">backward</span>() <span class="cm-comment">#back prop</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">optimizer</span>.<span class="cm-property">step</span>() <span class="cm-comment">#update weights </span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">batch_losses_val</span> = <span class="cm-variable">loss</span>.<span class="cm-property">item</span>() <span class="cm-operator">*</span> <span class="cm-variable">batch_size</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">report_loss</span> += <span class="cm-variable">batch_losses_val</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">cum_loss</span> += <span class="cm-variable">batch_losses_val</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">display_samples</span> += <span class="cm-variable">batch_size</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">total_samples</span> += <span class="cm-variable">batch_size</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">if</span> <span class="cm-variable">train_iter</span> <span class="cm-operator">%</span> <span class="cm-variable">display_num</span> == <span class="cm-number">0</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'epoch %d, iter %d, avg. loss %.2f, '</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'total samples %d, speed %.2f samples/sec, '</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-string">'time elapsed %.2f sec'</span> <span class="cm-operator">%</span> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> (<span class="cm-variable">epoch</span>, <span class="cm-variable">train_iter</span>, <span class="cm-variable">report_loss</span> <span class="cm-operator">/</span> <span class="cm-variable">display_samples</span>,</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">total_samples</span>, <span class="cm-variable">display_samples</span> <span class="cm-operator">/</span> (<span class="cm-variable">time</span>.<span class="cm-property">time</span>() <span class="cm-operator">-</span> <span class="cm-variable">train_time</span>),</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">time</span>.<span class="cm-property">time</span>() <span class="cm-operator">-</span> <span class="cm-variable">begin_time</span>), <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">train_time</span> = <span class="cm-variable">time</span>.<span class="cm-property">time</span>()</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">report_loss</span> = <span class="cm-variable">display_samples</span> = <span class="cm-number">0.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># perform validation</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">if</span> <span class="cm-variable">train_iter</span> <span class="cm-operator">%</span> <span class="cm-variable">valid_niter</span> == <span class="cm-number">0</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'epoch %d, iter %d, cum. loss %.2f, cum. examples %d'</span> <span class="cm-operator">%</span> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> (<span class="cm-variable">epoch</span>, <span class="cm-variable">train_iter</span>, <span class="cm-variable">cum_loss</span> <span class="cm-operator">/</span> <span class="cm-variable">total_samples</span>, <span class="cm-variable">total_samples</span>), <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">cum_loss</span> = <span class="cm-variable">total_samples</span> = <span class="cm-number">0.</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'begin validation ...'</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">valid_loss</span> = <span class="cm-variable">validation</span>(<span class="cm-variable">model</span>, <span class="cm-variable">df_val</span>, <span class="cm-variable">criterion</span>, <span class="cm-variable">bert_size</span>=<span class="cm-variable">bert_size</span>) </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'validation: iter %d, loss %f'</span> <span class="cm-operator">%</span> (<span class="cm-variable">train_iter</span>, <span class="cm-variable">valid_loss</span>), <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> </span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span class="cm-comment"># scheduler.step(valid_loss)</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">improved_loss</span> = <span class="cm-builtin">len</span>(<span class="cm-variable">valid_loss_hist</span>)==<span class="cm-number">0</span> <span class="cm-keyword">or</span> <span class="cm-variable">valid_loss</span> <span class="cm-operator"><</span> <span class="cm-builtin">min</span>(<span class="cm-variable">valid_loss_hist</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">valid_loss_hist</span>.<span class="cm-property">append</span>(<span class="cm-variable">valid_loss</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">if</span> <span class="cm-variable">improved_loss</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">patience</span> = <span class="cm-number">0</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'save currently the best model to [%s]'</span> <span class="cm-operator">%</span> <span class="cm-variable">args</span>[<span class="cm-string">'--model'</span>]<span class="cm-operator">+</span><span class="cm-string">'_model.bin'</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span>.<span class="cm-property">save</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--model'</span>]<span class="cm-operator">+</span><span class="cm-string">'_model.bin'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># also save the optimizers' state</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">torch</span>.<span class="cm-property">save</span>(<span class="cm-variable">optimizer</span>.<span class="cm-property">state_dict</span>(), <span class="cm-variable">args</span>[<span class="cm-string">'--model'</span>] <span class="cm-operator">+</span> <span class="cm-string">'.optim'</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">else</span>: <span class="cm-comment">#if valid loss did not improve</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">patience</span> += <span class="cm-number">1</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'hit patience %d out of %d'</span> <span class="cm-operator">%</span> (<span class="cm-variable">patience</span>, <span class="cm-builtin">int</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--patience'</span>])), <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">if</span> <span class="cm-variable">patience</span> <span class="cm-operator">></span>= <span class="cm-builtin">int</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--patience'</span>]):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">num_restarts</span> += <span class="cm-number">1</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'hit #%d restart out of max %d restarts'</span> <span class="cm-operator">%</span> (<span class="cm-variable">num_restarts</span>, <span class="cm-builtin">int</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--max-num-trial'</span>])), <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">if</span> <span class="cm-variable">num_restarts</span> <span class="cm-operator">></span>= <span class="cm-builtin">int</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--max-num-trial'</span>]):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'early termination!'</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">exit</span>(<span class="cm-number">0</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># decay lr, and restore from previously best checkpoint</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">lr</span> = <span class="cm-variable">optimizer</span>.<span class="cm-property">param_groups</span>[<span class="cm-number">0</span>][<span class="cm-string">'lr'</span>] <span class="cm-operator">*</span> <span class="cm-builtin">float</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--lr-decay'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'load previously best model and decay learning rate to %f'</span> <span class="cm-operator">%</span> <span class="cm-variable">lr</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># load model</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">params</span> = <span class="cm-variable">torch</span>.<span class="cm-property">load</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--model'</span>], <span class="cm-variable">map_location</span>=<span class="cm-keyword">lambda</span> <span class="cm-variable">storage</span>, <span class="cm-variable">loc</span>: <span class="cm-variable">storage</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">model</span>.<span class="cm-property">load_state_dict</span>(<span class="cm-variable">params</span>[<span class="cm-string">'state_dict'</span>])</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'restore parameters of the optimizers'</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">optimizer</span>.<span class="cm-property">load_state_dict</span>(<span class="cm-variable">torch</span>.<span class="cm-property">load</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--model'</span>] <span class="cm-operator">+</span> <span class="cm-string">'.optim'</span>))</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># set new lr</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">for</span> <span class="cm-variable">param_group</span> <span class="cm-keyword">in</span> <span class="cm-variable">optimizer</span>.<span class="cm-property">param_groups</span>:</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">param_group</span>[<span class="cm-string">'lr'</span>] = <span class="cm-variable">lr</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-comment"># reset patience</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">patience</span> = <span class="cm-number">0</span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-keyword">if</span> <span class="cm-variable">epoch</span> == <span class="cm-builtin">int</span>(<span class="cm-variable">args</span>[<span class="cm-string">'--max-epoch'</span>]):</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-builtin">print</span>(<span class="cm-string">'reached maximum number of epochs!'</span>, <span class="cm-variable">file</span>=<span class="cm-variable">sys</span>.<span class="cm-property">stderr</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"> <span class="cm-variable">exit</span>(<span class="cm-number">0</span>)</span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre><pre class=" CodeMirror-line " role="presentation"><span role="presentation" style="padding-right: 0.1px;"><span cm-text=""></span></span></pre></div></div></div></div></div><div style="position: absolute; height: 0px; width: 1px; border-bottom: 0px solid transparent; top: 5040px;"></div><div class="CodeMirror-gutters" style="display: none; height: 5040px;"></div></div></div></pre><h2><a name='header-n396' class='md-header-anchor '></a>7. Results</h2><p>We examine the accuracy and the Matthews correlation coefficient of all the models. Both BERT models out perform the baseline model with GloVE embedding, as expected. The LSTM BERT model, however, has a lower accuracy and MCC score than the base BERT sequence classification model. </p><figure><table><thead><tr><th>Model</th><th>Accuracy</th><th>Matthews Correlation Coefficient</th></tr></thead><tbody><tr><td>Baseline</td><td>0.6323</td><td>0.5674</td></tr><tr><td>Base BERT</td><td>0.6948</td><td>0.6401</td></tr><tr><td>LSTM BERT</td><td>0.6853</td><td>0.6311</td></tr></tbody></table></figure><p>We give the confusion matrix below. All models have trouble properly classifying the "not related or not informative" class. The accuracy for that class is around 50%. </p><p><span> </span><strong>Baseline Model Confusion Matrix</strong></p><p><img src='test_normalized_confusion_matrix.png' alt='Baseline Model Confusion Matrix' referrerPolicy='no-referrer' /></p><p> </p><p><span> </span><strong>BERT Sequence Classification Confusion Matrix</strong></p><p><img src='default_bert_test_normalized_confusion_matrix.png' alt='' referrerPolicy='no-referrer' /></p><p><span> </span></p><p> </p><p><span> </span><strong>LSTM + BERT Confusion Matrix</strong></p><p><img src='LSTM_bert_test_normalized_confusion_matrix.png' alt='' referrerPolicy='no-referrer' /></p></div>
</body>
</html>