-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
103 lines (102 loc) · 10.9 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
<!DOCTYPE html>
<!-- This file was auto-generated by exmd at 2023-04-03T09:47:22.679Z. Do NOT edit by hand! -->
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>🧪 WebR + Pyodide and Emscripten's Filesystem</title><meta property="og:title" content="🧪 WebR + Pyodide and Emscripten's Filesystem">
<meta property="twitter:title" content="🧪 WebR + Pyodide and Emscripten's Filesystem">
<meta name="description" content="Sharing Data Between R and Python Contexts">
<meta property="og:description" content="Sharing Data Between R and Python Contexts">
<meta property="twitter:description" content="Sharing Data Between R and Python Contexts">
<meta property="og:site" content="https://rud.is/w/webr-pyodide-fs">
<meta property="og:site_name" content="WebR Exeriments">
<meta property="og:image" content="https://rud.is/w/webr-pyodide-fs/preview.png">
<meta property="twitter:image" content="https://rud.is/w/webr-pyodide-fs/preview.png">
<meta property="og:image:width" content="1774">
<meta property="og:image:height" content="1256">
<meta property="og:image:alt" content="example">
<meta property="twitter:site_name" content="@hrbrmstr">
<meta property="twitter:domain" content="rud.is">
<meta property="twitter:card" content="summary_large_image">
<meta property="article:published_time" content="2023-04-03T09:47:22.680Z">
<link rel='apple-touch-icon' sizes='180x180' href='./favicon/apple-touch-icon.png'/>
<link rel='icon' type='image/png' sizes='32x32' href='./favicon/favicon-32x32.png'/>
<link rel='icon' type='image/png' sizes='16x16' href='./favicon/favicon-16x16.png'/>
<link rel='manifest' href='./favicon/site.webmanifest'/>
<link href='./src/index.css' rel='stylesheet'/>
<link href='./src/components.css' rel='stylesheet'/>
<script type='module' src='./src/main.js'></script>
</head>
<body>
<h1>🧪 WebR + Pyodide and Emscripten's Filesystem</h1>
<p><status-message id="webr-status" text="WebR Loading…"></status-message></p>
<h2>Can We Share Data Between R and Python Contexts Via Emscripten's Filesystem?</h2>
<p>Experiment hypothesis: It is not possible to share virtual filesystems across multiple WebR or Pyodide contexts.</p>
<p>Experiment parameters:</p>
<ul>
<li>Webr</li>
<li>Pyodide</li>
<li>Emscripten's Filesystem</li>
<li><code>R</code> template tag function</li>
<li><span class="pill">New!</span> <code>Python</code> template tag function</li>
<li>Lit (web components)</li>
<li>Vite (for building)</li>
</ul>
<h2>Emscripten All The Things</h2>
<p>If you're going to play in WebR, Pyodide, or other, similar WASM'd things, you're going to hear/see the word "<a href="https://emscripten.org/">Emscripten</a>" because that's the magic that makes it possible to turn native beasties into something that can be run in a WASM context.</p>
<p>Emscripten is an open-source compiler toolchain that allows you to compile C and C++ code — or any language that uses LLVM (Low Level Virtual Machine) — to WebAssembly or <code>asm.js</code>, which can be run in web browsers or other JavaScript contexts. The Emscripten toolchain is based on said LLVM, which is a popular compiler infrastructure project, and provides a way to port existing C/C++ code to the web without having to rewrite it in JavaScript. I mention LLVM <a href="https://dailyfinds.hrbrmstr.dev/archive?sort=search&search=llvm">quite a bit in my Daily Drops</a>.</p>
<p>More to the point of today's WebR experiment, Emscripten also provides a virtual file system that allows WASM-ified things to access files and directories in the browser or in other JavaScript contexts. This file system can be used to load and store data from the browser's local storage, to read and write files to the browser's <a href="https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API">IndexedDB</a>, and to access files that are part of the compiled code itself (such as assets and configuration files).</p>
<p><a href="https://emscripten.org/docs/api_reference/Filesystem-API.html?highlight=web_user">By default</a>, Emscrpiten creates <code>/home/web_user</code> and <code>/tmp</code>, so they <em>would</em> be common directories, if there was a single shared Emscripten in-browser filesystem.</p>
<h2>Experiment Results</h2>
<p>Given that browser WASM contexts are <a href="https://webassembly.org/docs/security/">sandboxed</a>, I figured that each got its own Emscripten filesystem. It makes sense that they wouldn't the same, single Emscripten filesystem, but it's also easy to prove one way or another.</p>
<p>Some new bits to check out in the code:</p>
<ul>
<li>This <code>py.js</code> now has a Python tag function similar to the one I made for R</li>
<li>This <code>r.js</code> now has <code>webR2</code>, a second, separate instantiation of WebR, so we can prove separate WebR contexts can't access each other</li>
</ul>
<p>Below, there are three output blocks, they contain a recursive directory listing of <code>/home</code> in each context.</p>
<p>To have something to list, we "touch" some files:</p>
<pre class="shiki " style="background-color: #0b0e14" tabindex="0"><code><span class="line"><span style="color: #ACB6BF8C; font-style: italic">// "touch" two files in Python. Pyodide uses `pyodide` as the home dir, but Emscripten's</span></span>
<span class="line"><span style="color: #ACB6BF8C; font-style: italic">// default `/home/web_user` is also there. R uses that default, so if this filesystem was</span></span>
<span class="line"><span style="color: #ACB6BF8C; font-style: italic">// shared, anything we create in it here would show up there.</span></span>
<span class="line"><span style="color: #FF8F40">await</span><span style="color: #BFBDB6"> </span><span style="color: #FFB454">Python</span><span style="color: #AAD94C">`os.close(os.open("/home/web_user/py-new-file-web-user", os.O_CREAT))`</span></span>
<span class="line"><span style="color: #FF8F40">await</span><span style="color: #BFBDB6"> </span><span style="color: #FFB454">Python</span><span style="color: #AAD94C">`os.close(os.open("/home/pyodide/py-new-file-pyodide", os.O_CREAT))`</span></span>
<span class="line"></span>
<span class="line"><span style="color: #ACB6BF8C; font-style: italic">// "touch" a file in R; again, if shared, Python would see this b/c it has `/home/web_user`</span></span>
<span class="line"><span style="color: #FF8F40">await</span><span style="color: #BFBDB6"> </span><span style="color: #FFB454">R</span><span style="color: #AAD94C">`writeLines("", "/home/web_user/r-new-file-web-user")`</span></span>
<span class="line"></span>
<span class="line"><span style="color: #ACB6BF8C; font-style: italic">// list files/dirs (recursively) in python</span></span>
<span class="line"><span style="color: #FF8F40">const</span><span style="color: #BFBDB6"> pyFiles </span><span style="color: #F29668">=</span><span style="color: #BFBDB6"> </span><span style="color: #FF8F40">await</span><span style="color: #BFBDB6"> </span><span style="color: #FFB454">Python</span><span style="color: #AAD94C">`glob.glob("/home/**", recursive = True)`</span></span>
<span class="line"></span>
<span class="line"><span style="color: #ACB6BF8C; font-style: italic">// notice how so much better R is than Python since you 100% semantically know</span></span>
<span class="line"><span style="color: #ACB6BF8C; font-style: italic">// what this call to `list.files()` is doing. 🙃</span></span>
<span class="line"><span style="color: #FF8F40">const</span><span style="color: #BFBDB6"> rFiles </span><span style="color: #F29668">=</span><span style="color: #BFBDB6"> </span><span style="color: #FF8F40">await</span><span style="color: #BFBDB6"> </span><span style="color: #FFB454">R</span><span style="color: #AAD94C">`list.files("/home", full.names=TRUE, include.dirs=TRUE, recursive=TRUE)`</span></span>
<span class="line"><span style="color: #FF8F40">const</span><span style="color: #BFBDB6"> r2Files </span><span style="color: #F29668">=</span><span style="color: #BFBDB6"> </span><span style="color: #FF8F40">await</span><span style="color: #BFBDB6"> (</span><span style="color: #FF8F40">await</span><span style="color: #BFBDB6"> </span><span style="color: #BFBDB6">webR2</span><span style="color: #F29668">.</span><span style="color: #FFB454">evalR</span><span style="color: #AAD94C">`list.files("/home", full.names=TRUE, include.dirs=TRUE, recursive=TRUE)`</span><span style="color: #BFBDB6">)</span><span style="color: #F29668">.</span><span style="color: #FFB454">toJs</span><span style="color: #BFBDB6">()</span></span>
<span class="line"></span></code></pre>
<p>And, the results are in!</p>
<p><strong>WebR's <code>/home</code></strong></p>
<pre class="shiki" id="r-fs"></pre>
<p><strong>Second instance of WebR's <code>/home</code></strong></p>
<pre class="shiki" id="r2-fs"></pre>
<p><strong>Pyodide's <code>/home</code></strong></p>
<pre class="shiki" id="py-fs"></pre>
<h2>Hack To Use WebR's "Data" In Pyodide</h2>
<p>I don't <em>think</em> there's (yet) an equivalent of Pyodide's <a href="https://pyodide.org/en/stable/usage/quickstart.html#accessing-javascript-scope-from-python">ability to access its parent JavaScript context from within Pyodide</a>. If I'm right about that, fear not, since it's almost a given that there will be an equivalent as WebR matures.</p>
<p>For now, you can hacky-access R's data from Pyodide this way:</p>
<pre class="shiki " style="background-color: #0b0e14" tabindex="0"><code><span class="line"><span style="color: #FF8F40">await</span><span style="color: #BFBDB6"> </span><span style="color: #FFB454">R</span><span style="color: #AAD94C">`hi <- "Hi from R!"`</span></span>
<span class="line"></span>
<span class="line"><span style="color: #BFBDB6">console</span><span style="color: #F29668">.</span><span style="color: #FFB454">log</span><span style="color: #BFBDB6">(</span><span style="color: #FF8F40">await</span><span style="color: #BFBDB6"> webPy</span><span style="color: #F29668">.</span><span style="color: #FFB454">runPythonAsync</span><span style="color: #BFBDB6">(</span><span style="color: #AAD94C">`</span></span>
<span class="line"><span style="color: #AAD94C">import js</span></span>
<span class="line"><span style="color: #AAD94C">res = await js.R("hi")</span></span>
<span class="line"><span style="color: #AAD94C">res + " Back at ya, from Python!"</span></span>
<span class="line"><span style="color: #AAD94C">`</span><span style="color: #BFBDB6">))</span></span>
<span class="line"></span></code></pre>
<p>Open up your browser's Developer Tools console to see that the above does, indeed, work.</p>
<h2>FIN</h2>
<p>The results were unsurprising, but hopefully this will save others some time down the road.</p>
<p>You can find the source for this experiment <a href="https://github.com/hrbrmstr/webr-pyodide-fs">on GitHub</a>.</p>
<p style="text-align: center">Brought to you by @hrbrmstr</p>
<!-- extra body bits -->
</body>
</html>