Skip to content

Latest commit

 

History

History
17 lines (15 loc) · 1.35 KB

spark-fsharp-mobius.md

File metadata and controls

17 lines (15 loc) · 1.35 KB
Raw

Implementing Spark Apps in F# using Mobius

Non-Interactive Apps

  1. Develop your application in a F# IDE using Mobius API. Refer to F# examples for sample code
  2. Use sparkclr-submit.cmd to run your Mobius-based Spark application implemented in F#

Interactive Apps

Using F# Interactive (fsi.exe)

  1. Run sparkclr-submit.cmd debug in a command prompt after setting necessary environment variables. Note that this debug parameter is a misnomer in this context and this command initializes .NET-JVM bridge similiar to running Mobius apps in debug mode.
  2. In Developer Command Prompt for VS, run fsi.exe --use:c:\temp\mobius-init.fsx. mobius-init.fsx has the initialization code that can be used to create SparkContext, SqlContext or SparkSession. You need to update the location of Mobius binaries referenced in the beginning of the script file. You may also need to update other configuration settings in the script.
  3. When the F# command prompt is available, Spark functionality can be invoked using Mobius API. For example, the following code can be used process JSON file.
let dataframe = sparkSession.Read().Json @"C:\temp\data.json";;
dataframe.Show();;
dataframe.ShowSchema();;
dataframe.Count();;