Skip to content

Installation Guide

Jump to bottom
Junrou Nishida edited this page Jun 24, 2023 · 81 revisions

This repository does not contain required libraries (e.g. libmediapipe_c.so, Google.Protobuf.dll, etc), so you need to build them first.

⚠️ Libraries that can be built differ depending on your environment.

Supported Platforms

⚠️ GPU mode is not supported on macOS and Windows.

Editor Linux (x86_64) macOS (x86_64) macOS (ARM64) Windows (x86_64) Android iOS WebGL
Linux (AMD64) [^1] ✔️ ✔️ ✔️
Intel Mac ✔️ ✔️ ✔️ ✔️
M1 Mac [^2] ✔️ ✔️ ✔️ ✔️
Windows 10/11 (AMD64) [^3] ✔️ ✔️ ✔️

[^1]: Tested on Arch Linux.
[^2]: Experimental, because MediaPipe does not support M1 Mac.
[^3]: Running MediaPipe on Windows is experimental.

Prerequisites

If Docker is not available, the below commands/tools/libraries are required.

  • Python >= 3.9.0
  • Bazelisk (tested against Bazel 5.0.0)
  • GCC/G++ >= 8.0.0 (Linux, macOS)
  • NuGet (tested against 5.10.0.7240)

Please go to the article for each OS for more details.

🔔 Run commands at the project root if not specified otherwise.

Linux

⚠️ If the GNU libc version in the target machine is less than the version of it in the machine where libmediapipe_c.so (a native library for Linux) is built, libmediapipe_c.so won't work.
For the same reason, if your target machine's GNU libc version is less than 2.27, you cannot use Docker[^5].

[^5]: You can still use Docker, but you need to write Dockerfile by yourself.

Docker

  1. Install Docker

    Make sure you can run docker command without using sudo.

  2. Build a Docker image

    • Ubuntu 20.04 image

      docker build --build-arg UID=$(id -u) -t mediapipe_unity:latest . -f docker/linux/x86_64/Dockerfile

  3. Run a Docker container

    # Run with `Packages` directory mounted to the container
docker run \
    --mount type=bind,src=$PWD/Packages,dst=/home/mediapipe/Packages \
    --mount type=bind,src=$PWD/Assets,dst=/home/mediapipe/Assets \
    -it mediapipe_unity:latest

  4. Run build command inside the container

    # Build native libraries for Desktop CPU.
# Note that you need to specify `--opencv cmake` because OpenCV is not installed in the container.
python build.py build --desktop cpu --opencv cmake -v

# Build native libraries for Desktop GPU and Android
python build.py build --desktop gpu --android arm64 --opencv cmake -v

If the command finishes successfully, the required files will be installed on your host machine.

Arch Linux

If you are using another distribution, please replace some of the commands.

⚠️ If your GNU libc version in the target machine is compatible with it in the container, always prefer Docker.

  1. Install yay

    # Run under your favorite directory

pacman -S --needed git base-devel
git clone https://aur.archlinux.org/yay.git
cd yay
makepkg -si

  2. Install required packages

    yay -Sy unzip mesa npm

    It is recommended to configure npm here (cf. https://docs.npmjs.com/cli/v8/configuring-npm/npmrc#files).

    # ~/.npmrc
prefix = ${HOME}/.npm-packages
    # ~/.bash_profile
export PATH=${HOME}/.npm-packages/bin:${PATH}

  3. (Optional) If you'd like to link OpenCV dynamically, install OpenCV.

    Skip this step if you want to link OpenCV statically.

    yay -S opencv3-opt

    By default, it is assumed that OpenCV 3 is installed under /usr (e.g. /usr/lib/libopencv_core.so).
    opencv3-opt will install OpenCV 3 to /opt/opencv3, so you need to edit WORKSPACE.

    # WORKSPACE
new_local_repository(
    name = "linux_opencv",
    build_file = "@//third_party:opencv_linux.BUILD",
    path = "/opt/opencv3",
)

    🔔 If you'd like to use OpenCV 4, you need to edit third_party/opencv_linux.BUILD, too.

  4. Install Bazelisk and NuGet

    npm install -g bazelisk
yay -S nuget

  5. Install NumPy

    pip install numpy --user

  6. (For Android) Install Android SDK and Android NDK

  7. Run build command

macOS

Intel Mac

  1. Install Homebrew

  2. Install OpenCV 3.

    🔔 It's essentially an optional step, but if you'd like to build libraries for iOS, it's necessary because of a bug.

    brew install opencv@3
brew uninstall --ignore-dependencies glog

  3. Install Python and NumPy

    If your Python version is < 3.9.0, install it here in your favorite way.

    brew install python
export PATH=$PATH:"$(brew --prefix)/opt/python/libexec/bin"

# Python version must be >= 3.9.0
python --version
# Python 3.9.x

pip3 install --user six numpy

  4. Install Bazelisk and NuGet

    brew install bazelisk
brew install nuget

  5. Install Xcode using App Store

    After that, install the Command Line Tools, too.

    sudo xcodebuild -license # if not agreed to the license yet
sudo xcode-select -s /Applications/Xcode.app
xcode-select --install

  6. (For iOS) Open mediapipe_api/objc/BUILD and modify bundle_id.

  7. (For Android) Install Android SDK and Android NDK

  8. Run build command

M1 Mac

⚠️ use UnityEditor (Apple silicon) (>= 2021.2.2f1) to open the project.

  1. Install Homebrew

  2. Install OpenCV 3.

    🔔 It's essentially an optional step, but if you'd like to build libraries for iOS, it's necessary because of a bug.

    brew install opencv@3
brew uninstall --ignore-dependencies glog

  3. Install Python and NumPy

    If your Python version is < 3.9.0, install it here in your favorite way.

    brew install python
export PATH=$PATH:"$(brew --prefix)/opt/python/libexec/bin"

# Python version must be >= 3.9.0
python --version
# Python 3.9.x

pip3 install --user six numpy

  4. Install Bazelisk

    brew install bazelisk

  5. Install NuGet

    /usr/sbin/softwareupdate --install-rosetta

# Install Homebrew using Rosetta 2
# Before running the command, please check https://brew.sh/ for the correct URL
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install NuGet using Rosetta 2
arch -x86_64 /usr/local/Homebrew/bin/brew install nuget

    If you've not included /usr/local/Homebrew/bin in PATH, do it here.

    # If you're using zsh
# ~/.zprofile

export PATH=/usr/local/Homebrew/bin:$PATH

# NOTE: The below command must be executed after `PATH` is exported.
#       Otherwise, when you run `brew`, `/usr/local/Homebrew/bin/brew` (x86_64 version) will be run.
eval "$(/opt/homebrew/bin/brew shellenv)"

# Set Python path
export PATH=$PATH:"$(brew --prefix)/opt/python/libexec/bin"

  6. Install Xcode using App Store

    After that, install the Command Line Tools, too.

    sudo xcodebuild -license # if not agreed to the license yet
sudo xcode-select -s /Applications/Xcode.app
xcode-select --install

  7. (For iOS) Open mediapipe_api/objc/BUILD and modify bundle_id.

  8. (For Android) Install Android SDK and Android NDK

  9. Run build command

Windows

⚠️ You MUST NOT set core.autocrlf in .gitconfig (run git config -l). To build libraries successfully, you need to check out the source code as it is without changing the line feed code.

Docker Windows Container

⚠️ Hyper-V backend is required (that is, Windows 10 Home is not supported).

  1. Install Docker Desktop

  2. Switch to Windows Containers

  3. Build a Docker image

    docker build -t mediapipe_unity:windows . -f docker/windows/x86_64/Dockerfile

    This process will hang when MSYS2 is being installed.
    If this issue occurs, remove C:\ProgramData\Docker\tmp\hcs*\Files\$Recycle.Bin\ manually (hcs* is random name).
    cf. https://github.com/docker/for-win/issues/8910

  4. Run a Docker container

    Rem Run with `Packages` directory mounted to the container
Rem Specify `--cpus` and `--memory` options according to your machine.
docker run --cpus=16 --memory=32g ^
    --mount type=bind,src=%CD%\Packages,dst=C:\mediapipe\Packages ^
    --mount type=bind,src=%CD%\Assets,dst=C:\mediapipe\Assets ^
    -it mediapipe_unity:windows

  5. Run build command inside the container

    python build.py build --desktop cpu --opencv cmake -vv
Rem or if you'd like to link OpenCV dynamically
python build.py build --desktop cpu -vv

If the command finishes successfully, the required files will be installed on your host machine.

Docker Linux Container

⚠️ This can be used only to build libraries for Android and you cannot build libraries for Windows.

  1. Install Docker Desktop

  2. Switch to Linux containers

  3. Build a Docker image

    docker build -t mediapipe_unity:linux . -f docker/linux/x86_64/Dockerfile

  4. Run a Docker container

    Rem Run with `Packages` directory mounted to the container
Rem Specify `--cpus` and `--memory` options according to your machine.
docker run --cpus=16 --memory=16g ^
    --mount type=bind,src=%CD%\Packages,dst=/home/mediapipe/Packages ^
    --mount type=bind,src=%CD%\Assets,dst=/home/mediapipe/Assets ^
    -it mediapipe_unity:linux

  5. Run build command inside the container

    python build.py build --android arm64 -vv

If the command finishes successfully, the required files will be installed on your host machine.

Windows 10

⚠️ You cannot build libraries for Android with the following steps.
If you use Window 10 Pro, go to Docker Windows Container. Otherwise, go to Docker Linux Container.

⚠️ Run commands using cmd.exe. It's known that some commands do not work properly with MSYS2.

  1. Install MSYS2 and edit the %PATH% environment variable.

    If MSYS2 is installed to C:\msys64, add C:\msys64\usr\bin to your %PATH% environment variable.

  2. Install necessary packages

    pacman -S git patch unzip

  3. Install Python >= 3.9.0 and allow the executable to edit the %PATH% environment variable.

    Download Python Windows executable from https://www.python.org/downloads/windows/ and install.

  4. Install Visual C++ Build Tools 2019 and WinSDK

    1. Download Build Tools from https://visualstudio.microsoft.com/visual-cpp-build-tools/ and run Visual Studio Installer.

    2. Select Desktop development with C++ and install it.

    Visual Studio Installer

  5. Install Bazelisk and add the location of the executable to %PATH% environment variable.

    Note that you need to rename the executable to bazel.exe.

  6. (Optional) Set Bazel variables.

    If you have installed multiple Visual Studios or Win SDKs, set environment variables here.
    Learn more details about “Build on Windows” in the Bazel official documentation.

    Rem Please find the exact paths and version numbers from your local version.

set BAZEL_VS=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools
set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC
set BAZEL_VC_FULL_VERSION=<Your local VC version>
set BAZEL_WINSDK_FULL_VERSION=<Your local WinSDK version>

  7. (Optional) If you'd like to link OpenCV dynamically, install OpenCV.

    You can skip this step if you want to link OpenCV statically.

    🔔 It can take a very long time to link OpenCV statically because you need to build OpenCV on your local machine.
    For reference, it takes about 5 - 6 minutes on Ryzen 3900x (12-core, 24-thread).

    By default, it is assumed that OpenCV 3.4.16 is installed under C:\opencv.
    If your version or path is different, please edit third_party/opencv_windows.BUILD and WORKSPACE.

  8. Install NuGet, and add the location of the NuGet executable to the %PATH% environment variable.

    nuget

  9. Install NumPy

    pip install numpy --user

  10. Run build command

    python build.py build --desktop cpu --opencv=cmake -v
Rem or if you'd like to use local OpenCV
python build.py build --desktop cpu -v

Android Configuration

  1. Install Android Studio

  2. Install Android SDK and NDK

    Open SDK Manager > SDK Tools and install Android SDK Build Tools and NDK.

    🔔 Note the following 2 points:

    • Bazel will use the newest version of Build Tools found automatically, but the latest Bazel (4.2.1) does not support Build Tools >= 31.0.0, so you need to uncheck these versions.
    • Bazel does not support NDK >= r22 yet

    Android Studio (SDK Tools)

  3. Set environment variables

    • Linux/macOS

      # Set ANDROID_HOME
# This directory should contain directories such as `platforms` and `platform-tools`.
export ANDROID_HOME=/path/to/SDK

# Set ANDROID_NDK_HOME
# This is usually like `$ANDROID_HOME/ndk/21.4.7075529
export ANDROID_NDK_HOME=/path/to/NDK

    • Windows

      Set them using GUI

  4. Set API Level

    To support older Android, you need to specify the API level for NDK when running build.py.
    Otherwise, some symbols in libmediapipe_jni.so cannot be resolved and DllNotFoundException will be thrown at runtime.

    python build.py build --android arm64 --android_ndk_api_level 21 -vv

Build Script

build.py supports the following commands.

Command Description
build Build and install required files (libraries, model files, C# scripts)
clean Clean cache directories (build, bazel-*)
uninstall Remove install files

Build Command

Run python build.py build --help for more details.

# Build for Desktop with GPU enabled (Linux only).
python build.py build --desktop gpu -vv

# If you've not installed OpenCV locally, you need to build OpenCV from sources for Desktop.
python build.py build --desktop gpu --opencv cmake -vv

# Build for Desktop with GPU disabled.
python build.py build --desktop cpu -vv

# Build a Universal macOS Library (macOS only).
python build.py build --desktop cpu --opencv cmake --macos_universal -vv

# Build for Desktop, Android, and iOS
python build.py build --desktop cpu --android arm64 --ios arm64 -vv

# Specify Android NDK level
python build.py build --android arm64 --android_ndk_api_level 21 -vv

# Specify required solutions
python build.py build --desktop gpu --solutions face_mesh hands pose -vv

You can also specify compilation mode and linker options.

# Build with debug symbols.
python build.py build -c dbg --android arm64 -vv

# Omit all symbol information.
# This can significantly reduce the library size.
python build.py build --android arm64 --linkopt=-s -vv

Known Issues

  1. (iOS) When building native libraries for iOS, OpenCV must be installed on your Mac.

    To be precise, /usr/local/opt/opencv@3 (Intel Mac) or /opt/homebrew/opt/opencv@3 directory must exist (even if it's empty).

  2. (iOS) In some situations, the iOS Framework won't be updated.

    Due to an issue with bazel, we cannot determine the output path beforehand.
    If you have built this plugin across versions, you may encounter this problem.
    In that case, run python build.py clean and try your build command again.

Troubleshooting

DllNotFoundException

This error can occur for a variety of reasons, so it is necessary to isolate the cause.

  1. Native libraries are not built yet

    If native libraries (libmediapipe_c.{so,dylib,dll} / mediapipe_android.aar / MediaPipeUnity.Framework) are not built yet, this error can occur because Unity cannot load them.

    If they don't exist under Packages/com.github.homuler.mediapipe/Runtime/Plugins, run build command first, and make sure that the command finishes successfully.

  2. Native libraries are incompatible with your device

    1. Libraries built on Linux machines won't work on your Windows machine (see Supported Platforms).
    2. If it occurs on your Android device, maybe

  3. Dependent libraries are not linked

    This error occurs when some symbols in the libraries are undefined and cannot be resolved at runtime, which is very typical when OpenCV is not configured properly.
    See opencv_linux.BUILD / opencv_windows.BUILD and check if the path is correct (if not, edit the BUILD file).
    You can also build and link OpenCV statically with --opencv cmake option instead.

    Tips
    In this case, when you check on Load on startup and click the Apply button, error logs like the following will be output.

    Plugins: Couldn't open Packages/com.github.homuler.mediapipe/Runtime/Plugins/libmediapipe_c.so, error:  Packages/com.github.homuler.mediapipe/Runtime/Plugins/libmediapipe_c.so: undefined symbol: _ZN2cv8fastFreeEPv

  4. Dependent libraries do not exist or are not loaded

    When you build an app and copy it to another machine, you need to bundle dependent libraries with it.

    For example, when you build libmediapipe_c.so with --opencv local, OpenCV is dynamically linked to libmediapipe_c.so.
    To use this on another machine, OpenCV must be installed on the machine too.
    In this case, the recommended way is to build libmediapipe_c.so with --opencv cmake (or bundle OpenCV with your app).

    If you are unsure of the cause, try checking the dependent libraries using such as ldd command.

  5. When you cannot identify the cause...

    If the error occurs on UnityEditor, try loading {lib}mediapipe_c.{so,dylib,dll} on startup.

    load-on-startup

    DllNotFoundException should be thrown right after that, and if you're lucky, the error message will be more verbose now.

    Otherwise, try loading the library (on your target device) by yourself using tools or code like below.

    Android 
    public void LoadLibrary()
{
    using (var system = new AndroidJavaClass("java.lang.System"))
    {
        system.CallStatic("loadLibrary", "mediapipe_jni");
    }
}
    Windows 
    public void LoadLibrary()
{
#if UNITY_EDITOR_WIN
    var path = Path.Combine("Packages", "com.github.homuler.mediapipe", "Runtime", "Plugins", "libmediapipe_c.dll");
#elif UNITY_STANDALONE_WIN
    var path = Path.Combine(Application.dataPath, "Plugins", "libmediapipe_c.dll");
#endif

    var handle = LoadLibraryW(path);

    if (handle != IntPtr.Zero)
    {
        // Success
        if (!FreeLibrary(handle))
        {
            Debug.LogError($"Failed to unload {path}: {Marshal.GetLastWin32Error()}");
        }
    }
    else
    {
        // Error
        // cf. https://docs.microsoft.com/en-us/windows/win32/debug/system-error-codes--0-499-
        var errorCode = Marshal.GetLastWin32Error();
        Debug.LogError($"Failed to load {path}: {errorCode}");

        if (errorCode == 126)
        {
            Debug.LogError("Check missing dependencies using [Dependencies](https://github.com/lucasg/Dependencies). If you're sure that required libraries exist, open the plugin inspector for those libraries and check `Load on startup`.");
        }
    }   
}

[DllImport("kernel32", SetLastError = true, ExactSpelling = true, CharSet = CharSet.Unicode)]
private static extern IntPtr LoadLibraryW(string path);

[DllImport("kernel32", SetLastError = true, ExactSpelling = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool FreeLibrary(IntPtr handle);
    Linux/macOS 
    public void LoadLibrary()
{
#if UNITY_EDITOR_LINUX
    var path = Path.Combine("Packages", "com.github.homuler.mediapipe", "Runtime", "Plugins", "libmediapipe_c.so");
#elif UNITY_STANDALONE_LINUX
    var path = Path.Combine(Application.dataPath, "Plugins", "libmediapipe_c.so");
#elif UNITY_EDITOR_OSX
    var path = Path.Combine("Packages", "com.github.homuler.mediapipe", "Runtime", "Plugins", "libmediapipe_c.dylib");
#elif UNITY_STANDALONE_OSX
    var path = Path.Combine(Application.dataPath, "Plugins", "libmediapipe_c.dylib");
#endif

    var handle = dlopen(path, 2);

    if (handle != IntPtr.Zero)
    {
        // Success
        var result = dlclose(handle);

        if (result != 0)
        {
            Debug.LogError($"Failed to unload {path}");
        }
    }    
    else 
    {
        Debug.LogError($"Failed to load {path}: {Marshal.GetLastWin32Error()}");
        var error = Marshal.PtrToStringAnsi(dlerror());
        // TODO: release memory

        if (error != null)
        {
            Debug.LogError(error);
        }
    }
}

[DllImport("dl", SetLastError = true, ExactSpelling = true)]
private static extern IntPtr dlopen(string name, int flags);

[DllImport("dl", ExactSpelling = true)]
private static extern IntPtr dlerror();

[DllImport("dl", ExactSpelling = true)]
private static extern int dlclose(IntPtr handle);

/bin/bash: line 1: $'\r': command not found

Probably you've set core.autocrlf=true in .gitconfig (run git config -l).
Check out the source code as it is without changing the line feed code.

See https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration#_formatting_and_whitespace for more details.

ERROR: PKGBUILD contains CRLF characters and cannot be sourced.

See the above.

ERROR [internal] load metadata for mcr.microsoft.com/windows/servercore:ltsc2019

You're building a Windows Docker image using a Linux daemon.
You need to switch to Windows containers first or you may be trying to create a Windows image by mistake when you meant to create a Linux image.

See https://docs.docker.com/desktop/windows/#switch-between-windows-and-linux-containers for more details.

no matching toolchains found for types @bazel_tools//tools/android:sdk_toolchain_type

To build libraries for Android, you need to set ANDROID_HOME and ANDROID_NDK_HOME. See Android Configuration for more details.

java.io.IOException: ERROR: src/main/native/windows/process.cc(202): CreateProcessW("command" ...

command is often git.
This error occurs when you're building libraries on Windows and the command path is not resolved.
Make sure the path to the desired command is set in your %PATH%.

java.io.IOException: Error downloading [...]: Checksum was xxx but wanted yyy

If you changed some URLs in the WORKSPACE file, you may have forgotten to change the corresponding sha256sum value.
Otherwise, the contents may have been tampered with.

cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'ios_arm64'

This error means bazel could not find Xcode on your machine.
If you've installed Xcode, check its path.

xcode-select --path

If the output is like /Library/Developer/CommandLineTools, then run the following command and try again.

# Change the path to suit your environment
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
python3 build.py clean

Undefined symbols for architecture arm64: "_glReadPixels"

Add OpenGLES to the Framework Dependencies.
See https://github.com/homuler/MediaPipeUnityPlugin/issues/385#issuecomment-997788547 for more details.

AttributeError: module 'argparse' has no attribute 'BooleanOptionalAction'

Install Python >= 3.9.0

SyntaxError: invalid syntax

Install Python >= 3.9.0

Clone this wiki locally