v0.14.4

Features

• Upgrade MediaPipe to v0.10.14 in #1200

Bug Fixes

• NullReferenceException occurs when rendering multiple face landmarks by @Eunji-new in #1175

Build Systems

• Update the Unity version (2022.3.34f1) in #1202

Full Changelog: v0.14.3...v0.14.4

v0.14.3

Temporarily, the Android library is built only for arm64 (armv7 is excluded). This is due to the convenience of building the newly added AudioClassifier. For those who want to build for armv7, please refer to #1143 (comment).
If you need a fat aar, either build individually for each platform or exclude the AudioClassifier from the build (e.g. python build.py build --android arm64 --android_fat_apk_cpu=armeabi-v7a,arm64-v8a --solutions=face_mesh).

Features

• implement AudioClassifier (#1137) (3753fdf)
• implement Packet.At (#1136) (d397429)

Bug Fixes

• memory leaks when the PoseLandmarker runs with segmentation mask enabled (#1166) (019a655)
• sample: the sync mode fails when trying to get the ImageFrame result (#1142) (b6961e5)

ATTENTION: For some reason, the iOS framework is no longer included in the unitypackage, so if you need the iOS library, please use the tarball version or build it by yourself using MediaPipeUnityPlugin-all-stripped.zip or MediaPipeUnityPlugin-all.zip, which contain the iOS library.

v0.14.1

Bug Fixes

• PoseLandmarker returns only one pose if numPoses > 1 (#1125) (73554f2)
• sample: Android build error (#1127) (77183c6)

ATTENTION: For some reason, the iOS framework is no longer included in the unitypackage, so if you need the iOS library, please use the tarball version or build it by yourself using MediaPipeUnityPlugin-all-stripped.zip or MediaPipeUnityPlugin-all.zip, which contain the iOS library.

v0.14.0

Yesterday, I declared the deprecation of the generic Packet, but embarrassingly, I'm going to resurrect it. Well, to be precise, I will rather make the Packet added in the previous release generic (Packet<TValue>). This will allow us to regain the type safety sacrificed in the previous release.

Since I'm removing the APIs deprecated in v0.13.1, transitioning from v0.12.0 will involve more breaking changes. Here's how the syntax will differ:

// v0.12.0
StringPacket stringPacket = new StringPacket("Hello World!", new TimeStamp(0));
var value = stringPacket.Get();
stringPacket.GetInt(); // compile error

var intPacket = new IntPacket();

// v0.13.1
Packet stringPacket = Packet.CreateStringAt("Hello World!", 0);
var value = stringPacket.GetString();
stringPacket.GetInt(); // runtime error

var intPacket = new Packet();

// v0.14.0
Packet<string> stringPacket = Packet.CreateStringAt("Hello World!", 0);
var value = stringPacket.Get();
stringPacket.GetInt(); // compile error

var intPacket = new Packet<int>();

ATTENTION: For some reason, the iOS framework is no longer included in the unitypackage, so if you need the iOS library, please use the tarball version or build it by yourself. MediaPipeUnityPlugin-all-stripped.zip also contains the iOS library.

⚠️ BREAKING CHANGES

• strongly-typed Packet, once again (#1120)
• remove deprecated APIs (#1119)

See CHANGELOG for more details.

v0.13.1

This is a release after six months, and please be aware that this version includes relatively significant changes.

ATTENTION: For some reason, the iOS framework is no longer included in the unitypackage, so if you need the iOS library, please use the tarball version or build it by yourself.

Summary

• Upgraded MediaPipe to version v0.10.9
• Implemented Task API (FaceDetector, FaceLandmarker, HandLandmarker, PoseLandmarker)
• Status API has been made internal
• StatusOr<T> API has been removed
• Packet<T> is deprecated and non-generic Packet is introduced

Details

Features

• Upgraded MediaPipe to version v0.10.9
• Implemented Task API (FaceDetector, FaceLandmarker, HandLandmarker, PoseLandmarker)

Sample code can be found under Assets/MediaPipeUnity/Samples/Scenes/Tasks. Note that there's a known bug in HandLandmarker sample where handedness is reversed for Right/Left.

At this stage, the mechanism to share GlContext with MediaPipe in the Task API is not yet implemented. Therefore, it is necessary to copy the input image on the CPU, even on Android.

Breaking Changes

• Status API has been made internal
• StatusOr<T> API has been removed

Specifically, APIs that used to return Status will no longer do so (void), and APIs that used to return StatusOr<T> will now return the value directly. If the status is not OK, a BadStatusException will be thrown.
To migrate, please remove calls to Status.AssertOk and StatusOr<T>.Value. If you haven't checked Status before, it will now throw an exception. However, I think there should be no problem if the compilation step succeeds.

// graph.StartRun().AssertOk();
graph.StartRun(); // throws BadStatusException if the status is not OK

Background

As an excuse, let me provide some background information.

Originally, this plugin was created with the goal of allowing people familiar with the C++ MediaPipe API to use MediaPipe in C# with a similar syntax. Therefore, for APIs that return abseil::Status in C++, it was designed to return Status in C# as well (I would like to add that there was an intention to slack off from writing documentation by doing so).

However:

• In the latest MediaPipe, the documentation for the Python API is rather more comprehensive.
• Status probably exists due to Google's unique circumstances, not wanting to throw exceptions, and there is no rationale to return Status in C#.
• After all, calling AssertOk can throw an exception.
• Marshalling StatusOr is a cumbersome process.

Given these circumstances, at this timing, I decided to abolish them.
Status itself is still internally present as it is necessary for communicating with MediaPipe and handling errors.

Deprecation

As a forewarning for future breaking changes, Packet<T> has been deprecated.
Instead, please use a non-generic Packet.
This Packet is implemented to be similar to the interface of packet_creator and packet_getter in the Python API.

// before
var stringPacket = new StringPacket("Hello World!", new TimeStamp(0));
var intPacket = new IntPacket(0);
var value = stringPacket.Get();

// after
var stringPacket = Packet.CreateStringAt("Hello World!", 0);
var intPacket = Packet.CreateInt(0);
var value = stringPacket.GetString();

As a trade-off, because the data type inside Packet can no longer be statically determined, the OutputStream API is no longer able to return a value. Instead, an OutputStream API that returns Packet has been added (I apologize for the lack of documentation at the moment).

// before
var multiFaceLandmarksStream = new OutputStream<NormalizedLandmarkListVectorPacket, List<NormalizedLandmarkList>>(calculatorGraph, "multi_face_landmarks");
if (multiFaceLandmarksStream.TryGetNext(out var multiFaceLandmarks, false)) // may block the main thread
{
    // ...
}

// after
var multiFaceLandmarksStream = new OutputStream(calculatorGraph, "multi_face_landmarks");
var result = await multiFaceLandmarksStream.WaitNextAsync(); // won't block the main thread
if (result.packet != null)
{
    var multiFaceLandmarks = packet.GetProtoList(NormalizedLandmarkList.Parser);
    // ...
}

For usage, refer to the sample app code, etc.

Background

Here is another somewhat excuse-like background.

Packet<T> was created to somehow introduce type safety into Packet. For example, without type checking, you can call an API that retrieves a string when the data inside is an int (and this API call causes a crash on Windows). However, there were the following issues:

• No guarantee of consistency between C++ and C# types, so incorrect APIs could be still called
• Increased redundancy in specifying type parameters
• Some API calls Reflection API internally, and their calls are heavy

Especially the last issue was significant, and as the Task API required calling that API frequently, I migrated to a non-generic API.

Miscellaneous

Finally, there is no impact on the plugin itself, but the sample app's configuration method has been changed.
Previously, settings were in the Bootstrap Component but have now moved to AppSetting.asset.
assetLoaderType defaults to Local, and if not changed, it will result in an error when executed on a real device, so please be careful .

It's been lengthy, but those are the major changes.
Since there are many changes, it would be helpful if you could report any bugs you find.

The documentation is not up to date, but I will address it when the motivation arises.
For those who want to use the Task API, I have tried to write documentation comments, so please refer to them.

P.S. The absence of v0.13.0 is simply because it has been a while since the last release, and I made a mistake in the release process, so please don't worry about it.

Other CHANGELOG

Features

• build Image/ImageFrame from Texture2D (#1083) (bcce0e9)
• sample: drop non-blocking sync mode (#1110) (589b335)
• sample: remove Start Scene (#1017) (059b0b2)

Bug Fixes

• proper autofit image if camera is rotated (#1043) (2d2863e) (by @Dender46)

See CHANGELOG for more details.

v0.12.0

Starting from this version, the symbols of the libraries included in the tarball and unitypackage have been stripped.
The libraries included in the MediaPipeUnityPlugin-all.zip still contain all symbols as before.

Please note that this version includes an upgrade to MediaPipe, but it does not incorporate the new features of MediaPipe.

⚠️ BREAKING CHANGES

• MediaPipe 0.10.1 (#924)
  • drop support for BoxTracking, Instant Motion Tracking and Objectron
• The minimum GLIBC version has changed (2.27 -> 2.31) (#921)
  • drop support for Ubuntu 18.04 (#922)

Bug Fixes

• include MediaPipeUnity.framework in unitypackage (#882)

Build

• remove the cmake_dynamic option (#932)

See CHANGELOG for more details.

v0.11.0

⚠️ BREAKING CHANGES

• make ResourceManager more customizable (#834) (1eac8b1)
  • hide StdString from developers

Features

• plugin: MediaPipe v0.9.1 (#840) (1a85630)

Build

• Upgrade the Unity version to 2021.3.18f1 (#856) (7679fad)

v0.10.3

From this version, MediaPipeUnityPlugin-all.zip, which contains all the source code and required libraries, is distributed.
That means you don't need to clone this repository and build libraries on your local machine to run the sample app.
(I deleted v0.10.2 because there was a mistake.)

Features

• Add FloatVectorPacket and MatrixPacket (#767) (391d7d9) (by @mgarbade)
• Make mask color transparent (#733) (f196f46)

Bug Fixes

• Wait longer time for the WebCam to start (#668) (436edb0) (by @DonghwanKIM0101)
• Avoid drawing holistic landmarks twice (#755) (30cd149) (by @Mids)
• Ignore core.autocrlf (#807) (f2bc51b)
• Stop running OnValidate on immutable prefabs (#810) (f545346)

Build

• The package layout has changed (#812) (4cd3a54)
• Export all the project files (#811) (a4c7077)

v0.10.1

In this version, pre-built packages are distributed for the first time (to know how it's built, see https://github.com/homuler/MediaPipeUnityPlugin/blob/v0.10.1/.github/workflows/package.yml).
GitHub Actions workflows are also set up, so once you fork this repository, I think you can build a package with your favorite options.

Features

objectron: Set confidence parameters from the UI (#605)

Bug Fixes

• docker build freezes when building a Docker Windows image (#631)
• OpenCV paths are wrong on macOS (#592)
• Some tests fail when built with GPU disabled (#634)
• Some tests fail on Windows (#644)

Build

• Upgrade the Bazel version to 5.2.0 (#623)
  • Now build fails with bazel 5.0.0
• Select only required solutions when building libraries (#606)
• Build a universal macOS library when --macos_universal is set (#625, #646)
• Build release packages on GitHub Actions (#647)

v0.10.0

I have been writing tutorials for the past few weeks and have finally finished the basic parts.
This version includes scenes that can be used as templates, so I hope it's easier for more people to try this plugin.
If you find any errors in the tutorial, I would appreciate it if you could let me know.

I also have found several bugs and difficulties in using the plugin, which are fixed in this version.

⚠️ BREAKING CHANGES

• Upgrade the Unity version to 2021.3.3f1 (#588)
• plugin: Logger.minLogLevel -> Logger.MinLogLevel (#580)
• NativePacketCallback and NativeGlStatusFunction return StatusArgs instead of IntPtr (#574)
• Change CoordinateSystem method names and I/Fs (#569)
  • e.g. ImageCoordinate#GetLocalPosition -> ImageCoordinate#ImageToLocalPoint
  • The first argument type is now UnityEngine.Rect, not UnityEngine.RectTransform

Features

• Add scenes for tutorial (#582)
• plugin: MediaPipe v0.8.10 (#585)
• plugin: Implement GpuManager (#579)
• plugin: Add missing Status factory methods (#573)

Bug Fixes

• plugin: OutputStream#TryGetNext throws if StartPolling is not called (#581)
• Some tests abort on Windows (#572)
• Status instances returned by callback functions can be GCed prematurely (#574)
• TransformAnnotation value in the Objectron sample is incorrect (#569)