v0.13.1

This is a release after six months, and please be aware that this version includes relatively significant changes.

ATTENTION: For some reason, the iOS framework is no longer included in the unitypackage, so if you need the iOS library, please use the tarball version or build it by yourself.

Summary

• Upgraded MediaPipe to version v0.10.9
• Implemented Task API (FaceDetector, FaceLandmarker, HandLandmarker, PoseLandmarker)
• Status API has been made internal
• StatusOr<T> API has been removed
• Packet<T> is deprecated and non-generic Packet is introduced

Details

Features

• Upgraded MediaPipe to version v0.10.9
• Implemented Task API (FaceDetector, FaceLandmarker, HandLandmarker, PoseLandmarker)

Sample code can be found under Assets/MediaPipeUnity/Samples/Scenes/Tasks. Note that there's a known bug in HandLandmarker sample where handedness is reversed for Right/Left.

At this stage, the mechanism to share GlContext with MediaPipe in the Task API is not yet implemented. Therefore, it is necessary to copy the input image on the CPU, even on Android.

Breaking Changes

• Status API has been made internal
• StatusOr<T> API has been removed

Specifically, APIs that used to return Status will no longer do so (void), and APIs that used to return StatusOr<T> will now return the value directly. If the status is not OK, a BadStatusException will be thrown.
To migrate, please remove calls to Status.AssertOk and StatusOr<T>.Value. If you haven't checked Status before, it will now throw an exception. However, I think there should be no problem if the compilation step succeeds.

// graph.StartRun().AssertOk();
graph.StartRun(); // throws BadStatusException if the status is not OK

Background

As an excuse, let me provide some background information.

Originally, this plugin was created with the goal of allowing people familiar with the C++ MediaPipe API to use MediaPipe in C# with a similar syntax. Therefore, for APIs that return abseil::Status in C++, it was designed to return Status in C# as well (I would like to add that there was an intention to slack off from writing documentation by doing so).

However:

• In the latest MediaPipe, the documentation for the Python API is rather more comprehensive.
• Status probably exists due to Google's unique circumstances, not wanting to throw exceptions, and there is no rationale to return Status in C#.
• After all, calling AssertOk can throw an exception.
• Marshalling StatusOr is a cumbersome process.

Given these circumstances, at this timing, I decided to abolish them.
Status itself is still internally present as it is necessary for communicating with MediaPipe and handling errors.

Deprecation

As a forewarning for future breaking changes, Packet<T> has been deprecated.
Instead, please use a non-generic Packet.
This Packet is implemented to be similar to the interface of packet_creator and packet_getter in the Python API.

// before
var stringPacket = new StringPacket("Hello World!", new TimeStamp(0));
var intPacket = new IntPacket(0);
var value = stringPacket.Get();

// after
var stringPacket = Packet.CreateStringAt("Hello World!", 0);
var intPacket = Packet.CreateInt(0);
var value = stringPacket.GetString();

As a trade-off, because the data type inside Packet can no longer be statically determined, the OutputStream API is no longer able to return a value. Instead, an OutputStream API that returns Packet has been added (I apologize for the lack of documentation at the moment).

// before
var multiFaceLandmarksStream = new OutputStream<NormalizedLandmarkListVectorPacket, List<NormalizedLandmarkList>>(calculatorGraph, "multi_face_landmarks");
if (multiFaceLandmarksStream.TryGetNext(out var multiFaceLandmarks, false)) // may block the main thread
{
    // ...
}

// after
var multiFaceLandmarksStream = new OutputStream(calculatorGraph, "multi_face_landmarks");
var result = await multiFaceLandmarksStream.WaitNextAsync(); // won't block the main thread
if (result.packet != null)
{
    var multiFaceLandmarks = packet.GetProtoList(NormalizedLandmarkList.Parser);
    // ...
}

For usage, refer to the sample app code, etc.

Background

Here is another somewhat excuse-like background.

Packet<T> was created to somehow introduce type safety into Packet. For example, without type checking, you can call an API that retrieves a string when the data inside is an int (and this API call causes a crash on Windows). However, there were the following issues:

• No guarantee of consistency between C++ and C# types, so incorrect APIs could be still called
• Increased redundancy in specifying type parameters
• Some API calls Reflection API internally, and their calls are heavy

Especially the last issue was significant, and as the Task API required calling that API frequently, I migrated to a non-generic API.

Miscellaneous

Finally, there is no impact on the plugin itself, but the sample app's configuration method has been changed.
Previously, settings were in the Bootstrap Component but have now moved to AppSetting.asset.
assetLoaderType defaults to Local, and if not changed, it will result in an error when executed on a real device, so please be careful .

It's been lengthy, but those are the major changes.
Since there are many changes, it would be helpful if you could report any bugs you find.

The documentation is not up to date, but I will address it when the motivation arises.
For those who want to use the Task API, I have tried to write documentation comments, so please refer to them.

P.S. The absence of v0.13.0 is simply because it has been a while since the last release, and I made a mistake in the release process, so please don't worry about it.

Other CHANGELOG

Features

• build Image/ImageFrame from Texture2D (#1083) (bcce0e9)
• sample: drop non-blocking sync mode (#1110) (589b335)
• sample: remove Start Scene (#1017) (059b0b2)

Bug Fixes

• proper autofit image if camera is rotated (#1043) (2d2863e) (by @Dender46)

See CHANGELOG for more details.