matrix classification via tflite model #656

mgarbade · 2022-07-15T11:54:17Z

example usage of tflite graph from Unity
accepts inputs other than image
returns a list of floats representing tflite classification results
introduces new
- input packet class: MatrixFramePacket
- output packet class: FloatVectorFramePacket
  -> the name "Frame" should resemble "ImageFramePacket" and should help to avoid confusions with similar sounding classes like
matrix.cc
FloatArrayPacket.cs

Missing:

not tested on Android
no tests written
unsure function naming in "matrix_frame.cc" and "float_vector_frame.cc"
when compiling particular solutions "TfliteConverterCalculator" seems to be missing

- "matrix_frame" as name in order to avoid confusion with matrix.cc - matrix_frame is a 2D input data modality that gets converted to Eigen::MatrixXf internally - suited for non-image input to tflite models

- "float_vector_frame" as name in order to avoid confusion with FloatArrayPacket - float_vector_frame is a 1D output data modality that accepts std::vector<float> outputs from a mediapipe / tflite graph - suited for tflite classification results packaged as vector of floats

- send MatrixData to C++ as byte array

- return std::vector<float> from C++ to Unity as List<float>

- driver code for the newly added MatrixFramePacket and FloatVectorFramePacket - feeds an example matrix of size [ 2 x 3 ] into a mediapipe graph - the graph runs a simple tflite model (adds +1 to every input) - then the graph returns the result back to Unity as List<float> - only tested on Unity-Editor-Mode on Windows 10 Pro

mgarbade · 2022-07-15T13:13:57Z

just testing around a bit (didn't excessively test after rebasing onto current master). I guess I somehow lost the

        "//mediapipe/calculators/tflite:tflite_converter_calculator",
        "//mediapipe/calculators/tflite:tflite_inference_calculator",
        "//mediapipe/calculators/tflite:tflite_tensors_to_floats_calculator",

dependency needed for the example graph somewhere along the way.
Will look into this...

mgarbade · 2022-07-15T13:20:18Z

ok so the problem was related to building only the pose solution

python build.py build --desktop cpu --opencv=cmake -v --verbose -c dbg --solution pose

it works when I build the full solution ( omit the using --solution pose option)

homuler

Thank you!
Will you add tests for MatrixPacket and FrameVectorPacket?
Please check out https://github.com/homuler/MediaPipeUnityPlugin/blob/6efa5e1b75b7c6aa487254ade15fe7cd49fc6842/Packages/com.github.homuler.mediapipe/Tests/EditMode/Framework/Packet/FloatArrayPacketTest.cs for more details.

homuler · 2022-07-16T00:09:47Z

Assets/MediaPipeUnity/Samples/Scenes/Matrix Classification/MatrixClassification.cs

@@ -0,0 +1,200 @@
+// TODO:


Sorry, since our policy is to add sample code only for official solutions, can you please remove this scene (files under Assets/MediaPipeUnity/Samples)?

Could move it to Assets/MediaPipeUnity/Tutorial !? I'm not sure where else to put it. I assume MatrixClassification.cs is quite illustrative in showcasing how to use

matrix data as input
-> tflite model
-> receive float vector as output 🤔

since our policy is to add sample code only for official solutions

I don't want to maintain this sample scene, so please remove the related files.

...ages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/FloatVectorFramePacket.cs

homuler · 2022-07-16T00:18:31Z

...ages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/FloatVectorFramePacket.cs

+    /// </summary>
+    /// 
+
+    private int _vectorLength;


I think we should not store the length of the internal std::vector since its length is variable.

I had to introduce this variable in order to make the Get () method working in the same class

I borrowed the structure from "FloatArrayPacket.cs" which also has a "lenght" variable for the same reason apparantly

should not be a problem as tflite models don't allow for "dynamic output sizes" -> a single tflite model will always have a fixed size output (vector)

homuler · 2022-07-16T00:22:26Z

...ages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/FloatVectorFramePacket.cs

+    public FloatVectorFramePacket(float[] value) : base()
+    {
+      UnsafeNativeMethods.mp__MakeFloatVectorFramePacket__PA_i(value, value.Length, out var ptr).Assert();
+      this.ptr = ptr;
+      _vectorLength = value.Length;
+    }
+
+    public FloatVectorFramePacket(float[] value, Timestamp timestamp) : base()
+    {
+      UnsafeNativeMethods.mp__MakeFloatVectorFramePacket_At__PA_i_Rt(value, value.Length, timestamp.mpPtr, out var ptr).Assert();
+      GC.KeepAlive(timestamp);
+      this.ptr = ptr;
+    }
+
+    public FloatVectorFramePacket At(Timestamp timestamp)
+    {
+      return At<FloatVectorFramePacket>(timestamp);
+    }


I think these constructors are unnecessary for now since FloatVectorPacket is only used to receive output from a CalculatorGraph.

I didn't have the constructors in the beginning until I realized that I needed them

Probably due to the fact that one needs to create a mediapipe packet in order to give it to the OutputStreamPoller (at least that's where those constructor are being called)

...ages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/FloatVectorFramePacket.cs

...muler.mediapipe/Runtime/Scripts/PInvoke/NativeMethods/Framework/Format/MatrixFrame_Unsafe.cs

homuler · 2022-07-16T01:19:15Z

...muler.mediapipe/Runtime/Scripts/PInvoke/NativeMethods/Framework/Format/MatrixFrame_Unsafe.cs

@@ -0,0 +1,20 @@
+


Can you add a license header to the top of each file?

Suggested change

// Copyright (c) 2021 homuler

//

// Use of this source code is governed by an MIT-style

// license that can be found in the LICENSE file or at

// https://opensource.org/licenses/MIT.

yes

what exactly should I put there?

I'm not familiar with licensing

should I add

// Copyright (c) 2021 homuler // // Use of this source code is governed by an MIT-style // license that can be found in the LICENSE file or at // https://opensource.org/licenses/MIT.

to the top of every newly added file?

should I add ... to the top of every newly added file?

Yes, exactly.
That's because if the license is not clear, it will be difficult for me to modify the contents.

....mediapipe/Runtime/Scripts/PInvoke/NativeMethods/Framework/Format/FloatVectorFrame_Unsafe.cs

mediapipe_api/framework/formats/matrix_frame.h

Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/MatrixFramePacket.cs

mgarbade · 2022-07-25T16:16:58Z

thx for feedback. will look into this asap (couple of days)

- rename variables - rename cs files - rename cc files

- rename variables - rename cs files - rename cc files matrix_frame -> matrix_data -> avoid matrix.cc as it is already used as a name in mediapipe

- does not represent an official solution - not sure where else to place this - MatrixClassification.cs is an important example for showcasing the usage of a tflite model with a matrix data input

- before it was byte[]

- delete FloatVector_Unsafe.cs

TODO: - implement GetFloatVector with vector size as argument

- using format file ".clang-format" in project root

…et_At__PKc_i_Rt

- build similar to FloatArrayPacketTest - not yet tested

fix: FloatVectorPacket [skip actions] (+1 squashed commits) Squashed commits: [69302b1] FloatVectorPacket - replace list by array - list is slow

[skip actions]

as per request: - deleted demo / tutorial scene that showcasts a simple tflite graph [skip actions]

mgarbade · 2022-09-27T09:45:23Z

deleting MatrixClassification in "Assets/../Tutorial Scene"

FloatVectorPacket still assumes the vector length to be set by Unity via constructor

run and test the suggested version

I force pushed some more changes:

is done
is done
at least the matrix classification tutorial scene was compiling and running on my machine again

all your requested changes should still be inside the current HEAD of "matrix_packet_rebased_v3"

homuler · 2022-09-30T11:54:11Z

@mgarbade Will you fix the failed tests?
If they all pass, I'll merge this PR.

mgarbade · 2022-09-30T13:43:54Z

I'll look into it when I have time. (about 1-2 weeks)

mgarbade · 2022-10-10T10:09:29Z

I'm sorry to bother you. But I cannot figure out from the reports what is the problem with the tests.
For example the test
"run-tests-on-linux / test (ubuntu-22.04, 2021.3.3f1)
failed 13 days ago in 6m 9s"

has the following log output at the end:

[ALLOC_PROFILER_EDITOR]
Peak usage frame count: [0-1.0 KB]: 56 frames
Requested Block Size 1.0 MB
Peak Block count 0
Peak Allocated memory 0 B
Peak Large allocation bytes 0 B
[ALLOC_PROFILER_BUCKET]
Large Block size 32.0 MB
Used Block count 1
Peak Allocated bytes 2.0 KB
##utp:{"type":"MemoryLeaks","version":2,"phase":"Immediate","time":1664285327254,"processId":4274,"allocatedMemory":4018038,"memoryLabels":[{"Default":8247},{"Permanent":16120},{"Thread":1083180},{"Manager":18061},{"VertexData":12},{"Geometry":280},{"Texture":160},{"Shader":69203},{"Material":24},{"GfxDevice":35520},{"Animation":312},{"Audio":3976},{"Physics":289},{"Serialization":672},{"Input":26792},{"JobScheduler":200},{"Mono":40},{"ScriptingNativeRuntime":43520},{"BaseObject":1620252},{"Resource":944},{"Renderer":1936},{"Transform":48},{"File":271412},{"WebCam":24},{"Culling":40},{"Terrain":1025},{"Wind":24},{"String":30299},{"DynamicArray":93028},{"HashMap":77574},{"Utility":8682},{"Curl":3751},{"PoolAlloc":32192},{"AI":40},{"TypeTree":2064},{"ScriptManager":368},{"RuntimeInitializeOnLoadManager":72},{"SpriteAtlas":112},{"GI":3784},{"Unet":16},{"Director":7816},{"WebRequest":720},{"VR":45529},{"SceneManager":624},{"Video":72},{"LazyScriptCache":32},{"NativeArray":12},{"Camera":25},{"Secure":6609},{"SerializationCache":624},{"APIUpdating":5872},{"Subsystems":384},{"VirtualTexturing":57560},{"StaticSafetyDebugInfo":327752},{"EditorGui":40},{"EditorUtility":37559},{"VersionControl":4},{"Undo":450},{"AssetDatabase":10586},{"RestService":1518},{"EditorGi":368},{"License":1056},{"UnityConnect":29688},{"Collab":785},{"Upm":1504},{"DrivenProperties":80},{"HubClient":32},{"LocalIPC":101},{"ProfilerEditor":20133},{"CoreBusinessMetrics":1402},{"TLS":2958},{"Licensing":1816},{"AssetReference":32}]}
Error: Process completed with exit code 2.

This tells me nothing. I don't know what "##utp" means and any of the words following that keyword seem not to relate to my changes. "type":"MemoryLeaks" might hint to the problem, however without any more context, I'm not sure what to look for.

I'll try to run the tests on my local machine and see if there is something failing either...

homuler · 2022-10-10T10:43:26Z

See https://github.com/homuler/MediaPipeUnityPlugin/actions/runs/3135556925/jobs/5092743904.
I'm rerunning the CI, and once it finishes, you can download the results from the summary page (retained only for a week and the last time results are already removed).

On Linux or macOS, it seems that the result.xml is not printed to the console, so I'll fix it.

mgarbade · 2022-10-10T11:44:41Z

Ok. I found some failing tests in "MatrixPacketTest" on my local windows machine.

On the other hand, all tests related to "FloatVectorPacketTest" were successful.

I cannot relate these failed tests to those encountered by github actions. But I will fix them anyway.

As before the output of the github action run test is not very explicit about the cause of the test failure (taken from the link you pasted above):

name="TryReadPixelData_ShouldReturnTrue_When_TheFormatIsSrgba64" fullname="Mediapipe.Unity.Tests.ImageFrameExtensionTest.TryReadPixelData_ShouldReturnTrue_When_TheFormatIsSrgba64" methodname="TryReadPixelData_ShouldReturnTrue_When_TheFormatIsSrgba64" classname="Mediapipe.Unity.Tests.ImageFrameExtensionTest" runstate="Runnable" seed="1791896299" result="Passed" start-time="2022-09-27 14:04:29Z" end-time="2022-09-27 14:04:29Z" duration="0.000819" asserts="0">
<properties />
</test-case>
<test-case id="2440" name="TryReadPixelData_ShouldReturnTrue_When_TheFormatIsVec32f1" fullname="Mediapipe.Unity.Tests.ImageFrameExtensionTest.TryReadPixelData_ShouldReturnTrue_When_TheFormatIsVec32f1" methodname="TryReadPixelData_ShouldReturnTrue_When_TheFormatIsVec32f1" classname="Mediapipe.Unity.Tests.ImageFrameExtensionTest" runstate="Runnable" seed="1147385770" result="Passed" start-time="2022-09-27 14:04:29Z" end-time="2022-09-27 14:04:29Z" duration="0.000764" asserts="0">
<properties />
</test-case>
</test-suite>
</test-suite>
</test-suite>
</test-suite>
</test-suite>
</test-suite>
</test-run>
Error: Process completed with exit code 1.

homuler · 2022-10-10T12:29:27Z

As before the output of the github action run test is not very explicit about the cause of the test failure (taken from the link you pasted above):

I agree that the output can be improved, but at least you can know which test cases are failing.

xmllint --xpath "//test-case[@result='Failed']" results.xml

            <test-case id="2864" name="At_ShouldReturnNewPacketWithTimestamp" fullname="Mediapipe.Tests.MatrixPacketTest.At_ShouldReturnNewPacketWithTimestamp" methodname="At_ShouldReturnNewPacketWithTimestamp" classname="Mediapipe.Tests.MatrixPacketTest" runstate="Runnable" seed="132040465" result="Failed" label="Error" start-time="2022-10-10 10:22:45Z" end-time="2022-10-10 10:22:45Z" duration="0.010864" asserts="0">
              <properties/>
              <failure>
                <message><![CDATA[System.NotImplementedException : The method or operation is not implemented.]]></message>
                <stack-trace><![CDATA[  at Mediapipe.MatrixPacket.Get () [0x00001] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/MatrixPacket.cs:65 
  at Mediapipe.Tests.MatrixPacketTest.At_ShouldReturnNewPacketWithTimestamp () [0x0001d] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Tests/EditMode/Framework/Packet/MatrixPacketTest.cs:57 
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <8516445456524a6cb20c4b3b7de67982>:0 ]]></stack-trace>
              </failure>
            </test-case>
<test-case id="2865" name="Consume_ShouldThrowNotSupportedException" fullname="Mediapipe.Tests.MatrixPacketTest.Consume_ShouldThrowNotSupportedException" methodname="Consume_ShouldThrowNotSupportedException" classname="Mediapipe.Tests.MatrixPacketTest" runstate="Runnable" seed="847120028" result="Failed" start-time="2022-10-10 10:22:45Z" end-time="2022-10-10 10:22:45Z" duration="0.003217" asserts="0">
              <properties/>
              <failure>
                <message><![CDATA[  Expected: <System.NotSupportedException>
  But was:  <System.NotImplementedException: The method or operation is not implemented.
  at Mediapipe.MatrixPacket.Consume () [0x00001] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/MatrixPacket.cs:70 
  at Mediapipe.Tests.MatrixPacketTest+<>c__DisplayClass4_0.<Consume_ShouldThrowNotSupportedException>b__0 () [0x00001] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Tests/EditMode/Framework/Packet/MatrixPacketTest.cs:79 
  at NUnit.Framework.Assert.Throws (NUnit.Framework.Constraints.IResolveConstraint expression, NUnit.Framework.TestDelegate code, System.String message, System.Object[] args) [0x00004] in <10b6135e63434fdba4fc6c109928ab3b>:0 >
]]></message>
                <stack-trace><![CDATA[at Mediapipe.Tests.MatrixPacketTest.Consume_ShouldThrowNotSupportedException () [0x00013] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Tests/EditMode/Framework/Packet/MatrixPacketTest.cs:79
]]></stack-trace>
              </failure>
            </test-case>
<test-case id="2861" name="Ctor_ShouldInstantiatePacket_When_CalledWithValue" fullname="Mediapipe.Tests.MatrixPacketTest.Ctor_ShouldInstantiatePacket_When_CalledWithValue" methodname="Ctor_ShouldInstantiatePacket_When_CalledWithValue" classname="Mediapipe.Tests.MatrixPacketTest" runstate="Runnable" seed="697797710" result="Failed" label="Error" start-time="2022-10-10 10:22:45Z" end-time="2022-10-10 10:22:45Z" duration="0.000617" asserts="0">
              <properties/>
              <failure>
                <message><![CDATA[System.NotImplementedException : The method or operation is not implemented.]]></message>
                <stack-trace><![CDATA[  at Mediapipe.Packet`1[TValue].ValidateAsType () [0x00001] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/Packet.cs:74 
  at Mediapipe.Tests.MatrixPacketTest.Ctor_ShouldInstantiatePacket_When_CalledWithValue () [0x0000f] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Tests/EditMode/Framework/Packet/MatrixPacketTest.cs:22 
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <8516445456524a6cb20c4b3b7de67982>:0 ]]></stack-trace>
              </failure>
            </test-case>

mgarbade · 2022-10-10T12:51:47Z

Thanks. Looks like the same tests that where failing in my local windows build.
(not sure where to download the "results.xml" file you mentioned above)

Anyway, I'll go ahead and try to fix the tests...

homuler · 2022-10-10T13:07:52Z

You can download it from the summary page (Test Results for ...).

mgarbade · 2022-10-11T13:18:15Z

I'm not sure how to write the "packet__Get()" function for MatrixPacket.cs which is needed for one of the test functions

in C++ the packet holds an Eigen::Matrix
in Unity the same data is of class "MatrixData" which is a protobuf format
--> I assume I have to convert the EigenMatrix to MatrixData inside the C++ part of the code before sending it back to Unity

However I'm not sure how to do that conversion. This is my first draft:

MP_CAPI(MpReturnCode) mp_Packet__GetMatrix(mediapipe::Packet* packet, mp_api::SerializedProto* value_out) {
  TRY
    // Get Eigen::Matrix from packet
    mediapipe::Matrix matrix;
    matrix = packet->Get<mediapipe::Matrix>();

    // Convert to format that can be send to Unity
    mediapipe::MatrixData* matrix_data;
    mediapipe::MatrixDataProtoFromMatrix(matrix, matrix_data);
    
    *value_out = matrix_data;  // <-- THIS IS WHERE THE CODE IS CRASHING

    // *value_out = packet->Get<bool>();
    RETURN_CODE(MpReturnCode::Success);
  CATCH_EXCEPTION
}

the compilation fails saying

ERROR: C:/users/garbade/git/homuler/mpup_pr/mediapipe_api/framework/formats/BUILD:67:11: Compiling mediapipe_api/framework/formats/matrix_data.cc failed: (Exit 2): cl.exe failed: error executing command D:\Programs\MicrosoftVisualStudio\2022\Community\VC\Tools\MSVC\14.30.30704\bin\HostX64\x64\cl.exe @bazel-out/x64_windows-dbg/bin/mediapipe_api/framework/formats/_objs/matrix_data/matrix_data.obj.params
mediapipe_api/framework/formats/matrix_data.cc(60): error C2679: binary '=': no operator found which takes a right-hand operand of type 'mediapipe::MatrixData *' (or there is no acceptable conversion)
C:\_bzl\p7phwaas\execroot\mediapipe_api\mediapipe_api/external/protobuf.h(22): note: could be 'mp_api::SerializedProto &mp_api::SerializedProto::operator =(mp_api::SerializedProto &&)'
C:\_bzl\p7phwaas\execroot\mediapipe_api\mediapipe_api/external/protobuf.h(22): note: or       'mp_api::SerializedProto &mp_api::SerializedProto::operator =(const mp_api::SerializedProto &)'
mediapipe_api/framework/formats/matrix_data.cc(60): note: while trying to match the argument list '(mp_api::SerializedProto, mediapipe::MatrixData *)'
Target //mediapipe_api:mediapipe_desktop failed to build

however I'm not sure how to convert the protobuf to serialized proto

mgarbade · 2022-10-11T16:06:50Z

moving forward. This code seems more reasonable but still crashes:

MP_CAPI(MpReturnCode) mp_Packet__GetMatrix(mediapipe::Packet* packet, mp_api::SerializedProto* value_out) {
  TRY
    // Get Eigen::Matrix from packet
    mediapipe::Matrix matrix;
    matrix = packet->Get<mediapipe::Matrix>();

    // Convert to format that can be send to Unity
    mediapipe::MatrixData* matrix_data;
    mediapipe::MatrixDataProtoFromMatrix(matrix, matrix_data);  // <-- CRASHES HERE
    
    // auto matrix_data_serialized = new mp_api::SerializedProto();
    SerializeProto(*matrix_data, value_out);

    RETURN_CODE(MpReturnCode::Success);
  CATCH_EXCEPTION
}

The exact line where it crashes:

void MatrixDataProtoFromMatrix(const Matrix& matrix, MatrixData* matrix_data) {
  const int rows = matrix.rows();
  const int cols = matrix.cols();
  matrix_data->set_rows(rows);
  matrix_data->set_cols(cols);
  matrix_data->clear_layout();
  proto_ns::RepeatedField<float>(matrix.data(), matrix.data() + rows * cols)  // <-- CRASHES HERE
      .Swap(matrix_data->mutable_packed_data());
}

It throws an exception in mediapipe which I'm unable to read...

this is what the console says:

The thread 0xaf910 has exited with code 0 (0x0).
The thread 0x325b8 has exited with code 0 (0x0).
The thread 0xa7b68 has exited with code 0 (0x0).
The thread 0x10d28 has exited with code 0 (0x0).
The thread 0x9a708 has exited with code 0 (0x0).
The thread 0xcb4a4 has exited with code 0 (0x0).
Exception thrown at 0x00007FFA8D454FD9 in Unity.exe: Microsoft C++ exception: google::protobuf::FatalException at memory location 0x0000004DEC7EDB90.
The thread 0x4d68 has exited with code 0 (0x0).
The thread 0x8dd88 has exited with code 0 (0x0).

the "_mp_return_code" is -327236016 (which I'm unable to google)
will look further into this. Hints are welcome...

mgarbade · 2022-10-12T09:36:38Z

nevermind. I found the problem! (some astrix to many ...)

MP_CAPI(MpReturnCode) mp_Packet__GetMatrix(mediapipe::Packet* packet, mp_api::SerializedProto* value_out) {
  TRY
    // Get Eigen::Matrix from packet
    mediapipe::Matrix matrix;
    matrix = packet->Get<mediapipe::Matrix>();

    // Convert to format that can be send to Unity
    mediapipe::MatrixData matrix_data;
    mediapipe::MatrixDataProtoFromMatrix(matrix, &matrix_data);
    
    // auto matrix_data_serialized = new mp_api::SerializedProto();
    SerializeProto(matrix_data, value_out);

    RETURN_CODE(MpReturnCode::Success);
  CATCH_EXCEPTION
}

- new GetMatrix function Caveat: - MatrixPacket: Consume throws NotSupportedException() -> not sure if this is a useful test, but such tests exists in similar classes as well (cherry picked from commit 707af5f454e87312a86b60deaebec18463e47ded)

mgarbade · 2022-10-12T10:17:35Z

all tests concerning the two new classes I introduced are running successfully on my Windows 10 Pro machine in Unity Editor mode:

I'm not 100% positive how useful these tests are, but they roughly follow the pattern of similar classes. Especially questionable is the test "Consume_ShouldThrowNotSupportedException" which just makes sure that the Consume function is not implemented. (I'm not sure what the Consume function is used for anyway)

homuler

Thx!

* cc: matrix_frame as input to graph - "matrix_frame" as name in order to avoid confusion with matrix.cc - matrix_frame is a 2D input data modality that gets converted to Eigen::MatrixXf internally - suited for non-image input to tflite models * cc: float_vector_frame as input to graph - "float_vector_frame" as name in order to avoid confusion with FloatArrayPacket - float_vector_frame is a 1D output data modality that accepts std::vector<float> outputs from a mediapipe / tflite graph - suited for tflite classification results packaged as vector of floats * MatrixFramePacket - c# helper functions - send MatrixData to C++ as byte array * FloatVectorFramePacket - c# helper functions - return std::vector<float> from C++ to Unity as List<float> * Unity: Matrix Classification - Example scene * Matrix Classification.cs - driver code for the newly added MatrixFramePacket and FloatVectorFramePacket - feeds an example matrix of size [ 2 x 3 ] into a mediapipe graph - the graph runs a simple tflite model (adds +1 to every input) - then the graph returns the result back to Unity as List<float> - only tested on Unity-Editor-Mode on Windows 10 Pro * refactor: rename FloatVectorFrame -> FloatVector - rename variables - rename cs files - rename cc files * refactor: rename MatrixFrame -> Matrix - rename variables - rename cs files - rename cc files matrix_frame -> matrix_data -> avoid matrix.cc as it is already used as a name in mediapipe * move MatrixClassification example scene to Tutorials - does not represent an official solution - not sure where else to place this - MatrixClassification.cs is an important example for showcasing the usage of a tflite model with a matrix data input * GetArrayPtr() - change access to private * MatrixPacket: accept MatrixData as input - before it was byte[] * add license * move native functions to Packet_Unsafe - delete FloatVector_Unsafe.cs * float_vector.cc -> faster vector allocation * float_vector.cc remove unused function - delete(...) * float_vector.h - remove unused headers * refactor: float_vector.cc TODO: - implement GetFloatVector with vector size as argument * removed unused headers * refactor: float_vector.h * refactor: apply autoformatter on cc files - using format file ".clang-format" in project root * refactor: mp__MakeMatrixFramePacket_At__PA_i_Rt -> mp__MakeMatrixPacket_At__PKc_i_Rt * FloatVectorPacketTest added - build similar to FloatArrayPacketTest - not yet tested * fix: float_vector.cc * fix: MatrixPacket.cs * fix: Test: FloatVectorPacketTest - Consume_ShouldThrowNotSupportedException * MatrixPacketTest - add - all tests involving packet.Get() do not work - function is not yet implemented * fix: Make MatrixClassification.cs run on Android - adding StreamingAssets to ResourceManager [skip actions] * Update mediapipe_api/framework/formats/matrix_data.h Co-authored-by: Junrou Nishida <[email protected]> * Apply suggestions from code review Co-authored-by: Junrou Nishida <[email protected]> * float_vector - return vector size (+2 squashed commit) Squashed commit: [e409b05] refactor: vector_float.cc - naming aligns with files like packet.cc [bad3cd6] float_vector - return vector size * fix: matrix_data.cc - wrong func name (+3 squashed commit) Squashed commit: [9245a37] fix: Revert "Apply suggestions from code review" - the below mentioned commit is not working - return value of inline function is invalid -> probably due to inline function This reverts commit c374d61. [f597e83] fix: remove duplicate cpp func [6def10c] fix: semicolon omitted * FloatVectorPacket - replace list by array fix: FloatVectorPacket [skip actions] (+1 squashed commits) Squashed commits: [69302b1] FloatVectorPacket - replace list by array - list is slow * Add license headers [skip actions] * Remove Tutorial Scene: MatrixClassification as per request: - deleted demo / tutorial scene that showcasts a simple tflite graph [skip actions] * fix: MatrixPacket tests - new GetMatrix function Caveat: - MatrixPacket: Consume throws NotSupportedException() -> not sure if this is a useful test, but such tests exists in similar classes as well (cherry picked from commit 707af5f454e87312a86b60deaebec18463e47ded) Co-authored-by: Junrou Nishida <[email protected]>

* matrix classification via tflite model (#656) * cc: matrix_frame as input to graph - "matrix_frame" as name in order to avoid confusion with matrix.cc - matrix_frame is a 2D input data modality that gets converted to Eigen::MatrixXf internally - suited for non-image input to tflite models * cc: float_vector_frame as input to graph - "float_vector_frame" as name in order to avoid confusion with FloatArrayPacket - float_vector_frame is a 1D output data modality that accepts std::vector<float> outputs from a mediapipe / tflite graph - suited for tflite classification results packaged as vector of floats * MatrixFramePacket - c# helper functions - send MatrixData to C++ as byte array * FloatVectorFramePacket - c# helper functions - return std::vector<float> from C++ to Unity as List<float> * Unity: Matrix Classification - Example scene * Matrix Classification.cs - driver code for the newly added MatrixFramePacket and FloatVectorFramePacket - feeds an example matrix of size [ 2 x 3 ] into a mediapipe graph - the graph runs a simple tflite model (adds +1 to every input) - then the graph returns the result back to Unity as List<float> - only tested on Unity-Editor-Mode on Windows 10 Pro * refactor: rename FloatVectorFrame -> FloatVector - rename variables - rename cs files - rename cc files * refactor: rename MatrixFrame -> Matrix - rename variables - rename cs files - rename cc files matrix_frame -> matrix_data -> avoid matrix.cc as it is already used as a name in mediapipe * move MatrixClassification example scene to Tutorials - does not represent an official solution - not sure where else to place this - MatrixClassification.cs is an important example for showcasing the usage of a tflite model with a matrix data input * GetArrayPtr() - change access to private * MatrixPacket: accept MatrixData as input - before it was byte[] * add license * move native functions to Packet_Unsafe - delete FloatVector_Unsafe.cs * float_vector.cc -> faster vector allocation * float_vector.cc remove unused function - delete(...) * float_vector.h - remove unused headers * refactor: float_vector.cc TODO: - implement GetFloatVector with vector size as argument * removed unused headers * refactor: float_vector.h * refactor: apply autoformatter on cc files - using format file ".clang-format" in project root * refactor: mp__MakeMatrixFramePacket_At__PA_i_Rt -> mp__MakeMatrixPacket_At__PKc_i_Rt * FloatVectorPacketTest added - build similar to FloatArrayPacketTest - not yet tested * fix: float_vector.cc * fix: MatrixPacket.cs * fix: Test: FloatVectorPacketTest - Consume_ShouldThrowNotSupportedException * MatrixPacketTest - add - all tests involving packet.Get() do not work - function is not yet implemented * fix: Make MatrixClassification.cs run on Android - adding StreamingAssets to ResourceManager [skip actions] * Update mediapipe_api/framework/formats/matrix_data.h Co-authored-by: Junrou Nishida <[email protected]> * Apply suggestions from code review Co-authored-by: Junrou Nishida <[email protected]> * float_vector - return vector size (+2 squashed commit) Squashed commit: [e409b05] refactor: vector_float.cc - naming aligns with files like packet.cc [bad3cd6] float_vector - return vector size * fix: matrix_data.cc - wrong func name (+3 squashed commit) Squashed commit: [9245a37] fix: Revert "Apply suggestions from code review" - the below mentioned commit is not working - return value of inline function is invalid -> probably due to inline function This reverts commit c374d61. [f597e83] fix: remove duplicate cpp func [6def10c] fix: semicolon omitted * FloatVectorPacket - replace list by array fix: FloatVectorPacket [skip actions] (+1 squashed commits) Squashed commits: [69302b1] FloatVectorPacket - replace list by array - list is slow * Add license headers [skip actions] * Remove Tutorial Scene: MatrixClassification as per request: - deleted demo / tutorial scene that showcasts a simple tflite graph [skip actions] * fix: MatrixPacket tests - new GetMatrix function Caveat: - MatrixPacket: Consume throws NotSupportedException() -> not sure if this is a useful test, but such tests exists in similar classes as well (cherry picked from commit 707af5f454e87312a86b60deaebec18463e47ded) Co-authored-by: Junrou Nishida <[email protected]> * refactor: move float vector packet API * FloatVectorPacket#Get returns List<float> * copy float arrays when getting the value * refactor MatrixPacket * refactor: use ParseFromStringAsProto * test: replace DebugTypeName tests with ValidateAsType tests Co-authored-by: Martin Garbade <[email protected]>

mgarbade added 6 commits July 15, 2022 13:15

cc: matrix_frame as input to graph

21bb5cc

- "matrix_frame" as name in order to avoid confusion with matrix.cc - matrix_frame is a 2D input data modality that gets converted to Eigen::MatrixXf internally - suited for non-image input to tflite models

MatrixFramePacket - c# helper functions

9e1334b

- send MatrixData to C++ as byte array

FloatVectorFramePacket - c# helper functions

69f705f

- return std::vector<float> from C++ to Unity as List<float>

Unity: Matrix Classification - Example scene

4d43489

homuler self-requested a review July 16, 2022 00:06

homuler requested changes Jul 16, 2022

View reviewed changes

homuler added the ci:run Run CI label Jul 16, 2022

github-actions bot removed the ci:run Run CI label Jul 16, 2022

mgarbade added 17 commits July 29, 2022 16:38

refactor: rename FloatVectorFrame -> FloatVector

0bf959d

- rename variables - rename cs files - rename cc files

refactor: rename MatrixFrame -> Matrix

3a444c9

- rename variables - rename cs files - rename cc files matrix_frame -> matrix_data -> avoid matrix.cc as it is already used as a name in mediapipe

move MatrixClassification example scene to Tutorials

ffd978e

- does not represent an official solution - not sure where else to place this - MatrixClassification.cs is an important example for showcasing the usage of a tflite model with a matrix data input

GetArrayPtr() - change access to private

a723eec

MatrixPacket: accept MatrixData as input

6622fcb

- before it was byte[]

add license

44202ab

move native functions to Packet_Unsafe

69caf3c

- delete FloatVector_Unsafe.cs

float_vector.cc -> faster vector allocation

3669c86

float_vector.cc remove unused function - delete(...)

ebbdddc

float_vector.h - remove unused headers

9328084

refactor: float_vector.cc

b8384bf

TODO: - implement GetFloatVector with vector size as argument

removed unused headers

3150340

refactor: float_vector.h

66f197b

refactor: apply autoformatter on cc files

9c0d9ba

- using format file ".clang-format" in project root

refactor: mp__MakeMatrixFramePacket_At__PA_i_Rt -> mp__MakeMatrixPack…

f9ca7e1

…et_At__PKc_i_Rt

FloatVectorPacketTest added

6f3d1b7

- build similar to FloatArrayPacketTest - not yet tested

fix: float_vector.cc

4e60769

FloatVectorPacket - replace list by array

a8ec2a3

fix: FloatVectorPacket [skip actions] (+1 squashed commits) Squashed commits: [69302b1] FloatVectorPacket - replace list by array - list is slow

mgarbade force-pushed the matrix_packet_rebased_v3 branch from a90a2b5 to 7e5e5f1 Compare September 27, 2022 09:38

Add license headers

8e7771d

[skip actions]

mgarbade force-pushed the matrix_packet_rebased_v3 branch from 7e5e5f1 to 8e7771d Compare September 27, 2022 09:39

Remove Tutorial Scene: MatrixClassification

829a365

as per request: - deleted demo / tutorial scene that showcasts a simple tflite graph [skip actions]

homuler added the ci:run Run CI label Sep 27, 2022

github-actions bot removed the ci:run Run CI label Sep 27, 2022

homuler added the ci:run Run CI label Oct 10, 2022

github-actions bot removed the ci:run Run CI label Oct 10, 2022

fix: MatrixPacket tests

861752f

- new GetMatrix function Caveat: - MatrixPacket: Consume throws NotSupportedException() -> not sure if this is a useful test, but such tests exists in similar classes as well (cherry picked from commit 707af5f454e87312a86b60deaebec18463e47ded)

homuler added the ci:run Run CI label Oct 12, 2022

github-actions bot removed the ci:run Run CI label Oct 12, 2022

homuler approved these changes Oct 15, 2022

View reviewed changes

homuler merged commit 089816e into homuler:feat/matrix-packet Oct 15, 2022

mgarbade deleted the matrix_packet_rebased_v3 branch November 25, 2022 10:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

matrix classification via tflite model #656

matrix classification via tflite model #656

mgarbade commented Jul 15, 2022 •

edited

Loading

mgarbade commented Jul 15, 2022

mgarbade commented Jul 15, 2022

homuler left a comment

homuler Jul 16, 2022

mgarbade Jul 29, 2022 •

edited

Loading

homuler Sep 4, 2022

homuler Jul 16, 2022

mgarbade Jul 29, 2022

homuler Jul 16, 2022

mgarbade Jul 29, 2022

homuler Jul 16, 2022

mgarbade Jul 29, 2022

homuler Sep 24, 2022

mgarbade commented Jul 25, 2022

mgarbade commented Sep 27, 2022

homuler commented Sep 30, 2022

mgarbade commented Sep 30, 2022

mgarbade commented Oct 10, 2022

homuler commented Oct 10, 2022

mgarbade commented Oct 10, 2022

homuler commented Oct 10, 2022

mgarbade commented Oct 10, 2022

homuler commented Oct 10, 2022

mgarbade commented Oct 11, 2022

mgarbade commented Oct 11, 2022 •

edited

Loading

mgarbade commented Oct 12, 2022 •

edited

Loading

mgarbade commented Oct 12, 2022 •

edited

Loading

homuler left a comment

+// Copyright (c) 2021 homuler
+//
+// Use of this source code is governed by an MIT-style
+// license that can be found in the LICENSE file or at
+// https://opensource.org/licenses/MIT.

matrix classification via tflite model #656

matrix classification via tflite model #656

Conversation

mgarbade commented Jul 15, 2022 • edited Loading

mgarbade commented Jul 15, 2022

mgarbade commented Jul 15, 2022

homuler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgarbade Jul 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgarbade commented Jul 25, 2022

mgarbade commented Sep 27, 2022

homuler commented Sep 30, 2022

mgarbade commented Sep 30, 2022

mgarbade commented Oct 10, 2022

homuler commented Oct 10, 2022

mgarbade commented Oct 10, 2022

homuler commented Oct 10, 2022

mgarbade commented Oct 10, 2022

homuler commented Oct 10, 2022

mgarbade commented Oct 11, 2022

mgarbade commented Oct 11, 2022 • edited Loading

mgarbade commented Oct 12, 2022 • edited Loading

mgarbade commented Oct 12, 2022 • edited Loading

homuler left a comment

Choose a reason for hiding this comment

mgarbade commented Jul 15, 2022 •

edited

Loading

mgarbade Jul 29, 2022 •

edited

Loading

mgarbade commented Oct 11, 2022 •

edited

Loading

mgarbade commented Oct 12, 2022 •

edited

Loading

mgarbade commented Oct 12, 2022 •

edited

Loading