-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Filtering feature #36
base: master
Are you sure you want to change the base?
Conversation
emscripten/ARToolKitJS.cpp
Outdated
transF[j][k] = trans[j][k]; | ||
} | ||
} | ||
if (arFilterTransMat(arc->ftmi, transF, 1) < 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The third argument of arFilterTransMat should a value that is zero or 1 as explained in the doc http://www.artoolkitx.org/artoolkit5/doc/apiref/arFilterTransMat_h/index.html#//apple_ref/c/func/arFilterTransMat
this needs to be changed.
i added a arLog to retrieve the value of the filterCutoffFrequency and it give me |
AR_FILTER_TRANS_MAT_SAMPLE_RATE_DEFAULT and AR_FILTER_TRANS_MAT_CUTOFF_FREQ_DEFAULT defined in arFilterTransMat.h: as i wrote in the top comment. Maybe i was receiving that value because i haven't added them to ARBindEM.cpp |
I will test it! @kalwalt does it work with threejs_gltf_worker example? |
works with all the nft examples but i removed the interpolation routine (that in the javscript code) in a dedicated example main_threejs_filter_worker.html try this and if you want the others. But i think i still need to fix few things to make some comparisons. |
@nicolocarpignoli @ThorstenBux I added a WIP example with gpu.js but i think that there are many variables in this approach. I may implement a kernel for the matrix linear interpolation. Probably in this case that is not very recommended and i will remove it. Another idea is to implement a more efficient routine (linear interpolation) inside the C++ code. Probably will result faster? |
@nicolocarpignoli @ThorstenBux i think the version with filtering and linear interpolation in the C++ code is much better, I'd like to know what you think 😄 |
@nicolocarpignoli yes with the model animation, seems worste. I need to undersand why. Also fps are very slow on my Mobile device. Tested with a Wiko, View model with Android 7.1.2 and Chrome browser 79.0.3945.136 |
I will also remoove my gpu.js attempt, i think i will try to optimize the code without it, even if it sound interesting. |
@@ -112,6 +120,8 @@ extern "C" { | |||
|
|||
float trans[3][4]; | |||
ARdouble transF[3][4]; | |||
ARdouble transFLerp[3][4]; | |||
memset( transFLerp, 0, 3 * 4 * sizeof(ARdouble) ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe this is not needed but not sure of this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is needed, without it the interpolation give weird results.
emscripten/ARToolKitJS.cpp
Outdated
@@ -160,6 +170,9 @@ extern "C" { | |||
if (arFilterTransMat(arc->ftmi, transF, reset) < 0) { | |||
ARLOGe("arFilterTransMat error with marker %d.\n", markerIndex); | |||
} | |||
|
|||
matrixLerp(transF, transFLerp, 24.0); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
try with another interpolationFactor, maybe 12.0 instead of 24.0?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes we have to try different values
I found in the BabylonAR repository this 'naive' implementation of the Kalman filter, I should try it, it looks very promising! |
Tested, duck is jittering like crazy |
@kalwalt #36 (comment) === this is a good way to go. The Kalmar becomes a linear function after running for the first time so it can be even implemented in JS for some few relevant points. Again, finding the right parameters might change by image and perhaps by device. But that I don't know. If that the case, finding the right parameters might require a search algorithm (I have done some statistics / DS, so that my reference). But then you need to clarify which functions to optimise. I haven't done that job yet. A target success/failure variable is required. The search can be automated if the right functions are formulated. The search might be slow and data intensive. There are few more things that can be done to improve jittering that I have been testing. I will make a video and share the link to show you the results when having time. The target identification (short distance only) and homography were very good using the artoolkit algo for NFT. I will check later which algo is being used, I am really curious now. Previous essays I made with a simpler algos were not as good in detection, especially if the target image was a bit blurred, etc. I will study some C++ shortly... |
Nice reference! Thanks! |
Yeah... the same here. |
Hi @evaristoc recently i added a combined lerp and filtering (with arFilterTransMat) to my JsartoolkitNFT (dev branch) webarkit/jsartoolkitNFT#35 see if this is better or not. Also take a look at this discussion in the artoolkitx/jsartoolkit5 issue tracker |
@kalwalt I check and let you know? Same testing link, I guess... |
@kalwalt still jittering. Specially from some positions of the marker 😞 |
@kalwalt what is exactly arFilterTransMat? It is almost obvious from the name that it refers to the transformation matrix, but what does exactly do? Do you know? EDIT: first order low pass filter: https://www.electronics-tutorials.ws/filter/filter_2.html: "the low pass filter only allows low frequency signals from 0Hz to its cut-off frequency, ƒc point to pass while blocking those any higher." It might be close to what it is done when practicing time series analysis in statistics. It looks like it is assuming a certain frequency (a sinusoidal signal) and implementing a linear model (first order) to clean it. It appears to work well for certain positions, likely because it can capture the signal. |
Hi @evaristoc have you tested which code? Probably i didn't applyed correctly,It seems that has no effect ( or very low). |
Hi @kalwalt I was editing my comment above: It might be close to what it is done when practicing time series analysis in statistics. It looks like it is assuming a certain frequency (a sinusoidal signal) and implementing a linear model (first order) to clean it. It appears to work well for certain positions, likely because it can capture the signal. |
You might need not one but two or more of those to complete the job... |
So maybe you can help, to find what is not correct in my approach. I Think it should filter the noise, but of course it should be done with a correct approach/method. |
@kalwalt let's see. I expect to find a roof after which we can not improve the method any more, either because affects on performance or because it will mean to redo some aspects of the project. Right now we are post-processing it. But of course I want to help! |
I would first advice to create a table or something visible of the inputs and outputs, @kalwalt? We should evaluate other filters too, as you have been mentioning in other repos. |
@kalwalt will later make a sort of list of all the fronts where you are dealing with this (they are a few). Let me know where you want to deal with this in particular. Here? Meanwhile if you can point me to the script where you implemented this particular filter, the low pass one? We can probably create a factory for using not one but different kind of filters. I suspect filters will have to be implemented case by case. I expect the jittering to change for different target images. I even expect that to change between devices, etc. But let's see, I need to research this more. |
You can open an issue in this repository and fill all the data there and link to this PR. I prefere here because much control than in artoolkitx/jsartoolkit5. Eventually if yu are interested we have created a new org https://github.com/webarkit. If you are interested to join us, let me know! 🙂 |
the arFilterTransMat is used in ARToolKitJS,cpp the code is wrapped in #if WITH_FILTERING
// code here
#endif So you are not obliged to compile it, with filtering. |
I will be having a look to different examples made with the same library. Example: http://jevois.org/ And keeping this one for further reading: https://answers.opencv.org/question/182075/convert-pose-from-artoolkit-into-opencv-screen-coordinates/ It seems to be some kind of overlap between the two. I have been doing some opencv recently too. |
@kalwalt apparently this is what ARfilterTransMat does The filter tries to correct orientation and position. For orientation it tries to keep the changes within a frequency. I think it is here where the actual low pass is done (EDIT: it is implemented in both, for orientation and for position, with the same calculated alpha, only that for a different equation (trig for orientation, linear for position)). For position is a simple lerp. It uses an alpha value for BOTH corrections and that for my perspective is a hmmm..... That means you are probably implementing a second lerp over a lerp when going this route. (EDIT: I was referring here to the use of a custom LERP in other parts of the code you had been implementing.) alpha is the corrector already explained in the material I passed to you. It looks quite simple, IMHO. I haven't checked at what point of the whole process flow it is implemented, but it is likely a post-processing, as we are all trying to do. |
Yes, nice thanks! |
Some additional observations about the ARfilterTransMat and jitter smoothing:
Similarly:
In both cases, y1Entry will be used as y0 for the next frame.
|
Thank you @evaristoc for the long deep comment, i will think about it. 😄 |
@kalwalt how is the progress on this feature? Is it the same we talked about in the WebArkit slack? |
@Carnaux haven't done some progress lately... Yes, this is in part what we talked in the WebARKit slack channel. |
This PR try to implement filtering of transpose matrix with ArFilterTransMat method. Probably with this code the interpolation routine that was implemented to smooth and remove the jittering of the model became un-necessary, i'm refering to this part of the code:
jsartoolkit5/examples/nft_improved_worker/threejs_worker.js
Lines 179 to 191 in 244b2b2
with this PR simply became:
filterCutoffFrequency i set to AR_FILTER_TRANS_MAT_CUTOFF_FREQ_DEFAULT that means has a default values of 5.0, in the nftSimple.c example instead is set to 15.0
Build: build with emscripten Docker (latest)