Expose cpufit API to python #114

casparvitch · 2022-10-09T10:53:19Z

Requirements to expose cpufit API to python:

Check cpufit builds fine, with cuda SDK but without a cuda compatible card - all good on linux
Cpufit tests pass without a cuda compatible card (few bugs in test suite, all sorted)
Add constrained fit to cpufit - this was actually already in master, by adrianjp88 (nice)
Decide if cpufit should be in pyGpufit or a new package (I think in pyGpufit makes more sense)
Decide how to expose cpufit. I kept the same API but added a keyword argument platform='gpu' by default, which could be changed to cpu fitting via platform='cpu'.

Now, assuming we expose cpufit in pyGpufit:

Add simple logic to gpufit.py
The Cpufit library needs to be copied to the python dist folder.

Changes made:

fix bugs in cpufit test functions
add test for correctness in Consistency.cpp.
add tests for cpufit alone (for when no gpu) to Consistency.cpp
add simple tests for constrained fit in Consistency.cpp
add cpufit api to pyGpufit

Related to #110

superchromix · 2022-10-09T13:30:39Z

I also tend to agree that it would make sense to include Cpufit within pyGpufit, because they are coming from the same package. However, there is a (possibly major) caveat that this is counter-intuitive for the user. Someone looking for a curve-fitting algorithm to run on their CPU might skip over a package called "pyGpufit".

casparvitch · 2022-10-09T13:43:40Z

I think that's a fair point. Although I think the main reason one would use Cpufit would be to maintain the same API and build system etc., which currently requires the cuda SDK, so they're probably looking for Gpufit at some point.

Cpufit could be copied (or moved) over to a new project? Or maybe the make/cmake config could be changed to allow for an only-Cpufit build, which would not required the cuda SDK? The former is probably too much hassle, and I don't know how to do the latter :)

superchromix · 2022-10-09T14:24:53Z

If the main use case for Cpufit is for testing or as a fallback option when no GPU is detected, there is no issue with the non-intuitive naming. I would say we go ahead with using the flag platform='gpu' or platform='cpu' within the API to specify whether to call Cpufit.

casparvitch · 2022-10-12T06:14:56Z

OK the cmake stuff wasn't so hard, python tests (+1 for cpufit) work for me.
NB: once again, I haven't tested on a gpu. I'll comment again if I do test on gpu.

P.S. the matlab/etc. bindings will probably be very easy as well from looking at the source, but I can't test those either.

casparvitch · 2022-10-18T05:56:08Z

Ok I got access to a windows machine with a gpu. MSVS is complaining about the new call to cpufit_constrained in the test suite (Gpufit\tests\Consistency.cpp) with the following:

Error	LNK2019	unresolved external symbol cpufit_constrained referenced in function "void __cdecl perform_cpufit_constrained_check(void (__cdecl*)(struct FitInput &))" (?perform_cpufit_constrained_check@@YAXP6AXAEAUFitInput@@@Z@Z)	Cpufit_Gpufit_Test_Consistency	C:\src\gpufit-build\tests\Consistency.obj	1

MSVS target 'Cpufit_Gpufit_Test_Consistency' -> Properties -> Linker -> Input -> Additional Dependencies contains Cpufit.lib, so I assume everything is pointing in the correct place.

Note that I can build with g++-11 on linux without error.
Any insights would be appreciated!

casparvitch · 2022-10-18T05:58:10Z

Cpufit/lm_fit_cpp.cpp

@@ -816,11 +816,11 @@ void LMFitCPP::calc_values_gauss2delliptic(std::vector<REAL>& gaussian)
    int const size_y = size_x;
    for (int iy = 0; iy < size_y; iy++)
    {
+        REAL const argy = (iy - parameters_[2]) * (iy - parameters_[2]) / (2 * parameters_[4] * parameters_[4]);


probably not necessary, just mirroring gpufit

casparvitch · 2022-10-18T05:59:18Z

Gpufit/info.cpp

@@ -1,5 +1,6 @@
 #include "info.h"
 #include <algorithm>
+#include <limits>


casparvitch · 2022-10-18T05:59:49Z

Gpufit/python/CMakeLists.txt

@@ -17,9 +17,9 @@ set( module_files
 )

 if ( USE_CUBLAS AND DEFINED CUBLAS_DLL )
-	set( binary $<TARGET_FILE:Gpufit> ${CUBLAS_DLL} )
+	set( binary $<TARGET_FILE:Gpufit> ${CUBLAS_DLL} $<TARGET_FILE:Cpufit> )


adding cpufit binary dependency to python release

casparvitch · 2022-10-18T06:00:15Z

Gpufit/python/pygpufit/gpufit.py

@@ -15,33 +15,43 @@
 package_dir = os.path.dirname(os.path.realpath(__file__))



duplicate all of the gpufit library wrapping for cpufit

casparvitch · 2022-10-18T06:00:37Z

Gpufit/python/pygpufit/gpufit.py

@@ -86,7 +96,7 @@ def _valid_id(cls, id):




add flag for cpu/gpu

casparvitch · 2022-10-18T06:01:32Z

Gpufit/python/pygpufit/gpufit.py

@@ -238,41 +251,45 @@ def fit_constrained(data, weights, model_id, initial_parameters, constraints=Non
    chi_squares = np.zeros(number_fits, dtype=np.float32)
    number_iterations = np.zeros(number_fits, dtype=np.int32)

+    # argtypes are grabbed from gpufit_func, but are the same for cpufit_func
+    atypes = gpufit_func.argtypes


just cleaning up the name so future devs don't think the argtypes are specific to gpufit

casparvitch · 2022-10-18T06:01:47Z

Gpufit/python/pygpufit/gpufit.py

-        states.ctypes.data_as(gpufit_func.argtypes[15]), \
-        chi_squares.ctypes.data_as(gpufit_func.argtypes[16]), \
-        number_iterations.ctypes.data_as(gpufit_func.argtypes[17]))
+    func = {"gpu": gpufit_func, "cpu": cpufit_func}[platform]


call the specific backend depending on the platform flag

casparvitch · 2022-10-18T06:02:36Z

Gpufit/python/tests/test_gaussian_fit_1d.py

-        assert (chi_squares < 1e-6)
-        assert (states == 0)
-        assert (number_iterations <= max_n_iterations)
+        self.assertTrue(chi_squares < 1e-6)


self.assertTrue is the py unittest way - it prints the results prettier :)

casparvitch · 2022-10-18T06:03:56Z

Gpufit/python/tests/test_gaussian_fit_1d.py

@@ -26,7 +26,7 @@ def generate_gauss_1d(parameters, x):

 class Test(unittest.TestCase):

-    def test_gaussian_fit_1d(self):


gauss_1d shows up in the test from the filename, no need to duplicate

casparvitch · 2022-10-18T06:04:42Z

Gpufit/python/tests/test_gaussian_fit_1d.py


-if __name__ == '__main__':
+    def test_cpufit(self):


new cpufit test

casparvitch · 2022-10-18T06:05:43Z

tests/Consistency.cpp

+	clean_resize(cpu.n_iterations, i.n_fits);
+
+	// call to cpufit, store output
+	int const cpu_status


MSVS is not happy about this

casparvitch · 2022-10-18T06:06:40Z

tests/utils.cpp

@@ -36,7 +36,7 @@ void generate_gauss_2d(std::vector< REAL > & v, std::vector< REAL > const & p)
        {
            REAL const argx = ((i - p[1]) * (i - p[1]));
            REAL const ex = exp(-(argx + argy) / (2 * p[3] * p[3]));
-            v[j * n + i] = p[0] * ex + p[3];
+            v[j * n + i] = p[0] * ex + p[4];


casparvitch · 2022-10-18T06:07:55Z

tests/utils.h

 	REAL tolerance;
 	int max_n_iterations;

 	std::vector< REAL > user_info_; // user info is REAL

+	std::vector< REAL> expected_parameters; // size zero to not check


this vector holds the true_parameters we set up with, then we compare fit result with this vector, to test for correctness.

casparvitch · 2022-10-18T23:32:58Z

The above were comments to help if anyone reads the PR.
Further info on windows build issue

MSVS is complaining about the new call to cpufit_constrained in the test suite (Gpufit\tests\Consistency.cpp)

MSVS builds with those functions commented out, but cpufit_constrained cannot be found by python in the cpufit dll.

casparvitch · 2022-11-17T00:43:11Z

Friendly bump, happy for this to not be merged as there's quite a lot of changes.

Would appreciate any tips to get it to link properly on windows though, so my lab can use it.

Cheers

Changes: 1. fix bugs in cpufit test functions 2. add test for correctness in Consistency.cpp. 3. add tests for cpufit alone (for when no gpu) to Consistency.cpp 4. add simple tests for constrained fit in Consistency.cpp 5. attempt (incomplete) to add cpufit api to pyGpufit Related to #110 amend cmake to copy cpufit library over to pygpufit clean up python/pyGpufit unit tests clean up of python/pyGpufit unit tests

casparvitch · 2023-09-02T05:30:11Z

Alternative approached merged in PR #125

superchromix requested a review from jkfindeisen October 9, 2022 13:27

superchromix self-assigned this Oct 9, 2022

casparvitch marked this pull request as ready for review October 12, 2022 06:11

casparvitch marked this pull request as draft October 18, 2022 05:57

casparvitch commented Oct 18, 2022

View reviewed changes

Gpufit/info.cpp

@@ -1,5 +1,6 @@

#include "info.h"

#include <algorithm>

#include <limits>

Copy link

Contributor Author

casparvitch Oct 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#106

casparvitch commented Oct 18, 2022

View reviewed changes

Gpufit/python/pygpufit/gpufit.py

@@ -86,7 +96,7 @@ def _valid_id(cls, id):

Copy link

Contributor Author

casparvitch Oct 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add flag for cpu/gpu

casparvitch commented Oct 18, 2022

View reviewed changes

Gpufit/python/tests/test_gaussian_fit_1d.py

if __name__ == '__main__':

def test_cpufit(self):

Copy link

Contributor Author

casparvitch Oct 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new cpufit test

casparvitch commented Oct 18, 2022

View reviewed changes

casparvitch closed this Sep 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose cpufit API to python #114

Expose cpufit API to python #114

casparvitch commented Oct 9, 2022 •

edited

Loading

superchromix commented Oct 9, 2022

casparvitch commented Oct 9, 2022

superchromix commented Oct 9, 2022

casparvitch commented Oct 12, 2022

casparvitch commented Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch Oct 18, 2022

casparvitch commented Oct 18, 2022

casparvitch commented Nov 17, 2022

casparvitch commented Sep 2, 2023

		@@ -15,33 +15,43 @@
		package_dir = os.path.dirname(os.path.realpath(__file__))

		@@ -26,7 +26,7 @@ def generate_gauss_1d(parameters, x):

		class Test(unittest.TestCase):

		def test_gaussian_fit_1d(self):

Expose cpufit API to python #114

Expose cpufit API to python #114

Conversation

casparvitch commented Oct 9, 2022 • edited Loading

superchromix commented Oct 9, 2022

casparvitch commented Oct 9, 2022

superchromix commented Oct 9, 2022

casparvitch commented Oct 12, 2022

casparvitch commented Oct 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

casparvitch commented Oct 18, 2022

casparvitch commented Nov 17, 2022

casparvitch commented Sep 2, 2023

casparvitch commented Oct 9, 2022 •

edited

Loading