BKM for intra-op vs. inter-op parallelism?

[moderato]

Hi, I'm experimenting on an application involving both the so-called intra-op and inter-op parallelism (having a std::thread-based program with each thread calling OpenBLAS functions). In my application, each thread calls a bf16 gemm kernel to compute a tile of a tensor and there are some other ops that follow.

My question is what is the best practice for my application? I suppose by default OpenBLAS is used by calling one function from one thread, while that function will spawn multiple threads and do the compute. In my case, I'm wondering if calling OpenBLAS (multithreaded) GEMM functions from multiple threads will cause the GEMM functions to interfere each other and/or compete for resources? Is there anything I should do during the build? Or should I call any level3 or level2 functions instead of just calling the default sbgemm one?

Thanks in advance!

[martin-frbg]

Yes, sounds like you'd get best performance from using a single-threaded OpenBLAS (build options USE_THREAD=0 USE_LOCKING=1) or by calling openblas_set_num_threads(1) (or maybe any other small number, if your computer is big or your program is not using many threads at that momemt) before entering a multithreaded code section that makes BLAS calls.

[moderato]

@martin-frbg Thank you for the answer. I tried both options and the latencies are almost the same. Seems like threads are being handled well so the bottleneck might be somewhere else. My machine is a Windows PC with 6 physical / 12 logical cores btw.