diff --git a/docs/how-to-use-and-FAQ/openmp-best-practice.md b/docs/how-to-use-and-FAQ/openmp-best-practice.md new file mode 100755 index 000000000..14adb48c8 --- /dev/null +++ b/docs/how-to-use-and-FAQ/openmp-best-practice.md @@ -0,0 +1,74 @@ +ncnn openmp best practice + +### CPU loadaverage is too high with ncnn. + + When inference the neural network with ncnn, the cpu occupancy is very high even all CPU cores occupancy close to 100%. + + If there are other threads or processes that require more cpu resources, the running speed of the program will drop severely. + +### The root cause of high CPU usage + +1. ncnn uses openmp API to speed up the inference compute. the thread count equals to the cpu core count. If the computing work need to run frequently, it must consume many cpu resources. + +2. There is a thread pool managed by openmp, the pool size is equal to the cpu core size. (the max vulue is 15 if there are much more cpu cores?) + Openmp need to sync the thread when acquiring and returning threads to the pool. In order to improve efficiency, almost all omp implementations use spinlock synchronization (except for simpleomp). + The default spin time of the spinlock is 200ms. So after a thread is scheduled, the thread need to busy-wait up to 200ms. + +### Why the CPU usage is still high even using vulkan GPU acceleration. + +1. Openmp is also used when loading the param bin file, and this part runs on cpu. + +2. The fp32 to fp16 conversion before and after the GPU memory upload is executed on the cpu, and this part of the logic also uses openmp. + +### Solution +``` +1. Bind to the specific cpu core. +``` + If you use a device with large and small core CPUs, it is recommended to bind large or small cores through ncnn::set_cpu_powersave(int). Note that Windows does not support binding cores. By the way, it's possible to have multiple threadpool using openmp. A new threadpool will be created for a new thread scope. +Suppose your platform is 2 big cores + 4 little cores, and you want to execute model A on 2 big cores and model B on 4 little cores concurrently. + +create two threads via std::thread or pthread + ``` + void thread_1() + { + ncnn::set_cpu_powersave(2); // bind to big cores + netA.opt.num_threads = 2; + } + + void thread_2() + { + ncnn::set_cpu_powersave(1); // bind to little cores + netB.opt.num_threads = 4; + } + ``` + +``` +2. Use fewer threads. +``` + Set the number of threads to half of the cpu cores count or less through ncnn::set_omp_num_threads(int) or change net.opt.num_threads field. If you are coding with clang libomp, it's recommended that the number of threads does not exceed 8. If you use other omp libraries, it is recommended that the number of threads does not exceed 4. +``` +3. Reduce openmp spinlock blocktime. +``` + You can modify openmp blocktime by call ncnn::set_kmp_blocktime(int) method or modify net.opt.openmp_blocktime field. + This argument is the spin time set by the ncnn API, and the default is 20ms.You can set a smaller value according to + the situation, or directly change it to 0. + + Limitations: At present, only the libomp library of clang is implemented. Neither vcomp nor libgomp have corresponding interfaces. + If it is not compiled with clang, this value is still 200ms by default. + If you use vcomp or libgomp, you can use the environment variable OMP_WAIT_POLICY=PASSIVE to disable spin time. If you use simpleomp, + It's no need to set this parameter. +``` +4. Limit the number of threads available in the openmp thread pool. +``` + Even if the number of openmp threads is reduced, the CPU occupancy rate may still be high. This is more common on servers with + particularly many CPU cores. + This is because the waiting threads in the thread pool use a spinlock to busy-wait, which can be reducedby limiting the number of + threads available in the thread pool. + + Generally, you can set the OMP_THREAD_LIMIT environment variable. simpleomp currently does not support this feature so it's no need to be set. + Note that this environment variable is only valid if it is set before the program starts. +``` +5. Disable openmp completely +``` + If there is only one cpu core, or use the vulkan gpu acceleration, it is recommended to disable openmp, just specify -DNCNN_OPENMP=OFF + when compiling with cmake. \ No newline at end of file diff --git a/docs/how-to-use-and-FAQ/openmp-best-practice.zh.md b/docs/how-to-use-and-FAQ/openmp-best-practice.zh.md new file mode 100755 index 000000000..0481cdb58 --- /dev/null +++ b/docs/how-to-use-and-FAQ/openmp-best-practice.zh.md @@ -0,0 +1,70 @@ +ncnn openmp 最佳实践 + +### ncnn占用过多cpu资源 + + 使用ncnn推理运算,cpu占用非常高甚至所有核心占用都接近100%。 + + 如果还有其它线程或进程需要较多的cpu资源,运行速度下降严重。 + +### cpu占用高的根本原因 + +1. ncnn使用openmp API控制多线程加速推理计算。默认情况下,线程数等于cpu内核数。如果推理需要高频率运行,必然占用大部分 + cpu资源。 + +2. openmp内部维护一个线程池,线程池最大可用线程数等于cpu内核数。(核心过多时最大限制是15?)获取和归还线程时需要同步。 + + 为了提高效率,几乎所有omp实现都使用了自旋锁同步(simpleomp除外)。自旋锁默认的spin time是200ms。因此一个线程被调度后, + 需要忙等待最多200ms。 + +### 为什么使用vulkan加速后cpu占用依然很高。 + +1. 加载参数文件时也使用了openmp,这部分是在cpu上运行的。 + +2. 显存上传前和下载后的 fp32 fp16转换是在cpu上执行的,这部分逻辑也使用了openmp。 + +### 解决方法 + +``` +1. 绑核 +``` + 如果使用有大小核cpu的设备,建议通过ncnn::set_cpu_powersave(int)绑定大核或小核,注意windows系统不支持绑核。顺便说一下,ncnn支持不同的模型运行在不同的核心。假设硬件平台有2个大核,4个小核,你想把netA运行在大核,netB运行在小核。 + 可以通过std::thread or pthread创建两个线程,运行如下代码: + + ``` + void thread_1() + { + ncnn::set_cpu_powersave(2); // bind to big cores + netA.opt.num_threads = 2; + } + + void thread_2() + { + ncnn::set_cpu_powersave(1); // bind to little cores + netB.opt.num_threads = 4; + } + ``` + +``` +2. 使用更少的线程数。 +``` + 通过ncnn::set_omp_num_threads(int)或者net.opt.num_threads字段设置线程数为cpu内核数的一半或更小。如果使用clang的libomp, + 建议线程数不超过8,如果使用其它omp库,建议线程数不超过4。 +``` +3. 减小openmp blocktime。 +``` + 可以修改ncnn::set_kmp_blocktime(int)或者修改net.opt.openmp_blocktime,这个参数是ncnn API设置的spin time,默认是20ms。 + 可以根据情况设置更小的值,或者直接改为0。 + + 局限:目前只有clang的libomp库有实现,vcomp和libgomp都没有相应接口,如果不是使用clang编译的,这个值默认还是200ms。 + 如果使用vcomp或libgomp, 可以使用环境变量OMP_WAIT_POLICY=PASSIVE禁用spin time,如果使用simpleomp,不需要设置这个参数。 +``` +4. 限制openmp线程池可用线程数量。 +``` + 即使减小了openmp线程数量,cpu占用率仍然可能会很高。这在cpu核心特别多的服务器上比较常见。这是因为线程池中的等待线程使用 + 自旋锁忙等待,可以通过限制线程池可用线程数量减轻这种影响。 + + 一般可以通过设置OMP_THREAD_LIMIT环境变量。simpleomp目前不支持这一特性,不需要设置。注意这个环境变量仅在程序启动前设置才有效。 +``` +5. 完全禁用openmp +``` + 如果只有一个cpu核心,或者使用vulkan加速,建议关闭openmp, cmake编译时指定-DNCNN_OPENMP=OFF即可。 \ No newline at end of file