基本思想：需要移植一个自训练的关键点模型到某开发板上,看到社区已经有了，记录一下流程ncnn/mnn框架流程，然后在移植开发板上，以备自己以后自训练使用
实验数据
链接: https://pan.baidu.com/s/1Bd8lLtmWLyX64Jzc5743EQ?pwd=yhgh 提取码: yhgh
一、配置环境
ubuntu@ubuntu:~$ pip install paddlepaddle-gpu==2.2.2.post101 -f https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html -i https://mirror.baidu.com/pypi/simple
下载源码
ubuntu@ubuntu:~$ git clone https://github.com/PaddlePaddle/PaddleDetection.git
Cloning into 'PaddleDetection'...
remote: Enumerating objects: 255570, done.
remote: Counting objects: 100% (63/63), done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 255570 (delta 22), reused 37 (delta 16), pack-reused 255507
Receiving objects: 100% (255570/255570), 414.49 MiB | 856.00 KiB/s, done.
Resolving deltas: 100% (208587/208587), done
下载模型
https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.4/configs/keypoint/tiny_pose
测试检测+识别
ubuntu@ubuntu:~/PaddleDetection$ python3 deploy/python/det_keypoint_unite_infer.py --det_model_dir=/home/ubuntu/PaddleDetection/picodet_v2_s_192_pedestrian --keypoint_model_dir=/home/ubuntu/PaddleDetection/tinypose_128x96 --image_file=/home/ubuntu/PaddleDetection/demo/hrnet_demo.jpg --device=cpu
keypoint visualize image saved to: output/hrnet_demo_vis.jpg
------------------ Inference Time Info ----------------------
total_time(ms): 55.599999999999994, img_num: 1
average latency time(ms): 55.60, QPS: 17.985612
preprocess_time(ms): 2.40, inference_time(ms): 53.20, postprocess_time(ms): 0.00
------------------ Inference Time Info ----------------------
total_time(ms): 23.8, img_num: 1
average latency time(ms): 23.80, QPS: 42.016807
preprocess_time(ms): 1.90, inference_time(ms): 15.10, postprocess_time(ms): 6.8
测试识别
ubuntu@ubuntu:~/PaddleDetection$ python3 deploy/python/keypoint_infer.py --model_dir=/home/ubuntu/PaddleDetection/tinypose_128x96 --image_file=./demo/hrnet_demo.jpg --device=CPU --threshold=0.5
二、先转一下官方的模型，自己训练的就很简单了
ubuntu@ubuntu:~/PaddleDetection$ pip install paddle2onnx -i https://pypi.mirrors.ustc.edu.cn/simple/
然后转模型
ubuntu@ubuntu:~/PaddleDetection$ paddle2onnx --model_dir /home/ubuntu/PaddleDetection/tinypose_128x96/infer_cfg.yml --model_filename /home/ubuntu/PaddleDetection/tinypose_128x96/model.pdmodel --params_filename /home/ubuntu/PaddleDetection/tinypose_128x96/model.pdiparams --opset_version 11 --save_file paddlepose.onnx
[Paddle2ONNX] Start to parse PaddlePaddle model...
[Paddle2ONNX] Model file path: /home/ubuntu/PaddleDetection/tinypose_128x96/model.pdmodel
[Paddle2ONNX] Paramters file path: /home/ubuntu/PaddleDetection/tinypose_128x96/model.pdiparams
[Paddle2ONNX] Start to parsing Paddle model...
[Paddle2ONNX] Use opset_version = 11 for ONNX export.
[Paddle2ONNX] PaddlePaddle model is exported as ONNX format now.
2023-03-07 13:43:40 [INFO] ===============Make PaddlePaddle Better!================
2023-03-07 13:43:40 [INFO] A little survey: https://iwenjuan.baidu.com/?code=r8hu2s
三、先转ncnn模型，发现不支持op存在
ubuntu@ubuntu:~/ncnn/build/install/bin$ python3 -m onnxsim ~/PaddleDetection/paddlepose.onnx ~/PaddleDetection/paddlepose_sim.onnx
Simplifying...
Finish! Here is the difference:
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ ┃ Original Model ┃ Simplified Model ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ Add │ 79 │ 79 │
│ ArgMax │ 1 │ 1 │
│ BatchNormalization │ 268 │ 0 │
│ Concat │ 52 │ 52 │
│ Constant │ 1511 │ 0 │
│ Conv │ 269 │ 269 │
│ Relu │ 144 │ 144 │
│ Reshape │ 99 │ 99 │
│ Resize │ 29 │ 29 │
│ Shape │ 3 │ 3 │
│ Slice │ 3 │ 3 │
│ Split │ 49 │ 49 │
│ Transpose │ 49 │ 49 │
│ Model Size │ 5.4MiB │ 5.1MiB │
└────────────────────┴────────────────┴──────────────────┘
ubuntu@ubuntu:~/ncnn/build/install/bin$ ./onnx2ncnn ~/PaddleDetection/paddlepose.onnx ~/PaddleDetection/paddlepose.param ~/PaddleDetection/paddlepose.bin
Shape not supported yet!
Unknown data type 0
Unsupported Resize scales and sizes are all empty!
Shape not supported yet!
Unknown data type 0
Unsupported Resize scales and sizes are all empty!
Shape not supported yet!
Unknown data type 0
Unsupported Resize scales and sizes are all empty!
ArgMax not supported yet!
# axis=-1
# keepdims=0
四、修正一下固定的输入大小
ubuntu@ubuntu:~$ git clone https://github.com/jiangjiajun/PaddleUtils.git
设置固定输入大小
ubuntu@ubuntu:~/PaddleUtils/paddle$ python3 paddle_infer_shape.py --model_dir /home/ubuntu/PaddleDetection/tinypose_128x96 --model_filename /home/ubuntu/PaddleDetection/tinypose_128x96/model.pdmodel --params_filename /home/ubuntu/PaddleDetection/tinypose_128x96/model.pdiparams --save_dir /home/ubuntu/PaddleDetection/tinypose_128x96/new_model --input_shape_dict="{'image':[1,3,128,96]}"
ubuntu@ubuntu:~/PaddleUtils/paddle$ paddle2onnx --model_dir /home/ubuntu/PaddleDetection/tinypose_128x96/infer_cfg.yml --model_filename /home/ubuntu/PaddleDetection/tinypose_128x96/new_model/model.pdmodel --params_filename /home/ubuntu/PaddleDetection/tinypose_128x96/new_model/model.pdiparams --opset_version 11 --save_file /home/ubuntu/PaddleDetection/tinypose_128x96/new_model/paddlepose.onnx然后sim一下，就可以在转ncnn试一下
ubuntu@ubuntu:~/PaddleUtils/paddle$ python3 -m onnxsim /home/ubuntu/PaddleDetection/tinypose_128x96/new_model/paddlepose.onnx /home/ubuntu/PaddleDetection/tinypose_128x96/new_model/paddlepose_sim.onnx
Simplifying...
Finish! Here is the difference:
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ ┃ Original Model ┃ Simplified Model ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ Add │ 79 │ 79 │
│ ArgMax │ 1 │ 1 │
│ BatchNormalization │ 268 │ 0 │
│ Concat │ 52 │ 49 │
│ Constant │ 1511 │ 0 │
│ Conv │ 269 │ 269 │
│ Relu │ 144 │ 144 │
│ Reshape │ 99 │ 99 │
│ Resize │ 29 │ 29 │
│ Shape │ 3 │ 0 │
│ Slice │ 3 │ 0 │
│ Split │ 49 │ 49 │
│ Transpose │ 49 │ 49 │
│ Model Size │ 5.4MiB │ 5.1MiB │
└────────────────────┴────────────────┴──────────────────┘
问题就很少了，只剩下一个函数不支持，就很容易解决了
ubuntu@ubuntu:~/ncnn/build/install/bin$ ./onnx2ncnn /home/ubuntu/Downloads/tinypose_128x96/a/paddlepose_sim.onnx /home/ubuntu/Downloads/tinypose_128x96/a/example.param /home/ubuntu/Downloads/tinypose_128x96/a/example.bin
ArgMax not supported yet!
# axis=-1
# keepdims=0
提取一下可用的onnx,重新转ncnn
提取有效
import onnx
input_path = "/home/ubuntu/Downloads/tinypose_128x96/a/paddlepose_sim.onnx"
output_path = "/home/ubuntu/Downloads/tinypose_128x96/a/paddlepose_sim_remove_argmax.onnx"
input_names = ["image"]
output_names = ["reshape2_98.tmp_0"]
onnx.utils.extract_model(input_path, output_path, input_names, output_names)
其实我只要conv2d_441/temp_1的数据，他是关键点数据，另一个个输出是heatmap数据，我并不需要～
然后在转ncnn模型
ubuntu@ubuntu:~/ncnn/build/install/bin$ ./onnx2ncnn /home/ubuntu/Downloads/tinypose_128x96/a/paddlepose_sim_remove_argmax.onnx /home/ubuntu/Downloads/tinypose_128x96/a/paddlepose_sim_remove_argmax.param /home/ubuntu/Downloads/tinypose_128x96/a/paddlepose_sim_remove_argmax.bin
五、写一段ncnn的逻辑呗
cmakelists.txt
cmake_minimum_required(VERSION 3.16)
project(prj)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -o3")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -o3")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fopenmp ")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fopenmp ")
set(CMAKE_CXX_STANDARD 11)
include_directories(${CMAKE_SOURCE_DIR}/include)
include_directories(${CMAKE_SOURCE_DIR}/include/ncnn)
find_package(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})
#导入ncnn
add_library(libncnn STATIC IMPORTED)
set_target_properties(libncnn PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/lib/libncnn.a)
add_executable(prj main.cpp)
target_link_libraries(prj ${OpenCV_LIBS} libncnn)
代码
#include <iostream>
#include <algorithm>
#include <vector>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/opencv.hpp>
#include "platform.h"
#include "net.h"
#include "omp.h"
#include "chrono"
#if NCNN_VULKAN
#include "gpu.h"
#endif // NCNN_VULKAN
using namespace std;
using namespace chrono;
struct Keypoints {
float x;
float y;
float score;
Keypoints() : x(0), y(0), score(0) {}
Keypoints(float x, float y, float score) : x(x), y(y), score(score) {}
};
void rotate_point(float *pt, float angle_rad, float *rotated_pt) {
float sn = sin(angle_rad);
float cs = cos(angle_rad);
float new_x = pt[0] * cs - pt[1] * sn;
float new_y = pt[0] * sn + pt[1] * cs;
rotated_pt[0] = new_x;
rotated_pt[1] = new_y;
}
void _get_3rd_point(cv::Point2f a, cv::Point2f b, float *direction) {
float direction_0 = a.x - b.x;
float direction_1 = a.y - b.y;
direction[0] = b.x - direction_1;
direction[1] = b.y + direction_0;
}
void get_affine_transform(float *center, float *scale, float rot, float *output_size, float *shift, bool inv,
cv::Mat &trans) {
float scale_tmp[] = {0, 0};
//scale_tmp[0] = scale[0] * 200.0;
//scale_tmp[1] = scale[1] * 200.0;
float src_w = scale[0];
float dst_w = output_size[0];
float dst_h = output_size[1];
float rot_rad = M_PI * rot / 180;
float pt[] = {0, 0};
pt[0] = 0;
pt[1] = src_w * (-0.5);
float src_dir[] = {0, 0};
rotate_point(pt, rot_rad, src_dir);
float dst_dir[] = {0, 0};
dst_dir[0] = 0;
dst_dir[1] = dst_w * (-0.5);
cv::Point2f src[3] = {cv::Point2f(0, 0), cv::Point2f(0, 0), cv::Point2f(0, 0)};
src[0] = cv::Point2f(center[0] + scale_tmp[0] * shift[0], center[1] + scale_tmp[1] * shift[1]);
src[1] = cv::Point2f(center[0] + src_dir[0] + scale_tmp[0] * shift[0],
center[1] + src_dir[1] + scale_tmp[1] * shift[1]);
float direction_src[] = {0, 0};
_get_3rd_point(src[0], src[1], direction_src);
src[2] = cv::Point2f(direction_src[0], direction_src[1]);
cv::Point2f dst[3] = {cv::Point2f(0, 0), cv::Point2f(0, 0), cv::Point2f(0, 0)};
dst[0] = cv::Point2f(dst_w * 0.5, dst_h * 0.5);
dst[1] = cv::Point2f(dst_w * 0.5 + dst_dir[0], dst_h * 0.5 + dst_dir[1]);
float direction_dst[] = {0, 0};
_get_3rd_point(dst[0], dst[1], direction_dst);
dst[2] = cv::Point2f(direction_dst[0], direction_dst[1]);
if (inv) {
trans = cv::getAffineTransform(dst, src);
} else {
trans = cv::getAffineTransform(src, dst);
}
}
void pretty_print(const ncnn::Mat &m, std::vector<float> &vec_heap) {
for (int q = 0; q < m.c; q++) {
const float *ptr = m.channel(q);
for (int z = 0; z < m.d; z++) {
for (int y = 0; y < m.h; y++) {
for (int x = 0; x < m.w; x++) {
// printf("%f ", ptr[x]);
vec_heap.emplace_back(ptr[x]);
}
ptr += m.w;
//printf("\n");
}
//printf("\n");
}
}
}
void
transform_preds(std::vector<cv::Point2f> coords, std::vector<Keypoints> &target_coords, float *center, float *scale,
int w, int h) {
float temp_scale[] = {scale[0] * 200, scale[1] * 200};
float rot=0;
bool inv=true;
float shift[2]={0};
cv::Mat trans;
float heap_wh[2]={(float)w,(float)h};
get_affine_transform(center, temp_scale, rot, heap_wh, shift, inv, trans);
//std::cout<<trans<<std::endl;
double mat_00=trans.at<double>(0,0);
double mat_01=trans.at<double>(0,1);
double mat_02=trans.at<double>(0,2);
double mat_10=trans.at<double>(1,0);
double mat_11=trans.at<double>(1,1);
double mat_12=trans.at<double>(1,2);
for (int i = 0; i < coords.size(); i++) {
target_coords[i].x = coords[i].x * mat_00 + coords[i].y *mat_01+1*mat_02;
target_coords[i].y = coords[i].x * mat_10 + coords[i].y *mat_11+1*mat_12;
}
}
int main(int argc, char **argv) {
float keypoint_score=0.3f;
cv::Mat bgr = cv::imread("/home/ubuntu/PaddleDetection/demo/hrnet_demo.jpg");
//cv::Mat rgb;
//cv::cvtColor(bgr, rgb, cv::COLOR_BGR2RGB);
float center_x=bgr.cols/2.0;
float center_y=bgr.rows/2.0;
float scale_x=bgr.cols;
float scale_y=bgr.rows;
float image_target_w = 128;
float image_target_h = 96;
float center[2] = {0, 0};
float scale[2] = {0, 0};
center[0]=center_x;
center[1]=center_y;
scale[0]=scale_x;
scale[1]=scale_y;
float rot = 0;
float shift[] = {0, 0};
bool inv = false;
float output_size[] = {image_target_h, image_target_w};
cv::Mat trans;
get_affine_transform(center, scale, rot, output_size, shift, inv, trans);
//std::cout << trans << std::endl;
cv::Mat detect_image;//= cv::Mat::zeros(image_target_w ,image_target_h, CV_8UC3);
cv::warpAffine(bgr, detect_image, trans, cv::Size(image_target_h, image_target_w), cv::INTER_LINEAR);
//std::cout << detect_image.cols << " " << detect_image.rows << std::endl;
//std::cout << detect_image<<std::endl;
ncnn::Net harnet;
harnet.load_param("/home/ubuntu/CLionProjects/untitled3/tinypose_128x96/model/paddlepose_sim_remove_argmax.param");
harnet.load_model("/home/ubuntu/CLionProjects/untitled3/tinypose_128x96/model/paddlepose_sim_remove_argmax.bin");
ncnn::Mat in = ncnn::Mat::from_pixels(detect_image.data, ncnn::Mat::PIXEL_BGR2RGB, detect_image.cols,
detect_image.rows);
// transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))
const float mean_vals[3] = {0.485f * 255.f, 0.456f * 255.f, 0.406f * 255.f};
const float norm_255[3] = {(1 / 0.229f / 255.f), (1 / 0.224f / 255.f), (1 / 0.225f / 255.f)};
in.substract_mean_normalize(mean_vals, norm_255);
fprintf(stderr, "input shape: %d %d %d %d\n", in.d, in.h, in.w, in.c);
auto start = chrono::high_resolution_clock::now(); //开始时间
ncnn::Extractor ex = harnet.create_extractor();
ex.input("image", in);//input 是 .param文件中输入节点名称
ncnn::Mat result;
ex.extract("conv2d_441.tmp_1", result);
//pretty_print(result);
auto end = chrono::high_resolution_clock::now(); //结束时间
auto duration = (end - start).count();
cout << "程序运行时间：" << setprecision(10) << duration / 1000000000.0 << "s"
<< "； " << duration / 1000000.0 << "ms"
<< "； " << duration / 1000.0 << "us"
<< endl;
fprintf(stderr, "output shape: %d %d %d %d\n", result.d, result.c, result.h, result.w);
int shape_d = result.d;
int shape_c = result.c;
int shape_w = result.w;
int shape_h = result.h;
std::vector<float> vec_heap;
pretty_print(result, vec_heap);
scale[0]=center[0]/100;
scale[1]=center[1]/100;
std::vector<Keypoints> all_preds;
std::vector<int> idx;
for (int i = 0; i < shape_c; i++) {
auto begin = vec_heap.begin() + i * shape_w * shape_h;
auto end = vec_heap.begin() + (i + 1) * shape_w * shape_h;
float maxValue = *max_element(begin, end);
int maxPosition = max_element(begin, end) - begin;
all_preds.emplace_back(Keypoints(0, 0, maxValue));
idx.emplace_back(maxPosition);
}
std::vector<cv::Point2f> vec_point;
for (int i = 0; i < idx.size(); i++) {
int x = idx[i] % shape_w;
int y = idx[i] / shape_w;
vec_point.emplace_back(cv::Point2f(x, y));
}
for (int i = 0; i < shape_c; i++) {
int px = floor(vec_point[i].x+0.5);
int py = floor(vec_point[i].y+0.5);
if (px > 1 && px < shape_w - 1 && py > 1 && py < shape_h - 1) {
float diff_0 = vec_heap[py * shape_w + px + 1] - vec_heap[py * shape_w + px - 1];
float diff_1 = vec_heap[(py + 1) * shape_w + px] - vec_heap[(py - 1) * shape_w + px];
vec_point[i].x += diff_0 == 0 ? 0 : (diff_0 > 0) ? 0.25 : -0.25;
vec_point[i].y += diff_1 == 0 ? 0 : (diff_1 > 0) ? 0.25 : -0.25;
}
}
center[0]=ceil(center[0]);
center[1]=ceil(center[1]);
transform_preds(vec_point, all_preds, center, scale, shape_w, shape_h);
int skeleton[][2] = {{0, 1}, {0, 2}, {1, 3}, {2, 4}, {3, 5}, {4, 6}, {5, 7}, {6, 8},
{7, 9}, {8, 10}, {5, 11}, {6, 12}, {11, 13}, {12, 14},
{13, 15}, {14, 16}, {11, 12}};
for (int i = 0; i < all_preds.size(); i++) {
if(all_preds[i].score>keypoint_score){
cv::circle(bgr, cv::Point(all_preds[i].x, all_preds[i].y), 3, cv::Scalar(0, 255, 120), -1);//画点，其实就是实心圆
}
}
for (int i = 0; i < sizeof(skeleton) / sizeof(sizeof(skeleton[1])); i++) {
int x0 = all_preds[skeleton[i][0]].x;
int y0 = all_preds[skeleton[i][0]].y;
int x1 = all_preds[skeleton[i][1]].x;
int y1 = all_preds[skeleton[i][1]].y;
cv::line(bgr, cv::Point(x0, y0), cv::Point(x1, y1),
cv::Scalar(0, 255, 0), 1);
}
cv::imwrite("demo.jpg",bgr);
cv::imshow("image", bgr);
cv::waitKey(0);
return 0;
}
测试结果
/home/ubuntu/CLionProjects/untitled3/cmake-build-debug/prj
input shape: 1 128 96 3
output shape: 1 17 32 24
程序运行时间：0.005697311s； 5.697311ms； 5697.311us
图片
六、mnn代码
ubuntu@ubuntu:~/MNN/build$ ./MNNConvert -f ONNX --modelFile /home/ubuntu/CLionProjects/untitled3/tinypose_128x96/model/paddlepose_sim.onnx --MNNModel /home/ubuntu/CLionProjects/untitled3/tinypose_128x96/model/paddlepose_sim.mnn --bizCode MNN
The device support dot:0, support fp16:0, support i8mm: 0
Start to Convert Other Model Format To MNN Model...
[17:16:06] /home/ubuntu/MNN/tools/converter/source/onnx/onnxConverter.cpp:40: ONNX Model ir version: 8
Start to Optimize the MNN Net...
inputTensors : [ image, ]
outputTensors: [ argmax_0.tmp_0, conv2d_441.tmp_1, ]
Converted Success!
cmkelists.txt
cmake_minimum_required(VERSION 3.16)
project(untitled22)
set(CMAKE_CXX_FLAGS "-std=c++11")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fopenmp ")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fopenmp")
set(CMAKE_CXX_STANDARD 11)
include_directories(${CMAKE_SOURCE_DIR})
include_directories(${CMAKE_SOURCE_DIR}/include)
include_directories(${CMAKE_SOURCE_DIR}/include/MNN)
find_package(OpenCV REQUIRED)
#message(STATUS ${OpenCV_INCLUDE_DIRS})
#添加头文件
include_directories(${OpenCV_INCLUDE_DIRS})
#链接Opencv库
add_library(libmnn SHARED IMPORTED)
set_target_properties(libmnn PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/lib/libMNN.so)
add_executable(untitled22 main.cpp)
target_link_libraries(untitled22 ${OpenCV_LIBS} libmnn )
测试代码
#include <iostream>
#include <algorithm>
#include <vector>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/opencv.hpp>
#include<MNN/Interpreter.hpp>
#include<MNN/ImageProcess.hpp>
using namespace std;
using namespace chrono;
struct Keypoints {
float x;
float y;
float score;
Keypoints() : x(0), y(0), score(0) {}
Keypoints(float x, float y, float score) : x(x), y(y), score(score) {}
};
struct Box {
float center_x;
float center_y;
float scale_x;
float scale_y;
float scale_prob;
float score;
Box() : center_x(0), center_y(0), scale_x(0), scale_y(0), scale_prob(0), score(0) {}
Box(float center_x, float center_y, float scale_x, float scale_y, float scale_prob, float score) :
center_x(center_x), center_y(center_y), scale_x(scale_x), scale_y(scale_y), scale_prob(scale_prob),
score(score) {}
};
void rotate_point(float *pt, float angle_rad, float *rotated_pt) {
float sn = sin(angle_rad);
float cs = cos(angle_rad);
float new_x = pt[0] * cs - pt[1] * sn;
float new_y = pt[0] * sn + pt[1] * cs;
rotated_pt[0] = new_x;
rotated_pt[1] = new_y;
}
void _get_3rd_point(cv::Point2f a, cv::Point2f b, float *direction) {
float direction_0 = a.x - b.x;
float direction_1 = a.y - b.y;
direction[0] = b.x - direction_1;
direction[1] = b.y + direction_0;
}
void get_affine_transform(float *center, float *scale, float rot, float *output_size, float *shift, bool inv,
cv::Mat &trans) {
float scale_tmp[] = {0, 0};
//scale_tmp[0] = scale[0] * 200.0;
//scale_tmp[1] = scale[1] * 200.0;
float src_w = scale[0];
float dst_w = output_size[0];
float dst_h = output_size[1];
float rot_rad = M_PI * rot / 180;
float pt[] = {0, 0};
pt[0] = 0;
pt[1] = src_w * (-0.5);
float src_dir[] = {0, 0};
rotate_point(pt, rot_rad, src_dir);
float dst_dir[] = {0, 0};
dst_dir[0] = 0;
dst_dir[1] = dst_w * (-0.5);
cv::Point2f src[3] = {cv::Point2f(0, 0), cv::Point2f(0, 0), cv::Point2f(0, 0)};
src[0] = cv::Point2f(center[0] + scale_tmp[0] * shift[0], center[1] + scale_tmp[1] * shift[1]);
src[1] = cv::Point2f(center[0] + src_dir[0] + scale_tmp[0] * shift[0],
center[1] + src_dir[1] + scale_tmp[1] * shift[1]);
float direction_src[] = {0, 0};
_get_3rd_point(src[0], src[1], direction_src);
src[2] = cv::Point2f(direction_src[0], direction_src[1]);
cv::Point2f dst[3] = {cv::Point2f(0, 0), cv::Point2f(0, 0), cv::Point2f(0, 0)};
dst[0] = cv::Point2f(dst_w * 0.5, dst_h * 0.5);
dst[1] = cv::Point2f(dst_w * 0.5 + dst_dir[0], dst_h * 0.5 + dst_dir[1]);
float direction_dst[] = {0, 0};
_get_3rd_point(dst[0], dst[1], direction_dst);
dst[2] = cv::Point2f(direction_dst[0], direction_dst[1]);
if (inv) {
trans = cv::getAffineTransform(dst, src);
} else {
trans = cv::getAffineTransform(src, dst);
}
}
void
transform_preds(std::vector<cv::Point2f> coords, std::vector<Keypoints> &target_coords, float *center, float *scale,
int w, int h) {
float temp_scale[] = {scale[0] * 200, scale[1] * 200};
float rot=0;
bool inv=true;
float shift[2]={0};
cv::Mat trans;
float heap_wh[2]={(float)w,(float)h};
get_affine_transform(center, temp_scale, rot, heap_wh, shift, inv, trans);
//std::cout<<trans<<std::endl;
double mat_00=trans.at<double>(0,0);
double mat_01=trans.at<double>(0,1);
double mat_02=trans.at<double>(0,2);
double mat_10=trans.at<double>(1,0);
double mat_11=trans.at<double>(1,1);
double mat_12=trans.at<double>(1,2);
for (int i = 0; i < coords.size(); i++) {
target_coords[i].x = coords[i].x * mat_00 + coords[i].y *mat_01+1*mat_02;
target_coords[i].y = coords[i].x * mat_10 + coords[i].y *mat_11+1*mat_12;
}
}
int main(int argc, char **argv) {
float keypoint_score=0.3f;
cv::Mat bgr = cv::imread("/home/ubuntu/PaddleDetection/demo/hrnet_demo.jpg");
//cv::Mat rgb;
//cv::cvtColor(bgr, rgb, cv::COLOR_BGR2RGB);
float center_x=bgr.cols/2.0;
float center_y=bgr.rows/2.0;
float scale_x=bgr.cols;
float scale_y=bgr.rows;
float image_target_w = 128;
float image_target_h = 96;
float center[2] = {0, 0};
float scale[2] = {0, 0};
center[0]=center_x;
center[1]=center_y;
scale[0]=scale_x;
scale[1]=scale_y;
float rot = 0;
float shift[] = {0, 0};
bool inv = false;
float output_size[] = {image_target_h, image_target_w};
cv::Mat trans;
get_affine_transform(center, scale, rot, output_size, shift, inv, trans);
std::cout << "trans= "<<trans << std::endl;
cv::Mat detect_image;//= cv::Mat::zeros(image_target_w ,image_target_h, CV_8UC3);
cv::warpAffine(bgr, detect_image, trans, cv::Size(image_target_h, image_target_w), cv::INTER_LINEAR);
//std::cout << detect_image.cols << " " << detect_image.rows << std::endl;
//std::cout << detect_image<<std::endl;
auto start = chrono::high_resolution_clock::now(); //开始时间
// MNN inference
auto mnnNet = std::shared_ptr<MNN::Interpreter>(MNN::Interpreter::createFromFile("/home/ubuntu/CLionProjects/untitled3/tinypose_128x96/model/paddlepose_sim.mnn"));
MNN::ScheduleConfig netConfig;
netConfig.type = MNN_FORWARD_CPU;
netConfig.numThread = 4;
auto session = mnnNet->createSession(netConfig);
auto input = mnnNet->getSessionInput(session, nullptr);
mnnNet->resizeTensor(input, {1, 3, (int) image_target_w, (int) image_target_h});
mnnNet->resizeSession(session);
MNN::CV::ImageProcess::Config config;
const float mean_vals[3] = {0.485f * 255.f, 0.456f * 255.f, 0.406f * 255.f};
const float norm_255[3] = {(1 / 0.229f / 255.f), (1 / 0.224f / 255.f), (1 / 0.225f / 255.f)};
std::shared_ptr<MNN::CV::ImageProcess> pretreat(
MNN::CV::ImageProcess::create(MNN::CV::BGR, MNN::CV::RGB, mean_vals, 3,
norm_255, 3));
pretreat->convert(detect_image.data, (int) image_target_h, (int) image_target_w, detect_image.step[0], input);
mnnNet->runSession(session);
auto heatmap = mnnNet->getSessionOutput(session, "conv2d_441.tmp_1");
MNN::Tensor heatmapHost(heatmap, heatmap->getDimensionType());
heatmap->copyToHostTensor(&heatmapHost);
std::vector<float> vec_heap;
for (int i = 0; i < heatmapHost.elementSize(); i++) {
//std::cout << heatmapHost.host<float>()[i] << " ";
vec_heap.emplace_back(heatmapHost.host<float>()[i]);
}
auto end = chrono::high_resolution_clock::now(); //结束时间
auto duration = (end - start).count();
cout << "程序运行时间：" << setprecision(10) << duration / 1000000000.0 << "s"
<< "； " << duration / 1000000.0 << "ms"
<< "； " << duration / 1000.0 << "us"
<< endl;
int shape_c = heatmapHost.channel();
int shape_w = heatmapHost.width();
int shape_h = heatmapHost.height();
fprintf(stderr, "output shape: %d %d %d %d\n", heatmapHost.dimensions(), shape_c, shape_h, shape_w);
scale[0]=center[0]/100;
scale[1]=center[1]/100;
std::vector<Keypoints> all_preds;
std::vector<int> idx;
for (int i = 0; i < shape_c; i++) {
auto begin = vec_heap.begin() + i * shape_w * shape_h;
auto end = vec_heap.begin() + (i + 1) * shape_w * shape_h;
float maxValue = *max_element(begin, end);
int maxPosition = max_element(begin, end) - begin;
all_preds.emplace_back(Keypoints(0, 0, maxValue));
idx.emplace_back(maxPosition);
}
std::vector<cv::Point2f> vec_point;
for (int i = 0; i < idx.size(); i++) {
int x = idx[i] % shape_w;
int y = idx[i] / shape_w;
vec_point.emplace_back(cv::Point2f(x, y));
}
for (int i = 0; i < shape_c; i++) {
int px = floor(vec_point[i].x+0.5);
int py = floor(vec_point[i].y+0.5);
if (px > 1 && px < shape_w - 1 && py > 1 && py < shape_h - 1) {
float diff_0 = vec_heap[py * shape_w + px + 1] - vec_heap[py * shape_w + px - 1];
float diff_1 = vec_heap[(py + 1) * shape_w + px] - vec_heap[(py - 1) * shape_w + px];
vec_point[i].x += diff_0 == 0 ? 0 : (diff_0 > 0) ? 0.25 : -0.25;
vec_point[i].y += diff_1 == 0 ? 0 : (diff_1 > 0) ? 0.25 : -0.25;
}
}
center[0]=ceil(center[0]);
center[1]=ceil(center[1]);
transform_preds(vec_point, all_preds, center, scale, shape_w, shape_h);
int skeleton[][2] = {{0, 1}, {0, 2}, {1, 3}, {2, 4}, {3, 5}, {4, 6}, {5, 7}, {6, 8},
{7, 9}, {8, 10}, {5, 11}, {6, 12}, {11, 13}, {12, 14},
{13, 15}, {14, 16}, {11, 12}};
for (int i = 0; i < all_preds.size(); i++) {
if(all_preds[i].score>keypoint_score){
cv::circle(bgr, cv::Point(all_preds[i].x, all_preds[i].y), 3, cv::Scalar(0, 255, 120), -1);//画点，其实就是实心圆
}
}
for (int i = 0; i < sizeof(skeleton) / sizeof(sizeof(skeleton[1])); i++) {
int x0 = all_preds[skeleton[i][0]].x;
int y0 = all_preds[skeleton[i][0]].y;
int x1 = all_preds[skeleton[i][1]].x;
int y1 = all_preds[skeleton[i][1]].y;
cv::line(bgr, cv::Point(x0, y0), cv::Point(x1, y1),
cv::Scalar(0, 255, 0), 1);
}
cv::imwrite("demo.jpg",bgr);
cv::imshow("image", bgr);
cv::waitKey(0);
mnnNet->releaseModel();
mnnNet->releaseSession(session);
return 0;
}
结果
七、openvino模型测试结果
首先转openvnino模型
ubuntu@ubuntu:~$ python3 /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo.py --input_model /home/ubuntu/CLionProjects/untitled3/tinypose_128x96/model/paddlepose_sim.onnx --output_dir /home/ubuntu/CLionProjects/untitled3/tinypose_128x96/model/FP16 --input_shape [1,3,96,128] --data_type FP16
或者
ubuntu@ubuntu:~$ python3 /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo.py --input_model /home/ubuntu/CLionProjects/untitled3/tinypose_128x96/model/paddlepose_sim.onnx --output_dir /home/ubuntu/CLionProjects/untitled3/tinypose_128x96/model/FP16 --input_shape [1,3,96,128] --mean_values=[123.675,116.28,103.53] --scale_values=[58.395,57.12,57.375] --data_type FP16 --reverse_input_channels
好像结果有点问题，蛀牙处在interpolate上，修正中，但是偶尔会存在检测正确的，结果很玄幻
cmake_minimum_required(VERSION 3.16.1)
set(CMAKE_CXX_STANDARD 14)
project(untitled6)
find_package(OpenCV REQUIRED)
find_package(ngraph REQUIRED)
find_package(InferenceEngine REQUIRED)
include_directories(
${OpenCV_INCLUDE_DIRS}
${CMAKE_CURRENT_SOURCE_DIR}
${CMAKE_CURRENT_BINARY_DIR}
)
add_executable(untitled6 main.cpp)
target_link_libraries(
untitled6
${InferenceEngine_LIBRARIES}
${NGRAPH_LIBRARIES}
${OpenCV_LIBS}
)
测试代码
#include <opencv2/opencv.hpp>
#include <inference_engine.hpp>
#include <iostream>
#include <chrono>
#include <opencv2/dnn/dnn.hpp>
#include <cmath>
using namespace std;
using namespace cv;
using namespace InferenceEngine;
using namespace std;
using namespace chrono;
struct Keypoints {
float x;
float y;
float score;
Keypoints() : x(0), y(0), score(0) {}
Keypoints(float x, float y, float score) : x(x), y(y), score(score) {}
};
struct Box {
float center_x;
float center_y;
float scale_x;
float scale_y;
float scale_prob;
float score;
Box() : center_x(0), center_y(0), scale_x(0), scale_y(0), scale_prob(0), score(0) {}
Box(float center_x, float center_y, float scale_x, float scale_y, float scale_prob, float score) :
center_x(center_x), center_y(center_y), scale_x(scale_x), scale_y(scale_y), scale_prob(scale_prob),
score(score) {}
};
void rotate_point(float *pt, float angle_rad, float *rotated_pt) {
float sn = sin(angle_rad);
float cs = cos(angle_rad);
float new_x = pt[0] * cs - pt[1] * sn;
float new_y = pt[0] * sn + pt[1] * cs;
rotated_pt[0] = new_x;
rotated_pt[1] = new_y;
}
void _get_3rd_point(cv::Point2f a, cv::Point2f b, float *direction) {
float direction_0 = a.x - b.x;
float direction_1 = a.y - b.y;
direction[0] = b.x - direction_1;
direction[1] = b.y + direction_0;
}
void get_affine_transform(float *center, float *scale, float rot, float *output_size, float *shift, bool inv,
cv::Mat &trans) {
float scale_tmp[] = {0, 0};
//scale_tmp[0] = scale[0] * 200.0;
//scale_tmp[1] = scale[1] * 200.0;
float src_w = scale[0];
float dst_w = output_size[0];
float dst_h = output_size[1];
float rot_rad = M_PI * rot / 180;
float pt[] = {0, 0};
pt[0] = 0;
pt[1] = src_w * (-0.5);
float src_dir[] = {0, 0};
rotate_point(pt, rot_rad, src_dir);
float dst_dir[] = {0, 0};
dst_dir[0] = 0;
dst_dir[1] = dst_w * (-0.5);
cv::Point2f src[3] = {cv::Point2f(0, 0), cv::Point2f(0, 0), cv::Point2f(0, 0)};
src[0] = cv::Point2f(center[0] + scale_tmp[0] * shift[0], center[1] + scale_tmp[1] * shift[1]);
src[1] = cv::Point2f(center[0] + src_dir[0] + scale_tmp[0] * shift[0],
center[1] + src_dir[1] + scale_tmp[1] * shift[1]);
float direction_src[] = {0, 0};
_get_3rd_point(src[0], src[1], direction_src);
src[2] = cv::Point2f(direction_src[0], direction_src[1]);
cv::Point2f dst[3] = {cv::Point2f(0, 0), cv::Point2f(0, 0), cv::Point2f(0, 0)};
dst[0] = cv::Point2f(dst_w * 0.5, dst_h * 0.5);
dst[1] = cv::Point2f(dst_w * 0.5 + dst_dir[0], dst_h * 0.5 + dst_dir[1]);
float direction_dst[] = {0, 0};
_get_3rd_point(dst[0], dst[1], direction_dst);
dst[2] = cv::Point2f(direction_dst[0], direction_dst[1]);
if (inv) {
trans = cv::getAffineTransform(dst, src);
} else {
trans = cv::getAffineTransform(src, dst);
}
}
void
transform_preds(std::vector<cv::Point2f> coords, std::vector<Keypoints> &target_coords, float *center, float *scale,
int w, int h) {
float temp_scale[] = {scale[0] * 200, scale[1] * 200};
float rot = 0;
bool inv = true;
float shift[2] = {0};
cv::Mat trans;
float heap_wh[2] = {(float) w, (float) h};
get_affine_transform(center, temp_scale, rot, heap_wh, shift, inv, trans);
//std::cout<<trans<<std::endl;
double mat_00 = trans.at<double>(0, 0);
double mat_01 = trans.at<double>(0, 1);
double mat_02 = trans.at<double>(0, 2);
double mat_10 = trans.at<double>(1, 0);
double mat_11 = trans.at<double>(1, 1);
double mat_12 = trans.at<double>(1, 2);
for (int i = 0; i < coords.size(); i++) {
target_coords[i].x = coords[i].x * mat_00 + coords[i].y * mat_01 + 1 * mat_02;
target_coords[i].y = coords[i].x * mat_10 + coords[i].y * mat_11 + 1 * mat_12;
}
}
int main(int argc, char const *argv[]) {
std::vector<float> meanVals = {-0.485f * 255.f * (1.0f / 0.229f) * (1.0 / 255.f),
-0.456f * 255.f * (1.0f / 0.224f) * (1.0 / 255.f),
-0.406f * 255.f * (1.0f / 0.225f) * (1.0 / 255.f)};
std::vector<float> normVals = {(1.0f / 0.229f) * (1.0 / 255.f),
(1.0f / 0.224f) * (1.0 / 255.f),
(1.0f / 0.225f) * (1.0 / 255.f)};
float keypoint_score = 0.3f;
cv::Mat bgr = cv::imread("/home/ubuntu/PaddleDetection/demo/hrnet_demo.jpg");
cv::Mat rgb;
cv::cvtColor(bgr, rgb, cv::COLOR_BGR2RGB);
float image_target_w = 128;
float image_target_h = 96;
float center_x = bgr.cols / 2.0;
float center_y = bgr.rows / 2.0;
float scale_x = bgr.cols;
float scale_y = bgr.rows;
float center[2] = {0, 0};
float scale[2] = {0, 0};
center[0] = center_x;
center[1] = center_y;
scale[0] = scale_x;
scale[1] = scale_y;
float rot = 0;
float shift[] = {0, 0};
bool inv = false;
float output_size[] = {image_target_h, image_target_w};
cv::Mat trans;
get_affine_transform(center, scale, rot, output_size, shift, inv, trans);
std::cout << "trans= " << trans << std::endl;
cv::Mat detect_image;//= cv::Mat::zeros(image_target_w ,image_target_h, CV_8UC3);
cv::Mat resize_img;
resize(rgb, resize_img, Size(image_target_h, image_target_w));
cv::warpAffine(resize_img, detect_image, trans, cv::Size(image_target_h, image_target_w), cv::INTER_LINEAR);
// std::vector<cv::Mat> rgb_channel(3);
// cv::split(detect_image, rgb_channel);
// for (int i = 0; i < rgb_channel.size(); i++) {
// rgb_channel[i].convertTo(rgb_channel[i], CV_32FC1, normVals[i], meanVals[i]);
//
// for (int j = 0; j < rgb_channel[i].rows; j++) {
// for (int k = 0; k < rgb_channel[i].cols; k++) {
// std::cout << rgb_channel[i].at<float>(j, k) << " ";;
// }
// std::cout << std::endl;
// }
// }
// cv::Mat image_dst;
//
// // norm_image.convertTo(image_dst, CV_32FC3, 1);
// // std::cout << "original image_dst type " << image_dst.depth() << std::endl; //CV_32FC3
// //没必要处理 ，因为rgb_channel 是CV_32FC1
// cv::merge(rgb_channel, image_dst);
// // std::cout << "process type " << image_dst.depth() << std::endl; //CV_32FC3
// // std::cout << "last convertTo bgr vaule" << std::endl;
for (int i = 0; i < image_dst.channels(); i++)
for (int j = 0; j < image_dst.rows; j++) {
for (int k = 0; k < image_dst.cols; k++) {
std::cout << (float) image_dst.at<cv::Vec3f>(j, k)[i] << " ";;
}
std::cout << std::endl;
}
std::cout << detect_image.cols << " " << " " << detect_image.rows << " " << std::endl;
auto start = chrono::high_resolution_clock::now();
ExecutableNetwork _network;
OutputsDataMap _outputinfo;
//参数区
Core ie;
auto cnnNetwork = ie.ReadNetwork(
"/home/ubuntu/CLionProjects/untitled3/tinypose_128x96/model/FP16/paddlepose_sim.xml");
//输入设置
InputsDataMap inputInfo(cnnNetwork.getInputsInfo());
InputInfo::Ptr &input = inputInfo.begin()->second;
string _input_name = inputInfo.begin()->first;
input->setPrecision(Precision::FP32);
input->getInputData()->setLayout(Layout::NCHW);
ICNNNetwork::InputShapes inputShapes = cnnNetwork.getInputShapes();
SizeVector &inSizeVector = inputShapes.begin()->second;
cnnNetwork.reshape(inputShapes);
//输出设置
_outputinfo = OutputsDataMap(cnnNetwork.getOutputsInfo());
for (auto &output : _outputinfo) {
output.second->setPrecision(Precision::FP32);
}
//获取可执行网络
//_network = ie.LoadNetwork(cnnNetwork, "GPU");
_network = ie.LoadNetwork(cnnNetwork, "CPU");
size_t img_size = image_target_h * image_target_w;
InferRequest::Ptr infer_request = _network.CreateInferRequestPtr();
Blob::Ptr frameBlob = infer_request->GetBlob(_input_name);
InferenceEngine::LockedMemory<void> blobMapped = InferenceEngine::as<InferenceEngine::MemoryBlob>(
frameBlob)->wmap();
float *blob_data = blobMapped.as<float *>();
//nchw
std::vector<cv::Mat> rgb_channel(3);
cv::split(detect_image, rgb_channel);
for (int i = 0; i < rgb_channel.size(); i++) {
rgb_channel[i].convertTo(rgb_channel[i], CV_32FC1, normVals[i], meanVals[i]);
for (int j = 0; j < rgb_channel[i].rows; j++) {
for (int k = 0; k < rgb_channel[i].cols; k++) {
//std::cout << rgb_channel[i].at<float>(j, k) << " ";;
blob_data[img_size * i + j * int(image_target_w) + k] =rgb_channel[i].at<float>(j, k);
}
//std::cout << std::endl;
}
}
// for (size_t row = 0; row < image_target_h; row++) {
// for (size_t col = 0; col < image_target_w; col++) {
// for (size_t ch = 0; ch < 3; ch++) {
// blob_data[img_size * ch + row * int(image_target_w) + col] =
// float(detect_image.at<Vec3f>(row, col)[ch]);
// // std::cout<<blob_data[img_size * ch + row * int(image_target_w) + col]<<" ";
// }
// //std::cout<<std::endl;
// }
// }
//执行预测
infer_request->Infer();
//获取各层结果
std::vector<float> vec_heap;
int shape_d = 0;//result.d;
int shape_c = 0;//result.c;
int shape_w = 0;//result.w;
int shape_h = 0;//result.h;
for (auto &output : _outputinfo) {
auto output_name = output.first;
std::cout << output_name << std::endl;
if (strcmp(output_name.c_str(), "conv2d_441.tmp_1") == 0) {
shape_d = output.second.get()->getDims()[0];//result.d;
shape_c = output.second.get()->getDims()[1];//result.c;
shape_w = output.second.get()->getDims()[2];//result.w;
shape_h = output.second.get()->getDims()[3];//result.h;
}
}
printf("output shape: %d %d %d %d\n", shape_d, shape_c, shape_w, shape_h);
Blob::Ptr blob = infer_request->GetBlob("conv2d_441.tmp_1");
LockedMemory<const void> blobMappedResult = as<MemoryBlob>(blob)->rmap();
const float *output_blob = blobMappedResult.as<float *>();
for (int i = 0; i < shape_c ; i++) {
for(int j=0;j<shape_h*shape_w;j++){
vec_heap.push_back(output_blob[i*shape_h*shape_w+j]);
//std::cout<<output_blob[i*shape_h*shape_w+j]<<" ";
}
//std::cout<<std::endl;
}
auto end = chrono::high_resolution_clock::now(); //结束时间
auto duration = (end - start).count();
cout << "程序运行时间：" << setprecision(10) << duration / 1000000000.0 << "s"
<< "； " << duration / 1000000.0 << "ms"
<< "； " << duration / 1000.0 << "us"
<< endl;
scale[0]=center[0]/100;
scale[1]=center[1]/100;
std::vector<Keypoints> all_preds;
std::vector<int> idx;
for (int i = 0; i < shape_c; i++) {
auto begin = vec_heap.begin() + i * shape_w * shape_h;
auto end = vec_heap.begin() + (i + 1) * shape_w * shape_h;
float maxValue = *max_element(begin, end);
int maxPosition = max_element(begin, end) - begin;
all_preds.emplace_back(Keypoints(0, 0, maxValue));
idx.emplace_back(maxPosition);
}
std::vector<cv::Point2f> vec_point;
for (int i = 0; i < idx.size(); i++) {
int x = idx[i] % shape_w;
int y = idx[i] / shape_w;
vec_point.emplace_back(cv::Point2f(x, y));
}
for (int i = 0; i < shape_c; i++) {
int px = floor(vec_point[i].x+0.5);
int py = floor(vec_point[i].y+0.5);
if (px > 1 && px < shape_w - 1 && py > 1 && py < shape_h - 1) {
float diff_0 = vec_heap[py * shape_w + px + 1] - vec_heap[py * shape_w + px - 1];
float diff_1 = vec_heap[(py + 1) * shape_w + px] - vec_heap[(py - 1) * shape_w + px];
vec_point[i].x += diff_0 == 0 ? 0 : (diff_0 > 0) ? 0.25 : -0.25;
vec_point[i].y += diff_1 == 0 ? 0 : (diff_1 > 0) ? 0.25 : -0.25;
}
}
center[0]=ceil(center[0]);
center[1]=ceil(center[1]);
transform_preds(vec_point, all_preds, center, scale, shape_w, shape_h);
int skeleton[][2] = {{0, 1}, {0, 2}, {1, 3}, {2, 4}, {3, 5}, {4, 6}, {5, 7}, {6, 8},
{7, 9}, {8, 10}, {5, 11}, {6, 12}, {11, 13}, {12, 14},
{13, 15}, {14, 16}, {11, 12}};
for (int i = 0; i < all_preds.size(); i++) {
if(all_preds[i].score>keypoint_score){
cv::circle(bgr, cv::Point(all_preds[i].x, all_preds[i].y), 3, cv::Scalar(0, 255, 120), -1);//画点，其实就是实心圆
}
}
for (int i = 0; i < sizeof(skeleton) / sizeof(sizeof(skeleton[1])); i++) {
int x0 = all_preds[skeleton[i][0]].x;
int y0 = all_preds[skeleton[i][0]].y;
int x1 = all_preds[skeleton[i][1]].x;
int y1 = all_preds[skeleton[i][1]].y;
cv::line(bgr, cv::Point(x0, y0), cv::Point(x1, y1),
cv::Scalar(0, 255, 0), 1);
}
cv::imwrite("demo.jpg",bgr);
cv::imshow("image", bgr);
cv::waitKey(0);
return 0;
}
测试结果就不放了～
训练和我之前写的mmpose制作数据集和训练方法一样，数据格式也是一样