So I'm trying to make an android application, that would open a video file frame by frame with OpenCV througn JNI and NDK, detect faces in each frame and display them (the end goal is to acclerate the algorithm, to better suit crappy mobile processing).
所以我想做一个android应用，用OpenCV通过JNI和NDK一帧一帧地打开一个视频文件，检测每一帧的人脸并显示它们(最终目标是增强算法，以更好地适应糟糕的移动处理)。
First method I attempted was MediaMetadataRetriever with no results, it just returned null.
我尝试的第一个方法是MediaMetadataRetriever，没有结果，它只返回null。
The second method was FFMpegMediaMetadataRetriever, which does work as expected, but is very slow (up to 2fps; too slow), partially because you have to convert from FFMpegMMR's bitmap to Mat then detect and draw and then convert back to bitmap, which is very caveman of me to even attempt.
第二种方法是FFMpegMediaMetadataRetriever，它可以像预期的那样工作，但是非常慢(最高可达2fps;太慢了)，部分原因是您必须从FFMpegMMR的位图转换到Mat，然后检测并绘制，然后再转换回位图，这是我的原始人甚至尝试。
The third method I'm currently working on is the VideoCapture grab() and retrieve(). I made a wrapper for the native code, mostly by copying the facedtector from OpenCV samples. I've also tried read() that supposedly combines the two, but it also causes fatal signal 11 (segmentation fault at some god forsaken level of OpenCV or Android platform).
我目前正在开发的第三种方法是VideoCapture grab()和retrieve()。我为本机代码做了一个包装，主要是从OpenCV示例复制facedtector。我也尝试过read()，据说它将两者结合在一起，但它也会导致致命的信号11(在OpenCV或Android平台的某些被上帝抛弃的级别上出现分割错误)。
Here's how the application get the absolute file path (note that it did work for FFMpegMMR):
下面是应用程序如何获得绝对文件路径(请注意，它确实为FFMpegMMR工作):
//The file name, file path and filerawresource are used in getting the path of the video file on device's disk
String videoFileName="samplemp4";
String videoFileType=".mp4";
int videoFileRawResourceId=R.raw.samplemp4;
public String getVideoFilePath()
{
InputStream is = getResources().openRawResource(videoFileRawResourceId);
File sampleDir = getDir(videoFileName, Context.MODE_PRIVATE);
File sampleFile= new File(sampleDir, videoFileName+videoFileType);
try {
FileOutputStream os = new FileOutputStream(sampleFile);
byte[] buffer = new byte[4096];
int bytesRead;
while ((bytesRead = is.read(buffer)) != -1) {
os.write(buffer, 0, bytesRead);
}
is.close();
os.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
File videoDir = getDir(videoFileName, Context.MODE_PRIVATE);
File videoFile = new File(videoDir, videoFileName+videoFileType);
return videoFile.getAbsolutePath();
}
This is where the java wrappers of the native methods are called and results used:
这就是调用本机方法的java包装器并使用结果的地方:
class AsyncPlay extends AsyncTask<String, Mat, Bitmap>
{
@Override
protected Bitmap doInBackground(String... params) {
//for(;play;)
{
if(play)
{
if(currentTime==0)
{
test=new Test();
test.startTime.setToNow();
test.type="Video";
frameGrabber.open(videoFilePath);
}
//publishProgress(retriever.getFrameAtTime(currentTime*1000+111,
// FFmpegMediaMetadataRetriever.OPTION_CLOSEST));
Mat tmp=new Mat();
//frameGrabber.read(tmp);
frameGrabber.grab();
frameGrabber.retrieve(tmp);
publishProgress(tmp);
currentTime+=111;
if(currentTime*1000>=duration*1000)
{
currentTime=0;
test.endTime.setToNow();
tester.publishResult(test);
frameGrabber.release();
}
}
try
{
Thread.sleep(111);
} catch (InterruptedException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
return null;
}
@Override
protected void onProgressUpdate(Mat... values) {
//face detection logic goes here
//Bitmap bmp=values[0];
//Mat rgbMat = new Mat();
//Utils.bitmapToMat(bmp, rgbMat);
DetectionResult detectionResult = openCVFaceDetector.detectFromImage(values[0], Imgproc.COLOR_RGB2GRAY);//Detecting with a native detector from OpenCV
test.addDetection(detectionResult.detection);
imageViewOriginal.setImageBitmap(detectionResult.detectedImage);
super.onProgressUpdate(values);
}
};`
Where FrameGrabber frameGrabber is the java wrapper of the natives and the signal gets sent when retrieve() or read() are called, note that the constructor, open() and grab() all work (or at least don't crash the application).
当FrameGrabber FrameGrabber是本机程序的java包装器，并且在调用retrieve()或read()时发送信号时，请注意，构造函数open()和grab()所有工作(至少不会导致应用程序崩溃)。
This is the wrapper (I have passed the VideoCapture object properly, which I learned was one of the causes of the fatal signal):
这是包装器(我已经正确地通过了VideoCapture，我知道这是致命信号的原因之一):
import org.opencv.core.Mat;
public class FrameGrabber
{
private long mNativeObj = 0;
private static native long nativeCreateObject(String fileName);
private static native void nativeDestroyObject(long thiz);
private static native boolean nativeOpen(long thiz, String fileName);
private static native boolean nativeGrab(long thiz);
private static native boolean nativeRetrieve(long thiz, Mat imageMat);
private static native boolean nativeRead(long thiz, Mat imageMat);
private static native void nativeRelease(long thiz);
public FrameGrabber(String fileName) {
mNativeObj = nativeCreateObject(fileName);
}
public void destroy() {
nativeDestroyObject(mNativeObj);
mNativeObj = 0;
}
public boolean open(String fileName)
{
return nativeOpen(mNativeObj, fileName);
}
public boolean grab()
{
return nativeGrab(mNativeObj);
}
public boolean retrieve(Mat imageMat)
{
return nativeRetrieve(mNativeObj, imageMat);
}
public boolean read(Mat imageMat)
{
return nativeRead(mNativeObj, imageMat);
}
public void release()
{
nativeRelease(mNativeObj);
}
}
And this is it's native part (the exact line causing the error is result = ((VideoCapture*)thiz)->retrieve((*((Mat*)imageMat)));, which is in the method nativeRetrieve()
这是它的本机部分(导致错误的确切行是result = (VideoCapture*)thiz)->retrieve(*(*(Mat*)imageMat);
#include <string>
#include <vector>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <FrameGrabber_jni.h>
#include <android/log.h>
#define LOG_TAG "FrameGrabber"
#define LOGD(...) ((void)__android_log_print(ANDROID_LOG_DEBUG, LOG_TAG, __VA_ARGS__))
using namespace std;
using namespace cv;
inline void jStringToString(JNIEnv * jenv, jstring jString, string stdName)
{
LOGD("Java_boris_springar_diplomska_FrameGrabber_jStringToString enter");
LOGD("grabber is Making jnamestr");//this is buggy as hell, the line serves as a debug tool
const char* jnamestr = jenv->GetStringUTFChars(jString, NULL);
string stdNameTmp(jnamestr);
stdName=stdNameTmp;
LOGD("grabber is releasing jnamestr");
jenv->ReleaseStringUTFChars(jString, jnamestr);
}
/*
* CONSTRUCTOR AND DESTRUCTOR
*/
JNIEXPORT jlong JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeCreateObject
(JNIEnv * jenv, jclass, jstring jFileName)
{
LOGD("Java_boris_springar_diplomska_FrameGrabber_nativeCreateObject enter");
string stdFileName;
jStringToString(jenv, jFileName,stdFileName);
jlong result = 0;
try
{
result = (jlong)new VideoCapture(stdFileName);
}
catch(cv::Exception& e)
{
LOGD("nativeCreateObject caught cv::Exception: %s", e.what());
jclass je = jenv->FindClass("org/opencv/core/CvException");
if(!je)
je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, e.what());
}
catch (...)
{
LOGD("nativeCreateObject caught unknown exception");
jclass je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.nativeCreateObject()");
return 0;
}
LOGD("Java_boris_springar_diplomska_FrameGrabber_nativeCreateObject exit");
return result;
}
//should work
JNIEXPORT void JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeDestroyObject
(JNIEnv * jenv, jclass, jlong thiz)
{
LOGD("Java_boris_springar_diplomska_FrameGrabber_nativeDestroyObject enter");
try
{
if(thiz != 0)
{
delete (VideoCapture*)thiz;
}
}
catch(cv::Exception& e)
{
LOGD("nativeestroyObject caught cv::Exception: %s", e.what());
jclass je = jenv->FindClass("org/opencv/core/CvException");
if(!je)
je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, e.what());
}
catch (...)
{
LOGD("nativeDestroyObject caught unknown exception");
jclass je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.nativeDestroyObject()");
}
LOGD("Java_boris_springar_diplomska_FrameGrabber_nativeDestroyObject exit");
}
/*
* CORE METHODS
*/
//Open function opens the filename for playing
JNIEXPORT jboolean JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeOpen
(JNIEnv * jenv, jclass,jlong thiz,jstring jFileName)
{
LOGD("Java_boris_springar_diplomska_FrameGrabber_open enter");
string stdFileName;
jStringToString(jenv, jFileName,stdFileName);
jboolean result = false;
try
{
result = ((VideoCapture*)thiz)->open(stdFileName);
}
catch(cv::Exception& e)
{
LOGD("frame grabber open exception caught cv::Exception: %s", e.what());
jclass je = jenv->FindClass("org/opencv/core/CvException");
if(!je)
je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, e.what());
}
catch (...)
{
LOGD("frame grabber open caught unknown exception");
jclass je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.open()");
return 0;
}
LOGD("Java_boris_springar_diplomska_FrameGrabber_open exit");
return result;
}
//grab grabs the next frame from file or camera
JNIEXPORT jboolean JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeGrab
(JNIEnv * jenv, jclass,jlong thiz)
{
LOGD("Java_boris_springar_diplomska_FrameGrabber_grab enter");
jboolean result = false;
try
{
result = ((VideoCapture*)thiz)->grab();
}
catch(cv::Exception& e)
{
LOGD("frame grabber grab exception caught cv::Exception: %s", e.what());
jclass je = jenv->FindClass("org/opencv/core/CvException");
if(!je)
je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, e.what());
}
catch (...)
{
LOGD("frame grabber grab caught unknown exception");
jclass je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.grab()");
return 0;
}
LOGD("Java_boris_springar_diplomska_FrameGrabber_grab exit");
return result;
}
//retrieve retrieves the next frame and writes it to the image matrix
JNIEXPORT jboolean JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeRetrieve
(JNIEnv * jenv, jclass,jlong thiz, jlong imageMat)
{
LOGD("Java_boris_springar_diplomska_FrameGrabber_retrieve enter");
jboolean result = false;
try
{
LOGD("grabber trying to retrieve");
result = ((VideoCapture*)thiz)->retrieve((*((Mat*)imageMat)));//should write the current frame to the image matrix
}
catch(cv::Exception& e)
{
LOGD("frame grabber retrieve exception caught cv::Exception: %s", e.what());
jclass je = jenv->FindClass("org/opencv/core/CvException");
if(!je)
je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, e.what());
}
catch (...)
{
LOGD("frame grabber retrieve caught unknown exception");
jclass je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.retrieve(fileName)");
return 0;
}
LOGD("Java_boris_springar_diplomska_FrameGrabber_retrieve exit");
return result;
}
//read combines grab and retrieve and writes the stuff to the image matrix
JNIEXPORT jboolean JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeRead
(JNIEnv * jenv, jclass,jlong thiz, jlong imageMat)
{
LOGD("Java_boris_springar_diplomska_FrameGrabber_read enter");
LOGD("grabber setting result to false");
jboolean result = false;
try
{
LOGD("grabber trying to read capture");
result = ((VideoCapture*)thiz)->read((*((Mat*)imageMat)));//should write the current frame to the image matrix
}
catch(cv::Exception& e)
{
LOGD("frame grabber read exception caught cv::Exception: %s", e.what());
jclass je = jenv->FindClass("org/opencv/core/CvException");
if(!je)
je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, e.what());
}
catch (...)
{
LOGD("frame grabber read caught unknown exception");
jclass je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.open(fileName)");
return 0;
}
LOGD("Java_boris_springar_diplomska_FrameGrabber_read exit");
return result;
}
//Release releases the resource it's using, I hope
JNIEXPORT void JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeRelease
(JNIEnv * jenv, jclass,jlong thiz)
{
LOGD("Java_boris_springar_diplomska_FrameGrabber_release enter");
jboolean result = false;
try
{
((VideoCapture*)thiz)->release();//should release
}
catch(cv::Exception& e)
{
LOGD("frame grabber read exception caught cv::Exception: %s", e.what());
jclass je = jenv->FindClass("org/opencv/core/CvException");
if(!je)
je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, e.what());
}
catch (...)
{
LOGD("frame grabber release caught unknown exception");
jclass je = jenv->FindClass("java/lang/Exception");
jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.open(fileName)");
}
LOGD("Java_boris_springar_diplomska_FrameGrabber_release exit");
}
And also the LogCat output:
还有LogCat输出:
07-22 17:36:24.886: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_nativeCreateObject enter
07-22 17:36:24.886: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_jStringToString enter
07-22 17:36:24.886: D/FrameGrabber(15359): grabber is Making jnamestr
07-22 17:36:24.886: D/FrameGrabber(15359): grabber is releasing jnamestr
07-22 17:36:24.886: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_nativeCreateObject exit
07-22 17:36:24.936: I/dalvikvm(15359): threadid=3: reacting to signal 3
07-22 17:36:24.936: I/dalvikvm(15359): Wrote stack traces to '/data/anr/traces.txt'
07-22 17:36:24.946: D/libEGL(15359): loaded /system/lib/egl/libEGL_mali.so
07-22 17:36:24.956: D/libEGL(15359): loaded /system/lib/egl/libGLESv1_CM_mali.so
07-22 17:36:24.976: D/libEGL(15359): loaded /system/lib/egl/libGLESv2_mali.so
07-22 17:36:25.006: D/OpenGLRenderer(15359): Enabling debug mode 0
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_open enter
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_jStringToString enter
07-22 17:36:27.336: D/FrameGrabber(15359): grabber is Making jnamestr
07-22 17:36:27.336: D/FrameGrabber(15359): grabber is releasing jnamestr
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_open exit
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_grab enter
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_grab exit
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_retrieve enter
07-22 17:36:27.336: D/FrameGrabber(15359): grabber trying to retrieve
07-22 17:36:27.336: A/libc(15359): Fatal signal 11 (SIGSEGV) at 0x1d400019 (code=1)
The problem with debugging this code is that it's JNI in Eclipse and I don't know how to even make debugging c++ in Eclipse work, that's why I used the log messages to find the bugs in the code. Those helped immensely, as open() and grab() didn't work and I found and squashed a bug, where I forgot to pass thiz to the methods. The problem with retrieve() is that I can not for the life of me find it's source. There's partial definitions in highgui.hpp, but no implementation, where I could put the log messages to help me debug.
调试这段代码的问题是，它是Eclipse中的JNI，我甚至不知道如何在Eclipse中调试c++，这就是为什么我使用日志消息来查找代码中的bug。这些方法非常有用，因为open()和grab()不起作用，我发现并销毁了一个bug，我忘记了将thiz传递给方法。retrieve()的问题是我无法找到它的源代码。在highgui中有部分定义。hpp，但是没有实现，我可以在其中放置日志消息来帮助我调试。
Possible solutions would be:
可能的解决方案是:
- a different file format that's supported by retrieve(), except that I don't know which file format that might be
- 检索()支持的另一种文件格式，但我不知道可能是哪种文件格式。
- use a non absolute file path? Although neither open() and grab() did not cause a fatal signal
- 使用非绝对文件路径?虽然open()和grab()都不会导致致命信号
- scrap the whole thing and use a PNG (I really want to figure this out though)
- 把所有的东西都扔掉，使用PNG(我确实想弄清楚)
So if anyone could tell me where I can find the implementations of retrieve() and read(), I'd be most grateful. Or if it's something obvious and stupid that I missed (I wish).
因此，如果有人能告诉我在哪里可以找到检索()和read()的实现，我将非常感激。或者如果我错过了一些明显而愚蠢的事情(我希望)。
In the meantime, I'll try to find a way to debug c++ in Eclipse and maybe try another format?
同时，我将尝试找到一种方法在Eclipse中调试c++，或者尝试另一种格式?
1 个解决方案
#1
1
Found it, finally, following Aleksander's suggestion, I managed to find the type mismatch on object that were passed from Java to JNI.
最后，根据Aleksander的建议，我找到了从Java传递到JNI的对象上的类型不匹配。
Turns out, that the error was in the java part, where the native methods were being declared. Instead of passing Mat, i had to pass Mat.getNativeObjAddr(), which is a long.
结果发现，错误出现在java部分，其中声明了本机方法。我必须通过Mat. getnativeobjaddr()，而不是传递Mat。
So instead of
而不是
private static native boolean nativeRetrieve(long thiz, Mat imageMat);
private static native boolean nativeRead(long thiz, Mat imageMat);
public boolean retrieve(Mat imageMat)
{
return nativeRetrieve(mNativeObj, imageMat);
}
public boolean read(Mat imageMat)
{
return nativeRead(mNativeObj, imageMat);
}
I used this:
我用这个:
private static native boolean nativeRetrieve(long thiz, long imageMat);
private static native boolean nativeRead(long thiz, long imageMat);
public boolean retrieve(Mat imageMat)
{
return nativeRetrieve(mNativeObj, imageMat.getNativeObjAddr());
}
public boolean read(Mat imageMat)
{
return nativeRead(mNativeObj, imageMat.getNativeObjAddr());
}
NOTE: passing the wrong type DID NOT trigger an exception in c++, even though the code was in try...catch. The try...catch might be written wrong, I don't know.
注意:传递错误的类型不会在c++中触发异常，即使代码在try…catch中。试一试……catch可能写错了，我不知道。
#1
1
Found it, finally, following Aleksander's suggestion, I managed to find the type mismatch on object that were passed from Java to JNI.
最后，根据Aleksander的建议，我找到了从Java传递到JNI的对象上的类型不匹配。
Turns out, that the error was in the java part, where the native methods were being declared. Instead of passing Mat, i had to pass Mat.getNativeObjAddr(), which is a long.
结果发现，错误出现在java部分，其中声明了本机方法。我必须通过Mat. getnativeobjaddr()，而不是传递Mat。
So instead of
而不是
private static native boolean nativeRetrieve(long thiz, Mat imageMat);
private static native boolean nativeRead(long thiz, Mat imageMat);
public boolean retrieve(Mat imageMat)
{
return nativeRetrieve(mNativeObj, imageMat);
}
public boolean read(Mat imageMat)
{
return nativeRead(mNativeObj, imageMat);
}
I used this:
我用这个:
private static native boolean nativeRetrieve(long thiz, long imageMat);
private static native boolean nativeRead(long thiz, long imageMat);
public boolean retrieve(Mat imageMat)
{
return nativeRetrieve(mNativeObj, imageMat.getNativeObjAddr());
}
public boolean read(Mat imageMat)
{
return nativeRead(mNativeObj, imageMat.getNativeObjAddr());
}
NOTE: passing the wrong type DID NOT trigger an exception in c++, even though the code was in try...catch. The try...catch might be written wrong, I don't know.
注意:传递错误的类型不会在c++中触发异常，即使代码在try…catch中。试一试……catch可能写错了，我不知道。