Warning: This library is pre-alpha, not under active development,
and not ready for production.
Please message Brandon Amos
(email: bamos@cs.cmu.edu, twitter: @brandondamos)
if you're interested in using a complete version of library.
Thanks!
Mobile devices have powerful processors and streaming and processing video in real time is becoming a reality by leveraging resource-rich cloudlets, as described in "The Case for VM-Based Cloudlets in Mobile Computing". Real-time video processing is becoming a reality, and filtering video frames can be helpful to identify objects in frames. This project provides an image and video frame filtering library for Android and Glass. For example, an augmented reality application can say "only send video frames with a brick wall like in these examples because I'm only interested in overlaying images on top of brick walls."
These filters are provided by the Diamond project, which provides interactive search of non-indexed data. This repository compiles the C filters from the cmusatyalab/diamond-core-filters repository for ARM and statically links their dependencies.
examples/face-detection
is an example Android application to illustrate the usage of a
Diamond filter with an Android application filtering video
in real time for faces.
The APK of this application is available as
facerecognition-release-unsigned.apk from the releases.
The Diamond face detection filter is implemented in filters/ocv_face of cmusatyalab/diamond-core-filters and detects faces using the OpenCV C library. An alternate to using a Diamond filters is to use the OpenCV Android library, which uses the JNI to interact with the native library.
Applications interact with the filter binaries through Filter objects.
The face detection example uses the rgbimg filter to convert images
into RGB format and the ocv_face filter to detect faces.
See the Filter Javadoc for more information about creating filters.
String[] faceFilterArgs = {"1.2", "24", "24", "1", "2"};
InputStream ocvXmlIS = context.getResources().openRawResource(R.raw.haarcascade_frontalface);
Filter rgbFilter, faceFilter;
try {
rgbFilter = new Filter(R.raw.rgbimg, context, "RGB", null, null);
byte[] ocvXml = IOUtils.toByteArray(ocvXmlIS);
faceFilter = new Filter(R.raw.ocv_face, context, "OCVFace",
faceFilterArgs, ocvXml);
} catch (IOException e1) {
Log.e(TAG, "Unable to create filter subprocess.");
e1.printStackTrace();
return;
}Video frames are passed to the isFace function as jpg images,
which uses the filters to detect faces.
Applications communicate with filters through the process function,
which uses a map for communication and returns a double.
The face recognition filter returns 1.0 if a face is detected
ad 0.0 otherwise.
Further details are described in the Filter Javadoc.
private boolean isFace(byte[] jpegImage, Filter rgbFilter, Filter faceFilter) throws IOException, FilterException {
final Map<String,byte[]> m = new HashMap<String,byte[]>();
Log.d(TAG, "Sending JPEG image to RGB filter.");
Log.d(TAG, "JPEG image size: " + String.valueOf(jpegImage.length) + " bytes.");
m.put("", jpegImage);
rgbFilter.process(m);
byte[] rgbImage = m.get("_rgb_image.rgbimage");
Log.d(TAG, "Obtained RGB image from RGB filter.");
Log.d(TAG, "RGB image size: " + String.valueOf(rgbImage.length) + " bytes.");
Log.d(TAG, "Sending RGB image to OCV face filter.");
double faceRecognized = faceFilter.process(m);
return Math.abs(faceRecognized-1.0d) < 1E-6;
}This repository compiles the following filters from cmusatyalab/diamond-core-filters for ARM. The arguments to the filters can be found in the filter sources.
| Filter | Description |
|---|---|
| dog_texture | Wikipedia: Difference of Gaussians |
| gabor_texture | Wikipedia: Gabor filter |
| img_diff | Pixel-wise image difference. |
| null | Null filter (for testing) |
| ocv_face | OpenCV face detection. |
| rgb_histogram | Wikipedia: Color histogram |
| rgbimg | Tries to convert numerous image formats into RGB. |
| shingling | Fingerprint images based on sliding windows. |
The performance application runs filter on a sample data set to estimate how long the filters will take to run in production. The runtime may vary depending on the device, CPU load, and the number of sample images are used.
Extract and move public-domain-landscapes.tgz from the releases
into /sdcard/ of the device.
tar xvfz public-domain-landscapes.tgz
adb push public-domain-landscapes /sdcard/public-domain-landscapes
Move and rename public-domain-landscape-cloud-filters.zip (as a zip archive)
from the releases to res/raw/filter_zip of
the performance application.
Filters should be located in the res/raw directory of your
Android application and can be obtained as binaries
in arm-filters.tgz from the releases.
The following portion describes how to build the filters
from source with the NDK and integrate with gradle in Android Studio.
This section was ported from a pre-Android Studio into gradle and does not use all of gradle and Android Studio's dependency management solutions.
build.gradle in the facerecognition application will invoke the NDK if an arm filter is not present with the following task and dependency.
task buildJni(type: Exec) {
commandLine 'sh', './jni/build.sh'
commandLine 'sh', './add-filters-to-res.sh'
}
compile files("src/main/res/raw/dog_texture") {
builtBy 'buildJni'
}The build.sh
script in the
diamond-android-library/jni
directory will automatically download libraries from source,
apply source modifications,
and build static filter executables.
The only prerequisite to running build.sh is to install the
Android NDK
and set the ANDROID_NDK environment variable to the installation path.
build.sh will create a
standalone toolchain
for cross compiling the ARM applications.
The prebuilt filters use the Android NDK version r9d on OSX,
and #31
describes a possible bug if the filters are compiled from Linux.
Once the filters are built, the
add-res-to-raw.sh,
script will extract the binaries into res/raw of the
library application, which should be moved into res/raw
of applications using Diamond Android.
The Diamond Android source is copyright Carnegie Mellon University and licensed under the Eclipse Public License v1.0. The following libraries have been modified as noted and are included in the statically linked filter binary artifacts of this repository. These portions are copyright their respective authors with the licenses listed.
This software is based in part on the work of the Independent JPEG Group.
By downloading, copying, installing or using the software you agree to the OpenCV license. If you do not agree to this license, do not download, install, copy or use the software.
| Project | Source Modified | License |
|---|---|---|
| cmusatyalab/diamond-core-filters | Yes | Eclipse Public License v1.0 |
| cmusatyalab/opendiamond | Yes | Eclipse Public License v1.0 |
| glib | Yes | LGPL |
| ivanra/getline | Public Domain | |
| libarchive | Yes | BSD 2-Clause |
| libjpeg-turbo | libjpeg-turbo License, IJG, and libjpeg/SIMD | |
| libpng | libpng License | |
| libtiff | MIT-like | |
| memstream-0.1 | MIT | |
| NimbusKit/memorymapping | Apache 2.0 | |
| OpenCV | OpenCV License |