evaluation summary.md

Below are the list of apps used in DroidPerm evaluation.

 

	f-droid.org id	Valid Method	Invalid Method	Undetected Method	Valid Field	Invalid Field	Undetected Field
1	com.easwareapps.transparentwidget	1
2	com.easytarget.micopi				9		1
3	com.example.tobiastrumm.freifunkautoconnect	1			1
4	com.github.yeriomin.dumbphoneassistant	1			6
5	com.iamtrk.androidexplorer	6
6	com.jarsilio.android.waveup	0			0
7	com.jmstudios.redmoon	2
8	com.menny.android.anysoftkeyboard	-
9	com.nutomic.ensichat	-
10	com.poinsart.votar				1
11	com.saladdressing.veterondo	2
12	com.simplemobiletools.camera	2
13	com.simplemobiletools.flashlight	1
14	com.thibaudperso.sonycamera					1
15	com.todobom.opennotescanner	5
16	com.tortel.syslog		2
17	com.vsmartcard.acardemulator			1
18	com.vsmartcard.remotesmartcardreader.app			1
19	de.bitsharesmunich.smartcoinswallet	2
20	de.schildbach.wallet	1
21	de.schildbach.wallet_test	-
22	eu.kanade.tachiyomi			1
23	im.vector.alpha	3		2	3
24	is.pinterjann.jaws				1
25	net.bierbaumer.otp_authenticator			1
26	net.sf.times	8
27	org.androidpn.client	1
28	org.liberty.android.fantastischmemo	1
29	org.osmocom.tacdatabaseclient	1
30	org.pixmob.freemobile.netstat	2		1
31	org.traccar.client	4
32	protect.budgetwatch				1
33	protect.card_locker			1
34	ru.valle.btc	1
35	trikita.obsqr	2
	Total	47	2	8	22	1	1

Total valid sensitives: 69

Total invalid detected sensitives: 3

Total valid undetected sensitives: 9

 

Method sensitives precision: 47 / (47+2) * 100% = 96%

Field sensitives precision: 22 / (22+1) * 100% = 96%

 

Method sensitives recall: 47 / (47+8) * 100% = 85%

Field sensitives recall: 22 / (22+1) * 100% = 96%

 

Total precision: 69 / (69+3) * 100% = 96%

Total recall: 69 / (69+9) * 100% = 88%

 

 

Setup

The apps were extracted from f-droid.org on 12/16/2016. Links to source repositories are on f-droid page of each app. The 3 apps that were excluded from evaluation due to build issues or issues running DroidPerm have value “-” in the 3rd column.

 

Additional data

Please see evaluation notes.txt for the list of code locations in this table and explanation for each invalid/undetected sensitive. There are also more detailed evaluation notes here and the raw log files produced by DroidPerm here.