I came across Xamarin claims that their Mono implementation on Android and their C# compiled apps are faster than Java code. Did anyone perform actual benchmarks on very similar Java and C# code on different Android platforms to verify such claims, could post the code and results?

Added June 18, 2013

Since there was no answer and could not find such benchmarks done by others, decided to do my own tests. Unfortunately, my question remains "locked" so I cannot post this as the answer, only edit the question. Please vote to re-open this question. For C#, I used Xamarin.Android Ver. 4.7.09001 (beta). The source code, all the data I used for testing and compiled APK packages are on GitHub:

Java: https://github.com/gregko/TtsSetup_Java

C#: https://github.com/gregko/TtsSetup_C_sharp

If someone would like to repeat my tests on other devices or emulators, I'd be interested to learn the results as well.

Results from my testing

I ported my sentence extractor class to C# (from my @Voice Aloud Reader app) and run some tests on 10 HTML files in English, Russian, French, Polish and Czech languages. Each run was performed 5 times on all 10 files, and the total time for 3 different devices and one emulator are posted below. I tested "Release" builds only, without debugging enabled.

HTC Nexus One Android 2.3.7 (API 10) - CyanogenMod ROM

Java: Grand total time (5 runs): 12361 ms, with file reading total: 13304 ms

C#: Grand total time (5 runs): 17504 ms, with file reading total: 17956 ms

Samsung Galaxy S2 SGH-I777 (Android 4.0.4, API 15) - CyanogenMod ROM

Java: Grand total time (5 runs): 8947 ms, with file reading total: 9186 ms

C#: Grand total time (5 runs): 9884 ms, with file reading total: 10247 ms

Samsung GT-N7100 (Android 4.1.1 JellyBean, API 16) - Samsung ROM

Java: Grand total time (5 runs): 9742 ms, with file reading total: 10111 ms

C#: Grand total time (5 runs): 10459 ms, with file reading total: 10696 ms

Emulator - Intel (Android 4.2, API 17)

Java: Grand total time (5 runs): 2699 ms, with file reading total: 3127 ms

C#: Grand total time (5 runs): 2049 ms, with file reading total: 2182 ms

Emulator - Intel (Android 2.3.7, API 10)

Java: 合計時間 (5 回実行): 2992 ミリ秒、ファイル読み取り合計: 3591 ミリ秒

C#: 合計時間 (5 回実行): 2049 ミリ秒、ファイル読み取り合計: 2257 ミリ秒

エミュレーター - Arm (Android 4.0.4、API 15)

Java: 合計時間 (5 回実行): 41751 ミリ秒、ファイル読み取り合計: 43866 ミリ秒

C#: 合計時間 (5 回実行): 44136 ミリ秒、ファイル読み取り合計: 45109 ミリ秒

簡単な議論

私のテスト コードには主にテキスト解析、置換、および正規表現検索が含まれていますが、おそらく他のコード (より多くの数値演算など) の場合は結果が異なります。ARM プロセッサを搭載したすべてのデバイスで、Java は Xamarin C# コードよりも優れたパフォーマンスを発揮しました。最も大きな違いは Android 2.3 で、C# コードは Java の約 70% の速度で実行されました。

Intel エミュレーター (Intel HAX テクノロジーを使用、エミュレーターは高速仮想モードで実行) では、Xamarin C# コードはサンプル コードを Java よりもはるかに高速に実行します (約 1.35 倍高速)。Mono 仮想マシン コードとライブラリは、ARM よりも Intel の方がはるかに最適化されているのでしょうか?

2013年7月8日編集

Oracle VirtualBox で実行される Genymotion Android エミュレーターをインストールしました。これもネイティブの Intel プロセッサを使用し、ARM プロセッサをエミュレートしていません。Intel HAX エミュレーターと同様に、ここでも C# ははるかに高速に実行されます。結果は次のとおりです。

Genymotion エミュレーター - Intel (Android 4.1.1、API 16)

Java: 合計時間 (5 回実行): 2069 ミリ秒、ファイル読み取り合計: 2248 ミリ秒

C#: 合計時間 (5 回実行): 1543 ミリ秒、ファイル読み取り合計: 1642 ミリ秒

その後、Xamarin.Android ベータ版のバージョン 4.7.11 にアップデートがあり、リリース ノートに Mono ランタイムの変更点も記載されていることに気付きました。いくつかの ARM デバイスを簡単にテストしてみることにしましたが、驚いたことに C# の数値が改善されました。

BN Nook XD+、ARM (Android 4.0)

Java: 合計時間 (5 回実行): 8103 ミリ秒、ファイル読み取り合計: 8569 ミリ秒

C#: 合計時間 (5 回実行): 7951 ミリ秒、ファイル読み取り合計: 8161 ミリ秒

すごい！C# は Java より優れているのでしょうか？Galaxy Note 2 でテストを繰り返すことにしました。

Samsung Galaxy Note 2 - ARM (Android 4.1.1)

Java: 合計時間 (5 回実行): 9675 ミリ秒、ファイル読み取り合計: 10028 ミリ秒

C#: 合計時間 (5 回実行): 9911 ミリ秒、ファイル読み取り合計: 10104 ミリ秒

ここでは C# がわずかに遅いように見えますが、これらの数値を見て私は考えました。Note 2 の方が高速なプロセッサを搭載しているのに、なぜ時間が Nook HD+ よりも長いのでしょうか? 答えは、省電力モードです。Nook では無効でしたが、Note 2 では有効でした。省電力モードを無効にしてテストすることにしました (有効にした場合と同様に、プロセッサ速度も制限されます)。

Samsung Galaxy Note 2 - ARM (Android 4.1.1)、省電力無効

Java: 合計時間 (5 回実行): 7153 ミリ秒、ファイル読み取り合計: 7459 ミリ秒

C#: 合計時間 (5 回実行): 6906 ミリ秒、ファイル読み取り合計: 7070 ミリ秒

驚くべきことに、ARM プロセッサ上でも C# は Java よりわずかに高速です。大きな進歩です!

2013年7月12日編集

We all know, that nothing beats native code for speed, and I was not satisfied with the performance of my sentence splitter in Java or C#, particularly that I need to improve it (and thus make it even slower). Decided to re-write it in C++. Here is a small (i.e. a smaller set of files than previous tests, for other reasons) comparison of the speed of native vs. Java on my Galaxy Note 2, with power saving mode disabled:

Java: Grand total time (5 runs): 3292 ms, with file reading total: 3454 ms

Native thumb: Grand total time (5 runs): 537 ms, with file reading total: 657 ms

Native arm: Grand total time (5 runs): 458 ms, with file reading total: 587 ms

Looks like for my particular test, the native code is 6 to 7 times faster than Java. Caveat: could not use std::regex class on Android, so had to write my own specialized routines searching for paragraphs breaks or html tags. My initial tests of the same code on a PC using regex, were about 4 to 5 times faster than Java.

Phew! Waking raw memory with char* or wchar* pointers again, I instantly felt 20 years younger! :)

Edit July 15, 2013

(Please see below, with edits of 7/30/2013, for much better results with Dot42)

With some difficulty, I managed to port my C# tests to Dot42 (version 1.0.1.71 beta), another C# platform for Android. Preliminary results show that Dot42 code is about 3x (3 times) slower than Xamarin C# (v. 4.7.11), on an Intel Android emulator. One problem is that System.Text.RegularExpressions class in Dot42 does not have the Split() function that I used in Xamarin tests, so I used Java.Util.Regex class instead, and Java.Util.Regex.Pattern.Split(), so in this particular place in the code, there is this small difference. Should not be a big problem though. Dot42 compiles to Dalvik (DEX) code, so it cooperates with Java on Android natively, does not need expensive interop from C# to Java like Xamarin.

Just for comparison, I also run the test on ARM devices - here the Dot42 code is "only" 2x slower than Xamarin C#. Here are my results:

HTC Nexus One Android 2.3.7 (ARM)

Java: Grand total time (5 runs): 12187 ms, with file reading total: 13200 ms

Xamarin C#: Grand total time (5 runs): 13935 ms, with file reading total: 14465 ms

Dot42 C#: Grand total time (5 runs): 26000 ms, with file reading total: 27168 ms

Samsung Galaxy Note 2, Android 4.1.1 (ARM)

Java: Grand total time (5 runs): 6895 ms, with file reading total: 7275 ms

Xamarin C#: Grand total time (5 runs): 6466 ms, with file reading total: 6720 ms

Dot42 C#: Grand total time (5 runs): 11185 ms, with file reading total: 11843 ms

Intel emulator, Android 4.2 (x86)

Java: Grand total time (5 runs): 2389 ms, with file reading total: 2770 ms

Xamarin C#: Grand total time (5 runs): 1748 ms, with file reading total: 1933 ms

Dot42 C#: Grand total time (5 runs): 5150 ms, with file reading total: 5459 ms

To me, it was also interesting to note that Xamarin C# is slightly faster than Java on a newer ARM device and slightly slower on the old Nexus One. If anyone would like to run these tests as well, please let me know and I'll update the sources on GitHub. It would be particularly interesting to see results from a real Android device with Intel processor.

Update 7/26/2013

Just a quick update, re-compiled by benchmark apps with the latest Xamarin.Android 4.8, and also with dot42 1.0.1.72 update released today - no significant changes from the results reported before.

Update 7/30/2013 - better results for dot42

Re-tested Dot42 with Robert's (from dot42 makers) port of my Java code to C#. In my C# port done initially for Xamarin, I replaced some native Java classes, like ListArray, with List class native to C#, etc. Robert did not have my Dot42 source code, so he ported it again from Java and used original Java classes in such places, which benefits Dot42, I guess because it runs in Dalvik VM, like Java, and not in Mono, like Xamarin. Now Dot42 results are much better. Here is a log from my testing:

7/30/2013 - Dot42 tests with more Java classes in Dot42 C#

Intel emulator, Android 4.2

Dot42, Greg's Code using StringBuilder.Replace() (as in Xamarin):
Grand total time (5 runs): 3646 ms, with file reading total: 3830 ms

Dot42, Greg's Code using String.Replace() (as in Java and Robert's code):
Grand total time (5 runs): 3027 ms, with file reading total: 3206 ms

Dot42, Robert's Code:
Grand total time (5 runs): 1781 ms, with file reading total: 1999 ms

Xamarin:
Grand total time (5 runs): 1373 ms, with file reading total: 1505 ms

Java:
Grand total time (5 runs): 1841 ms, with file reading total: 2044 ms

ARM, Samsung Galaxy Note 2, power saving off, Android 4.1.1

Dot42, Greg's Code using StringBuilder.Replace() (as in Xamarin):
Grand total time (5 runs): 10875 ms, with file reading total: 11280 ms

Dot42, Greg's Code using String.Replace() (as in Java and Robert's code):
Grand total time (5 runs): 9710 ms, with file reading total: 10097 ms

Dot42, Robert's Code:
Grand total time (5 runs): 6279 ms, with file reading total: 6622 ms

Xamarin:
Grand total time (5 runs): 6201 ms, with file reading total: 6476 ms

Java:
Grand total time (5 runs): 7141 ms, with file reading total: 7479 ms

I still think that Dot42 has a long way to go. Having Java-like classes (e.g. ArrayList) and a good performance with them would make porting code from Java to C# slightly easier. However, this is something I would not be likely to do a lot. I would rather want to use existing C# code (libraries etc.), which will use native C# classes (e.g. List), and that would perform slowly with the current dot42 code, and very well with Xamarin.

Greg