It was time to crunch again some updated numbers from the opengapps.org website and the pre-built packages downloads. To gather the stats I wrote some better tools (Python instead of my old shell scripts) that fetch the data from the GitHub API, so future updates will be more easy.
The average amount of unique daily visitors at opengapps.org is according to Google Analytics around 55k. The amount of ad-blockers used is way over 50%. No surprise there with our tech-savvy crowd, luckily some people do contribute by donating instead, for which I am very grateful! By multiplying the amount of downloads of each package variant with its size we were also able to find out how many bytes of packages are downloaded everyday: 30TiB, happy to be hosted by GitHub’s cloud! So, what packages are being downloaded? I compiled two sets of data, lifetime stats and those of March 2016, below.
In this chart we see the architectures. Clearly ARM is still most popular. But ARM64 is gaining ground and also x86 is becoming more frequent. x86_64 Doesn’t exist that long and has such little share that it is marked as Other by Google Fusion Tables.
It is visible that 6.0 gained ground very quickly in the total number of downloads. One major reason is that opengapps.org is still quite young (less than a year old) and 6.0 was released when the project was still gaining popularity. The 5.1 high tide has been missed.
Super has been introduced more recently and is still not as popular as the other packages. Easily explained by the fact that it is only meant for power users that want to use a gapps-config with some of the more exotic applications.
March 2016 Architectures
In the March architectures overview you can see there are no major shifts compared to the lifetime stats, but ARM64 is very slowly growing.
March 2016 APIs
In the March APIs it is visible that 6.0 is now the de-facto standard for ROMs. 5.0 is shrinking. Also 4.4 is becoming less popular.
March 2016 Variants
In the March variants overview it is visible that the smaller pico and nano packages are gaining. It seems that the growing size of Google’s apps makes stock not always fit on
/system/ and leads to people switching to smaller packages, especially compared to the popularity stock had on 5.1 in September last year. Also aroma did shrink compared to September 2015, very probably caused by the unmaintained ‘AROMA Installer’ and the more frequent issues it poses on various devices and recent recoveries. You can read more about that in Raul’s post.
Earlier I used to build busybox on my Android device itself, but it soon posed an obstacle, as I could only support ARM architecture. So I decided to delve in the cross compiling business. I use custom-built buildroot toolchains using uClibc C-library to get the smallest and full-featured busybox binaries possible. This helped me support all the Android architectures available.
Open GApps busybox integration
Initially mfonville reached out to me for building a
busybox binary with lzip support included as he was trying out lzma compression over xz, because xz embedded in the busybox binary wasn’t fully functional. Finally it was decided that the installer will stick with xz as lzip was much slower and provided almost the same level of compression compared to xz. But the xz-embedded code that is part of the busybox source tree, was more of a hindrance because of its limitations and errors. So I was asked if I could also build an
xzdec binary for the Open GApps project.
Bugs and Issues
Then came up an issue on github where Tegra 2 device owners were getting “illegal instructions” error. Soon we realized that NEON support was the culprit here. Apparently even a modern SoC such as Tegra 2 lacked something as subtle as NEON support in their devices. The toolchain I built was configured to have VFPV3 support with NEON by default. So as simple as it may sound the solution was to build a toolchain without NEON support and using that to build busybox which was much easier said than done.
I also assisted the project by building the infozip’s
zip binary for advanced APK manipulation. Soon there was a hiccup in the working of this zip binary. MFonville and I invested a lot of time in bug hunting and it all came down to a nasty little variable
$ZIP that was misleading the zip program. It was also addressed by XDA member osm0sis in his post.
Head over to my XDA thread for more information about my busybox binaries and their latest version.
For any readers visiting this post on or after April 2, check the post date before continuing :)
The Hague, Apr. 1, 2016 - The Open GApps Team is very excited and proud to announce it’s been acquired by Google Inc. Google’s CEO Sundar Pichai explains how this latest acquisition is part of Google’s updated strategy as an Alphabet company: “By strengthening the Android division we can encapsulate even more of the Android market share, by acquiring Open GApps we directly gain over 90% marketshare of the AOSP-based installs and is proof of our commitment to ‘be together, not the same’.”
The Open GApps developers are very happy with the acquisition. MFonville is tasked with heading Google’s new shell-scripting division and a GSoC-project for developing a new “gsh”-shell. Also MNBooze is very content: “I will be relocated to the Google Campus and will be creating a Python based crawler that searches the Google-Docs department to find their newest APKs. This will enable us to release APKs even before they will reach the Play Store!” Raulx222 will join the Material Design Lite Project to bring MDL to the AROMA Recovery.
We sometimes get questions asking about mirrors or people uploading the OpenGApps.org pre-built packages to mirrors, which violates our license terms. In this post we will try to give some insight behind the rationale of the no-mirror policy of the OpenGApps.org pre-built packages:
The very first reason it technical: we want users to use the latest, greatest and most up-to-date version. The rationale behind the project was to be able to do automated builds to achieve this goal and it would be weird to forfeit this. To make this easy, we made it possible to hotlink and automatically download the latest versions for e.g. ROM devs their OPs on XDA. This eliminates the need for mirrors which would be outdated within one day.
Support and educate users
The second reason is support and documentation. If just uploading the package to a filehost it isn’t accompanied by any documentation or access to a support forum. If offered via OpenGApps.org there are direct links to the Wiki, XDA Forum, the source code and existing bug reports. This helps users find their way and to benefit most from the various features offered by the package.
Another very import reason is fostering a community. Open source does not mean “just download and ignore where it comes from”. For open source projects it is critical that people are aware of where their software originates from, the rationale behind it, how to contribute and how to get involved. If users of the software can’t get acquainted with the project, the project won’t be able to attract new potential volunteers and would have a large chance to die. OpenGApps.org is a HUB for the users of Open GApps, that enables them to get acquainted with the project and support it.
Inclusive to all
Also this policy benefits the people who are using less popular architectures, apis and variants. The amount of ARM 5.1 and 6.0 downloads each day is huge, but some people use less popular builds, like 4.4 x86, that only has a few downloads each day. To be able to compile builds every day for everyone instead of just the popular few packages, the buildbot compiles and uploads more than 20GiB of new packages every day. That is not without costs and the ads and donations on opengapps.org support us to do this. If you download an ARM 6.0 pico on our website and you see an advertisement, you are i.e. funding the 4.4 x86 users’ download.
Acknowledge, credit and motivation
Another aspect of open source projects and voluntary work is that it is important to give credit where it is due and respecting people’s efforts: volunteering projects can only strive because of the many people involved that contribute with their knowledge and skills. If filehosts offer the downloads without any reference to the context to the people who made it possible, it neglects this important aspect of keeping people motivated to put in the many hours they do. That is why direct access to the e.g. the source code commits is important.
Trust and security
Trustworthiness, the packages from OpenGApps.org are signed with a custom certificate, to ensure people that they were built by us. People can choose to trust OpenGApps.org’s packages to be signed with this certificate had no changes compared to the GitHub source code and don’t contain APKs that were tampered with.
What if downloads are slow?
Sometimes people experience slow downloads from OpenGApps.org. The OpenGApps.org packages are hosted on GitHub Releases, which is hosted on the Amazon Web Services Cloud. Most of the time and in most places of the world their service is good and reliable, but for some users it is sometimes slow. If it is slow, you can write to GitHub to let them ask Amazon for a solution.
Compile; don’t copy!
If you want to upload your own Open GApps variant, you are free to do so. The source code of Open GApps is freely available and everybody can compile and upload their own builds and variants, which you can distribute under various terms (as long as in compliance with the Open GApps license). Please just don’t use copy-paste the pre-built ones from OpenGApps.org.
With Android Marshmallow Google released a new version “3” of their Google Camera app. This version of camera comes pre-bundled with most of their stock Marshmallow ROMs.
The updated application APK states that it requires SDK level 23 (which means Marshmallow 6.0) as minimal android level for installation. This result in the Open GApps package-building scripts to automatically pick the v3 of the Google Camera when creating installable zips for 6.0. Because the APK is deemed compatible.
Unfortunately many people did experience issues with Google Camera v3 on Marshmallow when recording video. Apparently with the introduction of the new Google Camera v3 it assumes certain device and firmware capabilities that many devices and ROMs cannot offer. This posed a problem for the Open GApps packages. E.g. if we would bundle the v2 version on Marshmallow (which would need an override in the package-building scripts), the people with a v3 compatible device would directly get an (unexpected) update from the Play store offered. But if we bundled v3, many people would experience video recording problems.
We did need to find a solution for this problem, that would provide Google Camera v2 for devices that were not v3-compatible, and v3 for the compatible ones.
Determining compatibility indicator(s)
First challenge is: which devices are actually v3-compatible? We looked at the Android Source code and Android Camera documentation, but any version information seems to be only available within the OS itself. E.g. the Google Play Services is able to assess and communicate the device’s information and capabilities to the Play Store. Based on that information it can decide to offer the device the Google Camera v3 update or not. But we, from the device’s recovery, have no access to this information (yet). We looked at the camera libraries in the ROM and the symbols it exposes, but we have not found any reliable indicator. So we are forced to work with a white-list with v3 compatible devices. If you know your device is compatible, but it isn’t on the white-list yet, please let us know or create a pull request with the devicename added.
Including new and legacy Google Camera
The second challenge: how to include both the v2 and v3 Google Camera and let only the compatible version be installed. For this we added functionality to the inc.buildhelper.sh that allows to set a maximum SDK level when looking for an APK to package. After that we created an extra entry in the
inc.buildtarget.sh script called
cameragooglelegacy with a maximum SDK level of 22. Next step was to add “cameragooglelegacy” as extra package-building rule for the 6.0 stock and super packages. In the update-binary installer script there are some logic checks that decides if “cameragoogle” should be installed or not (based on keywords used and your device’s prerequisites). An extra piece of logic was added, that if
cameragoogle is going to be installed, but your device is not on the white-list, it will use
Adding the extra Camera Google v2 to the 6.0 packages takes around 20MiB extra. So the download is a bit larger than before. Let’s hope that when Android N is released this situation to be improved, so that this trick won’t be necessary anymore.
If you do have any ideas how to figure out if a device is Google Camera v3 compatible from the recovery without using a white-list, please let us know in the comments or on the XDA forum!