Open GApps logo
Navigation
Buildbot and GitHub

Open GApps buildbot

Opengapps.org’s daily builds are generated by our own buildbot and served via GitHub’s releases feature to the end-users. The buildbot is hosted on mfonville’s server and pulls in all the latest Open GApps git commits at midnight UTC.

Trigger

After having pulled in all the latest commits the buildbot uses the report_sources function to generate a hash for each architecture. The buildbot checks if the hash has changed, indicating that there have been significant updates for the architecture. If these exist, the buildbot will start to create new builds.

Build

The builds are forged at the server and are build using some specific characteristics. All builds by the buildbot are signed with a specific OpenGApps.org signature. During the build process a buildlog is generated, that specifies all APKs and libraries used, which can be downloaded by the users when they click the version information button at opengapps.org. All also builds have their md5 checksum calculated, also downloadable by the users, that can be used by the recovery to verify the integrity of the Open GApps file downloaded. The buildbot also generates a generic sources report that lists all APKs that were that day available in the repository for that architecture and Android version.

Upload

The builds and other files are then uploaded to GitHub. This process is automated, but not as smooth as we would like. GitHub’s servers regularly give an error 502 during the upload process. Which GitHub still has not been able to resolve (even though multiple users are writing to them about it for a year). Such an error results in an incomplete upload and blocks when retrying to upload a file with the same file name. Thus the buildbot must detect which uploads did fail and remove those from GitHub. This was something we had to do manually in the past. But luckily, we were able to find an automated solution recently. It was necessary to list the assets per release id to list these incomplete uploads, while successful uploads where listed in the generic list of releases. Also we had to work around GitHub’s API pagination because it was initially not showing all uploaded assets.

Download

GitHub’s API is accessible with javascript’s JSON, a feature that we use for opengapps.org When you visit the site and select an architecture, an AJAX request is send to the GitHub API to query for the latest release of the respective architecture. Basic information about the selected package is also directly fetched using this query and is shown in the dialog. The download button directly forwards to GitHub’s release infrastructure that hosts the file.

Issues

One current issue is that GitHub’s latest version detection mechanism is unfortunately not without faults. If multiple releases/tags are based on the same git commit, GitHub prefers to mark the lowest version number as the latest. This is of course illogical and we’ve written to them about it a couple of times, but without any solution yet. Luckily this doesn’t impact the more popular architectures, that almost always have new commits and thus no shared tags, but sometimes x86_64 doesn’t show the factual latest release. You can always click at older versions to open the GitHub releases page to review if there are newer releases.

Google Dialer for all!

Google Dialer

Google enabled now the Google Dialer for many more phones in the Play Store so we will be abandon the whitelist that was mentioned in the initial Google Dialer blogpost. This means that from now on, Google Dialer framework will be installed by default on all phones. The Dialer can subsequently be installed from the Play Store, or it can directly be installed if using the Stock, Super or Aroma package.

You still need to set Google Dialer as the default Phone app manually, if you remove or disable the stock/AOSP dialer application, to prevent the phone from rebooting when making or receiving a phone call.

Instructions how to set the Google Dialer as default phone app

If you also want Google Dialer to be able to adjust your ringtone settings, you need to grant it access to your system settings. Otherwise you will receive a toast with a message it is not allowed to change those.

Instructions how to set grant Google Dialer permission to access system settings

The advisory to regularly flash an updated GApps package to update DialerFramework remains intact. It can only receive security updates if you update your Open GApps installation and not via the Play Store. Make sure to update your Open GApps installation at least once a month, after Google’s monthly security updates are added to the Open GApps repository.

Open GApps’ first anniversary

Android with cake

The very first initial Open GApps’ buildscript was published at May 2nd 2015. That means we can celebrate our first anniversary!

Start of the project

At very beginning the project was not yet branded as Open GApps, but released as a very rudimentary prototype of what could be done using shell scripts to rebuild a legacy PA GApps package. There was no separation of build-scripts, sources and processed APK storage yet. Neither was the code of the high quality and many errors existed at that point. But we have come a long way. By a lot of testing by the community, feedback and reading through many debug logs most bugs have been squashed.

Improving Open GApps

All code for the build-scripts have been overhauled since that first version, to find the a durable design that could be scaled well. We have come a long way, from supporting only stock 5.1 packages to supporting all variants, AROMA support, becoming the first GApps that supported building for non-arm platforms and currently supporting all platforms and all major Android versions. Even support for flashing updated GApps on stock Nexus ROMs is supported! But next to all these variants and platforms support, a lot of development goes into extending features. Both device-side, where we could build further upon the foundations laid by Osm0sis and TKruzze to introduce new features. But especially on the build-side, where the build-scripts and tools used have been greatly enhanced. E.g. the APKCrawler project by MNBooze has been of great value. Open GApps is the prime supplier of GoogleAPKs to APKMirror because of it, and the pre-built OpenGApps.org packages are always having the very latest APKs thanks to these great scripts. But also other features, like verifying APKs for authenticity, re-packaging Marshmallow APKs but also automatic detection whether a SetupWizard is made for phones or tablets has all been automatized and is now a matter of mere seconds!

OpenGApps.org

Also the project has greatly benefited from registering OpenGApps.org to make the project more visible and offering convenient downloads to users. This would not have been possible by help from many various contributors who all thought of possible improvements and took the initiative to provide a better user experience!

Future plans

So what is ahead? Currently we are busy overhauling the build environment, to improve the speed of the building process. That will allow for continuous integration and automatic testing to provide the build quality and faster release of new builds. Also dynamic builds will become feasible with this new build environment, enabling the possibility to build packages that can be flashed on any platform. Also we are still busy with developing a package for Android TV, but because I (mfonville) lack an Android TV device myself, I can’t test which makes it hard to debug some issues.

Google Dialer

Google Dialer

Since November 2015 Google updates their Google Dialer application via the Play Store. The Open GApps scripts already supported the inclusion of Google Dialer as a package already before that, in October 2015. But the Google Dialer app faced many problematic issues at the time. I.e. before Google released their Play Store version, installing the Google Dialer would not work on any custom ROM. Because of these issues Google Dialer was not included in the regular Open GApps packages.

In March 2016 we finally enabled, after some testing with the Play Store version, the Google Dialer as an optional extra. Using the DialerGoogle keyword users could install with the dialer to experiment on their device. The feedback from users was that on Nexus devices it worked, but on almost all non-Nexus devices there were problems. But even on Nexus devices, it was obligatory to manually set Google Dialer as the default Phone app, otherwise the device would reboot when making or receiving a phone call.

Instructions how to set the Google Dialer as default phone app

Fast forward to April 2016, and more changes have been applied to DialerGoogle. It ain’t a special keyword anymore, but now acts as a regular include/exclude keyword. Also DialerGoogle has been split into a supporting DialerFramework package, that includes the Framework-files. While DialerGoogle is only in stock and larger, these Framework-files are included in all package variants. The Framework-files are required to be able to install Google Dialer from the Play Store, without them any device is deemed incompatible. The DialerFramework is obligatory if you want to install the DialerGoogle package (if you are using a gapps-config).

DialerFramework is only installed on white-listed devices, currently this white-list only includes Nexus Devices. You can override the white-list using the forcedialer keyword.

It is important to note, that the DialerFramework only receives any security updates if you update your Open GApps installation and not via the Play Store. So make sure to update your Open GApps installation at least once a month, after Google’s monthly security updates are added to the Open GApps repository.

Statistics update

It was time to crunch again some updated numbers from the opengapps.org website and the pre-built packages downloads. To gather the stats I wrote some better tools (Python instead of my old shell scripts) that fetch the data from the GitHub API, so future updates will be more easy.

OpenGApps.org

The average amount of unique daily visitors at opengapps.org is according to Google Analytics around 55k. The amount of ad-blockers used is way over 50%. No surprise there with our tech-savvy crowd, luckily some people do contribute by donating instead, for which I am very grateful! By multiplying the amount of downloads of each package variant with its size we were also able to find out how many bytes of packages are downloaded everyday: 30TiB, happy to be hosted by GitHub’s cloud! So, what packages are being downloaded? I compiled two sets of data, lifetime stats and those of March 2016, below.

Lifetime Architectures

lifetime architectures

In this chart we see the architectures. Clearly ARM is still most popular. But ARM64 is gaining ground and also x86 is becoming more frequent. x86_64 Doesn’t exist that long and has such little share that it is marked as Other by Google Fusion Tables.

Lifetime APIs

lifetime apis

It is visible that 6.0 gained ground very quickly in the total number of downloads. One major reason is that opengapps.org is still quite young (less than a year old) and 6.0 was released when the project was still gaining popularity. The 5.1 high tide has been missed.

Lifetime Variants

lifetime variants

Super has been introduced more recently and is still not as popular as the other packages. Easily explained by the fact that it is only meant for power users that want to use a gapps-config with some of the more exotic applications.

March 2016 Architectures

march architectures

In the March architectures overview you can see there are no major shifts compared to the lifetime stats, but ARM64 is very slowly growing.

March 2016 APIs

march apis

In the March APIs it is visible that 6.0 is now the de-facto standard for ROMs. 5.0 is shrinking. Also 4.4 is becoming less popular.

March 2016 Variants

march variants

In the March variants overview it is visible that the smaller pico and nano packages are gaining. It seems that the growing size of Google’s apps makes stock not always fit on /system/ and leads to people switching to smaller packages, especially compared to the popularity stock had on 5.1 in September last year. Also aroma did shrink compared to September 2015, very probably caused by the unmaintained ‘AROMA Installer’ and the more frequent issues it poses on various devices and recent recoveries. You can read more about that in Raul’s post.

OpenGApps.org HomeSupport by DonationSource on GitHubCommunity on XDA ForumManual on WikiFollow via PushbulletDownload Open GApps AppOpen GApps BlogOpen GApps on FacebookOpen GApps on Google+Open GApps on TwitterOpen GApps on YouTubeOpen GApps App