Super charge build time in multi-module project with gradle’s dependency resolution
After implementing various projects with multi-module approach I can tell you that modular approach is the way to go if you can afford it.
If you haven’t read my blog on creating a custom gradle plugin to facilitate multi-module project I recommend you to read it (this is not needed to understand what I am going to talk in this post but it will save you a lot of time in future).
For those feeling adventurous (I strongly recommend to go through the post before checking out sample project) here is the link to the sample repository, focus on the project level build.gradle file and you are good to go.
Let’s start this story with why. Why do we need this?
Most of us have seen this graph where the build times drastically decreases when we start to increase the modules (specially incremental builds).
The another big factor that might effect build time is the memory usage of the gradle daemon. It’s no surprise that if your project is large then the memory usage of the gradle will increase drastically.
Unless you have system with 32 or 64GB of RAM you will probably run into problems.
History
In our company we had just finished the modularization of our project. This significantly reduced the build time(incremental) and most importantly code structure. It’s easy to navigate the source code and make necessary changes without understanding the entire project.
However the problem starts with the memory usage of the gradle daemons. Modularization of a project does decrease the build times but not the memory usage. Our project is quite big and in my laptop (with i7-4 cores, 16GB RAM and a spinning rust(Harddisk 😆)) the gradle daemons alone takes about 6–8GB of RAM. As there are many other processes like emulator, browsers and Android Studio itself, the system’s memory is almost full. This is a problem because when the RAM is full the system will start to swap out processes to disk and this takes a long long time. For example if I close all other programs and clean build our project, it takes about 4–5 minutes and hot build will take anywhere from 5–40 seconds (generally depending on how many modules are changed).
However if all programs are running in the background the clean build time remains unaffected and incremental build sometimes takes around 6–7 minutes (😵) as the system is busy swapping to disk. This just reversed the benefit of multi-module refactor that we had done.
And let’s not forget Android Studio’s indexing. From time to time we had to do clean build and man that would take pretty significant time.
So to mitigate this problem and to make development easy we tried to exploit our modular approach.
Workaround
So the rough ideas is this, we publish the modules (artifactory in our case) and in our app we depend on the published modules instead of local modules. Then we unload the local modules and build the project.
This drastically improved the build time as we only had few classes in app module and it was the only module that was being compiled.
But How to easily develop the software if all modules are published?
In our app we had a core module that every feature modules depend on. Let’s say we are changing some things in core and feature_a. For that we changed the dependency of feature_a module to project(':core') from the published version.
If we add some methods in core and call from feature_a it compiles. But at run time it crashes with exception Method not found. That was very unusual and after some digging through gradle dependency chain we found out that we were using published core instead of local core even if we had specified local core in feature_a. This makes sense as all other published feature modules were using published core and that caused gradle to resolve to published core.
This was quite a disappointment and we also had some other features and code refactor to do (the usual stuff 😂). So, for the time being we used a very simple method. Just created a centralized version.gradle file and added all the versions there. For local development we would swap all the published modules to local module in versions.gradle file. This will cause all the modules to compile as usual (with same high memory usage) but fixed our previous issue as all modules are being loaded locally. I can tell you it was not fun. Most of us would forget to reverse the dependency to published module before creating Merge Request(MR). So most of the MR would have a commit with message “changed dependency to published module” 😅.
Solution
We started to research on alternative approaches that will allow us to use both local and published modules simultaneously without depending on published modules at runtime (which would probably crash the app as it won’t find our local changes). We stumbled across gradle dependency resolution and after some research we came up with a solution that worked in our case.
So, the idea is pretty simple. We depend on published versions in all modules(we don’t use local project dependency). During gradle sync (or technically in gradle’s dependency resolution) we check if the current dependency that the gradle is resolving is our own module(by checking group id)). If the dependency is our own module and if that module is currently loaded in settings.gradle then we resolve to local version of the module. Otherwise we resolve to the published version.
Enough talk, let’s get our hand dirty. Add the following code to your project level build.gradle just below build script block.
The above code is pretty simple (but took me a quite a while to figure that out due to groovy 😜). First we are storing all the modules loaded in our project in loadedModules. Then during dependency resolution we are checking if the current dependency is our own module and if it’s present in loadedModules. If so we resolve to the local version of the module.
And in your modules build.gradle file depend on the published version instead of local version as usual.
Then to force the use of local module for development purpose just load the module that you are going to change in settings.gradle file(or more professionally just comment out the modules that you don’t want to modify 😆).
In the above case only core module is loaded locally and published versions of other modules will be used. This is pretty neat. Even if the published version (let’s say feature_a) has dependency on published version of core, that dependency would be replaced by local core during dependency resolution.
This drastically improved the build times as only few modules has to be compiled and also improved android studio’s performance as it has to do less indexing.
We tried to make this more professional by using the load/unload modules feature of Android Studio but this only unloads the module from Android Studio. The modules were still visible in gradle so this was not used. If anyone has any idea integrating this with Android Studio load/unload modules let me know in the comments down below.
Note: The sample project publishes the module to local maven for the sake of simplicity.
That’s it for this story. If you have any question or suggestions feel free to comment.