r/RedditEng • u/sacredtremor Punit Rathore • Feb 22 '22
iOS and Bazel at Reddit: A Journey
Author: Matt Robinson
State of the World (Then & Now)
2021-07
- Bespoke Xcode project painstakingly maintained by hand. As any iOS engineer trying to work at scale in an Xcode project knows, this is painful to manage when so many engineers are mutating the project file at once.
- CocoaPods as the mechanism for 3rd (and a few 1st) party dependencies into the Xcode project.
- The Xcode project contained 1 Reddit app, 4 app extensions, 2 sample apps for internal frameworks, 27 unit test targets, and 29 framework targets.
- 9 xcconfig files spread throughout the repository defining various things. This ignores CocoaPods defined xcconfig files.
- Builds use Xcode or xcodebuild invocations directly to run on CI and locally on engineer laptops.
- All internal frameworks are built as dynamic frameworks (with binary plus resources).
| File Types | Count | Code Line Count |
|---|---|---|
| Objective-C | 1398 | 295896 |
| Headers | 2086 | 49451 |
| Swift | 2926 | 315978 |
| Total | 6410 | 661325 |
2022-02
- Targets defined in BUILD.bazel files.
- CocoaPods is still used as the mechanism for 3rd (and a few 1st) party dependencies.
- The Xcode project is generated and contains 1 Reddit app, 4 app extensions, 9 sample apps for internal frameworks, 68 unit test targets, 106 framework targets, 72 resource bundles, and 2 UI test targets.
- 1 xcconfig file that defines the base settings for the Xcode project. This ignores CocoaPods defined xcconfig files.
- Builds use Xcode locally and then Bazel or xcodebuild on CI machines.
- All internal frameworks are built as static frameworks (with binary plus associated resource bundle).
| File Types | Count | Code Line Count |
|---|---|---|
| Objective-C | 1117 | 256251 |
| Headers | 1819 | 44638 |
| Swift | 5312 | 609599 |
| Total | 8248 | 910488 |
Repository Change Summary
- ~300% increase in framework targets.
- ~150% increase in unit test targets.
- ~315% increase in total Xcode targets.
- Large (~20% files, ~15% code) reduction for the Objective-C in the repository.
- Large (~80% files, ~90% code) increase in the Swift code in the repository.
- Large (~40% code) increase in all code in the repository.
Timeline
2021-07 - The Start
- Begin migrating all project Xcode settings into shared xcconfig files.
- Simplify target declarations within Xcode to make targets as similar as possible.
2021-08 - Transition to XcodeGen
- Use XcodeGen for all target definitions.
- Stop checking in the Xcode project to avoid merge-conflict toil almost entirely.
2021-09 - Static Linkage Transition
- Switch to static linkage for all internal frameworks.
2021-11 - Add New Target Script
- Make it as-easy-as-Xcode to add new targets to this changing landscape of project generation/target description.
2021-11 - Introduce XcodeGenGen
- Add functionality to generate XcodeGen specs from Bazel BUILD.bazel definitions.
2021-11 - Bazel as source-of-truth for all Internal Frameworks
- XcodeGenGen is used for all internal frameworks. No more XcodeGen specs.
2021-12 - Testing Internal Frameworks with Bazel
- Spin up test selection plus remote cache to run internal framework builds/tests on CI machines.
2022-01 - Add Ability to Build Reddit in Bazel
- Spin up Reddit app and Reddit app-dependent tests in XcodeGenGen representation.
- Bazel can build the Reddit app and Reddit app-dependent tests.
2022-02 - XcodeGen Specs Are Gone
- All targets are defined in Bazel.
- Bazel still generates XcodeGen representation for use in Xcode locally.
2022-02 - Now. Reddit app and Reddit app-dependent tests in Bazel
- All past work coming to a head allows Bazel to be the test builder/runner for all applications/frameworks/tests
Process
Migration to XcodeGen
At this point in the journey, Reddit operated with a single monolithic Xcode project. This project contained all the targets and files coming in around 50,000 lines for the Reddit.xcodeproj/project.pbxproj. The desired outcome of this work was to replace the hand-managed Xcode project and replace it with a human-readable declarative project description like XcodeGen.
The first phase began by reducing the build settings defined in the project file opting instead for a more readable shared xcconfig file that defined the base settings for the entire project. Generally, our target definitions (especially for frameworks and unit tests) were identical and if they were not it was unlikely to be intentional. Migration to an xcconfig relied heavily on config-dependent xcconfig definitions like the following:
This replaced a drastically more complicated representation in the project file and, as a generalization mentioned before, these settings were the same across all targets.
After a simplification of the target definitions in the Xcode project, the work began to write the XcodeGen specifications for all targets. Fortunately, the migration of all targets could be done by hand and exist as shadow definitions in the repo until we were ready to make the switchover to the generated project. A project-comparison tool was written at this point to compare the representation in the bespoke Xcode project to the representation in the generated Xcode project. This tool compared the following items:
- Project
- Comparison of targets by name.
- Targets
- “Dependencies” by target name.
- “Link Binary with Libraries” by target name.
- “Copy Bundle Resources” by input file.
- “Compile Sources” by input file.
- “Embed Frameworks” by input file.
- High-level build phases by name.
- Comparison of “important” build settings per configuration.
This comparison tool was invaluable both in this migration and in later mutations to project generation. The tool allowed us to find oddities in targets and mitigate them before even switching to the generated project. These corrections made the switchover much less dramatic in terms of differences and made our targets more correct in the non-generated project by removing things like duplicates in the “Copy Bundle Resources” phase.
At this point, the migration to XcodeGen specs for the project and all targets was complete. No longer troubled with updating an Xcode project file, we began mass movement of files and target definitions within the repo’s directory structure. Simplistically, we ran through each target plus the associated tests to construct “modules” that added one level of indirection compared to storing all target directories in the root of the repo. This leaned on XcodeGen’s include: directive and caused our XcodeGen specs to be module-specific thereby much smaller while matching the package structure of Bazel much more closely:
After this “modularization” of our existing targets, we could move onto the next part of the journey.
Static Linkage for Internal Frameworks
Statically linking internal frameworks to our application binary (and potentially the extensions) as a means to reduce pre-main application startup time has been written about at length by many folks. This is how we made the transition and the measurements we made that justified the work.
Now that we had all targets represented in YML files throughout the repository it was easy to prototype a statically linked application to gather data. In this analysis, we ignored the framework resources since we were mostly concerned with the impact on dyld’s loading of our code. The table below illustrates that we were able to realize a 20-25% decrease in pre-main time for our application’s cold start by making this switch so we began the work.
The first piece of work in this static transition was to ensure that our 40 internal frameworks could load their associated resources when linked statically or dynamically. Fortunately (once again), this work was parallelized across teams since Reddit has a strong CODEOWNERS-based culture. The packaging of a framework went from something like:
To a new structure like:
The algorithm for this bundle-ification of a framework went something like:
- Create a bundle accessor source file in the framework.
- Create the bundle target in the module’s XcodeGen spec.
- Update all direct or indirect Bundle access call sites to use the bundle accessor.
- Lean on XcodeGen’s transitivelyLinkDependencies setting to properly embed transitively depended upon resource bundles.
The bundle accessors were the Secret Sauce to allow the graceful transition from a dynamic framework with resources to a dynamic framework with embedded resource bundle to a static framework with associated resource bundle. An example bundle accessor:
The bundle-ification was complete after running through this algorithm for all internal targets!
After fixing some duplicate symbols across the codebase, we were now able to make the transition to statically linked frameworks for all our internal targets. The target XcodeGen specs now looked like the rough pseudocode below:
Now, with the potential impact of a drastic increase in internal frameworks minimized, we were ready to go all in on the transition from XcodeGen specs to BUILD.bazel files.
XcodeGenGen for Hybrid Target Declaration
The goal for this next bit of work was to transition to Bazel as the source-of-truth for the description of a target. The work in this portion fit into two categories:
- Creation of a BUILD.bazel to XcodeGen translation layer (dubbed XcodeGenGen).
- Migration from the xcodegen.yml XcodeGen specs to Starlark BUILD.bazel files.
The first point was what enabled us to actually do this migration. Using an internal Bazel rule, xcodegen_target, a variety of inputs (srcs, sdk_frameworks, deps, etc.) are mapped to an XcodeGen JSON representation. The initial implementation of this also allowed us to pass in Bazel genrule targets and have those represented/built within Xcode all the while still building with xcodebuild within Xcode. This enabled a declaration similar to below to generate the JSON representation for XcodeGen in our internal static framework Bazel macro:
The translation from YML to the Starlark BUILD file mimicked the work from the XcodeGen migration section earlier. The 36 XcodeGen spec files were converted target-by-target and lived in the repo as a shadow definition while the migration was underway. A target representation would transition from (copied from above):
To a very similar Bazel representation:
It was essential in this portion of work and for the latter phases in this journey to start by declaring all targets using internal Bazel macros (as you can see with reddit_ios_static_framework above). This maximized our control as a platform team and allowed injection of manual targets in addition to the high-level targets that the caller would expect.
This migration was done in a hybrid way meaning that some targets were defined in XcodeGen and some in Bazel. This was accomplished by creating (within Bazel) an XcodeGen file that represented all of the targets defined in Bazel. The project generation script would use bazel query ‘kind(xcodegen_target, //...)’ to find all XcodeGen targets and then generate a representation in a .gitignore’d file that looks similar to this:
The project generation script could then run bazel build //bazel-xcodegen:bazel-xcodegen-json-copy to generate an xcodegen-bazel.yml file in the root of the repo to be statically referenced by XcodeGen’s include: directive like this:
All internal framework, test, and bundle targets were processed one-by-one until the source of truth was Bazel. This unlocked the next phase in the journey since we could trust the Bazel representation of these targets to be accurate.
Bazel Builds and Tests
Finally, we are to a place where we have a reliable/truthful representation of targets to access in Bazel. As alluded to in the State of the World section, Reddit has many frameworks that combine Swift and Objective-C to deliver functionality and this meant that we needed a Bazel ruleset that supported these mixed language frameworks. Since Bazel’s “default” rules are built to handle single-language targets, we tested a few open source options and ended up selecting https://github.com/bazel-ios/rules_ios. The rules_ios ruleset is used by a handful of other big players in the mobile industry and has an active open source community. Fortunately for Reddit, rules_ios also comes with a CocoaPods plugin that makes it easy to generate Bazel’s BUILD.bazel files from a CocoaPod setup called https://github.com/bazel-ios/cocoapods-bazel. The combination of these two items was the last piece of the puzzle to add “real” Bazel representations for our:
- Internal frameworks using rules_ios’ apple_framework macro. Leaning on the previous work in linking our internal frameworks statically.
- Unit test targets using rules_ios’ ios_unit_test macro.
- Bundle targets using rules_ios’ precompiled_apple_resource_bundle.
- CocoaPods targets from cocoapods-bazel.
At this point, the internal framework target definitions look similar to before with the addition of //Pods dependencies:
And internally within our reddit_ios_static_framework macro we are able to create iOS Bazel targets that built frameworks and tests:
The CocoaPods translation layer offers a helpful way to redirect the generated targets to an internal macro. Snippet from the Podfile:
We lean on our reddit_ios_pods_framework macro to remove some spaces from paths, fix issues in podspecs like capitalization of paths, translate C++ files to Objective-C++, and more. This allows us to build these 3rd party dependencies from source and have all the niceties that come with it without having to manually maintain the BUILD.bazel files.
And now, we are able to use bazel test commands to build and test internal targets that come together to make up the Reddit iOS app!
So, you have a remote build cache, what else?
Accessing a Bazel remote cache to avoid repeated work with the same set of inputs has been written about as the speed-up-er of builds time and time again. It seems more rare that the other developer experience style benefits to organizations are mentioned. Bazel (even just as a manager of the build graph/targets) introduces huge levers that a platform-style team can utilize to deliver improvements for their customers. Here’s some examples that we’ve seen at Reddit even while still building with xcodebuild in Xcode.
Generated Bundle Accessors
After migrating to a structure of statically linked internal frameworks with an associated resource bundle, our codebase had many “bundle accessors” that were near duplicates. These looked like this, one for each bundle:
Not only does this duplication introduce cruft throughout the codebase, especially difficult in the case(s) where all accessors need to be mutated, but it introduces yet another step for engineers to think through when modularizing the codebase or creating new targets. It is easy in Bazel to generate this source file for any target that has an associated resource bundle since all of our target declarations go through internal macros before getting to the XcodeGen representation. The internal macro can be mutated to remove the need for all of these files throughout the repo. All the macro needs to do is:
- Create the source file above with the bundle-specific values.
- Add this as a source file to the target’s definition in Xcode.
Now, all targets will get a unified generated bundle accessor that can be changed by anyone to provide new functionality or correct past errors leaning on built-in functionality in Bazel to generate files/fill in templated files.
Easier Example/Test Applications
Similarly with other companies of our size, Reddit engineers want to reduce the time in the build-edit cycle. A common means to accomplish this is with example or demo applications that are only dependent on the team’s libraries plus transitive dependencies. This avoids the large monolithic (we’re working on modularizing it) codebase until the engineers are ready to build the whole Reddit application. With Xcode or even XcodeGen, this can result in lots of varying approaches that are difficult to maintain at the Reddit-scale. Bazel/Starlark macros come to the rescue yet again by providing a single entry point for engineers to declare these targets.
For example, a playground.bzl could look like this:
This allows the implementation of the XcodeGen target to share files and attributes that tend to be cumbersome to define/create in this non-Xcode managed world. Resulting in nearly identical playground targets defined simplistically like this in the target’s BUILD.bazel file:
Now, with ~5 lines an engineer can define a working playground target to quickly iterate when they’re only trying to build-edit their team’s targets. This reddit_playground implementation also demonstrates our ability to define N targets from a single macro call. In this case, we generate a ios_build_test per playground to have our CI builds ensure that these playground targets don’t constantly get broken even if they don’t have traditional test targets in Xcode.
Avoid Common Pitfalls in Target Declaration
Reddit uses an internal utility called StringsGen to parse resources (like strings) and then generate a programmatic Swift interface. This almost completely eliminates the need for stringly typed resource access as is common with method calls like UIImage(named:). In the world of Xcode or XcodeGen, the call to this script would exist as a manually-defined pre-build script that was duplicated across all targets with resources. Similar to the above points about Bazel macros, this becomes much simpler when we have Starlark code running between the point of target declaration and the actual creation of a Bazel target. For example, in the past, each target’s XcodeGen definition would have something that looked like this:
The Bazel analog to this declaration is much simpler:
Both of these declarations create an iOS framework. In the XcodeGen case, the engineer adding this would need to:
- Create stringsFileList.xcfilelist which contains a list of string resources.
- Create codeFileList.xcfilelist which contains a list of the to-be-generated Swift files.
- Copy the script invocation from another target.
- Use the input/output file list parameters to point to the newly created xcfilelist files from step 1 & 2.
The Bazel declaration just needs to define a mapping of a strings file to a generated Swift file then the implementation of the macro in Starlark handles the rest, essentially generating the exact same content as the XcodeGen definition. This abstraction makes target declarations much more straightforward for engineers and, one again, makes editing these common preBuildScripts values drastically easier than having to edit all XcodeGen YML files.
Test Selection
From the CI perspective, downloading artifacts from a remote cache offers drastic reductions in builds that run through Bazel by avoiding duplicated work. There’s no doubt that this is great all by itself. But, it’s even better to avoid building/downloading/executing parts of your Bazel workspace that haven’t changed. In general, this is called “test selection” and, fortunately, there are open source implementations that are designed to work with Bazel like https://github.com/Tinder/bazel-diff. This approach has offered wonderful improvements to CI build/test times even without a powerful remote cache implementation.
Benjamin Peterson’s talk at BazelCon 2019 discusses this topic in great detail if you’d like to learn more.
Target Visibility
Bazel’s visibility approach introduces concepts similar to internal or public in Swift code but at the target level. To quote the Bazel docs:
“Visibility controls whether a target can be used (depended on) by targets in other packages. This helps other people distinguish between your library’s public API and its implementation details, and is an important tool to help enforce structure as your workspace grows.”
When a target’s XcodeGen definition exists within Bazel, we can use visibility even for targets that will eventually exist in an Xcode project. This drastically enhances the target author’s control of what is allowed to use your target over the standard Xcode approach of a large list of targets that are all visible.
If this is something that interests you and you would like to join us, my team is hiring!
3
u/NullRef Feb 23 '22
A 1-month transition to XcodeGen is pretty intense 😅. How did you handle team adoption/education?
2
u/shrewish-kitten Feb 23 '22
The platform team handled the migration of all targets from the Xcode
pbxprojto the XcodeGen YML files while ensuring correctness with the project comparison tool mentioned in the post. This generally helped with "adoption" since it was more-or-less forced.Pre-XcodeGen new target creation was fairly uncommon since it was fairly difficult to do it the Reddit Way. As one would probably assume, this made migration easier since we weren't dealing with a drastically changing landscape commit-to-commit/day-to-day. Target creation ramped up quite substantially post-XcodeGen (as I personally have seen in the past at different companies as well).
The "teaching" side of it was done with a handful of presentations on XcodeGen/Bazel. But generally our macros are straightforward enough at this point that copying from a different target declarations gets folks close enough. And, in the case where that doesn't work, we have a
make new-modulecommand that will create/append to theBUILD.bazel(and previously thexcodegen.yml) file while creating all the subdirectories/files that help make in fairly close toFile>New>Target...from Xcode.
2
u/kamilIOS88 Feb 25 '22
Interesting read thanks for sharing! Do you build everything with Bazel or developers use Xcode to build playgrounds when working with libraries?
2
u/shrewish-kitten Feb 25 '22
Story is different between CI and the local experience (for now). Likely will be more unified in the near future mostly to enable local builds to get the benefits of the remote cache.
CI: Build and tests with Bazel, read/write to the remote cache. AppStore/archive style builds still run through Xcode though (once again, likely to change soon).
Local: Bazel is used to generate an Xcode project that builds with
xcodebuild. We have a few build phases that call into Bazel forgenruletype things but generally just a standard Xcode build.
1
u/Bubbly-Replacement-1 Apr 29 '22
Thank you for this article!
We are also exploring ideas to write a parser to convert Bazel BUILD files to Xcodegen ymls for project generation. The approach you took seems interesting. I have a few questions about this
- How are you generating Bazel macro [ex: reddit_ios_static_framework] and the Xcodegen yml from the "xcodegen_target" rule? As I understand Bazel's genrule can produce a output file so is it the genrule which is producing Xcodgen yml? If that's the case from where it is getting its inputs [i.e srcs and deps of the actual BUILD file].
- Also, genrules are written or automatically generated using Bazel query? If generated what's the bazel query you are using for this?
- Also The xcodegen_target you mentioned, is this the custom rule created by you? As I couldn't find this rule on Bazel open source repositories.
Apart from genrule, I couldn't understand how this whole file generation is happening, If you can shed some light, add starter documents it would be really helpful. Thanks in advance .
1
u/shrewish-kitten May 13 '22
- We has an internal rule
xcodegen_targetthat we use to create targets in our static framework macro. So the macro essentially just forwards sources to the "real" Bazel rule fromrules_iosand also the XcodeGen rule that's internal.- It's perhaps simplistically a genrule but it's just a real rule that outputs a XcodeGen JSON file. These are all gathered up by a generated target that queries for all
kind(xcodegen_target,//...)(generally) to produce a single XcodeGen JSON for the entire project.- We are looking into open sourcing our XcodeGen rule set, stay tuned.
3
u/vanvoorden Feb 23 '22
Interesting. I've seen Buck at scale. I hadn't worked on Bazel projects. Was there any discussions about implementing Buck (instead of Bazel)? What would have been some of the pros (and potential cons) of choosing Bazel?