Case study: Climbing fflib
This site shows an AI agent improving the Apex code of fflib-apex-common, a widely used Salesforce library. The agent tries one small change at a time. Each change is scored in a Salesforce scratch org on tests, code coverage, static analysis and governor limit use, and is kept only if the score goes up. One such run on a group of classes is called an ascent. The tooling is Climber, and each ascent's code is on its own branch of this fork.
The library was climbed twice: once with Salesforce Code Analyzer's default PMD rules, and once with the PMD brace rules turned off. Comparing the two sets shows how far two independent runs find the same improvements.
How to use this site:
- Explore the out of the box map and the map adjusted for style to see how the library is grouped and how each group scored before any ascent.
- Explore the out of the box ascents and the ascents adjusted for style for each group. Each ascent's report shows every change the agent tried and why it was kept or discarded, and its retrospective sums up the behaviour changes and patterns.
- Compare the two sets and read what they taught us about Climber.
- Read the summary of what the out of the box ascents taught us, both about fflib itself and about how to climb better.
Caveats:
- The out of the box set ran PMD with Salesforce Code Analyzer's default rules and settings. fflib often leaves out braces, which is a valid style choice, and PMD can be configured not to flag it. The map adjusted for style shows the scores without the brace rules.
- The headroom tests that measure performance were written for these climbs and are not part of fflib. They repeat the same input many times, so the performance findings reflect those tests, not real workloads.
- Each ascent ran in one scratch org, and the governor limit headroom score varies by about ±12 between runs, so small performance differences are within noise.
- No anchoring was done. Anchoring means adding tests that fix the expected behaviour before the agent changes the code. We are not fflib's maintainers or authors, so we did not presume what the correct behaviour or the right tests are.
- The scratch org had multi-currency enabled and no namespace, so behaviour in namespaced orgs was not tested.
- Every ascent used Claude models to plan, make and review the changes. A model has its own biases, so some of the agreement between the two sets may come from the model, not the code. A future set climbed with a different model of similar capability would test this, and the Venn diagram could then be drawn again with that set.
- The two sets ran on different versions of Climber. Six of the ten out of the box ascents ran before Climber started rejecting candidates that add static analysis suppressions, measuring noise margins from three baseline runs, re-measuring close discards, and telling the agents not to pad code to raise a score. So some differences between the sets, such as the out of the box set's 146 iterations against 75 and its coverage restructuring, may come from those changes and not only from the brace rules.
If you are a coding agent, use the retrospective JSON files: one per ascent, linked in the table, or all of them in retros.json.
Out of the box map
The library grouped by area, sized by lines of code and coloured by score. Scored before any ascent, with Salesforce Code Analyzer's default PMD rules, as its ascents were scored.
Out of the box ascents
Map adjusted for style
The same code, scored from the same test and analyzer run as the map above, but with PMD run without the four brace rules (IfStmtsMustUseBraces, IfElseStmtsMustUseBraces, ForLoopsMustUseBraces, WhileLoopsMustUseBraces), which flag fflib's choice to leave braces out of one-line bodies. This removes 198 of the 229 PMD violations. Six of the ten groups reach a PMD score of 100. SObjects, Unit of work and SObject describe stay lower because of OneDeclarationPerLine and EagerlyLoadedDescribeSObjectResult.
Ascents adjusted for style
The same groups climbed again with the four brace rules left out of PMD, from the lowest scoring group on the map above.
What the two sets taught us about Climber
Climber's goal is to find the best improvements and bug fixes reliably, so two runs on the same code should find the same ideas. The chart counts the distinct ideas kept by each set across all ten groups, leaving out brace fixes.
- Both sets kept the biggest performance win in every group, and the main static analysis fixes such as the Graph Engine null check and deferred describes.
- They agree on about a third of all kept ideas (22 of 68). Counting the 15 ideas that one set kept and the other tried and discarded, they agree on about half.
- Most disagreement comes from the keep decision, not from missed ideas. Limit headroom varies by about ±12 between runs against margins of 3 to 23, so the same change can be kept in one run and discarded in another.
- Longer ascents drift into work of little value. About half of the ideas only the out of the box set kept restructure code to raise line coverage without improving it. That set ran 146 iterations against 75.
- The sets often agree on what to fix but not how. Five shared findings were fixed differently, such as suppressing a PMD warning in one set and deleting the code it flags in the other.
- Retrospectives agree more than code changes do. The paired retrospectives share 49 findings, such as stale caches under
System.runAsand bypassed virtual hooks. - Brace fixes made up most of the out of the box score gains, yet removing them changed little else: the gains outside PMD were similar in both sets.
- Each set has one run per group, and the ideas were matched by reading the ascent histories, so treat these numbers as approximate.
Changes made to Climber from what these climbs taught us:
- 7877e4e: fix iterate rejecting every plan step when an ascent has holdout tests.
- b6ac130: reject candidates that add static analysis suppressions, measure noise margins from three baseline runs, and re-measure a close discard.
- fdbc25e: tell the agents never to pad code to raise a score.
- 142dfc5: let an ascent leave chosen PMD rules out of the score, as the second set did with the brace rules.
- fff4c7e: re-measure a close keep, so a lucky run on a noisy score is not kept.
- 2eee2e3: flag a keep whose coverage rose for review, since tests cannot change and the rise came from removing or restructuring code.
Data
ascents.json lists every ascent with its scores and links. retros.json collects every ascent's retrospective (behaviour changes, patterns, test gaps) in one file.