Skip to main content

Validating recipes at scale

Validating a recipe at scale means running it across many real repositories at once, so you can see how it behaves on real-world code before you roll it out. With the Moderne CLI, you can build LSTs for dozens or hundreds of repositories - and then run your in-development recipe against all of them in a single command. From there, you can see the aggregated results: what it changed, what it left alone, and where it errored.

This guide will walk you through everything you need to know to get started with this validation.

Why validate at scale​

Unit testing recipes proves that a recipe does the right thing on a small, curated scale. While these provide a considerable amount of benefit, real-world code is far messier than any snippet. A recipe that passes its test suite can still miss variations you didn't anticipate, change code you didn't mean to touch, or throw on an unfamiliar construct.

Running a recipe against a large, representative body of actual repositories is how you surface those cases while you're still iterating on the recipe.

Prerequisites​

This guide expects that you have:

Haven't written your recipe yet? Moderne ships skills for AI coding agents that help you create, run, and refine recipes, automating much of the loop this guide covers by hand.

Step 1: Assemble your repositories and LSTs​

To validate at scale, the CLI needs a set of repositories with their LSTs available. You can either download them from Moderne or build LSTs for your projects locally. Follow the below instructions depending on which one you'd prefer.

Option A: Download existing LSTs from Moderne​

The easiest option is to pull down a Moderne organization, which downloads the repositories, their code, and their pre-built LSTs all at once:

mod git sync moderne /path/to/your/workspace --organization "<organization-name>" --with-sources

See Syncing Moderne organizations for how to discover the organizations available to you.

Option B: Build LSTs locally​

If you'd rather assemble your own set, point the CLI at any folder containing one or more checked-out Git repositories and build the LSTs yourself:

mod build /path/to/your/workspace

A recipe skips any repository whose LST isn't built, so you don't need a perfectly clean build to start. If one repository refuses to build, ignore it and validate against the rest.

Step 2: Make your recipe available to the CLI​

In order for the CLI to run your recipe, you'll need to let it know where the recipe is. The command you'll need to run to let the CLI know this will change depending on the language your recipe is written in:

For Java and YAML recipes, you can take advantage of the active recipe feature in the CLI. This feature lets you run a recipe without needing to specify the name or location of it every time.

To set the active recipe for the CLI, run the following command (make sure you specify the path to your recipe):

mod config recipes active set src/main/java/com/example/MyRecipe.java

If your file contains multiple recipes, you can add a --recipe <recipeName> flag to use a specific one.

Confirm the active recipe has been set correctly by running:

mod config recipes active show

Using the Moderne IntelliJ plugin? You can set the active recipe from the IDE instead of the command line: right-click the recipe class and choose Set Active Recipe. See Debugging recipes for details.

Step 3: Run the recipe across every repository​

Now that you've pointed the CLI to your recipe, you can run it against every repository in your workspace at once:

mod run /path/to/your/workspace --active-recipe

Step 4: Review the results and iterate​

Once the recipe run finishes, the next step is to see how your recipe did across all of the repositories. There are three things worth taking a look at:

First, the changes. Command/Ctrl-click the Fix results link in the output to open the combined diff, then confirm the recipe touched everything you expected and left everything else alone.

If you want to apply the changes your recipe made to the actual repositories, you can do so by running:

mod git apply /path/to/your/workspace --last-recipe-run

Next, any errors. The run output flags any repository where the recipe threw an error. To see what happened across the whole set at once, open the recipe run analyzer:

mod trace runs analyze /path/to/your/workspace --last-run

Finally, the data tables. Many recipes emit data tables alongside their changes, and the CLI suggests a mod study command after each run to open them:

mod study /path/to/your/workspace --last-recipe-run --data-table RecipeRunStats

RecipeRunStats and SourcesFileResults are good places to start for a repository-by-repository view of how the recipe behaved.

When the results don't match what you expected, tweak the recipe and run it again. For Java and YAML recipes, rebuild and re-run mod run /path/to/your/workspace --active-recipe (you only need to set the active recipe again if the file or its classpath moves). For C#, Python, and JavaScript recipes, rebuild and re-install the package first.

Debug a recipe against real code​

If you run into issues with your recipe, keep in mind that you can set breakpoints that can be hit while the recipe runs against these real repositories. To do this, add a --debug flag to the run command:

mod --debug run /path/to/your/workspace --active-recipe

The CLI suspends on startup and waits for a debugger to attach, then stops at your breakpoints as the recipe runs against real code. Any JDWP-capable debugger works (IntelliJ IDEA, VS Code, Eclipse). See Debugging recipes for the full setup, including how to drive it entirely from the CLI without the Moderne IDE plugin.

Breakpoint debugging attaches a JVM debugger, so it only applies to Java and YAML recipes. Debug C#, Python, and JavaScript recipes with your language's usual tooling.