Skip to main content

Writing a Python refactoring recipe

Across a large codebase, you often need to make the same change many times over: renaming a deprecated function, swapping an import, modernizing a pattern. When you run into this situation, the best thing you can do is write a recipe so that these changes can be made consistently and accurately. Then, you can use Moderne to apply these changes to thousands of repositories at once.

While we've done our best to provide you with a wide variety of Python recipes, you may find it useful to write your own. In this guide, we will walk you through everything you need to know to get started with creating and publishing your own Python recipe.

This guide focuses on authoring recipes in Python. If you would rather write recipes in another language, there are companion guides for writing a Java refactoring recipe, writing a JavaScript refactoring recipe, and writing Kotlin recipes. The core concepts carry over closely, since Python recipes build on the same Java model those guides use.

If you would rather start from working code, the python-recipe-starter repository contains a complete, runnable version of the recipe you'll build below, along with a second worked example and the tests, packaging, and CI already set up. Clone it to follow along, or use it as a scaffold for your own recipe.

Prerequisites​

This guide assumes that:

How Python recipes work​

Before we dive into how to write your own recipe, it's a good idea to take a few minutes to learn about Python recipes at a high level.

OpenRewrite represents Python code as a Lossless Semantic Tree (LST): a tree that preserves the code's exact formatting and is type-attributed, so every element carries its resolved type. Working against that tree instead of the raw text is what lets a recipe make precise, type-aware changes.

Every recipe is a class that describes itself with a name, a display name, and a description, and that returns a visitor from its editor() method. The visitor traverses the LST and returns modified nodes wherever it wants to make a change. Anything it returns unchanged is left exactly as it was, formatting included.

The Python LST builds on the Java LST. Shared constructs such as method invocations, identifiers, literals, and blocks come from the Java model (the J namespace), while Python-specific constructs such as pass statements, imports, and comprehensions live in the Python model (the Py namespace). Because of this, a Python visitor works with familiar J node types for most transformations.

The Python LST, parser, and node model live in the rewrite-python module. It is a useful reference when you need to know how a particular Python construct is represented.

Setting up your project​

Let's start by creating a virtual environment and installing the openrewrite package. This package contains the recipe framework, the Python LST, and the testing helpers:

python3 -m venv .venv
source .venv/bin/activate
pip install openrewrite

You'll also want to install a test runner. For this guide, we'll use pytest - but, in your actual recipe, you can choose whatever testing framework you want:

pip install pytest

Outlining the recipe​

Before implementing any logic, it's a good idea to sketch out the recipe's general shape. For the sake of an example, let's write a recipe that renames calls to one function so that they use a different name. In order to support that, we'll need to define two configuration options: the old name and the new name.

Here is what a rough outline of this class might look like:

rename_function_call.py
from dataclasses import dataclass, field

from rewrite import ExecutionContext, Recipe, TreeVisitor, option
from rewrite.java import J
from rewrite.python.visitor import PythonVisitor


@dataclass
class RenameFunctionCall(Recipe):
    """Rename calls to a function from one name to another."""

    old_name: str = field(default="", metadata=option(
        display_name="Old function name",
        description="The name of the function whose calls should be renamed.",
        example="assertEquals",
    ))

    new_name: str = field(default="", metadata=option(
        display_name="New function name",
        description="The name to rename matching calls to.",
        example="assertEqual",
    ))

    @property
    def name(self) -> str:
        return "com.yourorg.RenameFunctionCall"

    @property
    def display_name(self) -> str:
        return "Rename a function call"

    @property
    def description(self) -> str:
        return "Rename calls to a function from one name to another."

    def editor(self) -> TreeVisitor[J, ExecutionContext]:
        class Visitor(PythonVisitor[ExecutionContext]):
            pass

        return Visitor()

A few things to call out here:

  • The recipe is a @dataclass that subclasses Recipe.
  • Each configuration option is a dataclass field whose metadata is built with option().
  • The name follows a reverse-domain convention (com.yourorg.RenameFunctionCall) so that it is globally unique.
    • This is the identifier you will use to run the recipe later.
  • For now, editor() returns a visitor that does nothing, so the recipe is a no-op. We will add the code for this after we finish writing tests.

Give every option a default value (e.g., default=""). The framework instantiates your recipe without arguments when it builds the recipe's descriptor, so a recipe whose options lack defaults cannot be discovered or run by the Moderne CLI.

Writing tests first​

OpenRewrite's testing harness parses a before snippet, runs your recipe, and asserts that the result matches an after snippet. Writing the tests first gives you a precise specification of what the recipe should do.

Here is what our tests might look like:

test_rename_function_call.py
from rewrite.test import RecipeSpec, python

from rename_function_call import RenameFunctionCall


def test_renames_a_bare_call():
    spec = RecipeSpec(recipe=RenameFunctionCall(
        old_name="assertEquals",
        new_name="assertEqual",
    ))
    spec.rewrite_run(
        python(
            """
            assertEquals(a, b)
            """,
            """
            assertEqual(a, b)
            """,
        )
    )


def test_renames_a_qualified_call():
    spec = RecipeSpec(recipe=RenameFunctionCall(
        old_name="assertEquals",
        new_name="assertEqual",
    ))
    spec.rewrite_run(
        python(
            """
            self.assertEquals(a, b)
            """,
            """
            self.assertEqual(a, b)
            """,
        )
    )


def test_leaves_other_calls_unchanged():
    spec = RecipeSpec(recipe=RenameFunctionCall(
        old_name="assertEquals",
        new_name="assertEqual",
    ))
    spec.rewrite_run(
        python(
            """
            assertTrue(x)
            """
        )
    )

The python() helper accepts either one or two arguments:

  • Two arguments (python(before, after)) assert that the recipe transforms before into after.
  • One argument (python(before)) asserts that the recipe makes no change.

Always include at least one no-change test, such as test_leaves_other_calls_unchanged above, so that you can be confident your recipe does not touch code it should not.

If we run the suite now, we'll see the expected starting state: the two renaming tests fail because the visitor doesn't do anything yet, while the no-change test already passes.

Implementing the visitor​

With the tests written, let's work on making them pass. We'll update the editor function to look for and replace code that matches the inputs provided to the recipe. As part of this, we will override visit_method_invocation (the visit method the LST uses for function and method calls):

rename_function_call.py
from dataclasses import dataclass, field

from rewrite import ExecutionContext, Recipe, TreeVisitor, option
from rewrite.java import J
from rewrite.java.tree import MethodInvocation
from rewrite.python.visitor import PythonVisitor


@dataclass
class RenameFunctionCall(Recipe):
    """Rename calls to a function from one name to another."""

    old_name: str = field(default="", metadata=option(
        display_name="Old function name",
        description="The name of the function whose calls should be renamed.",
        example="assertEquals",
    ))

    new_name: str = field(default="", metadata=option(
        display_name="New function name",
        description="The name to rename matching calls to.",
        example="assertEqual",
    ))

    @property
    def name(self) -> str:
        return "com.yourorg.RenameFunctionCall"

    @property
    def display_name(self) -> str:
        return "Rename a function call"

    @property
    def description(self) -> str:
        return "Rename calls to a function from one name to another."

    def editor(self) -> TreeVisitor[J, ExecutionContext]:
        old_name = self.old_name
        new_name = self.new_name

        class Visitor(PythonVisitor[ExecutionContext]):
            def visit_method_invocation(self, method: MethodInvocation, p: ExecutionContext) -> J:
                method = super().visit_method_invocation(method, p)
                if method.name.simple_name == old_name:
                    renamed = method.name.replace(_simple_name=new_name)
                    return method.replace(_name=renamed)
                return method

        return Visitor()

Here's what the visitor does, step by step:

  • It copies self.old_name and self.new_name into local variables that the nested visitor can read. Inside the visitor, self is the visitor instance rather than the recipe, so self.old_name would not be available there.
  • It calls super().visit_method_invocation(...), which visits the call's children before the call itself. Visiting from the bottom up is the safe default, because it lets nested calls transform before their parents.
  • It checks whether the call should be renamed by comparing method.name.simple_name (the name of the function being called) against old_name.
  • When the name matches, it builds a renamed identifier with method.name.replace(_simple_name=new_name) and returns a new method invocation via method.replace(_name=renamed). LST nodes are immutable, so .replace(...) returns a new copy instead of mutating in place.
  • Otherwise, it returns the original method unchanged, so the call is left exactly as it was.

Returning None from a visit method removes the node entirely - which is how recipes delete code.

Running the tests​

Now that we have the visitor coded, let's run the suite with pytest:

python -m pytest test_rename_function_call.py -v

All three tests should now pass:

test_rename_function_call.py::test_renames_a_bare_call PASSED            [ 33%]
test_rename_function_call.py::test_renames_a_qualified_call PASSED       [ 66%]
test_rename_function_call.py::test_leaves_other_calls_unchanged PASSED   [100%]

============================== 3 passed in 0.20s ===============================

Packaging and running with the Moderne CLI​

With our tests passing, let's now test our recipe against some real repositories. We'll use the Moderne CLI to run our recipe. However, in order for the Moderne CLI to discover our recipe, our project needs to expose an activate() function that registers it with the recipe marketplace.

Let's add that activate() function to rename_function_call.py:

rename_function_call.py
from dataclasses import dataclass, field

from rewrite import ExecutionContext, Recipe, TreeVisitor, option
from rewrite.java import J
from rewrite.java.tree import MethodInvocation
from rewrite.marketplace import RecipeMarketplace, Python
from rewrite.python.visitor import PythonVisitor

# ... the RenameFunctionCall class from the previous step ...

def activate(marketplace: RecipeMarketplace) -> None:
    marketplace.install(RenameFunctionCall, Python)

That second argument is the recipe's category path, which decides where it shows up in the marketplace. Python files it under the top-level Python category. To group your recipes under your own organization instead, check out categorizing your recipe below.

We also will need to describe the package in a pyproject.toml file:

pyproject.toml
[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "rename-function-call"
version = "0.1.0"
dependencies = ["openrewrite"]

[tool.setuptools]
py-modules = ["rename_function_call"]

[project.entry-points."openrewrite.recipes"]
rename-function-call = "rename_function_call:activate"

Two parts of that file are what make your recipe discoverable, and the install fails quietly without either one:

  • dependencies = ["openrewrite"] is required because the CLI installs each recipe package into its own virtual environment. A package that does not declare openrewrite cannot import rewrite when activate() runs.
  • The openrewrite.recipes entry point is how the CLI finds your activate() function. It points at module:function, so keep it in step with your module name if you rename either one.

With the activate() function and pyproject.toml file in place, the package is ready to install into the Moderne CLI.

Running your recipe locally​

Install the recipe straight from your project directory:

mod config recipes pip install /path/to/your/recipe-project

You should see Found 1 recipes if everything worked correctly - which confirms that your recipe was registered.

Found 0 recipes means the CLI installed your package but could not activate it. The install still reports success and writes nothing to its log, so the recipe count is the only signal you get.

The two usual causes are a missing openrewrite.recipes entry point and a missing openrewrite dependency. Check both in your pyproject.toml file before looking anywhere else.

Now run it against a repository whose Python LSTs you've already built, passing each option as a -P parameter:

mod run . --recipe=com.yourorg.RenameFunctionCall -Pold_name=assertEquals -Pnew_name=assertEqual

When you are done, you can remove the recipe from your local marketplace. A package installed from a local path is keyed by that path, so pass the same path you installed:

mod config recipes pip delete /path/to/your/recipe-project

Deleting by distribution name reports success without removing anything when the package was installed from a path. If your recipes are still in the marketplace afterwards, you passed the wrong key.

Publishing your recipe​

The openrewrite.recipes entry point you already declared is what lets anyone who installs your package pick up its recipes. Once the package is published to a package index, others can install it by name and run it exactly as you did locally:

mod config recipes pip install rename-function-call

Categorizing your recipe​

The marketplace organizes recipes into a tree of categories, and every recipe has to say where in that tree it belongs. In Python you declare that placement in activate(), as the category path you hand to marketplace.install().

Building a category path​

A category path is a list of CategoryDescriptor objects ordered from shallowest to deepest. The Python constant you imported is a one-element list, so you can splice it into a longer path:

from rewrite import CategoryDescriptor
from rewrite.marketplace import Python

Cleanup = [*Python, CategoryDescriptor(display_name="Cleanup")]

Installing a recipe with that path files it under Python > Cleanup:

def activate(marketplace: RecipeMarketplace) -> None:
    marketplace.install(RemoveRedundantPass, Cleanup)

Levels that don't exist yet are created on install. You can also install the same recipe twice to have it show up under two different paths.

Nesting under an existing category​

A category has no identifier of its own. It's keyed by its display name, so any two bundles that install recipes under the same name land in the same node of the tree, whatever language they were written in. There's no shared registry to import from. Each package declares its own descriptor, and matching ones merge.

Say your organization already publishes Java recipes named com.example.recipes.*. Those show up under a top-level Example category, since com is a root category that gets skipped and the next segment is capitalized into a display name. To file your Python recipes there too:

rename_function_call.py
from rewrite import CategoryDescriptor
from rewrite.marketplace import RecipeMarketplace

Example = [CategoryDescriptor(
    display_name="Example",
    description="Recipes maintained by the Example platform team.",
)]
ExamplePython = [*Example, CategoryDescriptor(display_name="Python")]


def activate(marketplace: RecipeMarketplace) -> None:
    marketplace.install(RenameFunctionCall, ExamplePython)

Your recipes now show up under Example > Python, alongside the Java ones under Example.

Copy the existing display name and description exactly, capitalization included. Whichever bundle is installed first owns the node. A later one can fill in a description that was left blank, but it can never change a display name or overwrite a description. Copying both is what makes the result the same no matter which order things are installed in.

If several of your own packages file recipes into the same category, you can publish the descriptors in a small shared package and depend on it from each one rather than copying them by hand. Give that package no openrewrite.recipes entry point, so the CLI treats it as a plain dependency instead of a recipe bundle.

Declaring the category on the recipe class​

Repeating the path in every install() call gets old once you have more than a few recipes. The @categorize decorator lets each recipe carry its own:

rename_function_call.py
from dataclasses import dataclass

from rewrite import CategoryDescriptor, Recipe, categorize

ExamplePython = [
    CategoryDescriptor(display_name="Example"),
    CategoryDescriptor(display_name="Python"),
]


@categorize(ExamplePython)
@dataclass
class RenameFunctionCall(Recipe):
    """Rename calls to a function from one name to another."""

    # ... the options, properties, and editor() from the previous step ...

Your activate() function can then read those paths back instead of hard-coding them:

__init__.py
import sys

from rewrite import RecipeMarketplace, discover_decorated_recipes_in_module


def activate(marketplace: RecipeMarketplace) -> None:
    for recipe_class, category in discover_decorated_recipes_in_module(sys.modules[__name__]):
        marketplace.install(recipe_class, category)

discover_decorated_recipes_in_module() finds every decorated Recipe subclass in a module and returns it paired with its path. Called from your __init__.py, it picks up everything that file imports, so a new recipe needs nothing but an import and a decorator. It looks at a single module, so recipes that live in subpackages have to be imported into __init__.py before it will find them.

@categorize records a path on the class. It doesn't install anything. If you decorate a recipe but leave a hard-coded path in its install() call, the decorator is ignored and activate() wins.

Next steps​

Now that you've written your first Python recipe, you can go deeper: