How to configure the Moderne CLI to work with internal tools

This guide walks you through configuring the Moderne CLI to work in environments with restricted internet access or internal artifact repositories. You'll learn how to set up the CLI to use your internal Maven repository, configure it to download recipe JARs from your internal systems, and prepare it to run OpenRewrite recipes against your repositories. The process ensures that all dependencies and recipe modules are properly accessible within your internal infrastructure.

Assumptions

• You have an internal mirror of Maven Central (or some other internal artifact repository)
• You have the ability to download and add JARs from Maven Central to your internal artifact repository

Installation and configuration

Step 0: Ensure recipes exist in your internal artifact repository

There are numerous OpenRewrite recipe modules available in Maven Central. Please ensure that those are copied over to your internal artifact repository (or your internal mirror of Maven Central).

You can find the full list of all OpenRewrite recipe JARs available here.

Step 1: Download the Moderne CLI JAR

Download the latest version of the Moderne CLI from Maven Central. Once downloaded, please add it to your internal mirror so that it's accessible for all users in your environment.

Step 2: (Optional - but recommended) Create an alias for the Moderne CLI JAR

While not required, you are strongly encouraged to set up an alias for running the CLI JAR. Below are some ways you might configure this depending on your OS and terminal:

Git Bash (Windows)

Add the following function to your .bashrc file:

mod() {
    java -jar "/path/to/mod.jar" "$@"
}

Bash or Zsh (Mac or Linux)

Add the following to your .bashrc or .zshrc file:

alias mod="java -jar /path/to/mod.jar"

Powershell

Use the Set-Alias command within a profile script.

If everything was configured correctly, you should be able to type mod into your terminal and see a list of commands:

mod command results

➜ mod

Moderne CLI 3.50.1

Usage:

mod [-h] [--version] [COMMAND]

Description:

Automated code remediation.

Options:

  -h, --help      Display this help message.
      --version   Display version info.
Commands:

  afterburner          (INCUBATING) Indexes built LSTs to accelerate recipe
                         execution.
  audit                (INCUBATING) Perform an audit of recent activity.
  build                Generates LST artifacts for one or more repositories.
  clean                Clean build and run artifacts produced by the CLI.
  config               Global configuration options that are required by some
                         CLI commands.
  devcenter            DevCenter operations.
  exec                 Execute an arbitrary shell command recursively on
                         selected repository roots.
  git                  Multi-repository git operations.
  log                  Manages a log aggregate.
  list                 Lists the repositories that can be built and published.
  monitor              (INCUBATING) Launches an HTTP server used to monitor the
                         CLI.
  publish              Publishes the LST artifacts for one or more projects.
  run                  Runs an OpenRewrite recipe locally on pre-built LSTS.
  run-history          Get information about the most recent recipe runs. This
                         will be transitioning to mod audit runs list
                         eventually. A deprecation notice will be added here
                         when we suggest adopting the alternative.
  study                Produces studies from OpenRewrite recipe data tables
                         locally.
  generate-completion  Generate bash/zsh completion script for mod.

MOD SUCCEEDED in 1s

Step 3: Configure the CLI to use your license

In order to run recipes, you'll need to ensure the CLI is configured with a license. You should have received a license from us. With that license, please run the following command:

mod config license edit <license-you-were-provided>

For more information on the Moderne CLI license, please see our license documentation.

(Optional) Step 4: Configure the CLI to point to your internal artifact repository

If your Maven settings file is not in the default location, please run the following command to point the CLI to your Maven settings file. If it is in the default location, skip to step 5.

mod config build maven settings edit /path/to/maven/settings/file

In order for the CLI to download dependencies/lookup versions as needed, it will need to be provided with the path to your Maven settings file. This likely exists on developer machines for the sake of redirecting requests from Maven Central to an internal artifact instance.

(Optional) Step 5: Configure the CLI to point to an internal Artifactory to download recipes

If you have an internal Artifactory instance where you plan on storing recipes, please direct the CLI to it by running the following command:

mod config recipes artifacts artifactory edit

Step 6: Install recipe JARs

The next thing you need to do is ensure your internal artifact repository has all of the recipe JARS (assuming that your artifact repository is not a pure remote proxy to Maven Central already or that there isn't some automatic procurement step at dependency resolution time).

With that done, you'll need to run the mod config recipes jar install command and provide it with the JARs you wish to install.

The latest version of every JAR and the CLI command to install those latest versions can be found at the bottom of the latest versions of every OpenRewrite module doc. This is automatically updated whenever we do a new release.

Step 7: Create a list of your repositories

In order for the CLI to run recipes against your code, you will need to provide it with a repos.csv file. The first row in the CSV file should be a header row that lists out the columns you intend to provide. After that, each row will represent a repository. At a minimum, you should include a URL to clone said repository – but you can also provide other columns as needed.

Here is an example of a simple CSV file for cloning some OpenRewrite repositories:

cloneUrl
https://github.com/openrewrite/rewrite-spring
https://github.com/openrewrite/rewrite-recipe-markdown-generator
https://github.com/openrewrite/rewrite-docs
https://github.com/openrewrite/rewrite

Columns you can provide in your repos.csv file:

Column name	Required	Description	Examples
cloneUrl	true	The URL of the repository that should be ingested.	git@github.com:google/guava.git or https://github.com/openrewrite/rewrite
branch	false	The branch of the above repository that should be ingested.	main
changeset	false	If provided, this will check out the repository at this specific commit SHA.	aa5f25ac0031
java	false	Configures the JDK to use.	17 or 17-tem or 17.0.6-tem
jvmopts	false	JVM options added to tools building LSTs. Must be configured before you can run the build command if non-standard VM options are required.	-Xmx4G
mavenargs	false	Build arguments are added to the end of the Maven command line when building LSTs.	-P fast
gradleargs	false	Build arguments that are added to the end of the Gradle command line when building LSTs.	-Dmyprop=myvalue
org*	false	If you want to configure an organizational hierarchy, you can provide one or more organization columns. Each column will specify an organization the repository should be part of. The column name should be org plus a number such as: org1,org2,org3. There is no limit for how many orgs you can define.	openrewrite

To assist with creating a repos.csv file, we've written some bash scripts that will generate a simple CSV file for you:

GitHub

Step 1: Install and configure the GitHub CLI if you haven't already done so.

Step 2: Create a github.sh file like:

github.sh

#!/bin/bash

if [ -z "$1" ]; then
  echo "Usage: $0 <org>"
  echo "Example: $0 openrewrite"
  exit 1
fi

organization=$1

gh repo list "$organization" \
    --json url,defaultBranchRef \
    --jq  '["cloneUrl","branch"], (.[] | [.url, .defaultBranchRef.name]) | @csv'

Step 3: Grant the script access to be run:

chmod +x github.sh

Step 4: Run the script and pipe it to a repos.csv file:

./github.sh YOUR_ORG_NAME > repos.csv

If everything was done correctly, you should have a repos.csv file that looks similar to the following:

"cloneUrl","branch"
"https://github.com/openrewrite/rewrite","main"
"https://github.com/openrewrite/rewrite-python","main"
"https://github.com/openrewrite/rewrite-openapi","main"
"https://github.com/openrewrite/rewrite-static-analysis","main"
"https://github.com/openrewrite/rewrite-java-dependencies","main"
"https://github.com/openrewrite/rewrite-docs","master"

GitLab

Step 1: Create a GitLab personal access token if you don't already have one.

Step 2: Create a gitlab.sh file like:

gitlab.sh

#!/bin/bash

while getopts ":g:h:" opt; do
    case ${opt} in
        g )
            GROUP=$OPTARG
            ;;
        h )
            GITLAB_DOMAIN=$OPTARG
            ;;
        \? )
            echo "Usage: gitlab.sh [-g <group>] [-h <gitlab_domain>]"
            exit 1
            ;;
    esac
done


if [[ -z $AUTH_TOKEN ]]; then
    echo "Please set the AUTH_TOKEN environment variable."
    exit 1
fi

# Default GITLAB_DOMAIN to gitlab.com
GITLAB_DOMAIN=${GITLAB_DOMAIN:-https://gitlab.com}

if [[ -z $GROUP ]]; then
    base_request_url="$GITLAB_DOMAIN/api/v4/projects?membership=true&simple=true&archived=false"
else
    base_request_url="$GITLAB_DOMAIN/api/v4/groups/$GROUP/projects?include_subgroups=true&simple=true&archived=false"
fi

page=1
per_page=100

echo '"cloneUrl","branch"'
while :; do
    # Construct the request URL with pagination parameters
    request_url="${base_request_url}&page=${page}&per_page=${per_page}"

    # Fetch the data
    response=$(curl --silent --header "Authorization: Bearer $AUTH_TOKEN" "$request_url")

    # Check if the response is empty, if so, break the loop
    if [[ $(echo "$response" | jq '. | length') -eq 0 ]]; then
        break
    fi

    # Process and output data
    echo "$response" | jq -r '(.[] | [.http_url_to_repo, .default_branch]) | @csv'

    # Increment page counter
    ((page++))
done

Step 3: Grant the script access to be run:

chmod +x gitlab.sh

Step 4: Run the script and pipe it to a repos.csv file:

AUTH_TOKEN=YOUR_AUTH_TOKEN ./gitlab.sh [-g <group>] [-h <gitlab_domain>] > repos.csv

info

The -g option specifies a group to fetch repositories from. The -h option specifies the GitLab domain (defaults to https://gitlab.com if not provided).

If everything was done correctly, you should have a repos.csv file that looks similar to:

"cloneUrl","branch"
"https://gitlab.com/moderneinc/moderne-docker-build.git","main"
"https://gitlab.com/moderneinc/spring-petclinic.git","main"
"https://gitlab.com/moderneinc/git-test.git","main"
"https://gitlab.com/moderneinc/moderne-gitlab-ingest.git","main"
...

Bitbucket Data Center

Step 1: Create a Bitbucket HTTP access token to provide to the command.

Step 2: Create a bitbucket-data-center.sh file like:

bitbucket-data-center.sh

#!/bin/bash

if [ -z "$1" ]; then
    echo "Usage: $0 <bitbucket_url>"
    echo "Example: $0 https://my-bitbucket.com/stash"
    exit 1
fi

bitbucket_url=$1
auth_header=""

if [ -n "$AUTH_TOKEN" ]; then
    auth_header="Authorization: Bearer $AUTH_TOKEN"
fi

ALL_REPOS=$(curl -s -X GET -H "Content-Type: application/json" -H "$auth_header" "$bitbucket_url/rest/api/1.0/repos"| jq -r '.values[] | [.slug, .project.key, (.links.clone[] | select(.name == "http").href)] | @csv')
if [ $? -ne 0 ]; then
    echo "Error occurred while retrieving repository list."
    exit 1
fi

echo "cloneUrl,branch"
for REPO in $ALL_REPOS; do
    IFS=',' read -r repo project cloneUrl <<< "$REPO"
    repo="${repo//\"/}"
    project="${project//\"/}"
    cloneUrl="${cloneUrl//\"/}"
    branch=$(curl -s -X GET -H "Content-Type: application/json" -H "$auth_header" "$bitbucket_url/rest/api/latest/projects/$project/repos/$repo/default-branch" | jq -r '.displayId')

    echo "$cloneUrl,$branch"
done

Step 3: Grant the script access to be run:

chmod +x bitbucket-data-center.sh

Step 4: Run the script and pipe it to a repos.csv file:

AUTH_TOKEN=YOUR_AUTH_TOKEN ./bitbucket-data-center.sh YOUR_BITBUCKET_URL > repos.csv

If everything was done correctly, you should have a repos.csv file that looks similar to:

cloneUrl,branch
https://bitbucket.your.place/stash/scm/greg/demo-multimodule.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo-multimodule.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo-multimodule-rename.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo_private.git,main

Bitbucket Cloud

Step 1: Create a Bitbucket Cloud username and password that the script can use to grab repositories.

Step 2: Create a bitbucket-cloud.sh file like:

bitbucket-cloud.sh

#!/bin/bash

usage() {
  echo "Usage: $0 -u <username> -p <password> <workspace>"
  exit 1
}

# Parse command-line arguments
while getopts ":u:p:" opt; do
  case ${opt} in
    u) username=$OPTARG;;
    p) app_password=$OPTARG;;
    *) usage;;
  esac
done
shift $((OPTIND -1))

# Set workspace from positional argument
workspace=$1

# Check if username and app_password are provided via command line or environment variables
if [ -z "$username" ]; then
    username=$BITBUCKET_USERNAME
fi

if [ -z "$app_password" ]; then
    app_password=$BITBUCKET_APP_PASSWORD
fi

if [ -z "$username" -o -z "$app_password" -o -z "$workspace" ]; then
    echo "Error: Please provide username, password, and workspace." >&2
    usage
fi

echo "cloneUrl,branch,org"

next_page="https://api.bitbucket.org/2.0/repositories/$workspace"

while [ "$next_page" ]; do
  response=$(curl -s -u "$username:$app_password" "$next_page")

  # Extract repository data and append to CSV file
  echo "$response" | jq -r '
    .values[] |
    (.links.clone[] | select(.name=="https") | .href) as $cloneUrl |
    .mainbranch.name as $branchName |
    .workspace.name as $organization |
    "\($cloneUrl),\($branchName),\($organization)"' |
  while IFS=, read -r cloneUrl branchName organization; do
    cleanUrl=$(echo "$cloneUrl" | sed -E 's|https://[^@]+@|https://|')
    echo "$cleanUrl,$branchName,$organization"
  done


  next_page=$(echo "$response" | sed -e "s:${username}@::g" | jq -r '.next // empty')
done

Step 3: Grant the script access to be run:

chmod +x bitbucket-cloud.sh

Step 4: Run the script and pipe it to a repos.csv file:

./bitbucket-cloud.sh -u username -p password <workspace> > repos.csv

If everything was done correctly, you should have a repos.csv file that looks similar to:

cloneUrl,branch
https://bitbucket.your.place/stash/scm/greg/demo-multimodule.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo-multimodule.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo-multimodule-rename.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo_private.git,main

Step 8: Clone your repositories

Create a directory somewhere on your machine where you'd like the CLI to clone the repositories to. Then navigate to that directory, copy the repos.csv file to it, and run the following command:

mod git sync csv . repos.csv

Step 9: Build your repositories

With all of the repositories cloned to your machine, you can then build LSTs for them by running the following command:

mod build .

With the LSTs built, you're ready to run recipes against them! Consider checking out the using the CLI section in the getting started guide to see some ways you can use the CLI.