How to install and configure the Moderne CLI in an air-gapped environment

If your company has restrictions on what you can access via the internet, you'll need to download and configure the Moderne CLI in a special way. This doc will walk you through everything you need to know for this. By the end, you will have the CLI downloaded and configured in your air-gapped environment.

Assumptions

You can't access app.moderne.io from your environment
You have an internal mirror of Maven Central (or some other internal artifact repository)
You have the ability to download and add JARs from Maven Central to your internal artifact repository

Installation and configuration

Step 1: Download the Moderne CLI JAR

Download the latest version of the Moderne CLI from Maven Central. Once downloaded, please add it to your internal mirror so that it's accessible for all users in your environment.

Step 2: (Optional - but recommended) Create an alias for the Moderne CLI JAR

While not required, you are strongly encouraged to set up an alias for running the CLI JAR. Below are some ways you might configure this depending on your OS and terminal:

Git Bash (Windows)
Bash or Zsh (Mac or Linux)
Powershell

Add the following function to your .bashrc file:

mod() {
    "java -jar /path/to/mod.jar" "$@"
}

Add the following to your .bashrc or .zshrc file:

alias mod="java -jar /path/to/mod.jar"

If everything was configured correctly, you should be able to type mod into your terminal and see a list of commands:

mod command results

➜ mod

Moderne CLI 3.26.5

Usage:

mod [-h] [--version] [COMMAND]

Description:

Automated code remediation.

Options:

  -h, --help      Display this help message.
      --version   Display version info.
Commands:

  build                Generates LST artifacts for one or more repositories.
  clean                Clean build and run artifacts produced by the CLI.
  config               Global configuration options that are required by some
                         CLI commands.
  exec                 Execute an arbitrary shell command recursively on
                         selected repository roots.
  git                  Multi-repository git operations.
  log                  Manages a log aggregate.
  list                 Lists the repositories that can be built and published.
  monitor              (INCUBATING) Launches an HTTP server used to monitor the
                         CLI.
  publish              Publishes the LST artifacts for one or more projects.
  run                  Runs an OpenRewrite recipe locally on pre-built LSTS.
  run-history          Get information about the most recent recipe runs.
  study                Produces studies from OpenRewrite recipe data tables
                         locally.
  generate-completion  Generate bash/zsh completion script for mod.

MOD SUCCEEDED in (0.01s)

Step 3: Configure the CLI to use your license

In order to run recipes, you'll need to ensure the CLI is configured with a license. You should have received a license from us. With that license, please run the following command:

mod config license edit <license-you-were-provided>

For more information on the Moderne CLI license, please see our license documentation

Step 4: Configure the CLI to point to your internal artifact repository

In order for the CLI to have access to read and publish LSTs, it will need to be provided with the path to your Maven settings file. This likely exists on developer machines for the sake of redirecting requests from Maven Central to an internal artifact instance.

Run the following command to point the CLI to your Maven settings file:

mod config build maven settings edit /path/to/maven/settings/file

After that, configure the connection to your artifact repository by running one of the following commands (depending on the type of artifact repository you're using):

Artifactory configuration
Maven configuration

mod config lsts artifacts artifactory edit <artifact-repository-url> --user <user> --password <password>

mod config lsts artifacts maven edit <artifact-repository-url> --user <user> --password <password>

Step 5: Install recipe JARs

For each of the GAV coordinates below, please ensure that the artifact has been added to your internal artifact repository (assuming that your artifact repository is not a pure remote proxy to Maven Central already or that there isn't some automatic procurement step at dependency resolution time):

mod config recipes jar install org.openrewrite:rewrite-core:LATEST
mod config recipes jar install org.openrewrite:rewrite-gradle:LATEST
mod config recipes jar install org.openrewrite:rewrite-hcl:LATEST
mod config recipes jar install org.openrewrite:rewrite-java:LATEST
mod config recipes jar install org.openrewrite:rewrite-json:LATEST
mod config recipes jar install org.openrewrite:rewrite-maven:LATEST
mod config recipes jar install org.openrewrite:rewrite-properties:LATEST
mod config recipes jar install org.openrewrite:rewrite-protobuf:LATEST
mod config recipes jar install org.openrewrite:rewrite-xml:LATEST
mod config recipes jar install org.openrewrite:rewrite-yaml:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-hibernate:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-java-dependencies:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-java-security:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-logging-frameworks:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-migrate-java:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-spring:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-sql:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-static-analysis:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-terraform:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-testing-frameworks:LATEST

Step 6: Create a list of your repositories

In order for the CLI to run recipes against your code, you will need to provide it with a repos.csv file. The first row in the CSV file should be a header row that lists out the columns you intend to provide. After that, each row will represent a repository. At a minimum, you should include a URL to clone said repository – but you can also provide other columns as needed.

Here is an example of a simple CSV file for cloning some OpenRewrite repositories:

cloneUrl
https://github.com/openrewrite/rewrite-spring
https://github.com/openrewrite/rewrite-recipe-markdown-generator
https://github.com/openrewrite/rewrite-docs
https://github.com/openrewrite/rewrite

Columns you can provide in your repos.csv file:

Column name	Required	Description	Examples
cloneUrl	`true`	The URL of the repository that should be ingested.	`git@github.com:google/guava.git` or `https://github.com/openrewrite/rewrite`
branch	`false`	The branch of the above repository that should be ingested.	`main`
changeset	`false`	If provided, this will check out the repository at this specific commit SHA.	`aa5f25ac0031`
java	`false`	Configures the JDK to use.	`17` or `17-tem` or `17.0.6-tem`
jvmopts	`false`	JVM options added to tools building LSTs. Must be configured before you can run the build command if non-standard VM options are required.	`-Xmx4G`
mavenargs	`false`	Build arguments are added to the end of the Maven command line when building LSTs.	`-P fast`
gradleargs	`false`	Build arguments that are added to the end of the Gradle command line when building LSTs.	`-Dmyprop=myvalue`
org*	`false`	If you have configured an organizations service, you can provide one or more organization columns. Each column will specify an organization the repository should be part of. The column name should be `org` plus a number such as: `org1,org2,org3`.	`openrewrite`

To assist with creating a repos.csv file, we've written some bash script that will generate a simple CSV file for you:

GitHub
GitLab
Bitbucket

Step 1: Install and configure the GitHub CLI if you haven't already done so.

Step 2: Create a github.sh file like:

github.sh
#!/bin/bash

if [ -z "$1" ]; then
  echo "Usage: $0 <org>"
  echo "Example: $0 openrewrite"
  exit 1
fi

organization=$1

gh repo list "$organization" \
    --no-archived --limit 1000 \
    --json url,defaultBranchRef \
    --template '{{"cloneUrl,branch\n"}}{{range .}}{{.url}}{{","}}{{.defaultBranchRef.name}}{{"\n"}}{{end}}'

Step 3: Grant the script access to be run:

chmod +x github.sh

Step 4: Run the script and pipe it to a repos.csv file:

./github.sh YOUR_ORG_NAME > repos.csv

If everything was done correctly, you should have a repos.csv file that looks similar to:

cloneUrl,branch
https://github.com/openrewrite/rewrite-spring,main
https://github.com/openrewrite/rewrite-recipe-markdown-generator,main
https://github.com/openrewrite/rewrite-docs,master
https://github.com/openrewrite/rewrite,main
https://github.com/openrewrite/rewrite-python,main
https://github.com/openrewrite/rewrite-migrate-java,main
https://github.com/openrewrite/rewrite-recommendations,main
https://github.com/openrewrite/rewrite-testing-frameworks,main
https://github.com/openrewrite/rewrite-gradle-tooling-model,main
https://github.com/openrewrite/rewrite-recipe-bom,main
...

Step 1: Create a GitLab personal access token if you don't already have one.

Step 2: Create a gitlab.sh file like:

#!/bin/bash

while getopts ":g:h:" opt; do
    case ${opt} in
        g )
            GROUP=$OPTARG
            ;;
        h )
            GITLAB_DOMAIN=$OPTARG
            ;;
        \? )
            echo "Usage: gitlab.sh [-g <group>] [-h <gitlab_domain>]"
            exit 1
            ;;
    esac
done


if [[ -z $AUTH_TOKEN ]]; then
    echo "Please set the AUTH_TOKEN environment variable."
    exit 1
fi

# Default GITLAB_DOMAIN to gitlab.com
GITLAB_DOMAIN=${GITLAB_DOMAIN:-https://gitlab.com}

if [[ -z $GROUP ]]; then
    base_request_url="$GITLAB_DOMAIN/api/v4/projects?membership=true&simple=true"
else
    base_request_url="$GITLAB_DOMAIN/api/v4/groups/$GROUP/projects?include_subgroups=true&simple=true"
fi

page=1
per_page=100

echo '"cloneUrl","branch"'
while :; do
    # Construct the request URL with pagination parameters
    request_url="${base_request_url}&page=${page}&per_page=${per_page}"

    # Fetch the data
    response=$(curl --silent --header "Authorization: Bearer $AUTH_TOKEN" "$request_url")

    # Check if the response is empty, if so, break the loop
    if [[ $(echo "$response" | jq '. | length') -eq 0 ]]; then
        break
    fi

    # Process and output data
    echo "$response" | jq -r '(.[] | [.http_url_to_repo, .default_branch]) | @csv'

    # Increment page counter
    ((page++))
done

Step 3: Grant the script access to be run:

chmod +x gitlab.sh

Step 4: Run the script and pipe it to a repos.csv file:

AUTH_TOKEN=YOUR_AUTH_TOKEN ./gitlab.sh -g YOUR_GROUP_NAME > repos.csv

If everything was done correctly, you should have a repos.csv file that looks similar to:

"cloneUrl","branch"
"https://gitlab.com/moderneinc/moderne-docker-build.git","main"
"https://gitlab.com/moderneinc/spring-petclinic.git","main"
"https://gitlab.com/moderneinc/git-test.git","main"
"https://gitlab.com/moderneinc/moderne-gitlab-ingest.git","main"
...

Step 1: Create a Bitbucket HTTP access token to provide to the command.

Step 2: Create a bitbucket-data-center.sh file like:

bitbucket-data-center.sh
#!/bin/bash

if [ -z "$1" ]; then
    echo "Usage: $0 <bitbucket_url>"
    echo "Example: $0 https://my-bitbucket.com/stash"
    exit 1
fi

bitbucket_url=$1
auth_header=""

if [ -n "$AUTH_TOKEN" ]; then
    auth_header="Authorization: Bearer $AUTH_TOKEN"
fi

ALL_REPOS=$(curl -s -X GET -H "Content-Type: application/json" -H "$auth_header" "$bitbucket_url/rest/api/1.0/repos"| jq -r '.values[] | [.slug, .project.key, (.links.clone[] | select(.name == "http").href)] | @csv')
if [ $? -ne 0 ]; then
    echo "Error occurred while retrieving repository list."
    exit 1
fi

echo "cloneUrl,branch"
for REPO in $ALL_REPOS; do
    IFS=',' read -r repo project cloneUrl <<< "$REPO"
    repo="${repo//\"/}"
    project="${project//\"/}"
    cloneUrl="${cloneUrl//\"/}"
    branch=$(curl -s -X GET -H "Content-Type: application/json" -H "$auth_header" "$bitbucket_url/rest/api/latest/projects/$project/repos/$repo/default-branch" | jq -r '.displayId')

    echo "$cloneUrl,$branch"
done

Step 3: Grant the script access to be run:

chmod +x bitbucket-data-center.sh

Step 4: Run the script and pipe it to a repos.csv file:

AUTH_TOKEN=YOUR_AUTH_TOKEN ./bitbucket-data-center.sh YOUR_BITBUCKET_URL > repos.csv

If everything was done correctly, you should have a repos.csv file that looks similar to:

cloneUrl,branch
https://bitbucket.your.place/stash/scm/greg/demo-multimodule.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo-multimodule.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo-multimodule-rename.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo_private.git,main

Step 7: Clone your repositories

Create a directory somewhere on your machine where you'd like the CLI to clone the repositories to. Then navigate to that directory, copy the repos.csv file to it, and run the following command:

mod git clone csv . repos.csv

Step 8: Build your repositories

With all of the repositories cloned to your machine, you can then build LSTs for them by running the following command:

mod build .

With the LSTs built, you're ready to run recipes against them! Consider checking out the using the CLI section in the getting started guide to see some ways you can use the CLI.

Assumptions​

Installation and configuration​

Step 1: Download the Moderne CLI JAR​

Step 2: (Optional - but recommended) Create an alias for the Moderne CLI JAR​

Step 3: Configure the CLI to use your license​

Step 4: Configure the CLI to point to your internal artifact repository​

Step 5: Install recipe JARs​

Step 6: Create a list of your repositories​

Step 7: Clone your repositories​

Step 8: Build your repositories​

Assumptions

Installation and configuration

Step 1: Download the Moderne CLI JAR

Step 2: (Optional - but recommended) Create an alias for the Moderne CLI JAR

Step 3: Configure the CLI to use your license

Step 4: Configure the CLI to point to your internal artifact repository

Step 5: Install recipe JARs

Step 6: Create a list of your repositories

Step 7: Clone your repositories

Step 8: Build your repositories