How to install and configure the Moderne CLI in an air-gapped environment
If your company has restrictions on what you can access via the internet, you'll need to download and configure the Moderne CLI in a special way. This doc will walk you through everything you need to know for this. By the end, you will have the CLI downloaded and configured in your air-gapped environment.
Assumptions
- You can't access app.moderne.io from your environment
- You have an internal mirror of Maven Central (or some other internal artifact repository)
- You have the ability to download and add JARs from Maven Central to your internal artifact repository
Installation and configuration
Step 1: Download the Moderne CLI JAR
Download the latest version of the Moderne CLI from Maven Central. Once downloaded, please add it to your internal mirror so that it's accessible for all users in your environment.
Step 2: (Optional - but recommended) Create an alias for the Moderne CLI JAR
While not required, you are strongly encouraged to set up an alias for running the CLI JAR. Below are some ways you might configure this depending on your OS and terminal:
- Git Bash (Windows)
- Bash or Zsh (Mac or Linux)
- Powershell
Add the following function to your .bashrc
file:
mod() {
"java -jar /path/to/mod.jar" "$@"
}
Add the following to your .bashrc
or .zshrc
file:
alias mod="java -jar /path/to/mod.jar"
Use the Set-Alias command within a profile script.
If everything was configured correctly, you should be able to type mod
into your terminal and see a list of commands:
mod command results
➜ mod
Moderne CLI 3.27.7
Usage:
mod [-h] [--version] [COMMAND]
Description:
Automated code remediation.
Options:
-h, --help Display this help message.
--version Display version info.
Commands:
build Generates LST artifacts for one or more repositories.
clean Clean build and run artifacts produced by the CLI.
config Global configuration options that are required by some
CLI commands.
exec Execute an arbitrary shell command recursively on
selected repository roots.
git Multi-repository git operations.
log Manages a log aggregate.
list Lists the repositories that can be built and published.
monitor (INCUBATING) Launches an HTTP server used to monitor the
CLI.
publish Publishes the LST artifacts for one or more projects.
run Runs an OpenRewrite recipe locally on pre-built LSTS.
run-history Get information about the most recent recipe runs.
study Produces studies from OpenRewrite recipe data tables
locally.
generate-completion Generate bash/zsh completion script for mod.
MOD SUCCEEDED in (0.01s)
Step 3: Configure the CLI to use your license
In order to run recipes, you'll need to ensure the CLI is configured with a license. You should have received a license from us. With that license, please run the following command:
mod config license edit <license-you-were-provided>
For more information on the Moderne CLI license, please see our license documentation
Step 4: Configure the CLI to point to your internal artifact repository
In order for the CLI to have access to read and publish LSTs, it will need to be provided with the path to your Maven settings file. This likely exists on developer machines for the sake of redirecting requests from Maven Central to an internal artifact instance.
Run the following command to point the CLI to your Maven settings file:
mod config build maven settings edit /path/to/maven/settings/file
After that, configure the connection to your artifact repository by running one of the following commands (depending on the type of artifact repository you're using):
- Artifactory configuration
- Maven configuration
mod config lsts artifacts artifactory edit <artifact-repository-url> --user <user> --password <password>
mod config lsts artifacts maven edit <artifact-repository-url> --user <user> --password <password>
Step 5: Install recipe JARs
For each of the GAV coordinates below, please ensure that the artifact has been added to your internal artifact repository (assuming that your artifact repository is not a pure remote proxy to Maven Central already or that there isn't some automatic procurement step at dependency resolution time):
mod config recipes jar install org.openrewrite:rewrite-core:LATEST
mod config recipes jar install org.openrewrite:rewrite-gradle:LATEST
mod config recipes jar install org.openrewrite:rewrite-hcl:LATEST
mod config recipes jar install org.openrewrite:rewrite-java:LATEST
mod config recipes jar install org.openrewrite:rewrite-json:LATEST
mod config recipes jar install org.openrewrite:rewrite-maven:LATEST
mod config recipes jar install org.openrewrite:rewrite-properties:LATEST
mod config recipes jar install org.openrewrite:rewrite-protobuf:LATEST
mod config recipes jar install org.openrewrite:rewrite-xml:LATEST
mod config recipes jar install org.openrewrite:rewrite-yaml:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-hibernate:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-java-dependencies:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-java-security:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-logging-frameworks:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-migrate-java:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-spring:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-sql:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-static-analysis:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-terraform:LATEST
mod config recipes jar install org.openrewrite.recipe:rewrite-testing-frameworks:LATEST
Step 6: Create a list of your repositories
In order for the CLI to run recipes against your code, you will need to provide it with a repos.csv
file. The first row in the CSV file should be a header row that lists out the columns you intend to provide. After that, each row will represent a repository. At a minimum, you should include a URL to clone said repository – but you can also provide other columns as needed.
Here is an example of a simple CSV file for cloning some OpenRewrite repositories:
cloneUrl
https://github.com/openrewrite/rewrite-spring
https://github.com/openrewrite/rewrite-recipe-markdown-generator
https://github.com/openrewrite/rewrite-docs
https://github.com/openrewrite/rewrite
Columns you can provide in your repos.csv
file:
Column name | Required | Description | Examples |
---|---|---|---|
cloneUrl | true | The URL of the repository that should be ingested. | git@github.com:google/guava.git or https://github.com/openrewrite/rewrite |
branch | false | The branch of the above repository that should be ingested. | main |
changeset | false | If provided, this will check out the repository at this specific commit SHA. | aa5f25ac0031 |
java | false | Configures the JDK to use. | 17 or 17-tem or 17.0.6-tem |
jvmopts | false | JVM options added to tools building LSTs. Must be configured before you can run the build command if non-standard VM options are required. | -Xmx4G |
mavenargs | false | Build arguments are added to the end of the Maven command line when building LSTs. | -P fast |
gradleargs | false | Build arguments that are added to the end of the Gradle command line when building LSTs. | -Dmyprop=myvalue |
org* | false | If you have configured an organizations service, you can provide one or more organization columns. Each column will specify an organization the repository should be part of. The column name should be org plus a number such as: org1,org2,org3 . | openrewrite |
To assist with creating a repos.csv
file, we've written some bash scripts that will generate a simple CSV file for you:
- GitHub
- GitLab
- Bitbucket Data Center
- Bitbucket Cloud
Step 1: Install and configure the GitHub CLI if you haven't already done so.
Step 2: Create a github.sh
file like:
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 <org>"
echo "Example: $0 openrewrite"
exit 1
fi
organization=$1
gh repo list "$organization" \
--json url,defaultBranchRef \
--jq '["cloneUrl","branch"], (.[] | [.url, .defaultBranchRef.name]) | @csv'
Step 3: Grant the script access to be run:
chmod +x github.sh
Step 4: Run the script and pipe it to a repos.csv
file:
./github.sh YOUR_ORG_NAME > repos.csv
If everything was done correctly, you should have a repos.csv
file that looks similar to:
"cloneUrl","branch"
"https://github.com/openrewrite/rewrite","main"
"https://github.com/openrewrite/rewrite-codemods-ng","main"
"https://github.com/openrewrite/rewrite-python","main"
"https://github.com/openrewrite/rewrite-openapi","main"
"https://github.com/openrewrite/rewrite-static-analysis","main"
"https://github.com/openrewrite/rewrite-java-dependencies","main"
"https://github.com/openrewrite/rewrite-docs","master"
Step 1: Create a GitLab personal access token if you don't already have one.
Step 2: Create a gitlab.sh
file like:
#!/bin/bash
while getopts ":g:h:" opt; do
case ${opt} in
g )
GROUP=$OPTARG
;;
h )
GITLAB_DOMAIN=$OPTARG
;;
\? )
echo "Usage: gitlab.sh [-g <group>] [-h <gitlab_domain>]"
exit 1
;;
esac
done
if [[ -z $AUTH_TOKEN ]]; then
echo "Please set the AUTH_TOKEN environment variable."
exit 1
fi
# Default GITLAB_DOMAIN to gitlab.com
GITLAB_DOMAIN=${GITLAB_DOMAIN:-https://gitlab.com}
if [[ -z $GROUP ]]; then
base_request_url="$GITLAB_DOMAIN/api/v4/projects?membership=true&simple=true"
else
base_request_url="$GITLAB_DOMAIN/api/v4/groups/$GROUP/projects?include_subgroups=true&simple=true"
fi
page=1
per_page=100
echo '"cloneUrl","branch"'
while :; do
# Construct the request URL with pagination parameters
request_url="${base_request_url}&page=${page}&per_page=${per_page}"
# Fetch the data
response=$(curl --silent --header "Authorization: Bearer $AUTH_TOKEN" "$request_url")
# Check if the response is empty, if so, break the loop
if [[ $(echo "$response" | jq '. | length') -eq 0 ]]; then
break
fi
# Process and output data
echo "$response" | jq -r '(.[] | [.http_url_to_repo, .default_branch]) | @csv'
# Increment page counter
((page++))
done
Step 3: Grant the script access to be run:
chmod +x gitlab.sh
Step 4: Run the script and pipe it to a repos.csv
file:
AUTH_TOKEN=YOUR_AUTH_TOKEN ./gitlab.sh [-g <group>] [-h <gitlab_domain>] > repos.csv
The -g
option specifies a group to fetch repositories from. The -h
option specifies the GitLab domain (defaults to https://gitlab.com if not provided).
If everything was done correctly, you should have a repos.csv
file that looks similar to:
"cloneUrl","branch"
"https://gitlab.com/moderneinc/moderne-docker-build.git","main"
"https://gitlab.com/moderneinc/spring-petclinic.git","main"
"https://gitlab.com/moderneinc/git-test.git","main"
"https://gitlab.com/moderneinc/moderne-gitlab-ingest.git","main"
...
Step 1: Create a Bitbucket HTTP access token to provide to the command.
Step 2: Create a bitbucket-data-center.sh
file like:
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 <bitbucket_url>"
echo "Example: $0 https://my-bitbucket.com/stash"
exit 1
fi
bitbucket_url=$1
auth_header=""
if [ -n "$AUTH_TOKEN" ]; then
auth_header="Authorization: Bearer $AUTH_TOKEN"
fi
ALL_REPOS=$(curl -s -X GET -H "Content-Type: application/json" -H "$auth_header" "$bitbucket_url/rest/api/1.0/repos"| jq -r '.values[] | [.slug, .project.key, (.links.clone[] | select(.name == "http").href)] | @csv')
if [ $? -ne 0 ]; then
echo "Error occurred while retrieving repository list."
exit 1
fi
echo "cloneUrl,branch"
for REPO in $ALL_REPOS; do
IFS=',' read -r repo project cloneUrl <<< "$REPO"
repo="${repo//\"/}"
project="${project//\"/}"
cloneUrl="${cloneUrl//\"/}"
branch=$(curl -s -X GET -H "Content-Type: application/json" -H "$auth_header" "$bitbucket_url/rest/api/latest/projects/$project/repos/$repo/default-branch" | jq -r '.displayId')
echo "$cloneUrl,$branch"
done
Step 3: Grant the script access to be run:
chmod +x bitbucket-data-center.sh
Step 4: Run the script and pipe it to a repos.csv
file:
AUTH_TOKEN=YOUR_AUTH_TOKEN ./bitbucket-data-center.sh YOUR_BITBUCKET_URL > repos.csv
If everything was done correctly, you should have a repos.csv
file that looks similar to:
cloneUrl,branch
https://bitbucket.your.place/stash/scm/greg/demo-multimodule.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo-multimodule.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo-multimodule-rename.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo_private.git,main
Step 1: Create a Bitbucket Cloud username and password that the script can use to grab repositories.
Step 2: Create a bitbucket-cloud.sh
file like:
#!/bin/bash
usage() {
echo "Usage: $0 -u <username> -p <password> <workspace>"
exit 1
}
# Parse command-line arguments
while getopts ":u:p:" opt; do
case ${opt} in
u) username=$OPTARG;;
p) app_password=$OPTARG;;
*) usage;;
esac
done
shift $((OPTIND -1))
# Set workspace from positional argument
workspace=$1
# Check if username and app_password are provided via command line or environment variables
if [ -z "$username" ]; then
username=$BITBUCKET_USERNAME
fi
if [ -z "$app_password" ]; then
app_password=$BITBUCKET_APP_PASSWORD
fi
if [ -z "$username" -o -z "$app_password" -o -z "$workspace" ]; then
echo "Error: Please provide username, password, and workspace." >&2
usage
fi
echo "cloneUrl,branch,org"
next_page="https://api.bitbucket.org/2.0/repositories/$workspace"
while [ "$next_page" ]; do
response=$(curl -s -u "$username:$app_password" "$next_page")
# Extract repository data and append to CSV file
echo "$response" | jq -r '
.values[] |
(.links.clone[] | select(.name=="https") | .href) as $cloneUrl |
.mainbranch.name as $branchName |
.workspace.name as $organization |
"\($cloneUrl),\($branchName),\($organization)"' |
while IFS=, read -r cloneUrl branchName organization; do
cleanUrl=$(echo "$cloneUrl" | sed -E 's|https://[^@]+@|https://|')
echo "$cleanUrl,$branchName,$organization"
done
next_page=$(echo "$response" | sed -e "s:${username}@::g" | jq -r '.next // empty')
done
Step 3: Grant the script access to be run:
chmod +x bitbucket-cloud.sh
Step 4: Run the script and pipe it to a repos.csv
file:
./bitbucket-cloud.sh -u username -p password <workspace> > repos.csv
If everything was done correctly, you should have a repos.csv
file that looks similar to:
cloneUrl,branch
https://bitbucket.your.place/stash/scm/greg/demo-multimodule.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo-multimodule.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo-multimodule-rename.git,main
https://bitbucket.your.place/stash/scm/~sjungling/demo_private.git,main
Step 7: Clone your repositories
Create a directory somewhere on your machine where you'd like the CLI to clone the repositories to. Then navigate to that directory, copy the repos.csv
file to it, and run the following command:
mod git clone csv . repos.csv
Step 8: Build your repositories
With all of the repositories cloned to your machine, you can then build LSTs for them by running the following command:
mod build .
With the LSTs built, you're ready to run recipes against them! Consider checking out the using the CLI section in the getting started guide to see some ways you can use the CLI.