Skip to main content

Creating a repos.csv file

The repos.csv file is a central configuration file that defines your repositories and their organizational structure. This file is commonly used for:

  1. Mass ingestion - Where you create this file and then use it to build LSTs for all of your repositories and publish them to your artifact repository
  2. Moderne Platform org hierarchy - Where you provide this file to the Moderne Connector that is then used to create organizations in the platform
  3. Local operations - Where you provide the CLI with this file so you can clone repositories or their LSTs to your local machine for things like recipe running or creating a repos-lock.csv file to share with others

This doc will provide you with everything you need to know about creating and using this file.

After creating your repos.csv file and running initial setup commands, the Moderne CLI will generate a repos-lock.csv file that captures the exact state of your repositories. This lock file can be shared with team members to ensure everyone has an identical setup.

Supported columns​

Column nameRequiredDescription
cloneUrltrueThe URL of the repository that should be ingested.

Example: git@github.com:google/guava.git or https://github.com/openrewrite/rewrite
branchfalseThe branch of the repository that should be checked out. If empty, the CLI uses the remote's default branch. Required when changeset is specified.

Example: main
origintrueThe host domain of the repository including any context root used by the server.

Example: github.com, gitlab.com, bitbucket.org, mycompany.com/bitbucket
pathtrueThe organization and repository name portion of the clone URL. This is typically the remainder of the cloneUrl after origin minus any leading symbols and trailing .git. For SCP-style URLs (like git@github.com:moderneinc/moderne-docs), the path is typically the portion after the :.

Example: moderne-inc/moderne-docs, openrewrite/rewrite, businessunit/teamA/subteamB/app
alternateCloneUrlfalseIf the cloneUrl doesn't work, then we will attempt to clone from this URL. This is commonly filled out if you want to provide both an HTTP link and an SSH link for a repository. Multiple columns with this name are supported for organizations that have additional clone URLs.
changesetfalseIf provided, this will check out the repository at this specific commit SHA. Requires a branch value on the same row.

Example: aa5f25ac0031
environmentfalseEnvironment variables the CLI passes to build tools when building LSTs. Write them as NAME=VALUE pairs with a space between each pair. During mod git sync csv, these are written to each repository's .moderne directory. They will replace all the variables that were previously saved for that repository. You can pass credentials by pointing to variables on the build machine.

Example: CI=true NPM_TOKEN=${CI_TOKEN}
gradleArgsfalseBuild arguments that are added to the end of the Gradle command line when building LSTs. During mod git sync csv, these values are written to each repository's .moderne directory and will overwrite any existing values. See configuration precedence for details.

Example: -Dmyprop=myvalue
gradleVersionfalseSelects a specific Gradle version to use when building LSTs. Takes precedence over the repository's Gradle wrapper; leave it empty to use the wrapper. The version must exactly match one of the Gradle installations known to the CLI (detected from $PATH, $GRADLE_HOME, SDKMAN, Homebrew, or manually configured via mod config build gradle installation edit). During mod git sync csv, this value is written to each repository's .moderne directory. Use mod config build gradle installation list to see available versions.

Example: 8.1.1
javafalseConfigures the JDK to use during LST generation.

Example: 17 or 17-tem or 17.0.6-tem
jvmoptsfalseJVM options added to tools building LSTs. Must be configured before you can run the build command if non-standard VM options are required.

Example: -Xmx4G
nodeOptionsfalseNode.js options set via the NODE_OPTIONS environment variable when building LSTs and running recipes. During mod git sync csv, these values are written to each repository's .moderne directory and will overwrite any existing values. See configuration precedence for details.

Example: --max-old-space-size=8192
nodeVersionfalseConfigures the Node.js version to use during LST generation. During mod git sync csv, this value is written to each repository's .moderne directory.

Example: 18 or 20.11.1
mavenArgsfalseBuild arguments that are added to the end of the Maven command line when building LSTs. During mod git sync csv, these values are written to each repository's .moderne directory and will overwrite any existing values. See configuration precedence for details.

Example: -Pfast
mavenVersionfalseSelects a specific Maven version to use when building LSTs. Takes precedence over the repository's Maven wrapper; leave it empty to use the wrapper. The version must exactly match one of the Maven installations known to the CLI (detected from $PATH, $MAVEN_HOME, $MVN_HOME, SDKMAN, Homebrew, or manually configured via mod config build maven installation edit). During mod git sync csv, this value is written to each repository's .moderne directory. Use mod config build maven installation list to see available versions.

Example: 3.9.6
org*falseDefines the organizational hierarchy. Each column specifies an organization the repository should be part of. The column name should be org plus a number such as: org1,org2,org3. There is no limit for how many orgs you can define. These columns are optional for local CLI use, but at least one org column is required when providing this file to the Moderne Connector.

Example: openrewrite
publishUrifalseA URL pointing to a pre-built LST artifact (typically hosted in Artifactory). When this column is present, mod git sync csv will download the LST directly from this URL instead of requiring you to clone sources and run mod build. This is useful for sharing pre-built LSTs with team members.

Example: https://artifactory.example.com/artifactory/moderne-lsts/myorg/myrepo/20251120044534864/myrepo-20251120044534864-ast.jar
alternatePublishUri*falseFallback URLs for LST downloads if the primary publishUri is unavailable. Multiple columns are supported using incrementing numbers (e.g., alternatePublishUri1, alternatePublishUri2). These can point to different artifact repositories or use custom URI schemes.

Example: https://backup-artifactory.example.com/moderne-lsts/myorg/myrepo/ast.jar

Understanding origin and path values​

The Moderne Platform uses a combination of origin, path, and branch to uniquely identify a specific repository. In general, all repositories coming from the same SCM server should have the same origin – whereas the path and branch will differ. The combination of origin, path, and branch must uniquely identify each row in your repos.csv. For that reason, we recommend setting branch explicitly when you provide this file to the Moderne Connector, even though the CLI can resolve it from the remote.

Different SCM providers format their clone URLs differently, making it impossible to automatically extract the correct origin and path values. You might also be running your SCM behind a reverse proxy or context root – which would further complicate this. Because of that, we require that you explicitly provide origin and path for each row.

Below you can find examples of what origin/path typically looks like for common SCM providers:

GitHub​

ProtocolClone URL PatternOriginPath
HTTPShttps://github.com/{org}/{repo}.gitgithub.com{org}/{repo}
SSHgit@github.com:{org}/{repo}.gitgithub.com{org}/{repo}

GitLab​

ProtocolClone URL PatternOriginPath
HTTPShttps://gitlab.com/{groups}/{repo}.gitgitlab.com{groups}/{repo}
SSHgit@gitlab.com:{groups}/{repo}.gitgitlab.com{groups}/{repo}
  • GitLab allows deeply nested folders to organize repositories. The path field must include the complete hierarchy of all group and subgroup folders. For example, if your repository is organized as groupa/groupb/groupc/repo, the entire path including all intermediate groups must be specified in the path field.

Bitbucket Cloud​

ProtocolClone URL PatternOriginPath
HTTPShttps://bitbucket.org/{org}/{repo}.gitbitbucket.org{org}/{repo}
SSHgit@bitbucket.org:{org}/{repo}.gitbitbucket.org{org}/{repo}

Bitbucket Data Center​

ProtocolClone URL PatternOriginPath
HTTPShttps://{host}/bitbucket/scm/{org}/{repo}.git{host}/bitbucket{org}/{repo}
SSHssh://{host}:7999/bitbucket/{org}/{repo}.git{host}/bitbucket{org}/{repo}
  • As a self-hosted option, Bitbucket Data Center is often deployed with a context root (e.g., the /bitbucket in the above examples). This context root should be included as part of the origin field, not the path field.
  • HTTPS clone URLs for Bitbucket Data Center include /scm/ between the host/context and the project/repo path. This /scm/ segment is part of Bitbucket's URL structure but must be excluded from the path field.
  • SSH clone URLs for Bitbucket Data Center use the ssh:// protocol prefix and typically run on port 7999 (the default port). Neither the protocol (ssh://) nor the port number (:7999) should be included in the origin or path fields.

Azure DevOps​

ProtocolClone URL PatternOriginPath
HTTPShttps://dev.azure.com/{org}/{project}/_git/{repo}dev.azure.com{org}/{project}/{repo}
SSHgit@ssh.dev.azure.com:v3/{org}/{project}/{repo}dev.azure.com{org}/{project}/{repo}
  • Azure DevOps inserts /_git/ into HTTPS clone URLs to distinguish git repositories from other project resources. This /_git/ segment must be excluded from the path field.
  • SSH clone URLs use a different host (ssh.dev.azure.com instead of dev.azure.com) and include :v3/ as a version identifier. Despite the different host in the SSH URL, the origin should still be dev.azure.com for consistency across both protocols. The :v3/ version identifier must be excluded from the path field.

On-premises SCM providers​

ProtocolClone URL PatternOriginPath
HTTPShttps://{host}/{context}/{org}/{repo}{host}{org}/{repo}
SSHgit@{host}:{org}/{repo} or ssh://{host}:{port}/{org}/{repo}{host}{org}/{repo}

For self-hosted providers such as GitHub Enterprise, Self-Managed GitLab, or Bitbucket Data Center or providers behind a custom domain, you'll need to identify any context root and include that in the origin and not the path. Depending on the SCM provider, you may need to alter the host or path in similar ways as noted above to remove provider-specific parts of the URL.

Using alternateCloneUrl to support multiple protocols​

You can provide both cloneUrl and alternateCloneUrl to support both your primary protocol (HTTPS or SSH) and a secondary protocol for maximum flexibility. In situations where your organization allows read access over one protocol but only allows commits through a different protocol, this enables the Moderne Platform to present a single repository while correctly supporting both workflows. This also allows matching when an LST is built manually from a git repository cloned with the secondary URL.

If the cloneUrl and alternateCloneUrl differ in their domain (as is the case with Azure DevOps above), you should set origin based on the HTTPS URL (since origins are used to build URLs for SCM API calls such as creating PRs) in a consistent way across HTTPS and SSH repositories for that SCM provider so that Moderne can successfully identify the repository in both cases. The origin should always be the root of the service on HTTP(S) but without the protocol and can most easily be inferred from the HTTP(S) clone URL.

Configuring build environment variables​

Some builds only work when certain environment variables are set, such as a registry token that npm reads. To set those variables for each repository, update the optional environment column.

Write each variable as NAME=VALUE, with no spaces around the =. If you have more than one variable, put a space between them. If a value has a space in it, wrap the whole NAME=VALUE pair in quotes (e.g., 'NAME=MY VALUE').

Let's take a look at an example. We'll turn on CI mode, pass a token from the build machine, and set a label that has a space in it:

cloneUrl,branch,origin,path,environment
https://github.com/example/service,main,github.com,example/service,CI=true NPM_TOKEN=${CI_TOKEN} 'BUILD_LABEL=Nightly build'

In this example, NPM_TOKEN=${CI_TOKEN} doesn't contain the token itself. Instead, it points to a variable called CI_TOKEN on the machine that runs mod build. The CLI saves that reference and will look up the real value when it builds the LST. You should point to a variable like this for any secret, since the CLI will save all variables as plain text in the repository's .moderne directory.

If CI_TOKEN isn't set on that machine, the CLI will leave NPM_TOKEN alone and will write a warning to the build log.

If the missing variable is only part of a longer value, such as TOOL_HOME=$MISSING/bin, the CLI will not warn you. It will pass $MISSING/bin to the build as written.

There are three different formats you can use to point to a variable: $CI_TOKEN, ${CI_TOKEN}, and %CI_TOKEN%. These will all look for the value of CI_TOKEN on the build machine.

The build log will hide the values of these variables. If you look in the logs for the environment, it will show each variable as something like NAME=********.

Which variables a repository ends up with after you run mod git sync csv depends on what its row has in the environment column:

  • If the row has variables, the CLI will save them in the repository's .moderne directory. They will replace all the variables it saved for that repository on an earlier sync.
  • If the row is blank, or your file has no environment column, the CLI will keep the variables it saved last time.
  • If the row has variables, the repository will ignore the ones you set globally with mod config build environment edit. The two lists will not be combined. If the repository still needs a global variable, you will need to add it to the row too.
More ways to write values

Commas

If a value has a comma in it, you will need to wrap the whole column value in double quotes. Otherwise, the CLI will drop everything after the comma without warning you.

cloneUrl,branch,origin,path,environment
https://github.com/example/service,main,github.com,example/service,"NO_PROXY=a.example.com,b.example.com CI=true"

If you also need double quotes inside the value, you will need to type each one twice.

Reusing a variable

A variable can use one you set earlier in the same row. For example, BASE=/opt/tools TOOL_HOME=$BASE/bin will set TOOL_HOME to /opt/tools/bin.

Using publishUri to share pre-built LSTs​

When you have pre-built LST artifacts stored in an artifact repository (such as Artifactory), you can include the publishUri column in your repos.csv to enable direct LST downloads. This is particularly useful for:

  1. Sharing LSTs with team members - Instead of everyone building LSTs locally, one person (or mass-ingest) can build and publish them, then share a repos.csv with publishUri values
  2. Faster onboarding - New team members can download pre-built LSTs immediately without cloning sources or running builds
  3. Offline/air-gapped environments - Pre-built LSTs can be distributed via artifact repositories

When mod git sync csv encounters a row with a publishUri:

  • It downloads the LST artifact from that URL
  • Source code is not cloned unless you also specify --with-sources

If you need both the LST and source code, use:

mod git sync csv /path/to/repos ./repos.csv --with-sources

Generating this file​

To kick-start a repos.csv file, we recommend using "repo fetchers". Repo fetchers are scripts that generate a CSV file of all of your repositories by making API calls to your source code manager.

Repo fetchers intentionally do not assign organizations, as how repositories map to organizations is a decision that belongs to you rather than your source code manager. After generating the base repository list, define your own hierarchy by adding org columns as described below.

Defining an organizational hierarchy​

An organizational hierarchy doesn't need to be complete from day one. A common way to start is with a single ALL organization that contains every repository, plus one organization for a single team. You can then expand to more organizations and more levels of hierarchy over time.

Org hierarchy is best explained with an example. Consider an organization that consists of two teams and two directors:

All
├── Director A
| └── Team 1
| └── Team 2
├── Director B
| └── apache

The following CSV file would represent this organizational structure:

"cloneUrl","branch","origin","path","org1","org2","org3"
"https://github.com/apache/maven-doxia","master","github.com","apache/maven-doxia","Team 1","Director A","ALL"
"https://github.com/Netflix/photon","main","github.com","Netflix/photon","Team 2","Director A","ALL"
"https://github.com/Netflix/ribbon","master","github.com","Netflix/ribbon","Director A","ALL"
"https://github.com/apache/maven-doxia","master","github.com","apache/maven-doxia","Director B","ALL"

In the above file, we define 5 organizations (ALL, Director A, Director B, Team 1, and Team 2) and 3 repositories. Notice that one repository can be defined multiple times if you want two distinct organizations to have access to it.

Also note that organizations on the left are children of organizations on the right. For instance, Team 1 is a child of Director A which is a child of ALL.

Comma-separated files (CSV) are best manipulated using a dedicated editor such as Microsoft Excel, Google Sheets, or Apple Numbers. Here is what the above CSV would look like on Apple Numbers:

Spreadsheet view of repos.csv with cloneUrl, branch, org1, org2, and org3 columns defining the hierarchy

Formatting details​

  • A single row may not have more columns than what is defined in the first row.
  • A single repository may be associated with multiple organizations by being referenced in multiple rows. In the above example, maven-doxia is associated with both Director B and Team 1.
  • You can add empty orgs to the beginning, middle, or end of a row to make them line up as desired (although you don't need to).
  • You can choose to surround elements with quotes or not depending on your preference – just keep in mind general CSV formatting rules.
  • You can order columns in any way.
  • Any columns not listed in the supported columns table are ignored. You can use these to hold internal notes such as team names, owners, or other metadata that is meaningful to your organization but not consumed by the Moderne CLI.

Spreadsheet showing repos.csv with repositories assigned to multiple org levels including AWS, FINOS, Netflix, and Spring
An example demonstrating various orgs lining up with blanks in between.