Skip to main content

Configure a Connector with a Google Cloud Storage organization source

If you publish LSTs to Google Cloud Storage, the bucket already holds everything the Connector needs. mod publish keeps a repos-lock.csv there that lists every repository and where its LST lives. Point the Connector at that file and it will discover your repositories and fetch their LSTs from the same bucket, with no separate CSV to host or keep in sync.

The organizational hierarchy comes from the org columns of the repos.csv file you publish from, as described in the repos.csv reference.

A row without a publishUri still shows up in your organizational hierarchy, but you won't be able to run recipes against it because the Connector has no LST to fetch.

For background on how the Connector uses CSV sources, please see how the Connector finds your repositories and their LSTs.

Prerequisites​

  • A repos-lock.csv file stored as a single object in a Cloud Storage bucket, alongside the LSTs it lists. The uri you configure must point at that object (e.g., gs://my-bucket/repos-lock.csv), not at a bucket or prefix.
  • A way for the Connector to authenticate, which will be either Application Default Credentials or a service account key. Both are covered in authentication options below.
  • A service account that can read the CSV object and the LST objects alongside it. The Connector only ever reads individual objects, so storage.objects.get is the only permission it needs. You can grant it through the predefined Storage Object Viewer (roles/storage.objectViewer) role, or through a custom role if you'd prefer. It never needs permission to list the bucket.

Authentication options​

You have two options here, and we'd recommend the first:

  • Application Default Credentials (ADC): attach a service account to the workload and configure only the uri. On GKE, that means Workload Identity, while on Compute Engine the attached service account is picked up for you. You can also point GOOGLE_APPLICATION_CREDENTIALS at a key file. Whichever you choose, no key material ends up in the Connector's configuration.
  • Service account key: set credentials-json to the contents of the key file. You should treat that value as a secret and supply it through your platform's secret mechanism rather than an inline environment variable wherever you can.

Which artifacts the Connector will fetch​

The Connector only fetches LST artifacts that live in the same bucket as your repos-lock.csv file. mod publish always writes that file at the root of the bucket (gs://my-bucket/repos-lock.csv), so every LST that mod publish uploads is within the Connector's reach.

Encrypting LSTs before they reach Moderne​

The Connector fetches your LSTs from the bucket, so it has to encrypt them before uploading them to Moderne. You will need to configure the Connector with a symmetric key, as described in step 1 of the Connector configuration guide.

Configuring the Moderne Connector​

The following table contains all the variables/arguments you need to add to your Moderne Connector run command in order for it to load a repository CSV from Cloud Storage. Please note that these variables/arguments must be combined with ones found in other steps in the Configuring the Moderne Connector guide.

You can configure multiple Cloud Storage sources by including multiple entries, each with a different {index}.

Environment variables:

Variable NameRequiredDefaultDescription
MODERNE_ORGANIZATION_SOURCES_GCS_{index}_URItrueThe Cloud Storage URI of the CSV object. Must start with gs:// and include the object name (e.g., gs://my-bucket/repos-lock.csv).
MODERNE_ORGANIZATION_SOURCES_GCS_{index}_CREDENTIALSJSONfalse (Required if not using ADC)The contents of a service account key file. Omit to use Application Default Credentials.
MODERNE_ORGANIZATION_SOURCES_GCS_{index}_PROJECTfalseThe project to bill requests to. Only needed for requester-pays buckets.
MODERNE_ORGANIZATION_SOURCES_GCS_{index}_ENDPOINTURLfalseOverrides the default storage.googleapis.com endpoint (e.g., a Private Service Connect endpoint).

Example using Application Default Credentials:

docker run \
# ... Existing variables
-e MODERNE_ORGANIZATION_SOURCES_GCS_0_URI=gs://my-bucket/repos-lock.csv \
# ... Additional variables

Example with a service account key:

docker run \
# ... Existing variables
-e MODERNE_ORGANIZATION_SOURCES_GCS_0_URI=gs://my-bucket/repos-lock.csv \
-e MODERNE_ORGANIZATION_SOURCES_GCS_0_CREDENTIALSJSON="$(cat service-account-key.json)" \
# ... Additional variables

Publishing LSTs to the bucket​

The Moderne CLI produces both the CSV and the LSTs it points at. Configure it to publish to the same bucket:

mod config lsts artifacts gcs edit gs://my-bucket
mod build /path/to/your/repos
mod publish /path/to/your/repos

mod publish uploads each LST under a yyyy/MM/dd/HH/ prefix and maintains a merged repos-lock.csv at the root of the bucket. That merged file is the object you'll want to point the Connector at.

To keep your LSTs up to date, you'll want to run mod publish every day through a mass ingest pipeline rather than by hand. Mass ingest runs mod publish --sync-csv, which reads the repos.csv you put at the root of the bucket, publishes each repository, and records the results in the same repos-lock.csv. The Connector re-reads that file every ten minutes by default. You can change that with the moderne.connector.organization.interval setting.

If you shard mass ingest across several machines, they can all publish to the same bucket at once. Each one adds its own rows to repos-lock.csv without touching the others'. If a machine can't get its rows in, its mod publish fails and leaves the file as the other machines wrote it.

See the repos-lock.csv guide for the full publishing workflow.