Skip to main content

Configure a Connector with an S3 organization source

If you publish LSTs to Amazon S3 or an S3-compatible service (e.g., MinIO), the bucket already holds everything the Connector needs. mod publish keeps a repos-lock.csv there that lists every repository and where its LST lives. Point the Connector at that file and it will discover your repositories and their LSTs in the same bucket, with no separate CSV to host or keep in sync.

The organizational hierarchy comes from the org columns of the repos.csv file you publish from, as described in the repos.csv reference.

A row without a publishUri still shows up in your organizational hierarchy, but you won't be able to run recipes against it because the Connector has no LST to fetch.

For background on how the Connector uses CSV sources and how S3 fits into the overall Connector configuration, please see how the Connector finds your repositories and their LSTs.

Prerequisites​

  • A repos-lock.csv file stored as a single object in an S3 bucket, alongside the LSTs it lists. The uri you configure must point at that object (e.g., s3://my-bucket/repos-lock.csv), not at a bucket or prefix.
  • One of the following authentication methods:
    • IAM role (when running on AWS infrastructure)
    • An AWS profile configured on the machine running the Connector
    • AWS access key ID and secret access key
  • If using a custom S3-compatible endpoint (e.g., MinIO), you will need the endpoint URL as well.
  • The credentials used must have the following permissions on your S3 bucket:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket",
                "arn:aws:s3:::my-bucket/*"
            ]
        }
    ]
}

The s3:GetObject permission lets the Connector read the CSV object. It also lets the Connector read the LST artifacts that the CSV references. When the CSV carries publishUri values that point into the same bucket, the Connector downloads those LSTs with the same credentials. Because the Connector only fetches objects under the parent prefix of the configured uri, you can scope the object ARN to that prefix. The examples on this page use my-bucket/*, since the uri sits at the bucket root.

The s3:ListBucket permission applies to the bucket itself. The Connector requires it because it runs a startup connectivity check to confirm that it can reach the bucket before it reads the object. If that check does not pass, the Connector fails to start.

Authentication options​

The Connector supports multiple authentication methods for S3:

  • Access key and secret key: Provide access-key and secret-key directly in the configuration. This is useful for non-AWS environments or when IAM roles are not available.
  • AWS profile: Specify a profile name that references credentials configured in your AWS credentials file (~/.aws/credentials).
  • IAM role: When running on AWS infrastructure (EC2, ECS, EKS), the Connector can use the instance's IAM role automatically. In this case, you only need to provide the uri and, if deployed to a different region than the bucket, the region.

Configuring the Moderne Connector​

The following table contains all the variables/arguments you need to add to your Moderne Connector run command in order for it to load a repository CSV from S3. Please note that these variables/arguments must be combined with ones found in other steps in the Configuring the Moderne Connector guide.

You can configure multiple S3 sources by including multiple entries, each with a different {index}.

Environment variables:

Variable NameRequiredDefaultDescription
MODERNE_ORGANIZATION_SOURCES_S3_{index}_URItrueThe S3 URI of the CSV object. Must start with s3:// and include the object key (e.g., s3://my-bucket/repos-lock.csv).
MODERNE_ORGANIZATION_SOURCES_S3_{index}_ENDPOINTURLfalseCustom endpoint URL for S3-compatible services (e.g., http://localhost:9000 for MinIO). Leave empty for standard AWS S3.
MODERNE_ORGANIZATION_SOURCES_S3_{index}_REGIONfalseThe AWS region where the bucket is located (e.g., us-east-1). Can be excluded if the Connector is deployed on AWS infrastructure in the same region as the bucket.
MODERNE_ORGANIZATION_SOURCES_S3_{index}_ACCESSKEYfalse (Required if not using profile or IAM role)The AWS access key ID for authentication.
MODERNE_ORGANIZATION_SOURCES_S3_{index}_SECRETKEYfalse (Required if using access key)The AWS secret access key for authentication.
MODERNE_ORGANIZATION_SOURCES_S3_{index}_PROFILEfalse (Alternative to access key/secret key)The AWS profile name from your credentials file.
MODERNE_ORGANIZATION_SOURCES_S3_{index}_SKIPSSLtrue (If using self-signed cert or non-HTTPS endpoint)falseSpecifies whether to skip SSL verification for connections to the S3 endpoint.

Example using IAM role authentication on AWS infrastructure:

docker run \
# ... Existing variables
-e MODERNE_ORGANIZATION_SOURCES_S3_0_URI=s3://my-bucket/repos-lock.csv \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_REGION=us-east-1 \
# ... Additional variables

Example with access key and secret key:

docker run \
# ... Existing variables
-e MODERNE_ORGANIZATION_SOURCES_S3_0_URI=s3://my-bucket/repos-lock.csv \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_REGION=us-east-1 \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_ACCESSKEY=AKIAIOSFODNN7EXAMPLE \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_SECRETKEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
# ... Additional variables

Example with AWS profile:

docker run \
# ... Existing variables
-e MODERNE_ORGANIZATION_SOURCES_S3_0_URI=s3://my-bucket/repos-lock.csv \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_REGION=us-east-1 \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_PROFILE=my-aws-profile \
-v ~/.aws:/root/.aws:ro \
# ... Additional variables

Example with MinIO (S3-compatible storage):

docker run \
# ... Existing variables
-e MODERNE_ORGANIZATION_SOURCES_S3_0_URI=s3://my-bucket/repos-lock.csv \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_ENDPOINTURL=http://minio.example.com:9000 \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_ACCESSKEY=minioadmin \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_SECRETKEY=minioadmin \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_SKIPSSL=true \
# ... Additional variables

Publishing LSTs to the bucket​

The Moderne CLI produces both the CSV and the LSTs it points at. Configure it to publish to the same bucket:

mod config lsts artifacts s3 edit s3://my-bucket
mod build /path/to/your/repos
mod publish /path/to/your/repos

mod publish uploads each LST under a yyyy/MM/dd/HH/ prefix and maintains a merged repos-lock.csv at the root of the bucket. That merged file is the object you'll want to point the Connector at.

To keep your LSTs up to date, you'll want to run mod publish every day through a mass ingest pipeline rather than by hand. Mass ingest runs mod publish --sync-csv, which reads the repos.csv you put at the root of the bucket, publishes each repository, and records the results in the same repos-lock.csv. The Connector re-reads that file every ten minutes by default. You can change that with the moderne.connector.organization.interval setting.

If you shard mass ingest across several machines, they can all publish to the same bucket at once. Each one adds its own rows to repos-lock.csv without touching the others'. If a machine can't get its rows in, its mod publish fails and leaves the file as the other machines wrote it.

See the repos-lock.csv guide for the full publishing workflow.

Using IAM roles on AWS​

When running the Connector on AWS infrastructure (EC2, ECS, EKS, Lambda), you can use IAM roles for authentication instead of providing access keys. To do this:

  1. Create an IAM role with the permissions shown in Prerequisites
  2. Attach the role to your compute resource (EC2 instance, ECS task, etc.)
  3. Configure the Connector with only the uri and optionally region

The AWS SDK will automatically detect and use the IAM role credentials.