Skip to main content

Configure a Connector with an S3 organization source

This guide explains how to point the Moderne Connector at a repository CSV stored in Amazon S3 or an S3-compatible service (e.g., MinIO). On each polling cycle, the Connector fetches that CSV to discover your repositories. When the CSV includes publishUri values, the Connector uses them to locate each repository's LST artifact.

Any of the CSV variants documented in the repos.csv reference work here: a minimal CSV with just origin, path, and branch columns; one that adds organization hierarchy columns; or a full repos-lock.csv produced by Mass Ingest's mod publish.

For background on how the Connector uses CSV sources and how S3 fits into the overall Connector configuration, please see how the Connector finds your repositories and their LSTs.

Prerequisites​

  • A repository CSV stored as a single object in an S3 bucket. The uri you configure must point at that object (e.g., s3://my-bucket/repos.csv), not at a bucket or prefix.
  • One of the following authentication methods:
    • IAM role (when running on AWS infrastructure)
    • An AWS profile configured on the machine running the Connector
    • AWS access key ID and secret access key
  • If using a custom S3-compatible endpoint (e.g., MinIO), you will need the endpoint URL as well.
  • The credentials used must have the following permissions on your S3 bucket:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/repos.csv"
            ]
        }
    ]
}

Authentication options​

The Connector supports multiple authentication methods for S3:

  • Access key and secret key: Provide accessKey and secretKey directly in the configuration. This is useful for non-AWS environments or when IAM roles are not available.
  • AWS profile: Specify a profile name that references credentials configured in your AWS credentials file (~/.aws/credentials).
  • IAM role: When running on AWS infrastructure (EC2, ECS, EKS), the Connector can use the instance's IAM role automatically. In this case, you only need to provide the uri and, if deployed to a different region than the bucket, the region.

Configuring the Moderne Connector​

The following table contains all the variables/arguments you need to add to your Moderne Connector run command in order for it to load a repository CSV from S3. Please note that these variables/arguments must be combined with ones found in other steps in the Configuring the Moderne Connector guide.

You can configure multiple S3 sources by including multiple entries, each with a different {index}.

Environment variables:

Variable NameRequiredDefaultDescription
MODERNE_ORGANIZATION_SOURCES_S3_{index}_URItrueThe S3 URI of the CSV object. Must start with s3:// and include the object key (e.g., s3://my-bucket/repos.csv).
MODERNE_ORGANIZATION_SOURCES_S3_{index}_ENDPOINTURLfalseCustom endpoint URL for S3-compatible services (e.g., http://localhost:9000 for MinIO). Leave empty for standard AWS S3.
MODERNE_ORGANIZATION_SOURCES_S3_{index}_REGIONfalseThe AWS region where the bucket is located (e.g., us-east-1). Can be excluded if the Connector is deployed on AWS infrastructure in the same region as the bucket.
MODERNE_ORGANIZATION_SOURCES_S3_{index}_ACCESSKEYfalse (Required if not using profile or IAM role)The AWS access key ID for authentication.
MODERNE_ORGANIZATION_SOURCES_S3_{index}_SECRETKEYfalse (Required if using access key)The AWS secret access key for authentication.
MODERNE_ORGANIZATION_SOURCES_S3_{index}_PROFILEfalse (Alternative to access key/secret key)The AWS profile name from your credentials file.
MODERNE_ORGANIZATION_SOURCES_S3_{index}_SKIPSSLtrue (If using self-signed cert or non-HTTPS endpoint)falseSpecifies whether to skip SSL verification for connections to the S3 endpoint.

Example using IAM role authentication on AWS infrastructure:

docker run \
# ... Existing variables
-e MODERNE_ORGANIZATION_SOURCES_S3_0_URI=s3://my-bucket/repos.csv \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_REGION=us-east-1 \
# ... Additional variables

Example with access key and secret key:

docker run \
# ... Existing variables
-e MODERNE_ORGANIZATION_SOURCES_S3_0_URI=s3://my-bucket/repos.csv \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_REGION=us-east-1 \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_ACCESSKEY=AKIAIOSFODNN7EXAMPLE \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_SECRETKEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
# ... Additional variables

Example with AWS profile:

docker run \
# ... Existing variables
-e MODERNE_ORGANIZATION_SOURCES_S3_0_URI=s3://my-bucket/repos.csv \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_REGION=us-east-1 \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_PROFILE=my-aws-profile \
-v ~/.aws:/root/.aws:ro \
# ... Additional variables

Example with MinIO (S3-compatible storage):

docker run \
# ... Existing variables
-e MODERNE_ORGANIZATION_SOURCES_S3_0_URI=s3://my-bucket/repos.csv \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_ENDPOINTURL=http://minio.example.com:9000 \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_ACCESSKEY=minioadmin \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_SECRETKEY=minioadmin \
-e MODERNE_ORGANIZATION_SOURCES_S3_0_SKIPSSL=true \
# ... Additional variables

Using IAM roles on AWS​

When running the Connector on AWS infrastructure (EC2, ECS, EKS, Lambda), you can use IAM roles for authentication instead of providing access keys. To do this:

  1. Create an IAM role with the permissions shown in Prerequisites
  2. Attach the role to your compute resource (EC2 instance, ECS task, etc.)
  3. Configure the Connector with only the uri and optionally region

The AWS SDK will automatically detect and use the IAM role credentials.