To help you understand how the Moderne platform works and how it interacts with your environment and services, this document will:
Moderne’s SaaS allows permitted users to run recipes on code in the repositories you've added to the platform. These recipes can yield pull requests (PRs) or commits that transform the code.
As the code changes (either due to recipe results being merged in or due to active development), an artifact that contains a serialized representation of the code's Lossless Semantic Tree (LST) will need to be generated and published (See the customer CI section for more information on how to publish artifacts and the Moderne agent section for more information on how these artifacts are encrypted and securely transmitted).
Once the artifact is published, the Moderne agent will send the changes to Moderne so that the internal state can be updated. After that happens, new recipes can be run against the new artifacts and the process will repeat.
Moderne + SDL
Below is a high-level architecture diagram that shows the flow of data between Moderne and a typical customer environment. Arrows indicate communication between components. The details of each component can be found in the following sections.
In order for Moderne to know the current state of your code, artifacts will need to be generated that contain a serialized representation of your code's LSTs. These artifacts must be put inside an artifact repository that the Moderne agent has access to.
There are currently two ways to generate artifacts. You can:
- 1.Set up a Jenkins ingest pipeline that runs at scheduled intervals to build and publish artifacts for the repositories you specify in a CSV (recommended)
- Easier to scale and does not require you to modify the build process
- 2.Apply a Gradle or Maven plugin directly to each repository that you want artifacts published for
- Faster to set up originally, but does not scale well for large numbers of repositories and requires the build process to be modified for each
Regardless of which option you choose, the LST artifacts will be published to your existing artifact repository alongside binary, source, and JavaDoc artifacts that are already created by your team's normal publishing mechanisms. Since the LST artifacts are added to your existing artifact repository, no additional credentials are needed.
At a high level, you can think of the Moderne agent as a bridge between your environment and Moderne. All data that Moderne needs to function will pass over this bridge and flow into the Moderne API Gateway. As this data is sent to Moderne, it's encrypted – with the key being kept in your environment. Whenever Moderne needs to access any data, it will request this key and the data will be decrypted for a short time before it's thrown away. If you decide you no longer want Moderne to have access to anything, you can raise the bridge (shut off the agent) and all of your data that Moderne has will no longer be decryptable.
There are a variety of tools and services that you can configure the agent to be connected to based on the needs of your team.
At a minimum, the agent will need to connect to:
- One or more of your artifact repositories so that changes to the Moderne LST artifacts kept in them can be sent to Moderne
- This is configured via the Artifactory Query Language or via Maven configuration. Only artifacts that match what you've configured will be sent to Moderne.
- Your SCM(s) so that PRs or commits can be created by approved users in Moderne
Your team may also wish to configure the agent to:
- Look in your artifact repositories for custom recipe JARs your team creates so that those recipes can be run in the Moderne SaaS
- Connect to your team's Organizations service so that the UI can provide a customized experience for some users or repositories
- Deploy a Moderne-provided OCI image adjacent to Artifactory
- Ensure the agent is configured with an Artifactory user that is authorized to make
findAQL requests against repositories containing LST artifacts
- Ensure the agent is configured with an Artifactory user capable of making
GETrequests to obtain the above LST artifacts
- Ensure that the deployed image is configured with an encryption key (stored in Hashicorp Vault or some other key management service)
- Ensure the agent can connect to your SCM(s) to authorize users to see code in the Moderne SaaS and to allow commits on their behalf
Multiple agents can be configured for high availability or to connect to only some of these services.
Agents initiate connections to the Moderne API gateway via the RSocket protocol. Moderne will never initiate an API call to the agent. Because of that, only egress from your environment needs to be open.
When you set up an agent, Moderne will share a token with you that you must configure in the Moderne agents you create. Moderne will reject any connection attempts from unauthorized agent instances. In this way, Moderne requires a minimum level of client (agent) verification as an extra security precaution.
The connection to Moderne is established over layer 7, so you may choose to route traffic from the agent through your own layer 7 gateway. This might be chosen to satisfy a desire for Moderne’s API gateway to perform client verification of an inbound agent connection using a mechanism like X.509 in addition to token-based verification.
These measures act in concert with techniques to limit IP addressability of the Moderne API gateway to enhance the overall security posture.
The Organizations service is an optional service that you can configure in your environment. It defines the organizational structure that a user or a repository belongs to. The Moderne SaaS will use this information to present different UIs or options for different users and repositories. Please see our Organizations service doc for more information.
- Ensure your service fulfills the GraphQL contract outlined in our Organizations service documentation
The Moderne API gateway serves as the entry point to Moderne. It talks with the Moderne agent to get data from your services to Moderne. It is the only component with a public IP address that can communicate with other Moderne services. The Moderne UI and Keycloak also have public IP addresses, but they can't communicate with other Moderne services.
The API gateway is responsible for:
- Handling API requests from your developers or your tools
- Handling API requests from the Moderne UI
- Handling encrypted LST artifacts from the Moderne agent(s)
- Handling encrypted custom recipe artifacts from the Moderne agent(s)
- Rate limiting as needed to guard Moderne services against overuse by a particular user
Authorized users in your company can access audit logs for this gateway via an API.
The Moderne API gateway is configured with a Moderne-managed SSL certificate.
- Ensure that https://api.TENANT.moderne.io is on the accept list for outbound HTTPS traffic from the Moderne agent
- Ensure that https://api.TENANT.moderne.io is on the accept list for outbound HTTPS traffic from the developer's machines
The Moderne UI provides a browser-based interface for:
- Executing search and transformation recipes across your codebase
- Issuing mass commits/PRs based on recipe runs
- Building new recipes based on other recipes
- Viewing audit logs
- Generating access tokens for interacting with the Moderne API
- Ensure that https://TENANT.moderne.io is on the accept list for outbound HTTPS traffic from the developer's machines
Keycloak is an open-source identity and access management system. Moderne services request authorization information for a user from Keycloak. Keycloak then calls out to your identity provider (such as LDAP, Okta, or Keycloak) to determine who is authorized for what.
- Ensure that https://login.TENANT.moderne.io is on the accept list for outbound HTTPS traffic from the developer's machines
As configuring identity providers between services can be quite complex, the setup for Keycloak is usually done over a Zoom meeting with Moderne and your company.
The Moderne artifact storage service is responsible for receiving pre-encrypted LST artifacts and recipe JARs and storing them in a private object store depending on the cloud provider you use (Azure Blob Storage, Google Cloud Storage, or AWS S3).
The artifact storage service will also write high-level information about where to find these artifacts and when they were last updated to our relationship database (RDS) so that the Moderne workers know where to go to obtain the artifacts they need.
The GraphQL federation is an internal Moderne microservice that composes the GraphQL schemas of individual Moderne microservices into one GraphQL schema. All GraphQL requests pass through the federation service on their way to the individual microservice that supports a piece of the GraphQL schema.
GraphQL federation uses Netflix Eureka to locate microservice instances to communicate with and to load-balance requests to those instances.
You can think of the Moderne recipe execution service as a manager that helps assign work, direct people on where to go, and provide high-level information. It knows all of the recipes that can be run and it chooses which workers are responsible for which repositories.
When a new recipe command comes in (such as run
Zoptions), the recipe execution service takes that command and stores all of the details in a database table that acts as a queue. Whenever a worker is free, it will query the database table and look for commands that haven't been started (for the repositories the worker is responsible for).
The recipe execution service is also responsible for providing the results of a recipe run by either calling the database to see what high-level information it has or by calling the worker directly to get the full results (diffs and data tables).
Moderne workers are responsible for running recipes and keeping their results. They interact with the Moderne recipe execution service to coordinate which repositories they should run recipes for before querying the database for a recipe to run. When a worker is stopped, all of the data and results from the recipes it has run are destroyed. If you need that data, you'll need to re-run the recipe again.
Worker instances are scaled horizontally in direct response to more code being ingested into the platform.
Workers decrypt LST and recipe artifacts by making a request to the Moderne Agent via the API Gateway for a customer-provided symmetric key. Workers discard this key at the end of every request.
Workers fetch a user’s SCM OAuth token via the API gateway in order to make authorization decisions about which repositories said user is allowed to read from. This ensures Moderne’s read access is aligned with a user's SCM access in real-time for every recipe run request.
The Moderne source code management service is responsible for handling all communication with your SCM(s).
The two primary responsibilities are:
- Creating commits, branches, forks, PRs, etc. in your SCM
- Coordinating authorization with your SCM to see what users are authorized to do or view
Please note that the requests to your SCM will appear to come from the Moderne agent. The Moderne source code management service will talk through the API gateway to the agent whenever it needs to interact with your SCM.
Authentication and authorization decisions are made in real-time to ensure that they are always up-to-date.
The Moderne audit log retrieves audit logs from our relational database for presentation to privileged users via the API gateway.
Audit logs can be retrieved via a paginated GraphQL API or via a REST call that responds in the CEF format.
Individual Moderne microservices are responsible for contributing to the audit log database when they perform any interaction on behalf of users.
The Moderne tokens service generates and retrieves access tokens tied to a particular user. Access tokens can be used to access the service via IDEs and custom tooling. Users can manage their access tokens via the Moderne UI. Tokens are only visible once at creation time and are hidden from that point forward, even from the user that created them.
Please see our token documentation for more information on how to create, work with, and revoke tokens.