Moderne architecture
To help you understand how the Moderne platform works and how it interacts with your environment and services, this document will:
How Moderne fits into the software development lifecycle
Moderne’s SaaS allows permitted users to run recipes on code in the repositories you've added to the platform. These recipes can yield pull requests (PRs) or commits that transform the code.
As the code changes (either due to recipe results being merged in or due to active development), an artifact that contains a serialized representation of the code's Lossless Semantic Tree (LST) will need to be generated and published.
Once the artifact is published, the Moderne agent will send the changes to Moderne so that the internal state can be updated. After that happens, new recipes can be run against the new artifacts and the process will repeat.
Architecture diagram
Below is a high-level architecture diagram that shows the flow of data between Moderne and a typical customer environment. Arrows indicate communication between components. The details of each component can be found in the following sections.
Key components
Mass ingest with mod CLI
In order for Moderne to know the current state of your code, artifacts will need to be generated that contain a serialized representation of your code's LSTs. These artifacts must be put inside an artifact repository that the Moderne agent has access to.
To do this, you'll want to use set up mass ingestion with the Moderne CLI. For instructions on how to do that, please read our Mass ingestion doc.
Moderne agent
At a high level, you can think of the Moderne agent as a bridge between your environment and Moderne. All data that Moderne needs to function will pass over this bridge and flow into the Moderne API Gateway. As this data is sent to Moderne, it's encrypted – with the key being kept in your environment. Whenever Moderne needs to access any data, it will request this key and the data will be decrypted for a short time before it's thrown away. If you decide you no longer want Moderne to have access to anything, you can raise the bridge (shut off the agent) and all of your data that Moderne has will no longer be decryptable.
There are a variety of tools and services that you can configure the agent to be connected to based on the needs of your team.
At a minimum, the agent will need to connect to:
One or more of your artifact repositories so that changes to the Moderne LST artifacts kept in them can be sent to Moderne
This is configured via the Artifactory Query Language or via Maven configuration. Only artifacts that match what you've configured will be sent to Moderne.
Your SCM(s) so that PRs or commits can be created by approved users in Moderne
Your team may also wish to configure the agent to:
Look in your artifact repositories for custom recipe JARs your team creates so that those recipes can be run in the Moderne SaaS
Connect to your team's Organizations service so that the UI can provide a customized experience for some users or repositories
You can find all of the documentation for configuring agents in your environment here.
Setup requirements
You must:
Deploy a Moderne-provided OCI image adjacent to Artifactory
Ensure the agent is able to make an outbound HTTPS request to
https://api.TENANT.moderne.ioEnsure the agent is configured with an Artifactory user that is authorized to make
findAQL requests against repositories containing LST artifactsEnsure the agent is configured with an Artifactory user capable of making
GETrequests to obtain the above LST artifactsEnsure that the deployed image is configured with an encryption key (stored in Hashicorp Vault or some other key management service)
Ensure the agent can connect to your SCM(s) to authorize users to see code in the Moderne SaaS and to allow commits on their behalf
Multiple agents can be configured for high availability or to connect to only some of these services.
Agent security
Agents initiate connections to the Moderne API gateway via the RSocket protocol. Moderne will never initiate an API call to the agent. Because of that, only egress from your environment needs to be open.
When you set up an agent, Moderne will share a token with you that you must configure in the Moderne agents you create. Moderne will reject any connection attempts from unauthorized agent instances. In this way, Moderne requires a minimum level of client (agent) verification as an extra security precaution.
The connection to Moderne is established over layer 7, so you may choose to route traffic from the agent through your own layer 7 gateway. This might be chosen to satisfy a desire for Moderne’s API gateway to perform client verification of an inbound agent connection using a mechanism like X.509 in addition to token-based verification.
These measures act in concert with techniques to limit IP addressability of the Moderne API gateway to enhance the overall security posture.
Organizations service
The Organizations service is an optional service that you can configure in your environment. It defines the organizational structure that a user or a repository belongs to. The Moderne SaaS will use this information to present different UIs or options for different users and repositories. Please see our Organizations service doc for more information.
Setup requirements
You must:
Ensure that the Organizations service is capable of receiving requests from the Moderne agent
Ensure your service fulfills the GraphQL contract outlined in our Organizations service documentation
Moderne API gateway
The Moderne API gateway serves as the entry point to Moderne. It talks with the Moderne agent to get data from your services to Moderne. It is the only component with a public IP address that can communicate with other Moderne services. The Moderne UI and Keycloak also have public IP addresses, but they can't communicate with other Moderne services.
The API gateway is responsible for:
Handling API requests from your developers or your tools
Handling API requests from the Moderne UI
Handling encrypted LST artifacts from the Moderne agent(s)
Handling encrypted custom recipe artifacts from the Moderne agent(s)
Rate limiting as needed to guard Moderne services against overuse by a particular user
Authorized users in your company can access audit logs for this gateway via an API.
The Moderne API gateway is configured with a Moderne-managed SSL certificate.
Setup requirements
You must:
Ensure that
https://api.TENANT.moderne.iois on the accept list for outbound HTTPS traffic from the Moderne agentEnsure that
https://api.TENANT.moderne.iois on the accept list for outbound HTTPS traffic from the developer's machines
Moderne user interface
The Moderne UI provides a browser-based interface for:
Executing search and transformation recipes across your codebase
Issuing mass commits/PRs based on recipe runs
Building new recipes based on other recipes
Viewing audit logs
Generating access tokens for interacting with the Moderne API
The Moderne UI is implemented with client-side Javascript. The Moderne UI is one of three components with a public IP address (the other two being the Moderne API gateway and Keycloak).
Setup requirements
You must:
Ensure that
https://TENANT.moderne.iois on the accept list for outbound HTTPS traffic from the developer's machines
Keycloak
Keycloak is an open-source identity and access management system. Moderne services request authorization information for a user from Keycloak. Keycloak then calls out to your identity provider (such as LDAP, Okta, or Keycloak) to determine who is authorized for what.
Setup requirements
You must:
Ensure that
https://login.TENANT.moderne.iois on the accept list for outbound HTTPS traffic from the developer's machines
As configuring identity providers between services can be quite complex, the setup for Keycloak is usually done over a Zoom meeting with Moderne and your company.
Moderne artifact storage
The Moderne artifact storage service is responsible for receiving pre-encrypted LST artifacts and recipe JARs and storing them in a private object store depending on the cloud provider you use (Azure Blob Storage, Google Cloud Storage, or AWS S3).
The artifact storage service will also write high-level information about where to find these artifacts and when they were last updated to our relationship database (RDS) so that the Moderne workers know where to go to obtain the artifacts they need.
Setup requirements
None
GraphQL federation
The GraphQL federation is an internal Moderne microservice that composes the GraphQL schemas of individual Moderne microservices into one GraphQL schema. All GraphQL requests pass through the federation service on their way to the individual microservice that supports a piece of the GraphQL schema.
GraphQL federation uses Netflix Eureka to locate microservice instances to communicate with and to load-balance requests to those instances.
Setup requirements
None
Moderne recipe execution
You can think of the Moderne recipe execution service as a manager that helps assign work, direct people on where to go, and provide high-level information. It knows all of the recipes that can be run and it chooses which workers are responsible for which repositories.
When a new recipe command comes in (such as run X recipe on Y repositories with Z options), the recipe execution service takes that command and stores all of the details in a database table that acts as a queue. Whenever a worker is free, it will query the database table and look for commands that haven't been started (for the repositories the worker is responsible for).
The recipe execution service is also responsible for providing the results of a recipe run by either calling the database to see what high-level information it has or by calling the worker directly to get the full results (diffs and data tables).
Setup requirements
None
Moderne worker
Moderne workers are responsible for running recipes and keeping their results. They interact with the Moderne recipe execution service to coordinate which repositories they should run recipes for before querying the database for a recipe to run. When a worker is stopped, all of the data and results from the recipes it has run are destroyed. If you need that data, you'll need to re-run the recipe again.
Worker instances are scaled horizontally in direct response to more code being ingested into the platform.
Workers decrypt LST and recipe artifacts by making a request to the Moderne Agent via the API Gateway for a customer-provided symmetric key. Workers discard this key at the end of every request.
Workers fetch a user’s SCM OAuth token via the API gateway in order to make authorization decisions about which repositories said user is allowed to read from. This ensures Moderne’s read access is aligned with a user's SCM access in real-time for every recipe run request.
Setup requirements
None
Moderne source code management
The Moderne source code management service is responsible for handling all communication with your SCM(s).
The two primary responsibilities are:
Creating commits, branches, forks, PRs, etc. in your SCM
Coordinating authorization with your SCM to see what users are authorized to do or view
Please note that the requests to your SCM will appear to come from the Moderne agent. The Moderne source code management service will talk through the API gateway to the agent whenever it needs to interact with your SCM.
Authentication and authorization decisions are made in real-time to ensure that they are always up-to-date.
Setup requirements
None
Moderne audit log
The Moderne audit log retrieves audit logs from our relational database for presentation to privileged users via the API gateway.
Audit logs can be retrieved via a paginated GraphQL API or via a REST call that responds in the CEF format.
Individual Moderne microservices are responsible for contributing to the audit log database when they perform any interaction on behalf of users.
Setup requirements
None
Moderne tokens
The Moderne tokens service generates and retrieves access tokens tied to a particular user. Access tokens can be used to access the service via IDEs and custom tooling. Users can manage their access tokens via the Moderne UI. Tokens are only visible once at creation time and are hidden from that point forward, even from the user that created them.
Please see our token documentation for more information on how to create, work with, and revoke tokens.
Setup requirements
None
Last updated
