Skip to main content

Estimate comprehension token usage

io.moderne.prethink.ComprehendCodeTokenCounter

Estimate the input token counts that would be sent to an LLM for method comprehension, without actually calling a model. Uses OpenAI's tokenizer locally. Outputs to the MethodDescriptions table with blank descriptions.

Recipe source​

GitHub: ComprehendCodeTokenCounter.java, Issue Tracker, Maven Central

This recipe is available under the Moderne Source Available License.

Used by​

This recipe is used as part of the following composite recipes:

Usage​

This recipe has no required configuration options. It can be activated by adding a dependency on org.openrewrite.recipe:rewrite-prethink in your build file or by running a shell command (in which case no build changes are needed):

  1. Add the following to your build.gradle file:
build.gradle
plugins {
    id("org.openrewrite.rewrite") version("latest.release")
}

rewrite {
    activeRecipe("io.moderne.prethink.ComprehendCodeTokenCounter")
    setExportDatatables(true)
}

repositories {
    mavenCentral()
}

dependencies {
    rewrite("org.openrewrite.recipe:rewrite-prethink:0.2.2")
}
  1. Run gradle rewriteRun to run the recipe.

See how this recipe works across multiple open-source repositories​

Run this recipe on OSS repos at scale with the Moderne SaaS.

The community edition of the Moderne platform enables you to easily run recipes across thousands of open-source repositories.

Please contact Moderne for more information about safely running the recipes on your own codebase in a private SaaS.

Data Tables​

Method descriptions​

io.moderne.prethink.table.MethodDescriptions

AI-generated descriptions of methods in the codebase with inference time and token usage metrics.

Column NameDescription
Source pathThe path to the source file containing the method.
Class nameThe fully qualified name of the class containing the method.
SignatureThe method signature including parameter types.
ChecksumSHA-256 checksum of the method source code for cache validation.
DescriptionAI-generated description of what the method does.
Return value descriptionAI-generated description of what the method returns.
Technique 1First programming technique or pattern used in the method.
Technique 2Second programming technique or pattern used in the method.
Technique 3Third programming technique or pattern used in the method.
Inference time (ms)Time taken for the LLM to generate the description, in milliseconds.
Input tokensNumber of tokens in the input prompt sent to the LLM.
Output tokensNumber of tokens in the response generated by the LLM.