🏃 Usage

[ 📋 Synopsis | 🧰 Commands | 📝 Settings File | 🤖 GitHub Action | 🎛️ Feature Parameters | 📤 Output Formats | 🚦 Exit Status | 🔧 Tuning the Search ]

📋 Synopsis

#

pelicode <COMMAND> [OPTIONS]

Commands:
  crawl      Fetches packages from a public repository (pypi, maven, go, rubygems, crates) and indexes them
  index      Extracts the functions of every supported source file under the given directories and stores their features in the database
  stats      Shows database statistics and the hashes shared by the most packages
  clean      Removes files and functions no package version refers to any more, and shrinks the database file
  snapshot   Writes a copy of the database for checking only, to be distributed
  pull       Fetches a snapshot from an OCI registry into the cache, and prints its path
  scan       Reports stored functions similar to the functions of the given code
  extract    Extracts the functions of the given files or directories and prints their features
  export     Prints the functions stored in the database
  languages  Lists the supported languages and their file extensions

A typical session builds a database once and scans many times:

pelicode crawl pypi --top 100        # build the reference database
pelicode stats --refresh             # count common hashes
pelicode scan ./src                  # check code

Every command that uses a database takes --db <FILE>; the default is the database of the settings file, else pelicode.db in the current directory. pelicode <COMMAND> --help lists all options of a command.

🧰 Commands

#

crawl

#

Fetches packages from a public repository and indexes them.

pelicode crawl [OPTIONS] <ECOSYSTEM> [PACKAGES]...

Without a version, the latest release is fetched.

It also takes the feature parameters.

pelicode crawl pypi requests numpy@2.1.0
pelicode crawl pypi --top 100 --contact https://example.org/about
pelicode crawl maven --list maven-packages.txt --refresh-stats
pelicode crawl go github.com/google/uuid
pelicode crawl rubygems rack thor@1.3.0
pelicode crawl crates --top 100 --contact https://example.org/about
pelicode crawl pypi --updates --contact https://example.org/about   # new versions of the packages in the database

What is fetched from each repository:

Good to know:

• Archives are read in memory and discarded. Nothing is unpacked to disk, and no source code is stored.
• Versions already indexed are skipped, so an interrupted crawl can simply be run again.
• --updates keeps a database current, one request per package already in it; the time of the last check is recorded per ecosystem and shown at the next one.
• A failed package does not stop the others. Failures are recorded in the database, and the exit status is 1 if any package failed.
• Requests are sent one at a time. Be gentle with public services: keep the delay, and set --contact when crawling many packages.
• For PyPI, the ranking used by --top comes from hugovk/top-pypi-packages, not from PyPI itself; for crates.io, it is the all-time download count of crates.io. Maven Central, the Go module proxy and RubyGems publish no ranking of their own, so --top asks ecosyste.ms for them: RubyGems by downloads, Maven Central and Go, which publish no download counts, by the number of repositories that depend on a package. The aim is well-known packages, not one ranking across repositories.
• The crates.io data access policy allows at most one API request per second and strongly suggests a way to contact you. Give --contact when crawling crates.io, and keep --delay-ms at 1000 or more.

index

#

Indexes local directories (or files) as one package.

pelicode index [OPTIONS] <PATHS>...

It also takes the feature parameters.

pelicode index ./my-project --name my-project --version 1.0 --license MIT

Hidden files and, in Git repositories, files ignored by .gitignore are skipped (this applies to scan and extract too). Indexing the same name and version again replaces the files recorded for it. When several directories are given, each file path is prefixed with the name of its directory.

stats

#

Shows the size of the database, recounts how many packages contain each hash, and lists the most common ones.

pelicode stats [OPTIONS]

4 ecosystems, 5 packages, 5 versions, 373 files, 6550 functions (1654 with fingerprints)
hash statistics: 2026-10-17T08:27:18Z; 21 exact hashes and 354 fingerprints in 2+ packages

most common fingerprints (N1):
       4 packages      22 functions  7d679b3567a1201f  pypi:requests@2.34.2 tests/test_requests.py:1510-1510 test_iter_content_wraps_exceptions

The counts are not updated when packages are added. Run pelicode stats --refresh, or pass --refresh-stats to crawl or index; scan warns when the counts are out of date.

Once counted, the statistics are refreshed incrementally: only the hashes of the files added to or removed from a version since the last count are recounted, which gives the same result as recounting everything in a fraction of the time. --refresh --full recounts everything anyway. The first count, and the first after an upgrade from a database whose statistics were out of date, are always full.

clean

#

Removes the files no package version refers to any more, with their functions and features, and shrinks the database file. Such files are left behind when a version is indexed again (index of the same name and version, or crawl --force).

pelicode clean [OPTIONS]

removed 3 files and 41 functions no longer referenced
database: 221.3 MB -> 213.5 MB

The hash statistics stay valid: they count only files that versions refer to.

snapshot

#

Writes a copy of the database for checking only, to be distributed: building a reference database takes hours, so it is built in one place and others only scan against a snapshot of it.

pelicode snapshot -o <FILE> [OPTIONS]

pypi.db: 26 packages, 26 versions, 77084 functions, 165.5 MB

A snapshot keeps everything scan needs, with the hash statistics recounted for what it holds, and drops the MinHash signatures, which only indexing and export use (export shows none for a snapshot). It is a single file; nothing can be indexed into it. The database it is made from is not changed.

Snapshots are published in quay.io as OCI artifacts, compressed with zstd (to about a third), tagged <NAME>-YYYYMMDD and <NAME>-latest. With ORAS and zstd installed:

pelicode snapshot -o pypi.db --ecosystem pypi --latest --annotations pypi.db.annotations.json
oras login quay.io
just snapshot_push pypi.db pypi                 # quay.io/tama5/pelicode-db:pypi-YYYYMMDD and :pypi-latest

The annotations say what is in the snapshot and how it was built, so that it can be checked before it is downloaded: the versions of pelicode, of the table layout and of the normalizer, the feature parameters, the fingerprint levels, the ecosystems, the numbers of packages, versions and functions (keys io.github.tamada.pelicode.*), and the creation time (org.opencontainers.image.created).

Elsewhere, scan fetches it by itself (see pull):

pelicode scan ./src --db oci://quay.io/tama5/pelicode-db:pypi-latest

pull

#

Fetches a snapshot from an OCI registry into the cache and prints the path of the database. scan, stats and export do the same for a --db of the form oci://REGISTRY/REPOSITORY:TAG (or @sha256:...), so pull is needed only to fetch ahead of time or to keep a copy.

pelicode pull <REFERENCE> [-o <FILE>]

• Verified: the download is checked against the digest of the layer in the manifest, then decompressed; neither ORAS nor zstd is needed.
• Cached: snapshots are kept by digest in $PELICODE_CACHE_DIR, else the cache directory of the user (~/.cache/pelicode on Linux, ~/Library/Caches/pelicode on macOS). When a tag still points to the same digest, only the manifest is fetched.
• Offline: when the registry cannot be reached, the snapshot the tag pointed to last time is used, with a warning.
• Checked first: a snapshot whose annotations show it cannot be used, because it was made by a newer pelicode or with another normalizer, is refused before it is downloaded.
• Public repositories need no login; pelicode asks the registry for an anonymous token.

fetched oci://quay.io/tama5/pelicode-db:pypi-latest (sha256:3f1c...)
/home/user/.cache/pelicode/snapshots/sha256-3f1c....db

scan

#

Searches the database for functions similar to those of the given code.

pelicode scan [OPTIONS] [PATHS]...

pelicode scan ./src
pelicode scan --git main..HEAD                    # functions changed between main and HEAD
pelicode scan --git main...feature                # changed on feature since it left main
pelicode scan --git HEAD                          # uncommitted changes in the working tree
pelicode scan --staged                            # staged changes, for a pre-commit hook
pelicode scan ./src --exclude local:my-project    # do not report the project itself
pelicode scan ./src -f json -o report.json
pelicode scan --git origin/main...HEAD -f sarif -o pelicode.sarif   # for GitHub code scanning

scan reads the feature parameters from the database, so your code is always hashed exactly like the stored code. The search runs in stages:

1. Exact matches, comparing the hash of the whole function after N0, N1 and N2 normalization, in that order.
2. Candidates from LSH buckets, which bring together functions with similar fingerprint sets.
3. Similarity of the fingerprint sets of each candidate, after removing common fingerprints; candidates below --threshold or sharing fewer than --min-shared fingerprints are dropped.

Dependencies are left out

#

Code that matches a library the checked project depends on is expected, so scan leaves it out. The dependencies are read from Cargo.lock, go.mod, go.sum, requirements*.txt, poetry.lock, uv.lock, Pipfile.lock, Gemfile.lock, pom.xml and gradle.lockfile found in the checked directories and below, and in the directories above them up to the root of the git repository (so scan ./src reads the lock files at the root of the project).

• A candidate found only in dependencies is not reported, and scan says how many were left out, on the standard error.
• A candidate found in a dependency and elsewhere is reported, without the occurrences in the dependency.
• --no-exclude-dependencies, or exclude_dependencies: false in the settings file, reports them all; for a license audit, it also shows code copied from dependencies into the repository.

extract

#

Prints the functions found in files or directories and their features, without using a database. Useful to see what pelicode sees.

pelicode extract [OPTIONS] <PATHS>...

It also takes the feature parameters.

pelicode extract upload.py --tokens N1

upload.py:6-43	python	total_length	function_definition	tokens=202	N1=87ea5b9cd905f4c2
    N1: def $ID ( $ID ) : $ID = None $ID = $NUM if $ID ( $ID , $ID ) : $ID = $ID . $ID ( $STR ) ...

Functions marked (too small) have fewer tokens than --min-tokens and get no fingerprints.

export

#

Prints every function in the database with its package, file, location, size and exact hashes.

pelicode export [OPTIONS]

languages

#

Lists the supported languages and their file extensions.

go       go
java     java
kotlin   kt, kts
python   py
ruby     rb
rust     rs

📝 Settings File

#

Options used every time, such as the database, the thresholds and what to leave out, can be written in a settings file at the root of the repository, in YAML or JSON5:

# .pelicode.yaml
version: 1
database: data/pelicode.db        # relative to this file; or oci://..., or a list
scan:
  languages: [python, rust]
  threshold: 0.6
  stop_packages: 10%
  fail_on_match: true
  exclude:
    packages: ["local:my-project"]
    paths: ["vendor/**", "**/testdata/**"]
crawl:
  contact: https://example.org/about

// .pelicode.json5
{
  version: 1,
  database: 'data/pelicode.db',
  scan: {
    languages: ['python', 'rust'],
    threshold: 0.6,
    stop_packages: '10%',
    fail_on_match: true,
    exclude: { packages: ['local:my-project'], paths: ['vendor/**', '**/testdata/**'] },
  },
  crawl: { contact: 'https://example.org/about' },
}

Where it is found: .pelicode.yaml, .pelicode.yml, .pelicode.json5 or .pelicode.json (read as JSON5), in the current directory or the nearest directory above it, up to the root of the git repository. Several in the same directory are an error. --config <FILE> names one explicitly, and --no-config ignores settings files; both work with every command.

Precedence: an option on the command line wins over the settings file, which wins over the built-in default. --exclude and --exclude-path are added to the lists of the settings file. Flags such as --idf and --fail-on-match can only switch a setting on; to switch off one set in the file, use --no-config or edit the file.

Unknown keys are errors, so that a misspelled key does not go unnoticed; the message names the file and, for YAML, the line.

The paths matched by exclude.paths are the paths scan reports: relative to the directory given, or to the repository with --git.

🤖 GitHub Action

#

pelicode-action is a GitHub Action that checks the functions changed by a pull request or a push, shows the candidates in the pull request through GitHub code scanning, and lists them in the summary of the job.

# .github/workflows/pelicode.yaml
name: pelicode
on: pull_request

permissions:
  contents: read
  security-events: write   # to upload the results to code scanning

jobs:
  pelicode:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0       # pelicode compares the pull request with its base
      - uses: tamada/pelicode-action@v0.1.0
        with:
          database-url: https://example.org/pelicode/pelicode.db

What it does:

1. Installs pelicode: the latest release for the runner’s platform, the release given by pelicode-version, or, with pelicode-version: source, pelicode built with Cargo at pelicode-ref.
2. Downloads the database from database-url, if given.
3. Scans the changes of the pull request (base...head) or of the push (before..after); other events, and pushes creating a branch, check the whole repository. The settings file of the repository applies.
4. Uploads the SARIF log to code scanning, where each candidate is a note on the function, and writes the candidates to the summary of the job.
5. Fails the step after the upload, only with fail-on-match: true.

The outputs are candidates, the number of functions with candidates, and sarif-file.

🎛️ Feature Parameters

#

crawl, index and extract take these options. They decide how functions are hashed, so a database records the values it was built with: adding to it with different values is refused, and scan uses the recorded values automatically.

📤 Output Formats

#

Text

#

The text output of scan lists each function that has candidates, followed by its candidates:

upload.py:6-43 total_length (tokens=202)
  similar(N1)  0.72  pypi:requests@2.34.2 src/requests/utils.py:160-228 [Apache-2.0]
               lines 9-23,26-43 <-> 164-189,199-228 (containment 0.86, 62 shared fingerprints)

The kind of match is exact(LEVEL) when the whole function is identical after normalization at that level, or similar(LEVEL) when their fingerprint sets are similar. A candidate found in several packages or versions shows the earliest published one followed by (+N more).

JSON

#

scan -f json writes an array with one object per checked function that has candidates:

extract -f json writes the full features of every function, including fingerprints, MinHash signatures and LSH buckets, which is useful for research.

SARIF

#

scan -f sarif writes a SARIF 2.1.0 log, which GitHub code scanning (with github/codeql-action/upload-sarif) and other tools read:

• one result per checked function with candidates, located at the function, with the level note, since a candidate is something to review rather than a defect;
• the rule exact-match or similar-match, after the best candidate;
• a message naming the best candidate, its package, version, file, lines and license, the matched lines, and the other candidates;
• the details of every candidate in properties.candidates;
• partialFingerprints, so that code scanning keeps one alert for a function across runs even when its lines move.

File paths are written relative to the current directory, which should be the root of the repository: pelicode scan ./src reports src/upload.py, not upload.py.

CSV

#

scan -f csv writes one row per candidate with the columns path, name, start_line, end_line, token_count, match, similarity, containment, shared_fingerprints, matched_lines, candidate_ecosystem, candidate_package, candidate_version, candidate_path, candidate_name, candidate_start_line, candidate_end_line, candidate_lines, candidate_license and occurrence_count.

extract -f csv and export -f csv write one row per function, with its location, size and the exact hashes n0 to n3.

🚦 Exit Status

#

🔧 Tuning the Search

#

• Fewer missed copies: lower --threshold (for example to 0.5) to catch code that was edited more heavily.
• Fewer false alarms from boilerplate: keep the hash counts up to date with stats --refresh, and lower --stop-packages. The default of 100 packages suits a large database; with a few dozen packages, give a percentage such as --stop-packages 40%.
• Short functions: functions under 50 tokens are not checked, because short code matches by chance too often. Lowering --min-tokens finds more, at the cost of more noise.
• Your own code: exclude your own package with --exclude local:NAME when it is in the database.
• Changing feature parameters such as --kgram requires a new database. Keep one database per set of parameters.