📖 Glossary

The terms below are the ones the commands, their --help and this site use. The order follows the work: code is collected, its functions are normalized and hashed, hashes are counted across packages, code is scanned against them, and databases are shared as snapshots.

📦 Collecting

#

Ecosystem

#

A public package repository: pypi, maven (Maven Central), go (the Go module proxy), rubygems or crates (crates.io). Projects registered with pelicode index belong to the ecosystem local unless another one is given.

Package

#

A named unit of an ecosystem: a PyPI project, a Maven groupId:artifactId, a Go module path, a gem or a crate. Hash counts are per package, so the versions of one package count once.

Version

#

One release of a package. crawl indexes one version per request, the latest unless a version is given.

Occurrence

#

A place where a stored function was found: a file of a version of a package, with its license, publication date and download URL. The same file in several versions is stored once and has several occurrences.

Reference database

#

The SQLite file, pelicode.db by default, holding the features of every indexed function together with its occurrences. It never holds source code.

Unreferenced file

#

A stored file that no version links to any more, left behind when a version is indexed again with other files. It never appears in results, but takes space; pelicode clean removes it with its functions.

🧩 Normalizing

#

Function

#

The unit pelicode compares: a function, method, constructor, lambda or block, extracted with Tree-sitter. Nested functions are extracted as well as the function around them. Functions with syntax errors are skipped.

Token

#

One lexical element of a function after comments and layout are removed: a keyword, an operator, an identifier or a literal. The size of a function is its number of tokens.

Normalization level

#

How much a function is abstracted before it is hashed:

Docstrings of Python functions are dropped at every level.

Clone types

#

The usual classification of similar code: Type-1 differs only in layout and comments, Type-2 also in identifiers and literals, Type-3 also by added, removed or changed statements, and Type-4 is equivalent in behavior with different syntax. pelicode targets types 1 to 3.

#️⃣ Hashing

#

Exact hash

#

A 128-bit hash of the whole token sequence of a function at one normalization level. Two functions with the same exact hash at N1 are identical except for names and literals.

k-gram

#

\(k\) consecutive tokens, 10 by default (--kgram). Every k-gram of a function is hashed.

Winnowing

#

The method that selects a subset of the k-gram hashes: in every window of \(w\) consecutive hashes (--window, 4 by default), the smallest is kept. Two functions that share a run of at least \(w + k - 1\) tokens are guaranteed to share a selected hash.

Fingerprint

#

A k-gram hash selected by winnowing, together with the source lines it covers. The fingerprints of a function form its fingerprint set, which is what similar matches compare.

MinHash

#

A fixed-length signature of a fingerprint set (--minhash-size, 128 by default) whose agreement with another signature estimates the Jaccard similarity of the two sets.

LSH

#

Locality-sensitive hashing: the MinHash signature is cut into bands (--bands, 32 by default), and each band is hashed into a bucket. Functions sharing a bucket become candidates, so that only a few stored functions need a full comparison.

📊 Counting

#

Package count

#

The number of packages containing a hash. Also called the document frequency (df) of the hash. Counted by pelicode stats --refresh.

Common hash

#

A hash found in many packages: getters, equals methods, small idioms. It is no evidence of copying.

Stop threshold

#

The package count from which a hash is ignored during a scan (--stop-packages): a number of packages, or a percentage of all packages.

IDF

#

Inverse document frequency: with --idf, each fingerprint is weighted by \(\ln(1 + N / \mathit{df})\), where \(N\) is the number of packages, so that rare fingerprints count more than common ones.

Stale statistics

#

Hash counts made before the last package was indexed. scan warns about them; pelicode stats --refresh brings them up to date.

Pending hashes

#

The hashes whose package counts may have changed since the last count: those of the files linked to or unlinked from a version since then. pelicode stats --refresh recounts only them (an incremental refresh); --full recounts everything.

🔍 Scanning

#

Candidate

#

A stored function reported as similar to a checked function: something for a person to review, not a verdict.

Exact match

#

A candidate whose exact hash equals that of the checked function at N0, N1 or N2. Reported as exact(N1) and so on.

Similar match

#

A candidate whose fingerprint set is similar enough to that of the checked function. Reported as similar(N1) and so on.

Similarity

#

For a similar match, the Jaccard similarity of the two fingerprint sets after common fingerprints are removed: the size of their intersection divided by the size of their union, from 0 to 1. With --idf, the sizes are sums of weights. Exact matches have similarity 1.

Containment

#

The share of the checked function’s fingerprints that are found in the candidate. A high containment with a lower similarity means the checked function is mostly inside a larger candidate.

Shared fingerprints

#

The fingerprints the two functions have in common, excluding common ones. A similar match needs at least --min-shared of them, 5 by default, so that one or two coincidences cannot make a match.

Matched lines

#

The line ranges covered by shared fingerprints, in the checked function and in the candidate (lines 9-23 <-> 164-189).

Dependencies

#

The packages the checked project depends on, read from its lock files (Cargo.lock, go.sum, requirements.txt, poetry.lock, Gemfile.lock, pom.xml and others). Code found only in them is expected, so it is not reported; scan says how many candidates were left out this way, and --no-exclude-dependencies shows them.

Settings file

#

.pelicode.yaml, .pelicode.yml, .pelicode.json5 or .pelicode.json, found in the current directory or above it, up to the root of the git repository. It holds the options of scan and crawl and the database, so that they can be committed with the project. Options on the command line win over it.

SARIF

#

The Static Analysis Results Interchange Format, a JSON format for the results of analysis tools (scan -f sarif). Each function with candidates is one result, at the level note, with the rule exact-match or similar-match.

Code scanning

#

The GitHub feature that shows SARIF results as notes on the lines of a pull request and lists them in the Security tab of the repository.

📤 Sharing

#

Snapshot

#

A copy of a reference database for checking only, written by pelicode snapshot: it keeps what scan needs and drops what only indexing uses. Nothing can be indexed into it. Snapshots are meant to be built in one place and checked against everywhere.

OCI artifact

#

A file stored in a container registry (such as quay.io or ghcr.io) like an image, under a repository and tags. Snapshots are published as OCI artifacts, compressed with zstd.

Snapshot reference

#

Where a published snapshot is, as oci://REGISTRY/REPOSITORY:TAG or oci://REGISTRY/REPOSITORY@sha256:.... It can be given wherever a database is (--db, database in the settings file).

Snapshot cache

#

The directory where fetched snapshots are kept by digest: $PELICODE_CACHE_DIR, else the cache directory of the user. A tag that still points to the same digest is not downloaded again, and a cached snapshot is used when the registry cannot be reached.