Secrets Detection Engine

Philosophy

Secrets detection is probabilistic: some secrets are easier to find than others. There is a trade-off between low number of false alerts and low number of missed credentials (precision/recall trade-off)

Our secrets detection engine has been running in production since 2017, analyzing billions of commits coming from GitHub. Since day one we began to train and benchmark our algorithms against the open source code. It allowed GitGuardian to build a language agnostic secrets detection engine, integrating new secrets or new way of declaring secrets really fast while keeping a really low number of false positives. We are also collecting feedback from the alerts we are sending including the pro bono alerts:

• Explicit feedback when a developer or security team marks an alert as a false alert.
• Implicit feedback when a developer takes down a public repository or deletes a public commit a few minutes after we sent an alert.

We are currently implementing two types of detectors:

• Specific detectors: we implement an algorithm dedicated to looking for one specific type of secret like AWS keys, Postgres URI or SMTP credentials. This type of detector aims at having high recall and precision for the targeted type of secret. The downside of this approach is that we have to implement a lot of specific detectors to have a good coverage of all existing secrets.

• Generic detectors: The purpose of these detectors is to catch what our specific detectors are missing with a generic approach. We will implement generic detectors capturing patterns such as secret=\{high_entropy_string\} or password=xxx, email=yyy@corp.com. We can thereby achieve a low number of secrets missed even if the precision of these detectors is slightly lower than the specific ones.

You can of course choose what detectors you want to allow or deny in our different web applications.

How it works

Our secrets detection engine takes as input a Document having as parameter a string (a Git Patch, a GitHub gist, a Slack message) and an optional parameter which is the filename (secrets.py, index.html, ...) Note that this detection engine can use the filename when available in a pre-validation step: for example we won't scan an image or a video file since nobody puts secrets in it.

We currently have the following ban-lists in place:

1. We do not scan binary files such as jpg, tar.*, ...
2. We exclude some filepaths using the following regexes:

node_modules(/|\\)
vendors?(/|\\)
top-1000\.txt$
\.sops$
\.sops\.yaml$

3. In most cases, we do not scan documents with the following extensions because they bring a lot of false positives and almost no real secrets: ["html", "css", "md", "lock", "storyboard", "xib"]

Tao of our secrets detection engine

• 🎯 High precision: We want to keep a low number of false positives to avoid alert fatigue.
• 🔐 High recall: We want to keep a low number of missed secrets to keep our customers safe.
• ⚡ Speed: While speed is less important than recall and precision our secrets detection engine is designed to be fast and scan a common Git repository history in less than a minute.
• 👥 Community and customer driven: Our engine is constantly trained and improved by feedback from hundreds of thousands of developers using our applications and by feedback from our customers.

Main features

Broad coverage with 650+ detectors

We have developed a library of 670 specific and generic detectors. See the list of supported credentials.

Detecting secrets with multiple matches and multi-line secrets

GitGuardian supports the detection of multi-matches secrets such as oauth client_id and client_secret, database credentials, SMTP credentials...

GitGuardian supports the detection of multi-line secrets such as private keys.

For example we are able to detect AWS keys from the following snippet and the output will be two matches: one for the client id and one for the client secret.

input: 'id=AKIAFJKR45SAWSZ5XDF3, client_secret: hjshnk5ex5u34565d4654HJKGjhz545d89sjkjka'
output:
  client_id: AKIAFJKR45SAWSZ5XDF3
  client_secret: hjshnk5ex5u34565d4654HJKGjhz545d89sjkjka

Example for database credentials:

input: |
  dbusername = admin
  dbpassword = 8095uohoiw4ur90
  dbhost = db-postgres-nyc1-1111-do-user-111111-0.db.ondigitalocean.com
  dbport = 25060
  dbdatabase = defaultdb
  dbsslmode = require
output:
  username: admin
  password: 8095uohoiw4ur90
  host: db-postgres-nyc1-1111-do-user-111111-0.db.ondigitalocean.com
  port: 25060

Automatic de-duplication

• If two detectors match the same characters and have the same matches we drop the output of the detectors bringing less information.

For example the following content slack_token="xoxp-198947049743-7861195093-830655328819-9d40a979cac97bccf1190afb660b37e1" triggers two detectors: our slack_user_token detector and our detector catching generic secrets (token={high_entropy_string}). We only keep the results of the slack_user_token detector in order to reduce alert fatigue and attach the maximum information to help with the remediation process.

Detecting prefixed Base64 encoded secrets

Recent frameworks like Kubernetes use Base64-encoded secrets and it is becoming pretty common to see Base64-encoded secrets. We are able to detect prefixed secrets encoded in Base64 like private keys.

How GitGuardian Filters False Positives

Before GitGuardian reports a secret, every candidate match passes through a series of filters (called PostValidators) that check its value and the code around it. Each detector combines several of them, tuned to how that provider's real credentials look, so placeholders, test data and code that merely resembles a key are not raised as incidents.

Filter families

The filters fall into seven families. Each detector uses the combination that fits its format.

Filter family	What it checks	False positives it removes
Randomness and character mix	Whether the value is as random as a real credential, and whether it has the expected mix of letters and digits	Repeated characters, simple sequences, and variable or function names that happen to fit a token's format
Natural-language detection	Whether the value reads like words rather than random characters, using a large dictionary and statistical language models	Values such as MySuperSecretPassword or default_api_key_value
Placeholder and test values	The value against curated lists of known placeholder, sample and test values	your_api_key, changeme, xxxx, ****, example, test, 12345, and well-known sample keys from vendor documentation
Code and template references	Whether the "value" is code that points to a secret rather than the secret itself	Environment variable references (${API_KEY}, process.env.TOKEN), template placeholders, function calls, file paths, URLs, dates and color codes
Surrounding context	The variable name and nearby text, compared with signals that the value is not a credential for this provider	Hashes, salts, signatures and IDs, values labelled as invalid or example, and credentials for a different service with a similar format
Provider confirmation	For formats that are not unique to one provider, whether the provider's name appears near the value	Generic-looking strings, such as UUIDs or hex values, that would otherwise be attributed to a provider wrongly
Structural validation	Built-in checksums and the parts of multi-part credentials (username, password, host)	Mistyped or invented tokens that fail the provider's checksum, connection strings to localhost, private IPs or example domains, admin:admin-style pairs, and encrypted or hashed values that cannot be used

How filters combine

The less distinctive a secret's format, the more filters its detector applies.

• Provider-specific keys with a recognizable prefix or structure mostly rely on randomness checks, plus checksum validation where the provider builds one into its tokens. The format alone already rules out most false positives.
• Provider keys without a distinctive format add the natural-language, context and provider-confirmation filters, so a random string is only reported when the code around it points to that provider.
• Credential pairs and connection strings apply separate checks to each part: placeholder lists on the username and password, non-routable or example hosts on the host, and a check that the username and password are not near-identical.
• Generic detectors (passwords and secrets with no provider-specific format) use the widest set: natural-language detection, code and template filters, placeholder lists, and rejection of encrypted values.

GitGuardian tunes these filters continuously against large volumes of real-world code, so detection stays precise as new formats and placeholder conventions appear.

Add insights to secrets to ease the work of security engineers and analysts

We attach insights to secrets so the application security analysts can have more context information for the remediation. Currently we support the following insights:

• test_file: We found the secret in a test context (test folder, test filename).
• sensitive_file: We found the secret in a document with a filename considered as sensitive (.env, credentials.json, ...).
• whitelisted: The secret has been white-listed by the developer that wrote the code.
• decoded value for encoded secret: We can attach the decoded secret value to encoded secret.

Limitations

1. We are not able to scan binary files (for now). Some binary filenames can contain secrets, such as .pyc files in Python for example.
2. By default, the number of secrets detected per detector and per file is capped at 8. For high-recall detectors that typically return a single match per document, this limit is increased to 50. These thresholds help prevent alert fatigue from noisy documents, while still handling files that legitimately contain many secrets (such as a file with dozens of Slack webhooks). We always recommend reviewing the full document when a secret is detected, as it may contain additional sensitive information that wasn't surfaced.

Architecture

Our secret detection engine is based on the following logic:

We use PreValidators to remove some filenames bringing false positives or to select a document based on the presence of a keyword. For example every SendGrid API key must start with the prefix SG., we can discard all documents that don't contain this prefix. You can find more information about pre-validators here.

Our Scanner is a collection of detectors that scans a Document and yields Secret candidates.

We then use PostValidators to validate if our secret candidates are real secrets. We configure PostValidators for each Detector so we can achieve the best trade-off between recall and precision. Post-validation rules filter out false positives by:

• Filtering example keys
• Filtering secrets with low entropy
• Filtering secrets with substrings from English dictionary
• Machine learning models to exclude false secrets based on context

You can find more information about all the PostValidators we currently use and support here.