Scaling

info

Looking for the GitGuardian legacy architecture Scaling page? Please visit the Scaling (legacy) page.

For information on the new architecture, as well as determining whether you are using the new or the legacy GitGuardian architecture, explore the New GitGuardian Architecture page.

Application topology

GitGuardian application consists of several Kubernetes resources. Here are key aspects based on the installation type:

KOTS-based Installations: Allows configuration of replicas, CPU, and memory requests/limits for main deployments/pods.

Helm-based Installations: Provides more comprehensive customization, including:

• Creation of new classes of workers.
• Customization of other resource types such as ephemeral storage, huge pages, etc.
• Setting nodeSelector, tolerations, and additional configurations.

For detailed insights into deployment/pod names, types, and their usage, visit the GitGuardian Application Topology page.

Configure scaling settings

KOTS-based installation

Navigate under Config > Advanced Options in the KOTS Admin Console, you will have access to the worker scaling options.

• Frontend: Scale nginx workers.
• Worker replicas: Scale worker workers.
• Scanning worker replicas: Scale scanners workers.
• Email worker replicas: Scale email workers.

Note: Changing these values doesn't affect the rollout upgrade strategy.

Note: Workers are configured to spread across nodes if there are multiple nodes. If you have configured your cluster for high availability, do not use less than 2 workers of each type.

Helm-based installation

Customize Helm applications using your local-values.yaml file, submitted with the helm command.

Configure deployments with replicas: use webapps.[name].replicas for web pods and celeryWorkers.[name].replicas for async workers. Additionally, set resources requests and limits as needed.

Example

migration:
  # Set resources for pre-deploy and post-deploy jobs
  resources:
    limits:
      cpu: 1000m
      memory: 500Mi
front:
  nginx:
    # Set resources for nginx init containers
    init:
      resources:
        limits:
          cpu: 1000m
          memory: 500Mi
    replicas: 2
    resources:
      limits:
        memory: 1Gi
webapps:
  public_api:
    replicas: 5
    resources:
      requests:
        cpu: 200m
        memory: 500Mi
      limits:
        memory: 4Gi
celeryWorkers:
  scanner:
    replicas: 8
    resources:
      requests:
        cpu: 200m
        memory: 4Gi
      limits:
        memory: 16Gi

See the values reference documentation for further details.

Scaling Recommendation

For optimal performance, consider scaling the following pods to a minimum of 2 replicas each: webapp-hook, webapp-internal-api-long, webapp-public-api, worker-email, worker-long-tasks, and worker-worker.

Scaling for historical scans of repositories up to 15GB in size

Performing your first Historical Scans

When you add a high number of sources, consider temporarily increasing the number pods for the time of the initial historical scan. Afterward, you can decrease those pods' replicas and resources.

When performing a historical scan, GitGuardian clones the git repository on the pod's ephemeral storage and will traverse all branches and commits in search of potential Policy Breaks. The full scan is done by a single Pod and can last from a few seconds to many hours depending on the repository size.

The more pods you'll add, the more historical scans can be done concurrently. When sizing your nodes, keep in mind that each Pod must have enough ephemeral storage and memory to run.

The following sizing has been tested for 7000 repositories up to 15GB with 16 pods:

Component	Required Capacity	Count
Compute nodes	8 vCPU 64 GB RAM 50GB ephemeral disk space, 500 GB persistent disk space	6
PostgreSQL Master	16 vCPU 64 GB memory 300 GB disk space	1
PostgreSQL Read Replica	8 vCPU 32 GB memory 300 GB disk space	1
worker-scanners Pods	Memory request and limit: 16GB	16

Additional tuning ephemeral storage

In certain scenarios, optimizing ephemeral storage configurations becomes essential for achieving better performance and stability, particularly in Helm-based installations for scanners workers. This section outlines additional configurations for fine-tuning ephemeral storage, focusing on leveraging "On Demand" nodes with nvme disks and integrating Generic Ephemeral Inline Volumes.

"On Demand" nodes with nvme disks

In the following example, we specify that scanners workers only use "On Demand" VMs with nvme disks and that pods' ephemeral storage will use these disks

celeryWorkers:
  scanners:
    replicas: 16
    localStoragePath: /nvme/disk # Used for pods ephemeral storage
    nodeSelector: # Must run on "On Demand" nodes with nvme disks
      eks.amazonaws.com/capacityType: ON_DEMAND
      local-nvme-ready: 'true'
    tolerations:
      - key: worker-highdisk
        operator: Equal
        value: 'true'
        effect: NoSchedule
    resources:
      requests:
        cpu: 200m
        memory: 16Gi
      limits:
        memory: 16Gi

Generic Ephemeral Inline Volumes

In the following example, we leverage Kubernetes' Generic Ephemeral Inline Volumes within Helm charts. This feature facilitates dynamic provisioning and reclamation of storage, particularly beneficial when dealing with small limits on Ephemeral storage. Note that it's supported starting Kubernetes 1.23. Learn more.

celeryWorkers:
  scanner:
    replicas: 8
    resources:
      requests:
        cpu: 200m
        memory: 4Gi
      limits:
        memory: 16Gi
    # -- Worker ephemeral storage
    ephemeralStorage:
      enabled: true
      size: 2Gi

Scaling real-time scans

Real-time scans are triggered by Push events sent by the VCS to GitGuardian. Those scan duration is often under a second and should always be under 3 seconds. But to be able to handle peaks of pushes, you may want to increase the count of worker Pods that are processing real-time scans.

We successfully tested peaks of 2000 pushes per hour with 8 worker Pods replicas, without changing default resources settings.

Was this page helpful?