Configure Quiver

Quiver is a core component of GoodData.CN responsible for caching analytical data and managing durable storage for both cached datasets and raw exports.

You configure Quiver exclusively through Helm values during deployment; Helm generates all internal configuration files automatically. This article walks you through the basic process.

Prerequisites

Before configuring Quiver, ensure you have:

• A running Kubernetes cluster (EKS, AKS, or similar)
• Access to your object storage service (AWS S3 or Azure Blob Storage)
• IAM role, IRSA, or Managed Identity configured for authentication
• A prepared bucket or container for durable storage and raw exports

Quick Start

The following example shows a minimal Helm configuration for Quiver using AWS S3 as durable storage:

quiver:
  durableStorageType: "S3"
  s3DurableStorage:
    s3Bucket: "my-quiver-bucket"
    s3BucketPrefix: "quiver/"
    s3Region: "us-east-1"
    authType: "aws_default"  # for IRSA or instance profile

For Azure Blob Storage:

quiver:
  durableStorageType: "ABS"
  absDurableStorage:
    absAccountName: "myaccount"
    absContainer: "my-quiver-container"
    absContainerPrefix: "quiver/"
    authType: "azure_default"  # for Managed Identity

After applying the Helm chart, Helm will generate the corresponding internal configuration for Quiver.

Select Storage Type

Quiver supports the following durable storage backends:

Only one durable storage type is active at a time. Setting durableStorageType automatically determines which configuration block (s3DurableStorage, absDurableStorage, or fsDurableStorage) is used. Changing this value requires an ETCD wipe to refresh configuration.

AWS S3

When using S3 storage, configure the following Helm values:

quiver:
  durableStorageType: "S3"
  s3DurableStorage:
    s3Bucket: "my-quiver-bucket"
    s3BucketPrefix: "quiver/"
    s3Region: "us-east-1"
    authType: "aws_default"     # or "aws_tokens"
    s3AccessKey: ""             # optional when using aws_default
    s3SecretKey: ""             # optional when using aws_default

Authentication Methods

• aws_default

  Uses the default AWS credentials chain (ideal for IRSA on EKS or EC2 instance profiles).

• aws_tokens

  Uses static access keys defined through s3AccessKey and s3SecretKey.
  You can also define them as environment variables:

  export AWS_ACCESS_KEY_ID=...
  export AWS_SECRET_ACCESS_KEY=...
  

An example using static credentials:

quiver:
  durableStorageType: "S3"
  s3DurableStorage:
    s3Bucket: "gooddata-quiver"
    s3BucketPrefix: "data/"
    s3Region: "us-east-1"
    authType: "aws_tokens"
    s3AccessKey: "AKIA..."
    s3SecretKey: "..."

Azure Blob Storage

When using Azure Blob Storage:

quiver:
  durableStorageType: "ABS"
  absDurableStorage:
    absAccountName: "myaccount"
    absContainer: "quiver-container"
    absContainerPrefix: "data/"
    authType: "azure_default"  # or "azure_tokens"
    absAccountKey: ""          # required only for azure_tokens
    absClientId: ""            # optional for user-assigned managed identity

Authentication Methods

• azure_default

  Uses Kubernetes-provided Managed Identity (recommended).

• azure_tokens

  Uses static account keys. Store them in a Kubernetes secret and reference them via environment variables.

Prefixes

Quiver uses prefixes (for example s3BucketPrefix or absContainerPrefix) to organize data within your storage backend.

• Prefixes separate datasets stored in the same bucket or container.
• A missing prefix can lead to collisions between different environments.
• You can use subprefixes such as "prod/quiver/", "dev/raw/", or "exports/" to isolate workloads.

Best practice is to use a dedicated bucket or container for each environment (for example dev or prod) and consistent prefixes for Quiver and the export-controller.

Integration with Export Controller

Quiver and the export-controller share the same durable storage for raw exports.

In Helm values, this is configured under the exportController section:

exportController:
  storageType: "s3"
  exportS3Storage:
    s3Bucket: "my-quiver-bucket"
    s3BucketPrefix: "raw-exports/"

Ensure that:

1. The export-controller and Quiver use the same bucket and prefix for raw exports.
2. The bucket/container exists before deployment (neither component creates it automatically).
3. The prefix defined under rawExportsFs.prefix (default: raw-exports) matches the one in exportController.

If the prefixes do not match, exports may appear as successful but will not be accessible through Quiver.

Operational Parameters

You can fine-tune Quiver’s behavior with these optional Helm values:

If you increase concurrency (for example concurrentPutRequests), also increase the memory resources for Quiver pods.

Common Deployment Patterns

S3 with IRSA (Recommended on EKS):

quiver:
  durableStorageType: "S3"
  s3DurableStorage:
    s3Bucket: "gooddata-prod"
    s3BucketPrefix: "quiver/"
    s3Region: "us-east-1"
    authType: "aws_default"

Azure Blob Storage with Managed Identity (recommended on AKS):

quiver:
  durableStorageType: "ABS"
  absDurableStorage:
    absAccountName: "gdstorage"
    absContainer: "gooddata-quiver"
    absContainerPrefix: "quiver/"
    authType: "azure_default"

Local Filesystem (for testing only):

quiver:
  durableStorageType: "FS"
  fsDurableStorage:
    storageClassName: "nfs-rwx"

Secrets Management

Avoid storing credentials directly in Helm values. Instead, use:

• IRSA (AWS) or Managed Identity (Azure) for production
• Kubernetes Secrets referenced in environment variables for testing

Example (AWS keys in Kubernetes Secret):

env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: aws-credentials
        key: accessKey
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: aws-credentials
        key: secretKey

Troubleshooting