Version: v1

Database storage

Your site's database reaches a developer's machine and CI in 1 of 3 ways. The fetch source in VORTEX_FETCH_DB_SOURCE picks the way. This page shows how fetching, refreshing and pushing the database work in each case. It also covers setting up an FTP server or an S3 bucket as a dump store, so developers get a sanitized dump without access to your hosting.

The page covers these commands:

ahoy fetch-db (vortex-fetch-db) downloads a dump into .data
vortex-refresh-db provisions a site from the fetched dump in CI and exports it again
vortex-push-db uploads the exported dump to a dump store on the nightly CI run

Fetch from the source

With the url, ftp, acquia, lagoon or s3 source, every environment downloads the dump from that source. Developers need credentials for it.

  Developer                          CI (every run with no cached dump)
  ─────────                          ──────────────────────────────────
  ahoy fetch-db                      Fetch DB
       │                                  │
       ▼                                  ▼
  .data/db.sql  ◀── source ──▶       .data/db.sql
       │                                  │
       ▼                                  ▼
  ahoy provision                     Export DB (vortex-refresh-db)
                                          │  provisions the dump and exports
                                          │  it as fetched
                                          ▼
                                     cache ──▶ build job

Fetch a database image

With the container_registry source, the database lives in a container image. Developers and CI pull the image. The nightly CI run rebuilds the image from a dump file and pushes it back. See Dump file source for a database image.

  Nightly CI
  ──────────
  Fetch DB ── VORTEX_REFRESH_DB_SOURCE (for example, lagoon)
       │
       ▼
  Export DB (vortex-refresh-db)
       │  imports the dump into a clean base image
       ▼
  Deploy DB image (vortex-push-db-image) ──▶ container registry

Fetch from a dump store

A dump store is an ftp or s3 source with an upstream source set in VORTEX_REFRESH_DB_SOURCE, such as lagoon. Developers and branch builds fetch from the store and need no hosting access. The nightly CI run refreshes the store from the upstream source:

  Nightly CI (VORTEX_PUSH_DB_PROCEED=1)
  ─────────────────────────────────────
  Fetch DB
       │  fetches from VORTEX_REFRESH_DB_SOURCE in place of the store
       ▼
  Export DB (vortex-refresh-db)
       │  provisions the dump, sanitizes it and exports it
       ▼
  Push DB (vortex-push-db)
       │  1. replaces db.sql with the new dump
       │  2. with VORTEX_PUSH_DB_TIMESTAMPED_COPY=1, adds a timestamped copy
       ▼
  FTP server or S3 bucket
  ├── db.sql                    ◀── ahoy fetch-db, branch builds
  ├── db_20261110-180000.sql    timestamped copies, opt-in
  └── db_20261111-180000.sql

db.sql always holds the latest dump, and fetching downloads it. With a file suffix, the latest dump is the suffixed file, such as db_gha.sql (see Tell pushes apart). By default, it's the only file a push creates. To keep a history of the pushed dumps, set VORTEX_PUSH_DB_TIMESTAMPED_COPY=1 in .env. Each push then also creates a copy named with its UTC time, identical to db.sql at that moment. Nothing removes the copies, so clear out the old ones yourself.

Every other CI run fetches from the store, so only the nightly run needs the credentials of the upstream source.

The pushed dump leaves your hosting and CI, so it's always sanitized. The nightly run stops with an error when VORTEX_PROVISION_SANITIZE_DB_SKIP is 1.

Multiple databases

The migration database pushes its own dump file to its own fetch source. Its files sit next to the primary dump in the same store:

  FTP server or S3 bucket
  ├── db.sql                     primary database (VORTEX_FETCH_DB_FTP_FILE)
  ├── db_20261111-180000.sql
  ├── db2.sql                    migration database (VORTEX_FETCH_DB2_FTP_FILE)
  └── db2_20261111-180000.sql

To make the migration source a dump store, set VORTEX_REFRESH_DB2_SOURCE to its upstream source. The Push migration DB step pushes the migration dump as fetched, because Vortex sanitizes only the Drupal database.

Set up an FTP dump store

Before you begin, create an FTP account that can write to a directory on the server. The push uploads the dump under a temporary name, then renames it into place, so a failed upload keeps the latest dump. The server must allow the DELE, RNFR and RNTO commands, which standard FTP servers do.

In .env, set the fetch source and the server:

VORTEX_FETCH_DB_SOURCE=ftp
VORTEX_FETCH_DB_FTP_HOST=ftp.example.com
VORTEX_FETCH_DB_FTP_PORT=21
VORTEX_FETCH_DB_FTP_FILE=your_site/db.sql

In .env, uncomment VORTEX_REFRESH_DB_SOURCE and set it to your upstream source, for example lagoon. Add the variables and secrets of that source from the table in Dump file source for a database image.
Add VORTEX_FETCH_DB_FTP_USER and VORTEX_FETCH_DB_FTP_PASS as CI secrets. GitHub Actions passes them to the database job. CircleCI reads them from the project environment variables.
Ask each developer to set VORTEX_FETCH_DB_FTP_USER and VORTEX_FETCH_DB_FTP_PASS in .env.local.
After the next nightly run, confirm that the Push DB step ends with Finished database push. and that the server holds db.sql.

To give CI an account with write access and developers a read-only one, set the CI account in VORTEX_PUSH_DB_FTP_USER and VORTEX_PUSH_DB_FTP_PASS. On GitHub Actions, map those 2 secrets in the env block of the database job.

FTP sends the credentials and the dump unencrypted. Use a server on a network you trust, or the S3 store.

Set up an S3 dump store

Before you begin, create a private S3 bucket with public access blocked.

Create an IAM user for CI with this policy. Replace your-bucket with your bucket name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::your-bucket/*"
    }
  ]
}

In .env, set the fetch source and the bucket:

VORTEX_FETCH_DB_SOURCE=s3
VORTEX_FETCH_DB_S3_BUCKET=your-bucket
VORTEX_FETCH_DB_S3_REGION=ap-southeast-2
VORTEX_FETCH_DB_S3_PREFIX=your_site

In .env, uncomment VORTEX_REFRESH_DB_SOURCE and set it to your upstream source, and add the variables and secrets of that source.
Add the access key of the IAM user as the VORTEX_FETCH_DB_S3_ACCESS_KEY and VORTEX_FETCH_DB_S3_SECRET_KEY CI secrets.
Create a second IAM user for developers with s3:GetObject only. Ask each developer to set its keys in .env.local.
After the next nightly run, confirm that the Push DB step ends with Finished database push. and that the bucket holds db.sql.

S3 makes the timestamped copy on its side, so the dump is uploaded once. S3 copies a dump of up to 5 GB in 1 request, so the timestamped copy fails for a larger dump. On FTP, the timestamped copy is a second upload.

Tell pushes apart

VORTEX_PUSH_DB_FILE_SUFFIX adds a suffix before the extension of the pushed files, such as db_gha.sql and db_gha_20261111-180000.sql. Use a different suffix in each place that pushes to the same store.

The push takes its remote filename from the fetch, then adds the suffix. To fetch a suffixed dump from an FTP store, set both filenames, so the suffix appears once:

# The push adds the suffix to this name.
VORTEX_PUSH_DB_FTP_FILE=your_site/db.sql
# The fetch reads the suffixed name.
VORTEX_FETCH_DB_FTP_FILE=your_site/db_gha.sql

The S3 fetch names the local dump after the remote file. To fetch a suffixed dump from an S3 store, rename the dump everywhere with VORTEX_DB_FILE, which the fetch, the refresh and the import all read. Then set the push remote filename to the name without the suffix:

# The fetch, the refresh and the import use the suffixed name.
VORTEX_DB_FILE=db_gha.sql
# The push adds the suffix to this name.
VORTEX_PUSH_DB_S3_REMOTE_FILE=db.sql

What's next

Variables reference lists every VORTEX_PUSH_DB_ variable
Provisioning covers importing the fetched dump
Continuous integration covers the database job and its caching

Fetch from the source​

Fetch a database image​

Fetch from a dump store​

Multiple databases​

Set up an FTP dump store​

Set up an S3 dump store​

Tell pushes apart​

What's next​