go-storage/README.md

Go Storage

A streaming storage abstraction for Go backup tools.

Supported backends:

• Local filesystem
• S3 and S3-compatible services (MinIO, Ceph, Wasabi, ...) via aws-sdk-go-v2
• Azure Blob Storage (and Azurite)
• Google Cloud Storage
• SFTP
• FTP / explicit FTPS

go get github.com/jkaninda/go-storage

Streaming API

Every backend implements the same Storage interface:

type Storage interface {
	Upload(ctx context.Context, localPath, remotePath string) error
	Download(ctx context.Context, remotePath, localPath string) error
	List(ctx context.Context, prefix string) ([]Item, error) // recursive
	Remove(ctx context.Context, remotePath string) error     // missing -> ErrNotFound
	Close() error

	Put(ctx context.Context, key string, r io.Reader, opts *PutOptions) (Item, error)
	Get(ctx context.Context, key string) (io.ReadCloser, Item, error)
	Stat(ctx context.Context, key string) (Item, error)
	PutIf(ctx context.Context, key string, r io.Reader, cond Condition, opts *PutOptions) (Item, error)
	ListDir(ctx context.Context, dir string) ([]Item, error) // non-recursive, files only
}

• Put streams a reader of unknown length with bounded memory (no temp files, no whole-file buffering). If the reader fails or the context is cancelled, Put returns an error and no object (and no partial object) is left at the key: S3 aborts the multipart upload, Azure never commits the block list, GCS cancels the resumable upload, and SFTP/FTP/local write to <key>.<random>.partial and rename on success.
• Get returns a streaming body; always Close it. On FTP the body holds the single control connection, so close it before making another call.
• PutIf is a conditional write for small files such as a backup index (optimistic concurrency): Condition{IfNoneMatch: true} creates only if the key does not exist, Condition{IfMatch: item.ETag} replaces only if the object is unchanged. It returns ErrPreconditionFailed when the condition does not hold and ErrConditionNotSupported when the backend cannot evaluate it (fall back to locking).
• Errors wrap the sentinels ErrNotFound, ErrPreconditionFailed and ErrConditionNotSupported; test them with errors.Is.
• Keys are relative to the backend base (bucket, container, remote directory or local base path), use / separators and may not escape the base (..).

Backend	Item.ETag	PutIf IfNoneMatch	PutIf IfMatch
S3	S3 ETag	yes	yes
Azure	blob ETag	yes	yes
GCS	object generation	yes	yes
Local	<mtime-ns hex>-<size hex>	yes (lock file)	yes (lock file)
SFTP	empty	yes (O_EXCL)	not supported
FTP	empty	not supported	not supported

S3 PutIf uses a single PutObject request and is limited to 16 MiB.

Example: dump, compress and upload without temp files

package main

import (
	"compress/gzip"
	"context"
	"errors"
	"io"
	"log"
	"os/exec"
	"strings"

	storage "github.com/jkaninda/go-storage"
)

func main() {
	ctx := context.Background()
	s, err := storage.NewStorage(storage.StorageTypeS3, storage.Config{
		S3Bucket:    "backups",
		S3Region:    "us-east-1",
		S3KeyID:     "...",
		S3Secret:    "...",
		S3Endpoint:  "https://minio.example.com", // optional, for S3-compatible services
		S3ForcePath: true,
	})
	if err != nil {
		log.Fatal(err)
	}
	defer s.Close()

	pr, pw := io.Pipe()
	go func() {
		gz := gzip.NewWriter(pw)
		cmd := exec.CommandContext(ctx, "pg_dump", "mydb")
		cmd.Stdout = gz
		err := cmd.Run()
		if err == nil {
			err = gz.Close()
		}
		pw.CloseWithError(err) // a non-nil error aborts the upload
	}()

	item, err := s.Put(ctx, "db/mydb-2026-01-01.sql.gz", pr, &storage.PutOptions{ContentType: "application/gzip"})
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("uploaded %s (%d bytes, etag %s)", item.Key, item.Size, item.ETag)

	// Restore: stream it back.
	body, _, err := s.Get(ctx, item.Key)
	if errors.Is(err, storage.ErrNotFound) {
		log.Fatal("backup not found")
	} else if err != nil {
		log.Fatal(err)
	}
	defer body.Close()
	zr, _ := gzip.NewReader(body)
	_, _ = io.Copy(io.Discard, zr)

	// Update an index file with optimistic concurrency.
	idx, err := s.Stat(ctx, "db/index.json")
	cond := storage.Condition{IfMatch: idx.ETag}
	if errors.Is(err, storage.ErrNotFound) {
		cond = storage.Condition{IfNoneMatch: true}
	}
	_, err = s.PutIf(ctx, "db/index.json", strings.NewReader(`{"backups":[]}`), cond, nil)
	if errors.Is(err, storage.ErrPreconditionFailed) {
		// someone else updated the index: re-read, merge and retry
	}
}

Configuration

storage.NewStorage(storageType, storage.Config{...}) with storageType one of s3, gcs, azure, sftp, ftp, local.

Field	Backend	Description
LocalPath	local	Base directory (default /backup)
S3Bucket	s3	Bucket (required)
S3Region	s3	Region (default from AWS config, else us-east-1)
S3KeyID, S3Secret, S3SessionToken	s3	Static credentials; when empty the default AWS chain is used (env, shared profile, IRSA/web identity, ECS/EC2 roles)
S3Profile	s3	Shared config profile
S3Endpoint	s3	Custom endpoint (MinIO etc.); without a scheme https:// is assumed (http:// with S3DisableTLS)
S3ForcePath	s3	Path-style addressing
S3DisableTLS	s3	Use http:// for scheme-less endpoints
S3StorageClass	s3	e.g. STANDARD_IA, GLACIER_IR
S3SSE, S3SSEKMSKeyID	s3	Server-side encryption: AES256 or aws:kms (+ key id)
S3PartSize	s3	Multipart part size (default 64 MiB, min 5 MiB)
S3Concurrency	s3	Parallel part uploads (default 2); memory ≈ part size × concurrency
AzureAccount, AzureKey	azure	Shared key credentials
AzureSASToken	azure	SAS token (used instead of the shared key when set)
AzureContainer	azure	Container (required)
AzureEndpoint	azure	Domain suffix (default blob.core.windows.net) or a full service URL such as http://127.0.0.1:10000/devstoreaccount1 (Azurite)
GCSBucket	gcs	Bucket (required)
GCSCredentialsFile, GCSCredentialsJSON	gcs	Credentials; default is Application Default Credentials
GCSEndpoint	gcs	Custom endpoint
SFTPHost, SFTPPort	sftp	Server (port default 22)
SFTPUsername, SFTPPassword	sftp	Password authentication
SFTPIdentityFile, SFTPIdentityPassphrase	sftp	Private key authentication (an SSH agent is used too when available)
SFTPKnownHostsFile	sftp	known_hosts file (default ~/.ssh/known_hosts); host keys are always verified unless...
SFTPIgnoreKnownHosts	sftp	...this is true (insecure)
SFTPDirectory	sftp	Remote base directory (default: login directory)
FTPHost, FTPPort, FTPUsername, FTPPassword	ftp	Server and credentials (port default 21)
FTPDirectory	ftp	Remote base directory (default: login directory)
FTPTLS	ftp	Explicit FTPS (AUTH TLS)

The library never reads environment variables itself (except what the cloud SDKs do for their default credential chains, and SSH_AUTH_SOCK for the SSH agent).

Testing

go test ./...                       # unit tests: local + in-process SFTP server
go test -tags integration ./...     # + MinIO, Azurite, fake-gcs-server, FTP via docker

Integration tests skip when docker is unavailable. Images can be overridden with GO_STORAGE_MINIO_IMAGE, GO_STORAGE_AZURITE_IMAGE, GO_STORAGE_FAKEGCS_IMAGE and GO_STORAGE_FTP_IMAGE.

Deprecated: pkg/ (v0.1 API)

The packages under pkg/ (pkg/local, pkg/s3, pkg/ssh, pkg/ftp, pkg/azure) are the legacy v0.1 API. They still compile but are deprecated and will be removed in a future release; use the root package instead.

License

MIT License, Copyright (c) 2024 Jonas Kaninda. See LICENSE.