transx

package module
v0.0.0-...-2e94d91 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 7, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

README

transx

A Go-based data migration library that supports multiple transfer methods for moving data between databases and storage systems.

Features

  • Error-Only Approach: Structured error handling for seamless integration with logging frameworks (zerolog, logrus, etc.)
  • Multi-Protocol Transfer: Core data transfer functionality with support for rsync and object-storage-api methods
  • Optional Data Processing: Backup and restore operations as supplementary features when needed
  • Explicit Configuration: User-specified transfer methods (no auto-detection)
  • Direct & Relay Modes: Flexible migration patterns supporting both direct transfers and relay node scenarios
  • Data Integrity: Built-in validation and verification with comprehensive error context

Quick Start

import "github.com/yunkon-kim/transx"

// Core transfer operation
err := transx.Transfer(dataModel)
if err != nil {
    log.Error().Err(err).Msg("Transfer failed")
}

// Complete migration with optional backup/restore
err = transx.MigrateData(dataModel)
if err != nil {
    // Rich error context available for logging frameworks
    log.Error().Err(err).Msg("Migration failed")
}

Transfer Methods

Method Description Use Case
rsync SSH-based file transfers using rsync Remote server migrations with authentication
object-storage-api HTTP-based transfers with Object Storage APIs S3-compatible storage (CB-Spider, AWS S3, etc.)

Note: File operations on the same host are handled automatically when endpoint is empty.

Architecture

The library implements a Transfer-Centric Data Migration Model with the following components:

Core Functionality:

  • Transfer: Move data between systems using specified transfer methods (rsync, object-storage-api)

Optional Operations (when backup/restore commands are provided):

  • Backup: Export data from source systems before transfer
  • Restore: Import transferred data into destination systems

The transfer operation is the primary focus, with backup and restore serving as optional pre/post-processing steps.

Transfer Modes
  • Direct Mode: Source → Destination (at least one endpoint is local)
  • Relay Mode: Source → Relay Node → Destination (both endpoints are remote)

Error Handling

The library implements an Error-Only Approach with unified error handling:

// Unified OperationError provides rich context for all operations
type OperationError struct {
    Operation   string            // "backup", "restore", "transfer"
    Method      string            // transfer method (for transfer operations)
    Source      string            // source path/endpoint
    Destination string            // destination path/endpoint
    Command     string            // executed command (for backup/restore)
    Output      string            // command output (for backup/restore)
    IsRelayMode bool              // relay mode flag (for transfer)
    Context     map[string]string // additional context information
    Err         error             // underlying error
}

// Simple interface for users - just use err.Error()
// Advanced users can access rich context via type assertion

The transfer operation always executes, while backup and restore are conditional based on configuration.

Examples

MariaDB Migration

See examples/mariadb-migration for a complete database migration example:

  • Direct mode migration (local ↔ remote)
  • Relay mode migration (remote ↔ remote)
  • Individual step execution (backup, transfer, restore)
  • Comprehensive testing and validation
Object Storage Migration

See examples/object-storage for Object Storage integration:

  • CB-Spider presigned URL integration
  • Bidirectional transfers (local ↔ object storage)
  • Cross-cloud migration scenarios
  • AWS S3 compatibility

Installation

go get github.com/yunkon-kim/transx

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Documentation

Index

Constants

View Source
const (
	TransferMethodRsync         = "rsync"
	TransferMethodObjectStorage = "object-storage"
)

Transfer method constants

View Source
const (
	ObjectStorageClientSpider = "spider" // CB-Spider presigned URL API (default)
	ObjectStorageClientMinio  = "minio"  // MinIO SDK for direct S3-compatible access
)

Object Storage client constants

Variables

This section is empty.

Functions

func Backup

func Backup(dmm DataMigrationModel) error

Backup executes the BackupCmd defined in the source EndpointDetails of the DataMigrationModel. * Note: This is an optional operation that runs only if BackupCmd is configured.

func MigrateData

func MigrateData(dmm DataMigrationModel) error

MigrateData manages the complete data migration workflow: 1. If Source.BackupCmd is available, perform Backup 2. Always perform Transfer 3. If Destination.RestoreCmd is available, perform Restore This provides a simple one-call approach to handle the entire data migration pipeline. * Note: Transfer is the optional operation that runs BackUp, Transfer, and Restore sequentially.

func Restore

func Restore(dmm DataMigrationModel) error

Restore executes the RestoreCmd defined in the destination EndpointDetails of the DataMigrationModel. * Note: This is an optional operation that runs only if RestoreCmd is configured.

func Transfer

func Transfer(dmm DataMigrationModel) error

Transfer runs the transfer command to transfer data as defined by the given DataMigrationModel. * Note: This is the core operation that always executes when called.

func Validate

func Validate(dmm DataMigrationModel) error

Validate checks if the fields of DataMigrationModel satisfy basic requirements for transfer tasks.

Types

type BucketInfo

type BucketInfo struct {
	Name        string       `xml:"Name"`
	Prefix      string       `xml:"Prefix"`
	Marker      string       `xml:"Marker"`
	MaxKeys     int          `xml:"MaxKeys"`
	IsTruncated bool         `xml:"IsTruncated"`
	Contents    []ObjectInfo `xml:"Contents"`
}

BucketInfo represents bucket listing information

type DataMigrationModel

type DataMigrationModel struct {
	Source                     EndpointDetails  `json:"source" validate:"required"`                     // Source endpoint configuration
	SourceTransferOptions      *TransferOptions `json:"sourceTransferOptions" validate:"required"`      // Source-specific transfer options
	Destination                EndpointDetails  `json:"destination" validate:"required"`                // Destination endpoint configuration
	DestinationTransferOptions *TransferOptions `json:"destinationTransferOptions" validate:"required"` // Destination-specific transfer options
}

DataMigrationModel defines a single data migration task supporting multiple protocols.

func (*DataMigrationModel) IsRelayMode

func (dmm *DataMigrationModel) IsRelayMode() bool

IsRelayMode determines if both source and destination endpoints are remote. This is used to identify relay migration scenarios where data needs to flow through the local machine as an intermediary between two remote endpoints.

type EndpointDetails

type EndpointDetails struct {
	// Endpoint configuration (auto-detects protocol based on provided fields)
	Endpoint string `json:"endpoint,omitempty"` // SSH host/IP or Object Storage API endpoint (e.g., "server.com", "http://localhost:1024/spider/s3")
	Port     int    `json:"port,omitempty"`     // Port for SSH host/IP (default: 22) or Object Storage API endpoint (default: 1024)

	// Data location (required)
	DataPath string `json:"dataPath" validate:"required"` // Local path, remote path, or Object Storage bucket path (e.g., "/data", "bucket/object-key")

	// Command execution
	BackupCmd  string `json:"backupCmd,omitempty"`  // Backup command string to be executed on this endpoint
	RestoreCmd string `json:"restoreCmd,omitempty"` // Restore command string to be executed on this endpoint
}

EndpointDetails defines the source/destination endpoint for data transfer and backup/restore operations. Simple unified structure supporting SSH-based rsync, Object Storage API endpoints, and local filesystem transfers.

func (*EndpointDetails) GetBucketAndObjectKey

func (e *EndpointDetails) GetBucketAndObjectKey() (string, string, error)

GetBucketAndObjectKey extracts bucket name and object key from DataPath. DataPath formats:

  • "bucket-name/object/key/path" (e.g., "spider-test-bucket/a/b/c/test.txt")
  • "bucket-name/" (download all objects in bucket)
  • "bucket-name/prefix/" (download objects with prefix)

func (*EndpointDetails) GetEndpoint

func (e *EndpointDetails) GetEndpoint() string

GetEndpoint returns the endpoint (SSH host/IP or Object Storage endpoint).

func (*EndpointDetails) GetPort

func (e *EndpointDetails) GetPort() int

GetPort returns the SSH port.

func (*EndpointDetails) GetRsyncPath

func (e *EndpointDetails) GetRsyncPath(options *TransferOptions) string

GetRsyncPath constructs the path string suitable for rsync (e.g., "user@host:/path" or "/local/path").

func (*EndpointDetails) IsLocal

func (e *EndpointDetails) IsLocal() bool

IsLocal determines if the EndpointDetails represent a local endpoint.

func (*EndpointDetails) IsObjectStorageEndpoint

func (e *EndpointDetails) IsObjectStorageEndpoint() bool

IsObjectStorageEndpoint checks if this endpoint is for Object Storage

func (*EndpointDetails) IsRemote

func (e *EndpointDetails) IsRemote() bool

IsRemote determines if the EndpointDetails represent a remote endpoint. Returns true if the endpoint has a non-empty host/endpoint configured.

type FilterOption

type FilterOption struct {
	Include []string `json:"include,omitempty"` // Patterns to include (e.g., "*.txt", "data/**")
	Exclude []string `json:"exclude,omitempty"` // Patterns to exclude (e.g., "*.log", "temp/**", ".git/**")
}

FilterOption defines file filtering options for both rsync and Object Storage transfers. Supports glob patterns with rsync-like filtering behavior: 1. If include patterns are specified, only matching files are included 2. Exclude patterns are applied after include patterns (exclude takes priority) 3. Supports ** for recursive directory matching

type MigrationError

type MigrationError struct {
	Stage string // "backup", "transfer", or "restore"
	Err   error
}

MigrationError represents an error during the migration process

func (*MigrationError) Error

func (e *MigrationError) Error() string

func (*MigrationError) Unwrap

func (e *MigrationError) Unwrap() error

type ObjectInfo

type ObjectInfo struct {
	Key          string `xml:"Key"`
	LastModified string `xml:"LastModified"`
	ETag         string `xml:"ETag"`
	Size         int64  `xml:"Size"`
	StorageClass string `xml:"StorageClass"`
}

ObjectInfo represents an object in the bucket

type ObjectStorageOption

type ObjectStorageOption struct {
	// Client selection
	Client string `json:"client,omitempty" default:"spider"` // Object storage client: "spider" (default) or "minio"

	// Common authentication (REQUIRED - must be provided by user)
	AccessKeyId     string `json:"accessKeyId" validate:"required"`      // AWS Access Key ID or CB-Spider connection name (REQUIRED)
	SecretAccessKey string `json:"secretAccessKey,omitempty"`            // AWS Secret Access Key (REQUIRED for minio client)
	Region          string `json:"region,omitempty" default:"us-east-1"` // AWS region (for minio client, default: "us-east-1")
	UseSSL          bool   `json:"useSSL,omitempty" default:"false"`     // Use HTTPS (default: true)

	// Presigned URL configuration (spider client only)
	ExpiresIn int `json:"expiresIn,omitempty" default:"3600"` // Presigned URL expiration time in seconds (default: 3600)

	// HTTP request configuration (optional)
	Timeout    int `json:"timeout,omitempty" default:"300"`  // HTTP request timeout in seconds (default: 300)
	MaxRetries int `json:"maxRetries,omitempty" default:"3"` // Maximum number of retry attempts (default: 3)

	// File filtering options (applied after listing objects, before upload/download)
	Filter *FilterOption `json:"filter,omitempty"` // File filtering options (include/exclude patterns) - use nested structure for better organization
}

ObjectStorageOption defines Object Storage transfer options. Supports two clients: 1. Spider client (Client = "spider" or empty, default for backward compatibility)

  • Endpoint: CB-Spider API endpoint (e.g., "http://localhost:1024/spider/s3")
  • AccessKeyId: CB-Spider connection name (e.g., "aws-config01")
  • Uses presigned URLs from CB-Spider for upload/download

2. MinIO client (Client = "minio")

  • Endpoint: S3-compatible storage endpoint (e.g., "s3.amazonaws.com", "play.min.io:9000")
  • AccessKeyId: AWS Access Key ID
  • SecretAccessKey: AWS Secret Access Key (required for minio client)
  • Region: AWS region (optional, default: "us-east-1")
  • UseSSL: Use HTTPS for connections (default: true)

type OperationError

type OperationError struct {
	Operation   string            // "backup", "restore", "transfer"
	Method      string            // transfer method (for transfer operations)
	Source      string            // source path/endpoint
	Destination string            // destination path/endpoint
	Command     string            // executed command (for backup/restore)
	Output      string            // command output (for backup/restore)
	IsRelayMode bool              // relay mode flag (for transfer)
	Context     map[string]string // additional context information
	Err         error             // underlying error
}

OperationError provides detailed context about transx operation failures This unified error type handles backup, restore, and transfer operations

func (*OperationError) Error

func (e *OperationError) Error() string

func (*OperationError) GetMethod

func (e *OperationError) GetMethod() string

GetMethod returns the transfer method (applicable to transfer operations)

func (*OperationError) GetOutput

func (e *OperationError) GetOutput() string

GetOutput returns the command output for debugging (applicable to backup/restore operations)

func (*OperationError) IsOperation

func (e *OperationError) IsOperation(operation string) bool

IsOperation checks if the error is for a specific operation type

func (*OperationError) Unwrap

func (e *OperationError) Unwrap() error

type PresignedUrlInfo

type PresignedUrlInfo struct {
	PresignedURL string `xml:"PresignedURL"`
	Expires      int    `xml:"Expires"`
	Method       string `xml:"Method"`
}

PresignedUrlInfo represents a presigned URL result from Object Storage API

type RsyncOption

type RsyncOption struct {
	// SSH connection & authentication options (integrated)
	Username          string `json:"username,omitempty"`          // SSH username
	SSHPrivateKeyPath string `json:"sshPrivateKeyPath,omitempty"` // SSH private key path

	// InsecureSkipHostKeyVerification, if true, relaxes host key checking for SSH connections.
	// Adds "-o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=/dev/null" options.
	// Warning: This can be a security risk and should only be used in trusted environments.
	InsecureSkipHostKeyVerification bool `json:"insecureSkipHostKeyVerification,omitempty" default:"false"`
	ConnectTimeout                  int  `json:"connectTimeout,omitempty" default:"30"` // SSH connection timeout in seconds

	// Transfer behavior options
	Verbose  bool `json:"verbose,omitempty" default:"false"`  // Enable verbose logging
	DryRun   bool `json:"dryRun,omitempty" default:"false"`   // Perform a trial run with no changes made
	Progress bool `json:"progress,omitempty" default:"false"` // Show progress during transfer

	// Rsync-specific options
	Compress  bool          `json:"compress,omitempty" default:"true"`   // -z, --compress: Compress file data during the transfer
	Archive   bool          `json:"archive,omitempty" default:"true"`    // -a, --archive: Archive mode; equals -rlptgoD (no -H,-A,-X)
	Delete    bool          `json:"delete,omitempty" default:"false"`    // --delete: Delete extraneous files from dest dirs
	RsyncPath string        `json:"rsyncPath,omitempty" default:"rsync"` // Path to the rsync executable (if empty, uses system PATH)
	Filter    *FilterOption `json:"filter,omitempty"`                    // File filtering options (include/exclude patterns) - use nested structure for better organization

	// TransferDirContentsOnly, if true, adds a trailing slash to source paths
	// to transfer only the contents of the directory and not the directory itself.
	TransferDirContentsOnly bool `json:"transferDirContentsOnly,omitempty" default:"false"`
}

RsyncOption defines rsync-specific transfer options and SSH connection options.

type TransferOptions

type TransferOptions struct {
	// Transfer method specification (required)
	Method string `json:"method" validate:"required"` // Transfer method: "rsync", "object-storage"

	// Rsync-specific options
	RsyncOptions *RsyncOption `json:"rsyncOptions,omitempty"`

	// Object Storage-specific options (CB-Spider, AWS S3, etc.)
	ObjectStorageOptions *ObjectStorageOption `json:"objectStorageOptions,omitempty"`
}

TransferOptions defines options for various data transfer methods.

Directories

Path Synopsis
examples
mariadb-migration command
filepath: /home/ubuntu/dev/yunkon-kim/transx/examples/mariadb-migration/main.go
filepath: /home/ubuntu/dev/yunkon-kim/transx/examples/mariadb-migration/main.go
object-storage command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL