Documentation
¶
Index ¶
- Constants
- func Backup(dmm DataMigrationModel) error
- func MigrateData(dmm DataMigrationModel) error
- func Restore(dmm DataMigrationModel) error
- func Transfer(dmm DataMigrationModel) error
- func Validate(dmm DataMigrationModel) error
- type BucketInfo
- type DataMigrationModel
- type EndpointDetails
- func (e *EndpointDetails) GetBucketAndObjectKey() (string, string, error)
- func (e *EndpointDetails) GetEndpoint() string
- func (e *EndpointDetails) GetPort() int
- func (e *EndpointDetails) GetRsyncPath(options *TransferOptions) string
- func (e *EndpointDetails) IsLocal() bool
- func (e *EndpointDetails) IsObjectStorageEndpoint() bool
- func (e *EndpointDetails) IsRemote() bool
- type FilterOption
- type MigrationError
- type ObjectInfo
- type ObjectStorageOption
- type OperationError
- type PresignedUrlInfo
- type RsyncOption
- type TransferOptions
Constants ¶
const ( TransferMethodRsync = "rsync" TransferMethodObjectStorage = "object-storage" )
Transfer method constants
const ( ObjectStorageClientSpider = "spider" // CB-Spider presigned URL API (default) ObjectStorageClientMinio = "minio" // MinIO SDK for direct S3-compatible access )
Object Storage client constants
Variables ¶
This section is empty.
Functions ¶
func Backup ¶
func Backup(dmm DataMigrationModel) error
Backup executes the BackupCmd defined in the source EndpointDetails of the DataMigrationModel. * Note: This is an optional operation that runs only if BackupCmd is configured.
func MigrateData ¶
func MigrateData(dmm DataMigrationModel) error
MigrateData manages the complete data migration workflow: 1. If Source.BackupCmd is available, perform Backup 2. Always perform Transfer 3. If Destination.RestoreCmd is available, perform Restore This provides a simple one-call approach to handle the entire data migration pipeline. * Note: Transfer is the optional operation that runs BackUp, Transfer, and Restore sequentially.
func Restore ¶
func Restore(dmm DataMigrationModel) error
Restore executes the RestoreCmd defined in the destination EndpointDetails of the DataMigrationModel. * Note: This is an optional operation that runs only if RestoreCmd is configured.
func Transfer ¶
func Transfer(dmm DataMigrationModel) error
Transfer runs the transfer command to transfer data as defined by the given DataMigrationModel. * Note: This is the core operation that always executes when called.
func Validate ¶
func Validate(dmm DataMigrationModel) error
Validate checks if the fields of DataMigrationModel satisfy basic requirements for transfer tasks.
Types ¶
type BucketInfo ¶
type BucketInfo struct {
Name string `xml:"Name"`
Prefix string `xml:"Prefix"`
Marker string `xml:"Marker"`
MaxKeys int `xml:"MaxKeys"`
IsTruncated bool `xml:"IsTruncated"`
Contents []ObjectInfo `xml:"Contents"`
}
BucketInfo represents bucket listing information
type DataMigrationModel ¶
type DataMigrationModel struct {
Source EndpointDetails `json:"source" validate:"required"` // Source endpoint configuration
SourceTransferOptions *TransferOptions `json:"sourceTransferOptions" validate:"required"` // Source-specific transfer options
Destination EndpointDetails `json:"destination" validate:"required"` // Destination endpoint configuration
DestinationTransferOptions *TransferOptions `json:"destinationTransferOptions" validate:"required"` // Destination-specific transfer options
}
DataMigrationModel defines a single data migration task supporting multiple protocols.
func (*DataMigrationModel) IsRelayMode ¶
func (dmm *DataMigrationModel) IsRelayMode() bool
IsRelayMode determines if both source and destination endpoints are remote. This is used to identify relay migration scenarios where data needs to flow through the local machine as an intermediary between two remote endpoints.
type EndpointDetails ¶
type EndpointDetails struct {
// Endpoint configuration (auto-detects protocol based on provided fields)
Endpoint string `json:"endpoint,omitempty"` // SSH host/IP or Object Storage API endpoint (e.g., "server.com", "http://localhost:1024/spider/s3")
Port int `json:"port,omitempty"` // Port for SSH host/IP (default: 22) or Object Storage API endpoint (default: 1024)
// Data location (required)
DataPath string `json:"dataPath" validate:"required"` // Local path, remote path, or Object Storage bucket path (e.g., "/data", "bucket/object-key")
// Command execution
BackupCmd string `json:"backupCmd,omitempty"` // Backup command string to be executed on this endpoint
RestoreCmd string `json:"restoreCmd,omitempty"` // Restore command string to be executed on this endpoint
}
EndpointDetails defines the source/destination endpoint for data transfer and backup/restore operations. Simple unified structure supporting SSH-based rsync, Object Storage API endpoints, and local filesystem transfers.
func (*EndpointDetails) GetBucketAndObjectKey ¶
func (e *EndpointDetails) GetBucketAndObjectKey() (string, string, error)
GetBucketAndObjectKey extracts bucket name and object key from DataPath. DataPath formats:
- "bucket-name/object/key/path" (e.g., "spider-test-bucket/a/b/c/test.txt")
- "bucket-name/" (download all objects in bucket)
- "bucket-name/prefix/" (download objects with prefix)
func (*EndpointDetails) GetEndpoint ¶
func (e *EndpointDetails) GetEndpoint() string
GetEndpoint returns the endpoint (SSH host/IP or Object Storage endpoint).
func (*EndpointDetails) GetPort ¶
func (e *EndpointDetails) GetPort() int
GetPort returns the SSH port.
func (*EndpointDetails) GetRsyncPath ¶
func (e *EndpointDetails) GetRsyncPath(options *TransferOptions) string
GetRsyncPath constructs the path string suitable for rsync (e.g., "user@host:/path" or "/local/path").
func (*EndpointDetails) IsLocal ¶
func (e *EndpointDetails) IsLocal() bool
IsLocal determines if the EndpointDetails represent a local endpoint.
func (*EndpointDetails) IsObjectStorageEndpoint ¶
func (e *EndpointDetails) IsObjectStorageEndpoint() bool
IsObjectStorageEndpoint checks if this endpoint is for Object Storage
func (*EndpointDetails) IsRemote ¶
func (e *EndpointDetails) IsRemote() bool
IsRemote determines if the EndpointDetails represent a remote endpoint. Returns true if the endpoint has a non-empty host/endpoint configured.
type FilterOption ¶
type FilterOption struct {
Include []string `json:"include,omitempty"` // Patterns to include (e.g., "*.txt", "data/**")
Exclude []string `json:"exclude,omitempty"` // Patterns to exclude (e.g., "*.log", "temp/**", ".git/**")
}
FilterOption defines file filtering options for both rsync and Object Storage transfers. Supports glob patterns with rsync-like filtering behavior: 1. If include patterns are specified, only matching files are included 2. Exclude patterns are applied after include patterns (exclude takes priority) 3. Supports ** for recursive directory matching
type MigrationError ¶
MigrationError represents an error during the migration process
func (*MigrationError) Error ¶
func (e *MigrationError) Error() string
func (*MigrationError) Unwrap ¶
func (e *MigrationError) Unwrap() error
type ObjectInfo ¶
type ObjectInfo struct {
Key string `xml:"Key"`
LastModified string `xml:"LastModified"`
ETag string `xml:"ETag"`
Size int64 `xml:"Size"`
StorageClass string `xml:"StorageClass"`
}
ObjectInfo represents an object in the bucket
type ObjectStorageOption ¶
type ObjectStorageOption struct {
// Client selection
Client string `json:"client,omitempty" default:"spider"` // Object storage client: "spider" (default) or "minio"
// Common authentication (REQUIRED - must be provided by user)
AccessKeyId string `json:"accessKeyId" validate:"required"` // AWS Access Key ID or CB-Spider connection name (REQUIRED)
SecretAccessKey string `json:"secretAccessKey,omitempty"` // AWS Secret Access Key (REQUIRED for minio client)
Region string `json:"region,omitempty" default:"us-east-1"` // AWS region (for minio client, default: "us-east-1")
UseSSL bool `json:"useSSL,omitempty" default:"false"` // Use HTTPS (default: true)
// Presigned URL configuration (spider client only)
ExpiresIn int `json:"expiresIn,omitempty" default:"3600"` // Presigned URL expiration time in seconds (default: 3600)
// HTTP request configuration (optional)
Timeout int `json:"timeout,omitempty" default:"300"` // HTTP request timeout in seconds (default: 300)
MaxRetries int `json:"maxRetries,omitempty" default:"3"` // Maximum number of retry attempts (default: 3)
// File filtering options (applied after listing objects, before upload/download)
Filter *FilterOption `json:"filter,omitempty"` // File filtering options (include/exclude patterns) - use nested structure for better organization
}
ObjectStorageOption defines Object Storage transfer options. Supports two clients: 1. Spider client (Client = "spider" or empty, default for backward compatibility)
- Endpoint: CB-Spider API endpoint (e.g., "http://localhost:1024/spider/s3")
- AccessKeyId: CB-Spider connection name (e.g., "aws-config01")
- Uses presigned URLs from CB-Spider for upload/download
2. MinIO client (Client = "minio")
- Endpoint: S3-compatible storage endpoint (e.g., "s3.amazonaws.com", "play.min.io:9000")
- AccessKeyId: AWS Access Key ID
- SecretAccessKey: AWS Secret Access Key (required for minio client)
- Region: AWS region (optional, default: "us-east-1")
- UseSSL: Use HTTPS for connections (default: true)
type OperationError ¶
type OperationError struct {
Operation string // "backup", "restore", "transfer"
Method string // transfer method (for transfer operations)
Source string // source path/endpoint
Destination string // destination path/endpoint
Command string // executed command (for backup/restore)
Output string // command output (for backup/restore)
IsRelayMode bool // relay mode flag (for transfer)
Context map[string]string // additional context information
Err error // underlying error
}
OperationError provides detailed context about transx operation failures This unified error type handles backup, restore, and transfer operations
func (*OperationError) Error ¶
func (e *OperationError) Error() string
func (*OperationError) GetMethod ¶
func (e *OperationError) GetMethod() string
GetMethod returns the transfer method (applicable to transfer operations)
func (*OperationError) GetOutput ¶
func (e *OperationError) GetOutput() string
GetOutput returns the command output for debugging (applicable to backup/restore operations)
func (*OperationError) IsOperation ¶
func (e *OperationError) IsOperation(operation string) bool
IsOperation checks if the error is for a specific operation type
func (*OperationError) Unwrap ¶
func (e *OperationError) Unwrap() error
type PresignedUrlInfo ¶
type PresignedUrlInfo struct {
PresignedURL string `xml:"PresignedURL"`
Expires int `xml:"Expires"`
Method string `xml:"Method"`
}
PresignedUrlInfo represents a presigned URL result from Object Storage API
type RsyncOption ¶
type RsyncOption struct {
// SSH connection & authentication options (integrated)
Username string `json:"username,omitempty"` // SSH username
SSHPrivateKeyPath string `json:"sshPrivateKeyPath,omitempty"` // SSH private key path
// InsecureSkipHostKeyVerification, if true, relaxes host key checking for SSH connections.
// Adds "-o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=/dev/null" options.
// Warning: This can be a security risk and should only be used in trusted environments.
InsecureSkipHostKeyVerification bool `json:"insecureSkipHostKeyVerification,omitempty" default:"false"`
ConnectTimeout int `json:"connectTimeout,omitempty" default:"30"` // SSH connection timeout in seconds
// Transfer behavior options
Verbose bool `json:"verbose,omitempty" default:"false"` // Enable verbose logging
DryRun bool `json:"dryRun,omitempty" default:"false"` // Perform a trial run with no changes made
Progress bool `json:"progress,omitempty" default:"false"` // Show progress during transfer
// Rsync-specific options
Compress bool `json:"compress,omitempty" default:"true"` // -z, --compress: Compress file data during the transfer
Archive bool `json:"archive,omitempty" default:"true"` // -a, --archive: Archive mode; equals -rlptgoD (no -H,-A,-X)
Delete bool `json:"delete,omitempty" default:"false"` // --delete: Delete extraneous files from dest dirs
RsyncPath string `json:"rsyncPath,omitempty" default:"rsync"` // Path to the rsync executable (if empty, uses system PATH)
Filter *FilterOption `json:"filter,omitempty"` // File filtering options (include/exclude patterns) - use nested structure for better organization
// TransferDirContentsOnly, if true, adds a trailing slash to source paths
// to transfer only the contents of the directory and not the directory itself.
TransferDirContentsOnly bool `json:"transferDirContentsOnly,omitempty" default:"false"`
}
RsyncOption defines rsync-specific transfer options and SSH connection options.
type TransferOptions ¶
type TransferOptions struct {
// Transfer method specification (required)
Method string `json:"method" validate:"required"` // Transfer method: "rsync", "object-storage"
// Rsync-specific options
RsyncOptions *RsyncOption `json:"rsyncOptions,omitempty"`
// Object Storage-specific options (CB-Spider, AWS S3, etc.)
ObjectStorageOptions *ObjectStorageOption `json:"objectStorageOptions,omitempty"`
}
TransferOptions defines options for various data transfer methods.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
examples
|
|
|
mariadb-migration
command
filepath: /home/ubuntu/dev/yunkon-kim/transx/examples/mariadb-migration/main.go
|
filepath: /home/ubuntu/dev/yunkon-kim/transx/examples/mariadb-migration/main.go |
|
object-storage
command
|