Moving Data with AzCopy in Azure

Introduction

When working with Azure, moving data to and from Azure Storage is a frequent task. Whether we're uploading large files, syncing data, or simply transferring files, AzCopy is the go-to tool. 

What is AzCopy?

AzCopy is a command-line utility designed to copy data efficiently between our on-premises environment and Azure Storage or even between Azure Storage accounts. It supports:

• Blobs (Block Blob, Page Blob, and Append Blob)

• Files (Azure File Share)

• Tables (Table Storage)

Why Use AzCopy?

1. Speed: Optimized for high-speed data transfer.

2. Flexibility: Works with different Azure Storage services.

3. Free: Comes at no extra cost.

4. Automation-Friendly: Perfect for scripts and scheduled tasks.

Getting Started with AzCopy

1. Download AzCopy

You can download the latest version of AzCopy for your platform (Windows, Linux, or macOS) here.

2. Install AzCopy

Simply extract the downloaded file and add its location to your system’s PATH environment variable for easy access from any command prompt or terminal.

3. Authenticate AzCopy

To start using AzCopy, you need to authenticate:

azcopy login

This will redirect you to a web page where you can log in with your Azure account.

Special Note: If we are executing the command from the Cloud Shell, all of the previous three steps can be skipped. If we are using SAS, login can also be skipped.

Basic Scenarios/Commands

Here are the most common AzCopy commands we’ll use:

Upload Files to Blob Storage

azcopy copy "<local-file-path>" "<blob-url/blob-sas-url>" --recursive

• Replace <local-file-path> with the path to your file or folder.

• Replace <blob-url/blob-sas-url> with our blob's URL or Blob SAS URL.

• Use --recursive for folders.

Download Files from Blob Storage

azcopy copy "<blob-url/blob-sas-url>" "<local-destination-path>" --recursive

Copy Files  Across Storage Accounts

azcopy copy "<blob-url/blob-sas-url for source container>" "<blob-url/blob-sas-url for destination container>" --recursive

Sync Local and Azure Storage

azcopy sync "<local-path>" "<container-url>" --recursive

This ensures only changed or new files are copied.

List Files in a Container

azcopy list "https://<storage-account-name>.blob.core.windows.net/<container-name>"

Tips for Success

1. Use SAS Tokens for Secure Access: If we prefer not to log in directly, we can use a Shared Access Signature (SAS) token for secure and temporary access. Example:

   azcopy copy "<local-path>" "https://<storage-account-name>.blob.core.windows.net/<container-name>?<sas-token>" --recursive

2. Monitor Transfers: AzCopy provides progress updates during transfers. For detailed logs, check the ~/.azcopy directory.

3. Automate with Scripts: Integrate AzCopy commands into batch scripts, shell scripts, or CI/CD pipelines for automation.

Common Use Cases

• Backup and Restore: Quickly backup local files to Azure or restore them from Azure.

• Data Migration: Migrate data between storage accounts or regions.

• Sync Operations: Keep local and Azure Storage in sync.

Troubleshooting

If you encounter issues, check:

• Network Connectivity: Ensure your internet connection is stable.

• Permissions: Verify that your Azure account or SAS token has the necessary permissions.

• Logs: Refer to AzCopy’s logs for detailed error messages.

Conclusion

AzCopy is a powerful, lightweight tool that simplifies data transfer in Azure. Its speed, flexibility, and ease of use make it indispensable for Azure administrators and developers alike.