Moving .X12 Files with Azure Data Factory

Moving .X12 Files Between Folders in Azure Data Factory

This guide will provide the necessary information and C# code to move files with the .X12 extension from one folder to another using Azure Data Factory's built-in functions and custom activities.

Prerequisites

1. Azure Data Factory instance.
2. Source and destination storage accounts or blob containers.
3. Understanding of C# and Azure Data Factory Pipelines.

C# Code for Moving Files

Here is a sample C# code to move .X12 files between folders:

// Necessary Imports
using System;
using System.IO;
using Microsoft.Azure.Management.DataFactory.Models;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;

public static class MoveX12Files
{
    /// 

    /// Azure Function to move .X12 files from source to destination folder.
    /// 
    /// Timer trigger for periodic execution
    /// Logger to record execution process
    /// Throws exception if file operation fails
    [FunctionName("MoveX12FilesFunction")]
    public static void Run([TimerTrigger("0 */5 * * * *")] TimerInfo timer, TraceWriter log)
    {
        // Input Validation: Source and Destination paths
        string sourceFolder = Environment.GetEnvironmentVariable("SourceFolder");
        string destinationFolder = Environment.GetEnvironmentVariable("DestinationFolder");
        
        if (string.IsNullOrEmpty(sourceFolder) || string.IsNullOrEmpty(destinationFolder))
        {
            throw new ArgumentException("Source or Destination folder path is missing.");
        }
        
        // Blob Storage Connection String
        string storageConnectionString = Environment.GetEnvironmentVariable("AzureBlobStorageConnectionString");

        CloudStorageAccount storageAccount;
        if (CloudStorageAccount.TryParse(storageConnectionString, out storageAccount))
        {
            CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
            CloudBlobContainer sourceContainer = blobClient.GetContainerReference(sourceFolder);
            CloudBlobContainer destinationContainer = blobClient.GetContainerReference(destinationFolder);

            try
            {
                // List all .X12 files in the source folder
                foreach (IListBlobItem blobItem in sourceContainer.ListBlobs(useFlatBlobListing: true))
                {
                    if (blobItem is CloudBlob blob && blob.Name.EndsWith(".X12"))
                    {
                        CloudBlockBlob sourceBlob = sourceContainer.GetBlockBlobReference(blob.Name);
                        CloudBlockBlob destinationBlob = destinationContainer.GetBlockBlobReference(blob.Name);

                        // Move file: Copy to destination and delete from source
                        destinationBlob.StartCopy(sourceBlob);
                        sourceBlob.Delete();
                        
                        log.Info($"File moved: {blob.Name}");
                    }
                }
            }
            catch (Exception ex)
            {
                log.Error($"Error moving files: {ex.Message}", ex);
                throw;
            }
        }
        else
        {
            throw new Exception("Failed to connect to Azure Blob Storage.");
        }
    }
}

Code Description

• Imports: Necessary libraries for Azure Storage and logging.
• Run Function:
  • Parameters: timer for periodic execution and log for logging.
  • Input Validation: Checks if source and destination paths are valid.
  • Blob Operation:
    • Connect to Azure Blob Storage.
    • List .X12 files in the source folder.
    • Copy each file to the destination folder.
    • Delete the original file from the source folder.
• Exception Handling: Gracefully handles and logs any exceptions during blob operations.

Usage Example

1. Configuration: Set the following environment variables in your Azure Function App settings:

   • SourceFolder: Path of the source folder in the blob container.
   • DestinationFolder: Path of the destination folder in the blob container.
   • AzureBlobStorageConnectionString: Connection string of your Azure Blob Storage.

2. Deployment: Deploy this C# function as an Azure Function in your Data Factory pipeline.

Best Practices

• Utilize the Azure Data Factory's monitoring capabilities to oversee the execution.
• Handle exceptions and edge cases to prevent partial data movement.

For more advanced data engineering and data factory techniques, consider exploring the Enterprise DNA Platform for comprehensive courses and resources.