English 中文(简体)
利用AWS DMS从CSV(S3)向DynamoDB迁移的数据
原标题:Data Migration from CSV (S3) to DynamoDB using AWS DMS
  • 时间:2022-06-30 14:18:24
  •  标签:

我在把一张由<JSON组成的表格移入一个栏目,并插入时,正面临一个问题。 DyanmoDB 地图。 源数据采用以下格式:

<table border="1"><tr><th>userId</th><th>brandId</th><th>createdBy</th><th>modifiedBy</th><th>preferences</th><th>version</th></tr><tr><td>TU0001</td><td>TEST BRAND</td><td>{"channel":{"S":"website"},"date":{"S": "2022-06-13T08:16:26.300Z"},"userId":{"S":"TU0001"}}</td><td>{"channel":{"S":"website"},"date":{"S": "2022-06-13T015:26:10.200Z"},"userId":{"S":"TU0001"}}</td><td>{"Colour": {"S": "Red" },"Size":{"S": "XL" }}</td><td>1</td></tr></table>

位于DynamoDB的表格结构中,只有JSON的数值被储存为MAP。 当我们利用人口与健康调查进行移徙时,它就将JSON作为扼杀性价值,而不是按预期的MAP。

我将变革规则定义为:

"attribute-mappings": [
      {
        "target-attribute-name": "userId",
        "attribute-type": "scalar",
        "attribute-sub-type": "string",
        "value": "${userId}"
      },
      {
        "target-attribute-name": "brandId",
        "attribute-type": "scalar",
        "attribute-sub-type": "string",
        "value": "${brandId}"
      },
      {
        "target-attribute-name": "createdBy",
        "attribute-type": "document",
        "attribute-sub-type": "dynamodb-map",
        "value": {
          "M": {
            "S": "${createdBy}"
          }
        }
      },
      {
        "target-attribute-name": "modifiedBy",
        "attribute-type": "document",
        "attribute-sub-type": "dynamodb-map",
        "value": {
          "M": {
            "S": "${modifiedBy}"
          }
        }
      },
      {
        "target-attribute-name": "preferences",
        "attribute-type": "document",
        "attribute-sub-type": "dynamodb-map",
        "value": {
          "M": {
            "S": "${preferences}"
          }
        }
      },
      {
        "target-attribute-name": "version",
        "attribute-type": "scalar",
        "attribute-sub-type": "number",
        "value": "${version}"
      }
    ]

我也试图将地图添加如下,并在达扬莫亚的一张空洞地图。

"value": {
          "M": "${preferences}"
        }

希望的人能够提供帮助。

问题回答

我认为,DMS不支持将JSON的护卫直接改装为Dynamo地图。 这里的MS是,将JSON数据列为示意图,而不是转换为DynamoDB地图或清单类型。 但是,在移民之后,你可以在

<>Before Migration:

  • Process data in source db to convert JSON strings to a format that DMS can handle.
  • Write a script (JS/Python) to read data from your source, transform JSON into DynamoDB map and then write to DynamoDB.

<><>><>>>>>>>>

<>After Migration:

  • Read the data from DynamoDB, and then parse JSON string into Map, and then write back to DynamoDB.
  • Write a script which itereate over every time in the DynamoDB table, retrieve the string data, convert it into a Map, and then update item in the DynamoDB.

下面是AWS SDK后处理案例的Nodejs。

const AWS = require( aws-sdk );

AWS.config.update({
  region: "your_region",
  // Add your credentials
});

const dynamodb = new AWS.DynamoDB.DocumentClient();
const tableName =  your_table ; // replace with your table name

const scanAndProcessItems = async (startKey) => {
  try {
    const params = {
      TableName: tableName,
      ExclusiveStartKey: startKey
    };
    
    const data = await dynamodb.scan(params).promise();
    
    if (data.Items) {
      for (let item of data.Items) {
        // To parse JSON strings into DynamoDB Maps
        item.createdBy = JSON.parse(item.createdBy);
        item.modifiedBy = JSON.parse(item.modifiedBy);
        item.preferences = JSON.parse(item.preferences);
    
        // To write the item back to the table
        await dynamodb.put({ TableName: tableName, Item: item }).promise();
      }
    }
    
    // If there s more data to scan, it will recursively call this function
    if (data.LastEvaluatedKey) {
      await scanAndProcessItems(data.LastEvaluatedKey);
    }
    
  } catch (err) {
    console.error("An error occurred", err);
  }
};

scanAndProcessItems();

希望这将有助于你们。





相关问题
热门标签