Question

我已探索了有效途径,获取S3桶的元数据,如其大小和档案数量。见http://serverfault.com/questions/84815/how-can-i-get-the-size-of-an-amazon-s3-bucket>。联系讨论此类问题。但是,使用云层观察,它需要PHP和 a。我想知道,有几只 j瓦皮,以 f取3 b子元数据吗?

增编

Answer 1

你可以在这里找到AWS S3 Java图书馆的广泛文件:

rel=“nofollow” http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/overview-summary.html

在回答你的问题时,你可以利用S3号物体的大小,并且你可以把所有文件整理起来,以找到你的包裹。

S3并不支持直接收集磁盘使用(并非通过所有项目进行存储),但你可以利用云层收集你希望使用的数据。

例:

aws cloudwatch get-metric-statistics --namespace AWS/S3 --start-time 2016-01-01T10:00:00 --end-time 2016-02-12T01:00:00 --period 86400 --statistics Average --region us-east-1 --metric-name BucketSizeBytes --dimensions Name=BucketName,Value=www.streambrightdata.com Name=StorageType,Value=StandardStorage

回返:

{
    "Datapoints": [
        {
            "Timestamp": "2016-02-05T10:00:00Z",
            "Average": 54027423.0,
            "Unit": "Bytes"
        },
        {
            "Timestamp": "2016-02-03T10:00:00Z",
            "Average": 52917504.0,
            "Unit": "Bytes"
        },
        {
            "Timestamp": "2016-02-04T10:00:00Z",
            "Average": 53417421.0,
            "Unit": "Bytes"
        },
        {
            "Timestamp": "2016-02-07T10:00:00Z",
            "Average": 54949563.0,
            "Unit": "Bytes"
        },
        {
            "Timestamp": "2016-02-01T10:00:00Z",
            "Average": 24951965.0,
            "Unit": "Bytes"
        },
        {
            "Timestamp": "2016-02-02T10:00:00Z",
            "Average": 28254636.0,
            "Unit": "Bytes"
        },
        {
            "Timestamp": "2016-02-06T10:00:00Z",
            "Average": 54577328.0,
            "Unit": "Bytes"
        }
    ],
    "Label": "BucketSizeBytes"
}

AWS Java SDK forumi Watch:

Answer 2

awsdk Java2.x

    Set<String> fileTypes = new HashSet<>();
    ListObjectsResponse listObjResp = amazonS3Client.listObjects(ListObjectsRequest.builder().bucket(bucketName).build());
    int iCount=1;
    //********************************************************************//
    log.info("listObjResp.isTruncated() : "+listObjResp.isTruncated());
    String nextMarker = null;
    do {
        String sKey = null;
        List<S3Object> s3ObjList = listObjResp.contents();
        for (S3Object s3Obj: s3ObjList) {
            sKey = s3Obj.key();
            String[] sKeyValues = sKey.split("\.");
            if(sKeyValues.length==2) {
                fileTypes.add(sKeyValues[1]);
            }else {
                fileTypes.add(NO_FILE_EXT);
            }
            ++iCount;
        }
        nextMarker = listObjResp.nextMarker();
        log.debug("listObjResp.nextMarker() : "+nextMarker);
        listObjResp = amazonS3Client.listObjects(ListObjectsRequest.builder().bucket(bucketName).marker(nextMarker).build());
    } while (nextMarker !=null);
    
    log.info("iCount of  "+bucketName+" : "+(iCount-1));

Answer 3

您可使用<代码>MinioAdminClient及其getDataUsageInfo(方法,以获得你们所需要的所有信息。

It can be connected via this link and configured in the same way as MinioClient by passing parameters with the help of a builder instead of a constructor.

@Bean
public MinioClient minioClient(
        @Value("${aws.endPoint}") String endPoint,
        @Value("${aws.accessKey}") String accessKey,
        @Value("${aws.secretKey}") String secretKey) throws InvalidPortException, InvalidEndpointException {

    return new MinioClient(endPoint, accessKey, secretKey);
}

@Bean
public MinioAdminClient minioAdminClient(
        @Value("${aws.endPoint}") String endPoint,
        @Value("${aws.accessKey}") String accessKey,
        @Value("${aws.secretKey}") String secretKey){

    return MinioAdminClient
            .builder()
            .endpoint(endPoint)
            .credentials(accessKey, secretKey)
            .build();
}

与Gite Hub Repo的档案链接:https://github.com/minio-java/blob/master/adminapi/rc/main/java/minio/admin/MinioAdminClient.java3

Answer 4

你们需要这个桶子,但这一解决办法依赖于AWS云层,因为S3目前并不直接支持getting水的大小(Feb 2024)。

With Amazon CloudWatch to get the stats of an s3 bucket (which includes size as a dimension) it would be something like this (needs some refactoring):

  package com.allpalabs.haces.config;

import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
import com.amazonaws.services.cloudwatch.AmazonCloudWatchClientBuilder;
import com.amazonaws.services.cloudwatch.model.*;

import java.util.ArrayList;
import java.util.Collections;
import java.util.Date;
import java.util.List;

public class AwsCloudWatchMetricClient {
    public static final AmazonCloudWatch cw = AmazonCloudWatchClientBuilder.defaultClient();
    public static final String namespace = "AWS/S3";
    public static final String metricName = "BucketSizeBytes";

    public static double getAwsCloudWatchS3Metric(String bucketName) throws ResourceNotFoundException
    {
        double bucketStorageUsedInBytes = -1;

        final GetMetricDataRequest getMetricDataRequest = new GetMetricDataRequest();
        final MetricDataQuery metricDataQuery = new MetricDataQuery();
        metricDataQuery.setId("s3StorageMetric");
        final MetricStat metricStat = new MetricStat();
        final Metric metric = new Metric();
        metric.setNamespace(namespace);

        final Dimension dimensionStorageType = new Dimension();
        dimensionStorageType.setName("StorageType");
        dimensionStorageType.setValue("StandardStorage");

        final Dimension dimensionBucketName = new Dimension();
        dimensionBucketName.setName("BucketName");
        dimensionBucketName.setValue(bucketName);

        List<Dimension> dimensions = new ArrayList<>();
        dimensions.add(dimensionBucketName);
        dimensions.add(dimensionStorageType);

        metric.setDimensions(dimensions);
        metric.setMetricName(metricName);
        metricStat.setMetric(metric);
        metricStat.setPeriod(86400);
        metricStat.setStat("Maximum");
        metricDataQuery.setMetricStat(metricStat);
        getMetricDataRequest.setMetricDataQueries(Collections.singletonList(metricDataQuery));
        final Date endTime = new Date();
        getMetricDataRequest.setEndTime(endTime);
        getMetricDataRequest.setStartTime(new Date(endTime.getTime() - (3600*100000)));


        GetMetricDataResult response = cw.getMetricData(getMetricDataRequest);

        boolean done = false;


        while(!done) {
//            ListMetricsResult response = cw.listMetrics(request);

            for(MetricDataResult metricGathered : response.getMetricDataResults() ) {
                System.out.printf(
                        "Retrieved metric data is %s", metricGathered.getValues());
                System.out.println(metricGathered.toString());
                System.out.println(metricGathered.getMessages());

                List<Double> vals = metricGathered.getValues();

                if(!vals.isEmpty()) {
                    bucketStorageUsedInBytes = vals.get(vals.size() - 1);
                    return bucketStorageUsedInBytes;
                }else {
                    throw new ResourceNotFoundException("Problem adquiring S3 Maximum storage values for this bucket.");
                }

            }

            response.setNextToken(response.getNextToken());

            if(response.getNextToken() == null) {
                done = true;
            }
        }

        return bucketStorageUsedInBytes;
    }

}

友情链接