2022-10-21

Getting file from S3 and copying to stream is taking ages

I am trying to get a 3MB file from a S3 bucket and copying it to a MemoryStream because I will need to check what is inside this file at a later point. However, when CopyToAsync gets called, it takes at least 1 minute to complete.

Here's the code I have right now:

using var s3Object = await _client.GetObjectAsync(getRequest);

var ms = new MemoryStream();

// taking ages
await s3Object.ResponseStream.CopyToAsync(ms);

return ms;

I tried to use the transfer utility to open the stream or download the file but it takes roughly the same amount of time.

Am I doing something wrong? Is there a different approach?

From my understanding the s3Object variable is receiving metadata about the s3 object + the beginning of the stream of the mentioned file. So when CopyToAsync gets called it actually "downloads" the complete file.

The file looks something like this:

DEPT    .M                                      :1     Index curve
GR      .GAPI                                   :2     L05-07
DT      .US/F                                   :3     L05-07
RHOB    .G/C3                                   :4     L05-07
DRHO    .G/C3                                   :5     L05-07
NPHI    .V/V                                    :6     L05-07
~Ascii Log Data
      63.1000   -999.250000   -999.250000   -999.250000   -999.250000   -999.250000
      63.2000   -999.250000   -999.250000   -999.250000   -999.250000   -999.250000
      63.3000   -999.250000   -999.250000   -999.250000   -999.250000   -999.250000
      63.4000   -999.250000   -999.250000   -999.250000   -999.250000   -999.250000
      63.5000     11.094475   -999.250000   -999.250000   -999.250000   -999.250000
      63.6000     11.473166   -999.250000   -999.250000   -999.250000   -999.250000
      63.7000     12.066418   -999.250000   -999.250000   -999.250000   -999.250000
      63.8000     12.302490   -999.250000   -999.250000   -999.250000   -999.250000
      63.9000     12.272800   -999.250000   -999.250000   -999.250000   -999.250000
      64.0000     11.762026   -999.250000   -999.250000   -999.250000   -999.250000
      64.1000     10.414513   -999.250000   -999.250000   -999.250000   -999.250000
      64.2000      9.588345   -999.250000   -999.250000   -999.250000   -999.250000

Thanks for any help!



No comments:

Post a Comment