-
-
Notifications
You must be signed in to change notification settings - Fork 8
Architecture
The library is designed in multiple layers so that you have full flexibility and control without compromising on simplicity. Each individual abstraction layer has it's own responsibilities, but they are also designed to ensure you don't have to do a lot to extend or bugfix the library, even without changing a single line of code in the library itself.
I'll start with the lowest level of abstraction and work my way up. Note that each layer's features are accumulative, which means a layer inherits the features from the underlying layer.
The idea behind this interface is to split network functionality from the rest of the library. SimpleS3 provides you with an existing implementation called SimpleS3.Extensions.HttpClient, but you are free to implement a network driver yourself. If you decide to do so, you have to live up to the following requirements:
- Map request headers to HTTP headers
- Set the stream as the body of the HTTP request.
- Send the request over the network
- Map the HTTP response headers to a
IDictionary<string, string> - Return the HTTP status code, response headers, and HTTP body.
I would recommend you take a look at the source code of HttpClientNetworkDriver to see how to handle those requirements.
HttpClientNetworkDriver d = new HttpClientNetworkDriver(...);
await d.SendRequestAsync(HttpMethod.GET, "https://bucket.s3.amazonaws.com/myobject", new Dictionary<string, string>(), Stream.Null);The built-in HttpClientNetworkDriver comes with the following features
- Dynamically react to DNS changes in S3 every 5 minutes (provided by HttpClientFactory)
- Advanced retry/timeout policies if you add the Nuget package Microsoft.Extensions.Http.Polly
- Fully asynchronous send/receive functions.
- Different levels of logging through Microsoft.Extensions.Logging thanks to HttpClient.
This layer is responsible for converting in-memory S3 requests to HTTP requests and vice-versa. The default implementation of IRequestHandler in SimpleS3 is called DefaultRequestHandler and it utilizes a few friends to make it easier to extend the library going forward.
The first friend is the IMarshalFactory interface. MarshalFactory is the default SimpleS3 implementation that utilizes a set of IRequestMarshal and IResponseMarshal to map requests and responses. This way SimpleS3 can easily be extended with new request and response types, or fix a bug in the way mapping occurs, all without changing the source code of SimpleS3. Just build an implementation of IRequestMarshal/IResponseMarshal and add it to MarshalFactory to use it.
The second friend is IValidationFactory, which is responsible for giving DefaultRequestHandler a validator for a given request. This ensures all requests are validated before we send it to the S3 API server, but only if there is a validator registered to the request. Once a request is marshaled and validated, it is sent to the INetworkDriver.
The third friend is IRequestStreamWrapper/IResponseStreamWrapper. These 2 interfaces provide you with an extensibility point where you can wrap the request/response stream. The SimpleS3 library uses this feature to support streaming signatures, but you can also use it yourself if you want.
It is entirely possible for you to create your own request and send it through the DefaultRequestHandler. This is how you create a GetObject request with customer encryption key:
DefaultRequestHandler handler = new DefaultRequestHandler(...);
GetObjectRequest getReq = new GetObjectRequest("bucket", "key");
getReq.SseCustomerAlgorithm = SseCustomerAlgorithm.Aes256;
getReq.SseCustomerKey = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 };
getReq.SseCustomerKeyMd5 = MD5.Create().ComputeHash(getReq.SseCustomerKey);
await handler.SendRequestAsync<GetObjectRequest,GetObjectResponse>(getReq);- Validates all requests before sending to S3
- Supports both VirtualHost and Path-based bucket naming conventions
- Support streaming signatures via the IRequestStreamWrapper extensibility point
These two interfaces are designed to create a convenient way for you to get an overview of what functionality there is in SimpleS3. You don't have to look through hundreds of requests/responses and map them yourself, instead, you simply use IObjectOperations (implemented in ObjectOperations). It is built with the naming convention used by the Amazon S3 documentation, so if they have a "DeleteObject", then IObjectOperations contains a DeleteObjectAsync method.
I consider this the lowest layer that users can use. Only advanced users who wish to control everything themselves should use this layer.
The example is much like the one for IRequestHandler, except you don't need to do the type mapping of requests and responses yourself.
ObjectOperations o = new ObjectOperations(...);
GetObjectRequest getReq = new GetObjectRequest("bucket", "key");
getReq.SseCustomerAlgorithm = SseCustomerAlgorithm.Aes256;
getReq.SseCustomerKey = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 };
getReq.SseCustomerKeyMd5 = MD5.Create().ComputeHash(getReq.SseCustomerKey);
await o.GetObjectAsync(getReq);- It provides a convenient way to call operations in Amazon S3 using Amazon's naming convention.
- Object and Bucket operations are implemented in their own interfaces, which provides a clean API for you to call.
This layer lies directly above IObjectOperations and IBucketOperations respectively. The purpose of the clients is for me to provide you with an easy to use API with advanced features such as async requests and request pooling. While the S3 API is very powerful in itself, it is often not used to its full potential. The S3ObjectClient and S3BucketClient give you the full potential out of the box.
This example gets a file using encryption with a customer key. As you can see, you don't create the request, the API does that for you. To set properties on the request, you do it via a delegate as shown below.
S3ObjectClient client = new S3ObjectClient(...);
client.GetObjectAsync("bucket", "key", req =>
{
req.SseCustomerAlgorithm = SseCustomerAlgorithm.Aes256;
req.SseCustomerKey = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 };
req.SseCustomerKeyMd5 = MD5.Create().ComputeHash(req.SseCustomerKey);
});- Low memory usage due to request object pooling
- Multiple concurrent requests via asynchronous IO completion ports for optimum performance
- Can be created using dependency injection via the
AddSimpleS3Core()extension to Microsoft.Extensions.DependencyInjection.
This class is designed to allow users that have no knowledge or interest in dependency injection and how it works. Some users just want to create a new instance of a client and immediately start downloading objects. They don't care about logging abstractions, configuration, validation etc.
Working with this layer is extremely simple, but also quite limited in flexibility. It inherits from IS3ObjectClient and IS3BucketClient, so you will always be able to control both through this layer.
S3Client client = new S3Client("KeyId", "AccessKey", AwsRegion.EuWest1);
await client.GetObjectAsync("bucket","key");- Instance invocation
- Can be created using dependency injection via the
AddSimpleS3()extension.
This layer provides an extremely easy to use API for downloading and uploading objects. It is designed with usability in mind, and the fluent writing style gives a convenient way to succinctly describe a lot of properties.
In this example, we download a file using encryption with a customer key. If you look closely, we don't have to set the encryption algorithm or an MD5 of the key - this is all taken care of by SimpleS3.
await client.Transfer
.Download("MyBucket", "MyObject")
.WithEncryptionCustomerKey(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 })
.ExecuteAsync();- Fluent syntax for upload/download of objects
- All methods designed for ease of use
- Supports normal or multipart upload/download