Files Upload Strategies
As I’m working on one of my side projects. I have a feature that requires image/file uploading. and how to store them to the cloud with each Pros and Cons as far as I know. I will provide the code examples in the next articles as soon as I finish them. feel free to suggest or correct me as I’m learning ;).
Uploading Strategies:
1- upload files to server Hard Drive
2- upload files to server Hard Drive as temp then store them to cloud file storage
3- upload files from client to cloud file storage using pre-signed URLs
upload files to server Hard Drive:
uploading files to our server Hard Drive is the most common style of dealing with file/image uploading in most of the tutorials.
the main idea is the user will upload files/images using the multipart form which will be sent to the server.
then we will validate them and store them as shown in the figures below.
Pros:
1- easy to implement if we are using Nodejs we can use Multer package for that.
Cons:
1- we are responsible for Hard Drive size provisioning.
2- a poor design for scaling. as we want to scale our servers as shown in the figure below. the load balancer may route us to a server that not contains the needed image.
Sure we can solve this problem with various solutions as we can use Hashing Strategies like Consistent hashing or rendezvous hashing. or using a shared Hard Drive between servers as shown below.
So as we concluded that to solve that problem we can use the shared Hard Drive. why won’t we use a cloud file store which will be shared between our servers and we won’t bother provisioning Hard Drive size ourselves? which will lead us to the next strategy.
upload files to server disk as temp then store them to cloud file storage:
As discussed with the last solution we can replace the shared Hard Drive with one of Cloud file stores like (AWS-S3, Google-Cloud Storage, Azure-File Storage) as shown in the figure below.
The user will upload the image to the server. we will store it in our Hard Drive temporarily then we will upload it to Amazon S3.
Also, we can go one step further by decoupling our image uploader from the user’s request to avoid blocking the user till the uploading to s3 finishes. especially if we need to make some image processing before uploading the image like creating thumbnails, compressing images …etc.
Pros: scalable solution compared with the first one.
Cons: we are using server resources to upload the image as we can solve that problem with the third strategy.
upload files from client to cloud file storage using pre-signed URLs:
This solution simply gives the user permission to upload files directly to our cloud file store.
Pre signed URL:
This is the one people have usually heard of. A Presigned URL is just the URL of an S3 object with a bunch of query string parameters. The query string parameters are the magic part. They contain the signature and other security-related data.
we can use the flow below to achieve our last strategy.
Pros: no processing power for image uploads as we are not needing to process images or something
cons: we have to give a reasonable expire date for the PostPresigned URL. and don’t forget about CORS. these are the concerns as far as I know.
Conclusion:
It seems like file uploads are more complex than I thought. it’s not about storing an URL to the DB of the image. also, there is no right and wrong strategy. As one of my friends always says to me. it depends on what you need.
