Aawaara

Getting amazon SNS ans SQS feelers with some benchmarks

We are exploring Amazon SNS and SQS services to build our notifications system at Pratilipi. To get started and have some initial idea about whether it works the way it has been promised, I decided to do some basic benchmarking. Please note that I did this on my local machine to know whether the technologies fit our usecase, or we can reject them outrightly. I am sharing these in the hope that someone else in a similar situation might find this helpful. These are not production benchmarks, and the sole purpose is to get some firsthand feelers.

Below are a couple of Node.js apps built using sails that I created to carry out these benchmarks:

1. https://github.com/amitsaxena/sns_experiment (used to send a notification to SNS)
2. https://github.com/amitsaxena/sqs_experiment (used to send and receive jobs to and from SQS)

I used the good old apache bench to generate concurrency and load. The requests were being served by a single node server instance on my local machine with an average internet connection. Below are the test results:

50K requests at 100 concurrency sending messages to SNS:

50K requests at 100 concurrency receiving messages from SQS:

To also account for network latency, I hit one of our ECS container instance in the same AWS region to get an average round trip time (time spent by our request/response in transferring data over network) for our requests.

5K requests with 100 concurrency doing a health check

Below are my observations:

• The average time per request for sending messages to SNS is 474ms
• The average round trip time for our requests (health check) in the same AWS region is 538ms
• Looking at the above, feels like SNS is going to give us < 10 millisecond latency when we are going to trigger events and notifications from our app servers in the same AWS region as the SNS topics
• The average time taken to retrieve a message from SQS and delete it as well is 660ms
• Taking into account the network latency, it looks like it will take around 150ms to retrieve and delete a job from SQS (from our app servers)
• There were 0 failures when sending 50K messages
• There were 0 failures when retrieving 50K messages
  

Based on the above data it feels like that both SNS and SQS are good pieces of technology for the problems we are trying to solve here. We are fine with a few seconds of lag in our notifications system and are planning to go ahead with these.

If you have any questions or thoughts to share, please feel free to shoot a comment. How has your experience with SNS and SQS been?

If you use Google cloud with Docker and are suddenly seeing an invalid_grant error, restart docker first!

May be this helps someone from hours of frustration and pointless debugging! A working piece of code suddenly started throwing ‘invalid_grant’ errors and after debugging for hours (IAM permissions, matching secret keys, extreme logging, and what not!) I finally realized that the time in docker container was behind my system time, and that was causing the errors. Looks like it is some docker bug which people are talking about here:
https://forums.docker.com/t/time-in-container-is-out-of-sync/16566

Restarting docker fixed it instantaneously. So if you ever find yourself in the above situation (or similar situation where a third party API call suddenly starts failing withing a docker container), restart docker before doing anything else.

How to return a 404 response code for CloudFront requests (S3 origin) of a non-existent S3 object

If you are here, you already know that you get a 403 response code (unauthorized) for this scenario. I don’t want you to get drowned in the sea of confusing Amazon documentation. Below is how you can do it.

Open the AWS management console, and go to your specific bucket and click properties tab

Now under “Permissions” click “Edit bucket policy“

Your policy should look like something below. If you don’t have one, add a new one. Please note that this policy is used to access private content (via CloudFront) on S3, but it probably will work for public content as well.