# Reuse ECR Login Token and/or retry login **URL:** \ **Category:** Bug Reports\ **Tags:** aws, docker\ **Created:** [April 24, 2018, 5:04pm UTC](https://discuss.circleci.com/t/reuse-ecr-login-token-and-or-retry-login/21931 "2018-04-24T17:04:33Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![wesm](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/wesm/32/9726_2.png) [@wesm](https://discuss.circleci.com/u/wesm)\ **Post date:** [April 24, 2018, 5:04pm UTC](https://discuss.circleci.com/t/reuse-ecr-login-token-and-or-retry-login/21931/1 "2018-04-24T17:04:33Z")
We often encounter build failures due to ECR login throttling Build-agent version 0.0.4869-fac853b (2018-04-17T20:59:55+0000) error authentication with ECR: ThrottlingException: Rate exceeded status code: 400, request id: c7b48609-475c-11e8-9f28-77bef5fdcc37 This happens during `Spin Up Environment` step. So there’s nothing we can do on our side to mitigate this failure ---
**Author:** ![halfer](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/halfer/32/9083_2.png) [@halfer](https://discuss.circleci.com/u/halfer)\ **Post date:** [April 24, 2018, 6:53pm UTC](https://discuss.circleci.com/t/reuse-ecr-login-token-and-or-retry-login/21931/2 "2018-04-24T18:53:43Z")
How many times is that being tried, and in what timeframe? Is it being tried a lot because it is happening once per job, and you have many jobs in a workflow? Can you apply to AWS to get the throttle limits changed for your account on ECR? ---
**Author:** ![wesm](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/wesm/32/9726_2.png) [@wesm](https://discuss.circleci.com/u/wesm)\ **Post date:** [April 25, 2018, 4:47pm UTC](https://discuss.circleci.com/t/reuse-ecr-login-token-and-or-retry-login/21931/3 "2018-04-25T16:47:22Z")
We have a workflow which has 32 jobs, each job uses 6 docker images from AWS ECR which means we are doing 192 logins at the same time for 1 build. Multiply that by N number of concurrent builds, and it’s easy to see how we’d run into throttling limits. Per the documentation (below): - the throttle for the GetAuthorizationToken action is 4 transaction per second (TPS), with up to a 200 TPS burst allowed - the throttle on the GetAuthorizationToken operation cannot be increased on a per-account basis - To handle throttling errors, implement a retry function with incremental backoff into your code. - To avoid needing to retry, the token should be reusable for a certain period of time. [https://docs.aws.amazon.com/AmazonECR/latest/userguide/common-errors.html#error-429-too-many-requests](https://docs.aws.amazon.com/AmazonECR/latest/userguide/common-errors.html#error-429-too-many-requests) ---
**Author:** ![halfer](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/halfer/32/9083_2.png) [@halfer](https://discuss.circleci.com/u/halfer)\ **Post date:** [April 25, 2018, 5:11pm UTC](https://discuss.circleci.com/t/reuse-ecr-login-token-and-or-retry-login/21931/4 "2018-04-25T17:11:12Z")
Gotcha. I’ve not used them, but could persist your images using workspaces? [https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs](https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs) However, I admit I don’t know how compatible that is with Circle’s native way of instantiating containers, since you may not be able to set anything up prior to Circle doing its thing. I have a greater flexibility with my own (perhaps unusual) configuration as I do manual pulls of the images I use, and then start them all with Docker Compose inside a single Docker image. This means I could, if I wished, park the images in a workspace with `docker save` and then `docker load` them when required. I can also put sleeps between pulls, if the registry provider throttling is triggered. ---
**Author:** ![wesm](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/wesm/32/9726_2.png) [@wesm](https://discuss.circleci.com/u/wesm)\ **Post date:** [April 25, 2018, 6:27pm UTC](https://discuss.circleci.com/t/reuse-ecr-login-token-and-or-retry-login/21931/5 "2018-04-25T18:27:54Z")
This is certainly on option, but I think failing on throttling is just a bug. I’ve filed a bug report with Circle. ---
**Author:** ![rohara](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/rohara/32/5744_2.png) [@rohara](https://discuss.circleci.com/u/rohara)\ **Post date:** [April 30, 2018, 8:28pm UTC](https://discuss.circleci.com/t/reuse-ecr-login-token-and-or-retry-login/21931/6 "2018-04-30T20:28:30Z")
You can’t persist images using workspaces or cache. This is a pretty bad issue, and it’s certainly complicated. I’m very confident you’re the first to hit this issue. Talking to AWS about your rate-limit could be a faster turnaround. I don’t know what’s going to be involved in fixing this from our end… but it’s a very legitimate (and frustrating!) bug report so we’ll certainly work on addressing it. ---
**Author:** ![wesm](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/wesm/32/9726_2.png) [@wesm](https://discuss.circleci.com/u/wesm)\ **Post date:** [May 1, 2018, 4:34pm UTC](https://discuss.circleci.com/t/reuse-ecr-login-token-and-or-retry-login/21931/7 "2018-05-01T16:34:40Z")
Per the documentation (below): - the throttle for the GetAuthorizationToken action is 4 transaction per second (TPS), with up to a 200 TPS burst allowed - the throttle on the GetAuthorizationToken operation **cannot** be increased on a per-account basis - To handle throttling errors, implement a retry function with incremental backoff into your code. - To avoid needing to retry, the token should be reusable for a certain period of time. [https://docs.aws.amazon.com/AmazonECR/latest/userguide/common-errors.html#error-429-too-many-requests](https://docs.aws.amazon.com/AmazonECR/latest/userguide/common-errors.html#error-429-too-many-requests) The last 2 points are something that Circle can be handling which would resolve this problem. ---
**Author:** ![system](https://global.discourse-cdn.com/circleci/original/3X/5/1/511608acda8e85906e15909b57bd0a46c1dbe291.png) [@system](https://discuss.circleci.com/u/system)\ **Post date:** [July 30, 2018, 4:34pm UTC](https://discuss.circleci.com/t/reuse-ecr-login-token-and-or-retry-login/21931/8 "2018-07-30T16:34:44Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.