API, Credits Used, and Being a Good Citizen

API
1

All,

I’m trying to write a script using the CircleCI API that queries all the repos I have on CircleCI and retrieves the amount of credits that repo has used. At the moment I’m using Python (pycircleci) because I think I saw an example using it? (I hadn’t yet seen this post.)

Now, the code (essentially) is:

from pycircleci.api import Api
circleci = Api(circleci_api_key)

gh_repos = []
# get project list from current user info
user_info = circleci.get_user_info()
for gh_repo_url in user_info['projects']:
    if gh_org in gh_repo_url:
        gh_repo_name = urlparse(gh_repo_url).path.rsplit('/')[-1]
        gh_repos.append(gh_repo_name)

credits = {}
for repo in gh_repos:
    insights = circleci.get_project_workflows_metrics(gh_org,repo)
    try:
        credits[repo] = extract_values(insights,'total_credits_used')[0]
    except IndexError:
        # I think this means no credits were used???
        credits[repo] = 0
    print(f'Repo: {repo:10} -- Credits: {credits[repo]:<10d}')
    time.sleep(10)

Now this does seem to work, but if I don’t have a time.sleep() at the end, I get a 429 Client Error: Too Many Requests for url because, well, that loop is executing so fast, CircleCI thinks I’m spamming.

I guess my question is: is there a good way to determine the wait needed? Or, perhaps, is there some way to do this “better”? I’m not great at Python and even worse at APIs/JSON/etc. I’m a Fortran programmer so a program accessing the Internet seems crazy to me! :smiley:

Note that a 10-second wait between calls isn’t too bad in truth as, if I get this working, the script might run once a day at night. I just figured I’d try and learn!

1 Like
2

This is interesting. Typically speaking, you should need to do a lot of API requests very close together to trigger rate limiting. That for loop (for repo in gh_repos) does not feel like something that should trigger the rate limit but it’s possible it is close.

It looks like you are making use of the pycircleci library. It appears that the library is not taking 429 errors into account when performing retries, so I submitted a quick PR for it:

In the meantime, you might be able to modify the library to do the retry on a 429:

from pycircleci.api import Api
circleci = Api(circleci_api_key)
# Add 429 to status_forcelist tuple
circleci._session = circleci._request_session(status_forcelist=(408, 429, 500, 502, 503, 504, 520, 521, 522, 523, 524))

That should patch the instance _session to have 429 added and might just get around the rate limit without needing the sleep. If not, you could try a low value, like sleep(0.05) along with the status_forcelist change above as well.

3

@mike Ooh. Well, that maybe did something? I’m not sure because now I can’t even duplicate my original issue (if I remove the sleep()). Now, I am on a VPN coupled with meh cable modem speed, so it’s possible at the moment things are slow enough I can never hit the rate limit?

I’ll give it a go early tomorrow morning when I might get enough speed to trigger it.