Offset not working beyond a certain number https://circleci.com/api/v1.1/project/github/username/reponame/?limit=100&offset=400

API
1

I am trying to get all builds of “bbyars/mountebank” repo. It stops after something.
Could someone help me?

2

It would be worth you moving to the v2 API rather than trying to fix issues with the v1.1 API.

This is because the v1.1 API has been marked with the rather unhelpful message " v1 APIs will be deprecated within the next year. We recommend switching to the v2 APIs." for a very long time now. I say unhelpful as the message provides no date information and really means the v1.1 API.

In terms of the v1.1 request can you post the exact code you are using as someone maybe able to spot any issue.

3

Here is my code. Pagination stops half way through. Please help me to fix this! I really appreciate it!

import requests
import json
import pandas as pd
import time
import os

def get_circleci_data(owner_and_repo, limit=100, offset=100):
    api_endpoint = "https://circleci.com/api/v1.1/project/github/" + owner_and_repo + "?limit=" + str(
        limit) + "&circle-token=" + circleci_token

    # Make an initial API request to get the first page of builds data
    response = requests.get(api_endpoint)
    builds = []

    # Extract the builds data from the response
    if response.ok:
        data = response.json()
        if (len(data) == 0):
            print("No builds at all.")
        else:
            builds += data
            print("len(builds)", len(builds))
            n_pages = int(data[0]['build_num'] / offset)  # remaning number of pages
            next_offset_page = 1
            print("n_pages", n_pages)
            print(data[0]['build_num'])

            # Continue making API requests with the page-token parameter to get all builds data

            while next_offset_page <= n_pages:
                new_api_endpoint = api_endpoint + "&offset=" + str(next_offset_page * offset)
                print(new_api_endpoint)
                response = requests.get(new_api_endpoint)
                if response.ok:
                    data = response.json()
                    if (len(data) == 0):
                        print("No more builds left.")
                        break
                    print(data[0]['build_num'])
                    builds += data
                    next_offset_page += 1
                    print("Got {} builds, next offset_page: {}".format(len(builds), next_offset_page))
                else:
                    print("Error:", response.status_code, response.reason)
    else:
        print("Error:", response.status_code, response.reason)

    return builds

# run the code
project_name = "bbyars/mountebank"
print(get_circleci_data(project_name))
4

Thanks for @rit1010 for the comment. I will check with API v2 as well.
I also posted my python code.

5

OK I may be missing something here as I am trying to follow your code by reading it but at the moment you

  • set the basic request with

    api_endpoint = "https://circleci.com/api/v1.1/project/github/" + owner_and_repo + "?limit=" + str(
  limit) + "&circle-token=" + circleci_token

  • As you start to page the results you then use

    new_api_endpoint = api_endpoint + “&offset=” + str(next_offset_page * offset)

This would seem to result in requests that look like the following

https://circleci.com/api/v1.1/project/github/<owner_and_repo>/?limit=100&&circle-token=<token value>&offset=400

As I’ve not worked with the v1.1 API at all I can not say if this would work, but it risks problems as all the (limited) examples have the circle-token being passed as a customized header value, rather than part of the query. This does not mean that the API may not accept the value being placed at the end of the query, but you then start to append the offset to the query and the circle-token is now within the query string. The end result is that you become very dependent on how the application code processes the information held within the query.

1 Like