Pagination
Cursor-based pagination is the method used to navigate through the large datasets offered by the GitGuardian API. This method allows you to retrieve items in chunks (pages) by using cursors, which are pointers to a specific item in a dataset. The cursor indicates the position in the dataset, making it easier to navigate back and forth between pages.
How cursor-based pagination works
- Initial Request: When you make the first request to an endpoint, you receive the first set of results
- Subsequent Requests: Use the response's
link
header to request the next set of results.link: <https://api.staging.gitguardian.tech/v1/members?cursor=cD03Mjc%3D>; rel="next"
- No more data: If there's no more data to retrieve, the response will have no
link
header.
Examples
Python
import os
import requests
BASE_URL = "https://api.gitguardian.com/v1"
API_KEY = os.environ['GITGUARDIAN_API_TOKEN']
HEADERS = {"Authorization": f"Token {API_KEY}"}
endpoint_url = f"{BASE_URL}/members?per_page=10"
all_members = []
while True:
response = requests.get(endpoint_url, headers=HEADERS)
assert response.status_code == 200, response.json()
all_members += response.json()
if "next" not in response.links:
# final page was reached
break
endpoint_url = response.links["next"]["url"]
Bash/cURL
#!/bin/bash
URL="https://api.gitguardian.com/v1/members?per_page=10"
# Check if jq is installed
if ! command -v jq &> /dev/null; then
echo "jq could not be found, please install it to parse JSON."
exit 1
fi
MEMBERS="[]"
while [ "${URL}" ]; do
RESP=$(curl -i -Ss -H "Authorization: Token $API_KEY" "${URL}")
# Check for curl error
if [ $? -ne 0 ]; then
echo "Error: Failed to retrieve data from ${URL}"
exit 1
fi
# Extract HTTP status code
HTTP_STATUS=$(echo "$RESP" | grep HTTP | awk '{print $2}')
if [ "$HTTP_STATUS" != "200" ]; then
echo "Error: Received HTTP status $HTTP_STATUS"
exit 1
fi
# Retrieve the body of the response and parse it with jq
BODY=$(echo "$RESP" | sed -n '/^\r$/,$p' | sed '1d' | jq '.')
# Append to members list
MEMBERS=$(echo "${MEMBERS}" | jq ". + ${BODY}")
# Retrieve the next url from the response's headers
URL=$(echo "$RESP" | grep -i '^link:' | sed -n -E 's/^link:.*<(.*)>; rel="next".*/\1/p')
done
# Output the members list formatted with jq
echo "${MEMBERS}" | jq '.'