Performance, Limits and Offset Pagination

Authentication Error Limits

After 40 failed authentication attempts from the same IP address in less than 2 min, the soure requestor IP address will be blocked for 2 hours. All requests to authentication endpoints will be blocked.

TrackTik Customer Support and DevOps can be contacted to have the IP ban lifted sooner than the 2 hour window.

Default Number of Items Returned

By default, if no requested number of result items is requested with a filter parameter, 100 items will be returned.

Maximum Number of Returned Items

The maximum number of returned items is 1000, and you can request up to it using the limit filter:

[GET] /zone-clients?limit=1000

"meta": { "count": 3357, "request": [ "zone-clients" ], "itemCount": 1000, "security": { "granted": true, "requested": "admin:zone-clients:view", "grantedBy": "admin:*", "scope": null }, "debug": null, "resource": "zone-clients", "limit": 1000, "offset": 0 }

Notice how there are 3357 possible items to return (count value), but that with the limit=1000, and looking at the itemCount value, only 1000 of the 3357 are returned.

Paginating Items Returned (offset)

If you want to see a different subset of the possible items that can be returned, you can use the offset filter to request “the next interval of items as defined by the limit value in the API request”.

Let’s say you’re looking at a sorted list of employees in groups of 5 (for performance, or some tool with a GUI). You would request the first 5 with:

[GET] /employees?limit=5&fields=id,name,email&sort=id

{ "id": 1000, "name": "TrackTIk Support (HQ)", "email": "E@E.com" }, { "id": 1001, "name": "TrackTik Support (NE)", "email": "" }, { "id": 1002, "name": "TrackTik Support (NW)", "email": "" }, { "id": 1003, "name": "TrackTik Support (SE)", "email": "" }, { "id": 1004, "name": "TrackTik Support (SW)", "email": "" }

Now let’s add an offset to get the next 5, not including the previous items already returned:

{ "id": 1005, "name": "Joan Slater", "email": "" }, { "id": 1006, "name": "John Mills", "email": "BMills@prioritysecurity.com" }, { "id": 1007, "name": "Jake Bond", "email": "JBond@prioritysecurity.com" }, { "id": 1008, "name": "Molly Sutherland", "email": "MSutherland@prioritysecurity.com" }, { "id": 1009, "name": "Mike Brown", "email": "mbrown@prioritysecurity.com" }

 

Quick Summary

You already know about limits:
[GET] /employees?limit=5&fields=id,name,email&sort=idA request like that will get the first 5 records stored in the database (because of the primary key sorting of sort=id parameter)

But to get the next 5 after that, you can use an offset value. The offset value is the number of rows to skip over before beginning the count up to the limit. So if the previous call returned the first 5, to get the next 5 after that would be:

[GET] /employees?limit=5&fields=id,name,email&sort=id&offset=5

And for the next 5 after the first 10:
[GET] /employees?limit=5&fields=id,name,email&sort=id&offset=10

When developing a call loop to essentially paginate like this, you would set the limit variable to 5, and the offset to the limit variable times the number of pages to skip ahead to.

First (no pages) 5: limit 5, offset 0 (offset = 5 x 0)
Second 5 (page 1): limit 5, offset 5 (offset = 5 x 1)
Third 5 (page 2): limit 5, offset 15 (offset = 5 x 2)

Performance is More Often About Record Counts in the Database

In day to day integrations with customers, we’ve noticed that even if you filter and offset, when there is an endpoint that requires at least one search filter, like the reportDateTime value of reports accessible via /reports, the back-end data retrieval process still has to crawl through the entire recordset finding matches with the required filter.

This means that if a customer database has 17 million reports (like our customer HSS does), that any little request to /reports in that dataset is going to take several minutes. This can be improved by looking for ranges of primary keys instead of indexed fields, but it will still take a while.

When working with large data sets, it’s important to manage expectations. Filters can only do so much.

API Limits at the Firewall

Currently the API solution is not trapping requests as a limit to return a “429 Too Many Requests” HTTP status code. Instead, the cloud level Web Application Firewall blocks source IP addresses that generate more requests than 4000 requests per 5 minutes.

Once the limit is exceed, the IP address is blocked. The requests that continue to be made will receive a response of:

403 Forbidden

After being blocked, the requests are still noticed and counted to see if the rate is reduced from 4k/5min.

Once the rate is reduced in the next 5 minute interval, the IP address is unblocked.

Relation Lists Include Limits

When an entity’s list items include a relation list, they cannot be included in the basic GET call to a list of the entity’s records. You can’t call /reports for example and include report-fields (same with /shifts and the relation list to notes).

You can however call a single record of an entity, and include the relation list records for it:

E.g. /reports/id?include=reportFields

:warning: But, upon review 2022-08-05, there is a hard limit to how many relation list items can be included.
The limit is 100 items.

Therefore, the best practice is that if you need to access all the items of the relation list, you should obtain that list through its dedicated endpoint:

Instead of : /reports/id?include=reportFields
Use: /report-fields

Instead of: /shifts/id?include=notes
Use: /shift-notes

When a solution using the API needs to obtain a list of parent entities, and lists of items related to it, the burden is on the integrator to first generate a list of parent entity IDs, and then for each of those, call the separate API endpoint for list items and filter on the parent entity, iterating through each parent.

Pseudocode of an approach:

  • Get list of Report IDs → Store in an array where the array index is the report id for each: Report_ID[report_id]
  • For each Report_ID[report_id] → Get data from API for Report Fields (/report-fields?report=report_id)
  • Store the Report Fields data as a new associative array:
    Report_Data[report_id] = Report_ID[report_id]
    Report_Data[report_id]['fields'] = Report Fields JSON object converted to array element
Was this article helpful?
0 out of 0 found this helpful

Articles in this section