Hey atw, that Censys Python git repo you linked is only compatible with our Legacy Search product. It will not work with Platform API creds, which are what you have.

You should be using the Platform API endpoints provided in the docs page you linked. An example search query cURL including your orgID and PAT would be as follows:

curl --request POST \
     --url 'https://api.platform.censys.io/v3/global/search/query?organization_id=1234' \
     --header 'accept: application/json' \
     --header 'authorization: 5678' \
     --header 'content-type: application/json' \
     --data '
{
  "query": "host.ip: 1.1.1.1/16"
}
'

Additional developer support for Platform in the form of SDKs and a tool similar to Censys Python are coming soon.

Thanks ​@MattK_Censys for the clarification. I have a script now that is working, but I cannot get it to capture more than 500 results from a Search query. On the web search, this query returns over 1,000 results.

I’m doing pagination with a page size of 100 with the API, but it ends after 500 every time. Do you know why this might be?

Same problem, nothing is clear, could have written clear documentation

Hey atw and Senior Shade, Starter tier users are limited to 500 results per search query. This is detailed on the pricing page on the website, but it could have been clearer--I am following up with our team about that. I also just updated the docs to make this clearer as well.

I recommend focusing your queries using filters to get under the 500 result limit. If you’re going to frequently run a query to identify any changes, I recommend using collections.

I do want to use Collections to identify changes, but just trying to get the initial data set into a CSV.
Any hack you can think of to split up the results somehow? I’m thinking maybe filtering by the first 2 IP octets to try to build multiple queries in order to get the desired data, or some stupid hack like that.

Also https://censys.com/resources/pricing doesn’t say anything about an API 500 result limit.

My hack to work around the odd 500 limitation was to create queries that filtered “web.scan_time” with smaller chunks of time, was able to export the ~3,000 results I needed this way.