How to get RAW DATA (JSON) via the Censys Platform API

Hi KetSC, looking into this for you. I just tried running a search API that returns your example IP from your screenshot and the data is returned in JSON format. I also ran a lookup API against that IP and it returned the same data in JSON. What is missing from your responses?

Hi MattK_Censys, thanks for your response!

I agree that the output from the Search API and the Get a Host API is almost the same.
However, when I compare those outputs with the RAW DATA (JSON) shown in the Web UI, I notice two things:

• The order of fields is different.
• There are some differences in certain fields, such as "extended_key_usage".

I understand that, due to the nature of JSON, the order of fields does not matter. However, this makes it difficult for me to confirm whether the API returns exactly the same fields and values as the RAW DATA.

According to the specification, should all data available in the Web UI RAW DATA also be retrievable via the Get a Host API or the Search API?

Hey KetSC, the API will omit boolean values that are empty or false from the response.

In this case, many of the extended_key_usage values for the certs on services on this host are false (as shown in the UI), so they are not returned in the API. If the fields or their data object is missing, then you can assume all values must be false.

Hope this helps - let me know if you have additional questions.

Hello MattK_Censys,

Thank you again for your previous clarification regarding the omission of false and empty values in the API responses. That explanation was very helpful and aligns with what I've observed.

I would like to follow up with a couple of additional points I noticed while comparing the Web UI RAW DATA and the API outputs:

• Duplicate fields in RAW DATA

In the RAW DATA, I noticed that some fields appear to be duplicated. For example, both "services.endpoints.body" and "services.endpoints.http.body" are present and seem to contain the same value.
In contrast, the Get a host API response only includes "services.endpoints.http.body".
I'm wondering if there is a specific reason for this duplication in the UI. Is there a semantic difference between these two fields, or is it simply a matter of presentation?

• Truncated body field in Search API

I also observed that the body field returned by the Search API appears to be truncated.
For instance, in this host, the HTML content ends abruptly in the Search API response, while the same host's data in the RAW DATA and the Get a Host API includes the full HTML body.
I assume this is due to the Search API returning a preview or summary version of the data. Could you confirm if this is expected behavior?

Please let me know if there is any documentation that explains these behaviors in more detail.

Thanks again for your support!

Hi ​@KetSC , I’m not able to replicate what you’re seeing with the duplicate fields. Could you provide a screenshot?

Regarding the truncated body data, only the first 2 kilobytes of HTTP headers and bodies are indexed and searchable in the Platform. As such, responses from the search API will only return the first 2 kilobytes. However, you can see more of the body when you use a lookup endpoint or view the entire body value in the raw data table in the UI.

This nuance is described in the documentation here and here but it is not clear how this works in the API - I’ll add that.

Hello MattK_Censys,

I paste an image below, this one is another example for the duplicated fields.

As you can see the yellow colored values, there are two “headers” fields.
The one is “endpoints.http.headers.Content-Length.headers”, the other one is “endpoints.headers.Content-Length.headers” and both fields have the same value.

Those two fields are summerized as “endpoints.http.headers.Content-Length.headers” in lookup API.

This is what I have asked you and I am not sure why there are those differences.

Thanks for your help.

Thanks for the screenshots KetSC. I’m unable to reproduce your issue - could you please send me a DM with the exact API call you’re making and the response, with all sensitive info redacted? Thank you.