Forbidden - no user agent has been specified in the request.

[wKovacs64]

Symptom:

hibp requests made outside of a browser (e.g. Node.js) result in the following:

Error: Forbidden - no user agent has been specified in the request.

Synopsis:

As first noticed on 2018-06-26, the remote API seems to have started enforcing a long documented but never enforced rule requiring a User-Agent header be included in all API requests. hibp does not currently include this header (see below for reasons why).

Backstory:

I (original author of this library) had every intention of honoring this requirement, but ran into what I believe is a misconfiguration or partial implementation of CORS on the API side of things which prevented consumers from sending a custom User-Agent request header in the browser. I contacted the API author/maintainer and provided a very detailed analysis of my findings, but the CORS configuration for the API remained the same. As such, hibp does not currently send a User-Agent header in its requests, as I noticed the rule was not being enforced anyway.

Proposed Solution:

hibp can check if it's running outside the browser and include a custom User-Agent header in its requests. In a browser environment, it will continue to allow the browser to include its own User-Agent and thus avoid CORS.

[wKovacs64]

Update (2018-06-27):

After reaching out to the remote API owner again, he tracked down the CORS issue and the configuration has been fixed! 🎉 That said, he does not expect the User-Agent to be anything other than the one the browser normally sends when the request is made in a browser environment, so the code changes made here to only set it when running outside the browser can remain.

[tmackness]

I am also getting this error when trying in runkit. Maybe add a global option to set a header in axios e.g. const hibp = require('hibp')(my-agent-name); or as an object: const hibp = require('hibp')({useragent: "my-agent-name"}); or maybe const hibp = require('hibp')({headers: ["User-Agent: my-name"]});

[wKovacs64]

Hmm, something has changed. Investigating.

[kierxn]

This is currently causing us issues as well.

[wKovacs64]

I'm reaching out to Troy, something seems to have changed on his end and request are being denied with a message in the response body indicating the requester has been blocked.

[kierxn]

Looks to be a recent thing on the breached accounts endpoint.

I could previously visit the example link on the API documentation here and it would return the response, now it returns a message that I've been blocked.

Hopefully this can be sorted soon :)

[wKovacs64]

Oh, good catch! So that pretty much confirms it's nothing on this end. I'll leave this issue open for now, but I doubt there is anything we can do but wait.

[kierxn]

I've been in contact with Troy today and he said he's made some tweaks of the back of my messages and everything is looking good on my side now.

Could you test and let me know?

I can curl the endpoint fine right now but I'm still seeing:

Error: Forbidden - no user agent has been specified in the request.

The code is currently run server side as well. I think going forward it would be great to be able to choose our own user-agent when requiring the library.

[wKovacs64]

I'm still being blocked in the browser (visiting the example URL) on basically every network I have access to, but library usage is working again.

His API documentation states 403 Forbidden responses are indicative of failing to include a User-Agent request header, which is why you see that message. This particular problem is not a matter of omitting the User-Agent, but rather being blocked for other reasons (abuse prevention, likely Cloudflare) which also respond with 403 Forbidden. So I think the message should be modified to be more generic (maybe just "Forbidden - access denied" or similar, or possibly go the other way and make it more detailed (perhaps a link to somewhere explaining you might have been blocked by the remote API). I don't see exposing the User-Agent as an option to add much value (not adamantly opposed, but would need a little more convincing as to its usefulness).

[kierxn]

I spoke to him and if I made a GET request to the page it failed along with a CURL request.

After he made some tweaks I've since been able to CURL the page with a User-Agent set.

$ curl -H "User-Agent: kieran-hibp-integration" https://haveibeenpwned.com/api/v2/breachedaccount/test@example.com
[{"Name":"000webhost","Title":"000webhost","Domain":"000webhost.com"... <snip>

Otherwise I get:

curl -H "User-Agent: usecure-hibp-integration" https://haveibeenpwned.com/api/v2/breachedaccount/test@example.com



...
<div class="cf-error-details cf-error-1020">
  <h1>Access denied</h1>
  <p>This website is using a security service to protect itself from online attacks.</p>
  <ul class="cferror_details">
    <li>Ray ID: 49b01be2db8a365f</li>
    <li>Timestamp: 2019-01-18 09:41:58 UTC</li>
    <li>Your IP address: <snip></li>
    <li class="XXX_no_wrap_overflow_hidden">Requested URL: haveibeenpwned.com/api/v2/breachedaccount/test@example.com </li>
    <li>Error reference number: 1020</li>
    <li>Server ID: FL_63F12</li>
    <li>User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
</li>
  </ul>
</div>

I think adding the ability to set a custom user-agent will do the job for server side. Other than that I'm not entirely sure you'd fix it for client facing applications.

[wKovacs64]

What user-agent would you set that you feel would help? I already set one if you aren't in a browser.

[kierxn]

The way I see it is, if I use this library and the default UA gets blocked because somebody else using the UA abuses the service or accidentally does something they shouldn't have then blocks it for everyone.

If I can customise it, it reduces the risk and lets me identify my app to the endpoint where Troy can relax rules for it/help debug issues.

Saves me the headache of having to rewrite this from scratch for myself when there's another problem!

[wKovacs64]

It sounds like you're under the impression the blocking is happening based purely on UA. Is that the case? That means every user in the world using the latest stable version of Chrome on a particular OS is blocked right now if one bad actor used that popular combination to abuse the service? That doesn't seem feasible. 🤔

If that is indeed the case, then yeah, I can see the benefit of exposing a UA option. I guess we could add it regardless, I just want to be careful with exposing too many non-HIBP related options.

[wKovacs64]

Note: a custom UA would not currently work when using hibp in the browser as Troy's API does not include User-Agent in the Access-Control-Allow-Headers header field, so attempting to use a custom UA will be blocked. See here for a demonstration of this:

https://www.test-cors.org/#?client_method=GET&client_credentials=false&client_headers=User-Agent%3A%20test123&server_url=https%3A%2F%2Fhaveibeenpwned.com%2Fapi%2Fv2%2Fbreachedaccount%2Ftest%40example.com&server_enable=true&server_status=200&server_credentials=false&server_tabs=remote

[kierxn]

It sounds like you're under the impression the blocking is happening based purely on UA. Is that the case? That means every user in the world using the latest stable version of Chrome on a particular OS is blocked right now if one bad actor used that popular combination to abuse the service? That doesn't seem feasible. 🤔

Yeah, I absolutely have no idea but that's the only way I can seem to replicate the issue. I did a bit more testing on this and it seems to be entirely UA based:

curl -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" https://haveibeenpwned.com/api/v2/breachedaccount/test@example.com

...
<h1>You have been blocked from accessing this resource on Have I Been Pwned.</h1>
...

And with a custom UA:

curl -H "User-Agent: kieran-usec-hibp" https://haveibeenpwned.com/api/v2/breachedaccount/test@example.com
[{"Name":"000webhost","Title":"000webhost","Domain":"000webhost.com","BreachDate":"2015-03-01", ...

If that is indeed the case, then yeah, I can see the benefit of exposing a UA option. I guess we could add it regardless, I just want to be careful with exposing too many non-HIBP related options.

I think having a custom UA is definitely a HIBP option though considering the documentation tells you to use a custom one that explains what your app is.

[wKovacs64]

It definitely appears to be UA related, but not 100%. For example, here is a user being blocked, but the same command works fine for me. Same UA.

Edit: And again, adding an option to customize the UA is worthless at the moment as that is not currently allowed when running in the browser.

[wKovacs64]

Closing in favor of #60 as this issue was originally for a different problem that has since been resolved but was emitting the same (misleading) error message. We can continue the discussion there.