Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

403 Forbidden Error #69

Open
warthog3 opened this issue Apr 25, 2024 · 48 comments
Open

403 Forbidden Error #69

warthog3 opened this issue Apr 25, 2024 · 48 comments

Comments

@warthog3
Copy link

Describe the problem
Seeing this new error when trying to start odmpy

`odmpy libby
odmpy Interactive Client for Libby

An unexpected error has occurred
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/libby.py", line 357, in make_request
res.raise_for_status()
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://sentry-read.svc.overdrive.com/chip/sync

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/odm.py", line 846, in run
synced_state = libby_client.sync()
^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/libby.py", line 457, in sync
res: Dict = self.make_request("chip/sync")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/libby.py", line 366, in make_request
ErrorHandler.process(http_err)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/libby_errors.py", line 103, in process
raise ClientError(
odmpy.libby_errors.ClientError: <odmpy.libby_errors.ClientError; http_status=403, msg='403 Client Error: Forbidden for url: https://sentry-read.svc.overdrive.com/chip/sync', error_response='{"result":"missing_chip"}''>
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/libby.py", line 357, in make_request
res.raise_for_status()
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://sentry-read.svc.overdrive.com/chip/sync

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.12/bin/odmpy", line 8, in
sys.exit(main())
^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/main.py", line 27, in main
run()
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/odm.py", line 846, in run
synced_state = libby_client.sync()
^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/libby.py", line 457, in sync
res: Dict = self.make_request("chip/sync")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/libby.py", line 366, in make_request
ErrorHandler.process(http_err)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/odmpy/libby_errors.py", line 103, in process
raise ClientError(
odmpy.libby_errors.ClientError: <odmpy.libby_errors.ClientError; http_status=403, msg='403 Client Error: Forbidden for url: https://sentry-read.svc.overdrive.com/chip/sync', error_response='{"result":"missing_chip"}''>`

To Reproduce
odmpy libby

Version/Environment
odmpy 0.8.1 [Python 3.12.3-darwin]

@wtanksleyjr
Copy link

wtanksleyjr commented Apr 25, 2024 via email

@leomem
Copy link

leomem commented Apr 25, 2024

The reset did not work for me. After reset and reentering the code, downloading got the following error.
python3.9/site-packages/odmpy/processing/odm.py", line 135, in process_odm raise ValueError("Unable to find Metadata in ODM") ValueError: Unable to find Metadata in ODM

The ODM file's content shows the error.

`

<title>Access Error</title> <style>pre { font: 12px/18px monospace; }</style>
{
  "result": "missing_chip",
  "policies": {
    "privacy": "https://company.cdn.overdrive.com/policies/privacy-policy",
    "terms": "https://company.cdn.overdrive.com/policies/terms-and-conditions.htm",
    "jobs": "https://company.overdrive.com/careers"
  }
}
<script type="text/javascript" src="/scripts/bridge.js"></script> <script type="text/javascript">setTimeout(function(){BRIDGE.transmit({name:'sentry:access:error',dest:'client',error:{origin:'Sentry'}});},500);</script> ~ `

@wtanksleyjr
Copy link

wtanksleyjr commented Apr 25, 2024 via email

@leomem
Copy link

leomem commented Apr 25, 2024

That's correct. I run odmpy libby and see my checked out books in the list. When I tried to download any of them, I got the error. The generated .odm file contained the above error. I have tried forceful reinstall too.

@wtanksleyjr
Copy link

Whew! Sorry about that. I'll have to step back and hope one of the experts can look at this, @ping maybe.

@jahlborn
Copy link

see #70

@wtanksleyjr
Copy link

OK, I have to confirm this - I'm now getting the same result. I kind of doubt they're attacking odmpy specifically, since they could do that just by sending GitHub a takedown notice; but for now it's the same effect.

@wtanksleyjr
Copy link

wtanksleyjr commented Apr 25, 2024

If you're in a situation where you're about to have a book expire while waiting for this, please note that the existing overdrive app for Windows should still work (if you're in a position to run it, of course). You'll have to download it from their site (paste this link into your browser press enter and then approve the security warning, don't try to click it or it'll just close the tab due to being an MSI file): http://app.overdrive.com/downloads/ODMediaConsoleSetup.msi; this is their official version so it's as safe as it was back when it was released. Go to your library's Overdrive site (usually something like lapl.overdrive.com). Sign in with your library card and last 4 digits of your phone number (or last 5 digits, or whatever). From there you should be able to get a download file, and that app can open it and use it to get the usual mp3 files. Not as nice as what we've become used to, but that's life!

If you need more metadata, you can download some nice JSON from an URL of the form https://thunder.api.overdrive.com/v2/media/9722081 (I picked a random libby identifier there). It'll work even if you don't have the book checked out, so long as you have the book ID. Use odmpy libby --exportloans loans.json to find the book ID of the books you currently have checked out.

@leomem
Copy link

leomem commented Apr 25, 2024

The overdrive windows program works. Thanks a lot.
Not sure how to use the JSON metadata yet.

@cerinawithasea
Copy link

cerinawithasea commented Apr 25, 2024

This is The error I am getting on both my Mac and windows laptop reset is not fixing it
The error will come after something similar to

Opening audiobook "When Trying to Return Home"...
Downloaded odm to "when-trying-to-return-home-9350409.odm"
An unexpected error has occurred
Then the error

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/odmpy/odm.py", line 1210, in run
process_odm(
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/odmpy/processing/odm.py", line 135, in process_odm
raise ValueError("Unable to find Metadata in ODM")
ValueError: Unable to find Metadata in ODM

I have tried different books

The reset did not work for me. After reset and reentering the code, downloading got the following error. python3.9/site-packages/odmpy/processing/odm.py", line 135, in process_odm raise ValueError("Unable to find Metadata in ODM") ValueError: Unable to find Metadata in ODM

The ODM file's content shows the error.

`

<title>Access Error</title> <style>pre { font: 12px/18px monospace; }</style> { "result": "missing_chip", "policies": { "privacy": "https://company.cdn.overdrive.com/policies/privacy-policy", "terms": "https://company.cdn.overdrive.com/policies/terms-and-conditions.htm", "jobs": "https://company.overdrive.com/careers" } } <script type="text/javascript" src="/scripts/bridge.js"></script> <script type="text/javascript">setTimeout(function(){BRIDGE.transmit({name:'sentry:access:error',dest:'client',error:{origin:'Sentry'}});},500);</script> ~ `

@mitrian7
Copy link

Go to your library's Overdrive site (usually something like lapl.overdrive.com). Sign in with your library card and last 4 digits of your phone number (or last 5 digits, or whatever). From there you should be able to get a download file, and that app can open it and use it to get the usual mp3 files. Not as nice as what we've become used to, but that's life!

So, this is cool, thank you. But also worth noting, you don't even need the desktop app. Just download the odm file from the overdrive site as you described, then you can use that odm file with odmpy directly. For example, if you want the benefits of merging with chapter metadata that odmpy does so nicely, you can run something like this:

odmpy dl "title.odm" -m --mergeformat m4b --mergecodec aac

... where title.odm is what you downloaded from the overdrive site.

@moviebrain
Copy link

Is this something that can even be fixed? The browser extensions stopped working about 48 hours prior to this stopping, so it seems like one by one these tools are being targeted specifically. Honestly I’d love if this just happens to be a back end change that needs a tweak but I’d fully understand any and all contributors not wanting to engage in a cat and mouse endless game of “fix the authentication”

If it can’t be fixed I’ll greatly miss the ability to play library audiobooks on the unsupported kids-boxy- speaker-things we have, it’s genuinely reignited my their love of stories. Thanks @ping for everything you did, in advance.

@mitrian7
Copy link

Is this something that can even be fixed?

While I do hope there's a solution, I think even if it can't be fixed, the workaround (see the post above) is mostly painless, it just adds the extra step of downloading the .odm manually from OverDrive. You can still get all the power and benefits of odmpy here.

@jahlborn
Copy link

Is this something that can even be fixed?

While I do hope there's a solution, I think even if it can't be fixed, the workaround (see the post above) is mostly painless, it just adds the extra step of downloading the .odm manually from OverDrive. You can still get all the power and benefits of odmpy here.

i certainly hope there is a solution as well. the workaround is manual, and the power of this library is that i can automate the process.

maybe this library could be patched to grab the odm files from the overdrive website...

@bckelly1
Copy link

I've been taking a look at this and to me it looks like they've changed how the JWT token is structured. The old JWT is very different than the one the web UI is using. Not sure if it's a red herring but it may be that the python client needs to use a different method to fetch the JWT.

Not an expert, and I'll look deeper when I get some time.

@jahlborn
Copy link

jahlborn commented Apr 26, 2024

in theory, this should be fixable. if the libby app and the libby webpage can load the data, then it should be possible to determine what api calls they are making and reproduce them. granted, they can use a ton of obfuscation to make that difficult...

@Prajna-Karuna
Copy link

From there you should be able to get a download file

I have looked everywhere and can't find a way to download the odm. Neither from the overdrive.com site nor from the installed msi file (which doesn't allow me to login, so it is just an empty overdrive GUI)
Can you be more specific on how to download the odm?

@wtanksleyjr
Copy link

wtanksleyjr commented Apr 26, 2024 via email

@moviebrain
Copy link

Prajnah, would you mind identifying which library you are using, and I'll find the right site for you? For example: I'm using Los Angeles Public Library, so to get ODMs I go to lapl.overdrive.com (notice the abbreviation for the library's name, it's not always that simple but sometimes is), log in, click "My Account", choose "Loans", and on any specific audiobook title, click "Have Overdrive?", and you'll see a button appear, click it and you'll download the ODM. BTW you can open that in the Windows Overdrive app, but if you're using odmpy already it turns out you can just use odmpy to do all the downloading using the same file. See above for instructions.

-Wm

My local branch overdrive site hid the download link for the odm because I was using a hardware/software combo that the Overdrive for (win/mac) didn’t support. I had to change the user-agent string in Firefox about:config to an appropriate OS version and then the “download” button appeared again.

@bckelly1
Copy link

bckelly1 commented Apr 27, 2024

in theory, this should be fixable. if the libby app and the libby webpage can load the data, then it should be possible to determine what api calls they are making and reproduce them. granted, they can use a ton of obfuscation to make that difficult...

I just did this and I noticed a couple things. The JWT in use actually changes at least 3 times. Also, the JWT is really weird. It's not a standard JWT format that I recognize. They are much longer and they have literally ... in the middle. It doesn't parse as a valid JWT.

Anyway, I did actually get it to work for books, but audiobooks has a problem that I have to debug (still the missing chip). You can splice the contents/results of two different calls and merge them into the libby.json replacing the chip, identity and __libby_sync_code fields.

@ziferius
Copy link

Prajnah, would you mind identifying which library you are using, and I'll find the right site for you? For example: I'm using Los Angeles Public Library, so to get ODMs I go to lapl.overdrive.com (notice the abbreviation for the library's name, it's not always that simple but sometimes is), log in, click "My Account", choose "Loans", and on any specific audiobook title, click "Have Overdrive?", and you'll see a button appear, click it and you'll download the ODM. BTW you can open that in the Windows Overdrive app, but if you're using odmpy already it turns out you can just use odmpy to do all the downloading using the same file. See above for instructions.

-Wm

My local branch overdrive site hid the download link for the odm because I was using a hardware/software combo that the Overdrive for (win/mac) didn’t support. I had to change the user-agent string in Firefox about:config to an appropriate OS version and then the “download” button appeared again.

You can always go to https://www.overdrive.com..... in the top right there is a 'Find Library' link. Click that, and key in your zip and it'll find your library.

@jahlborn
Copy link

looks like some of the extensions have been fixed. just used the latest version of the firefox libby downloader extension and it worked great.

@wtanksleyjr
Copy link

wtanksleyjr commented Apr 28, 2024 via email

@jahlborn
Copy link

I just did this and I noticed a couple things. The JWT in use actually changes at least 3 times. Also, the JWT is really weird. It's not a standard JWT format that I recognize. They are much longer and they have literally ... in the middle. It doesn't parse as a valid JWT.

@bckelly1 i did a little digging online and i saw some stuff mentioning "opaque" tokens for oauth2. for instance, auth0 has an opaque token which has two periods in the middle of it, which is not a valid jwt. not sure if that actually helps you in anyway, but just thought i'd throw it out there.

https://www.reddit.com/r/aws/comments/v16z4b/if_a_jwt_contains_two_consecutive_period/
https://community.auth0.com/t/why-my-token-has-2-dots-in-it/106186

@bckelly1
Copy link

bckelly1 commented May 7, 2024

I think there is a deeper change that will be required, though it looks easy maybe? I stumbled across this while looking for answers and it seems to mention similar topics. However, I don't see how the code actually fixed the problem so I'll have to dig a little deeper when I get some time.

@jahlborn
Copy link

jahlborn commented May 8, 2024

some progress appears to have been made in #71

@jahlborn
Copy link

@bckelly1 have you been able to figure anything else out?

@bckelly1
Copy link

bckelly1 commented Jun 3, 2024

@bckelly1 have you been able to figure anything else out?

Not much, I haven't put a lot of effort into it. I did find that this workaround in the firefox plugin is working, but it is highly manual. In theory, this could work with odmpy but I have yet to make it work for me. They embedded the spine info into an image, but I must not understand the instructions because it hasn't worked for me.

@wtanksleyjr
Copy link

Tonight my odmpy libby just fell out of sync, and there doesn't seem to be a way to get the 8-digit-code to make it come back into sync. So from here on out, it's odm downloads only.

@j-antunes
Copy link

j-antunes commented Aug 11, 2024

@wtanksleyjr - I tried it today and I was able to get the 8 digit code, by clicking on "Are you connecting sonos speakers"
image

image

@DaveAlton
Copy link

@wtanksleyjr - I tried it today and I was able to get the 8 digit code, by clicking on "Are you connecting sonos speakers"

Did this allow you to auto download your audiobooks?

@coloradoruss
Copy link

coloradoruss commented Aug 11, 2024 via email

@codyc1515
Copy link

@wtanksleyjr - I tried it today and I was able to get the 8 digit code, by clicking on "Are you connecting sonos speakers"

Same here. Looks like the way forward is using Passkeys.

@DaveAlton
Copy link

Same here. Looks like the way forward is using Passkeys.

For the record, I believe this ticket is to record that there's an issue auto-downloading audiobooks.

This solves an issue, but not the one originally ticketed.

@wtanksleyjr
Copy link

wtanksleyjr commented Aug 12, 2024 via email

@j-antunes
Copy link

@wtanksleyjr - How are you getting the odm file?

@coloradoruss
Copy link

coloradoruss commented Aug 12, 2024 via email

@ziferius
Copy link

ziferius commented Aug 13, 2024 via email

@jahlborn
Copy link

intersting, looks like someone made some progress here, i wonder if it is helpful to this project: bookbonobo/libby-download-extension#240

@wtanksleyjr
Copy link

Libby sign-in is failing with a new error, blocking Sonos and Android choices now. It also seems to destroy existing signins, the error I see is a certificate authentication failure, Libby seems to be sending an HTTPS certificate that doesn't match the machine it's coming from. This is reported in #78 .

@wtanksleyjr
Copy link

Regarding #78 someone has posted a fix at https://github.com/jordkh/odmpy , this does not fix the major problem here but it does allow Libby login once again - you still have to manually download the odm file from your library's Loans page on overdrive.com, but you can again use Libby's list of checked out books.

@wtanksleyjr
Copy link

The command I just ran to install this was python3 -m pip install git+https://git@github.com/jordkh/odmpy --upgrade --force-reinstall

@jahlborn
Copy link

jahlborn commented Oct 7, 2024

@wtanksleyjr can you clarify specifically what should be working now? being able to download files with odmpy given an odm file has been working the whole time.

@wtanksleyjr
Copy link

wtanksleyjr commented Oct 7, 2024 via email

@wtanksleyjr
Copy link

There are people working on making this work within browser extensions.

For example: bookbonobo/libby-download-extension#240

I don't think we've actually lost yet - the problem is that nobody here has the expertise to work on the problem.

@Ritergeek
Copy link

As of today, Overdrive has disabled the option to Download in Overdrive for Windows format on the Overdrive site for my library and apparently all subscribing libraries. This apparently means odm files. Does this totally tank the Libby Downloader potential?

@wtanksleyjr
Copy link

It doesn't change anything about the Libby downloader - specifically, we haven't figured out how to get its login token. Once the login token is fetched (which currently can only be done in a web browser), the files downloaded are the same MP3 files we always received before.

However, for now odmpy can only do one thing: show you what files you have checked out (odmpy libby). Everything else is completely disabled.

@jahlborn
Copy link

#78 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests