Self-host: Unable to upload photos

[andygeorge]

I've gotten a self-hosted Ente instance running, and I was able to build an Android APK that points at my instance. I am able to create a user and log in, but photo uploading doesn't seem to work.

I can see server logs showing my mobile client trying to do something, but no uploads seem to complete:

INFO[2115]request_logger.go:83 func1 incoming                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 query=sinceTime=1709654028970465&source=fg req_body= req_id=0690002e-0814-4f3b-abbf-745bdb9706ad req_method=GET req_uri=/collections/v2 ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009)
INFO[2115]request_logger.go:100 func1 outgoing                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 h_latency=4.654866ms latency_time=4.654866ms query=sinceTime=1709654028970465&source=fg req_body= req_id=0690002e-0814-4f3b-abbf-745bdb9706ad req_method=GET req_uri=/collections/v2 status_code=200 ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009) user_id=1580559962386438
INFO[2115]request_logger.go:83 func1 incoming                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 query=sinceTime=0 req_body= req_id=4d28acec-8db1-4b89-b9c8-40943f872d3b req_method=GET req_uri=/trash/v2/diff ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009)
INFO[2115]request_logger.go:100 func1 outgoing                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 h_latency=1.393221ms latency_time=1.393221ms query=sinceTime=0 req_body= req_id=4d28acec-8db1-4b89-b9c8-40943f872d3b req_method=GET req_uri=/trash/v2/diff status_code=200 ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009) user_id=1580559962386438
INFO[2115]request_logger.go:83 func1 incoming                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 query=count=4 req_body= req_id=6c74695b-ecd6-48fc-8b75-23e03494fe4d req_method=GET req_uri=/files/upload-urls ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009)
INFO[2115]file.go:253 GetUploadURLs Returning objectKeys: 1580559962386438/7cc03af0-e0ff-4538-b316-27df16830042, 1580559962386438/f920dc56-7071-4dae-a37a-3315cf4b7da6, 1580559962386438/b3bea574-6092-4899-b60d-688a6d958c62, 1580559962386438/a55f7c4a-fc32-4fc0-b1e1-77bede86737d
INFO[2115]request_logger.go:100 func1 outgoing                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 h_latency=7.018051ms latency_time=7.018051ms query=count=4 req_body= req_id=6c74695b-ecd6-48fc-8b75-23e03494fe4d req_method=GET req_uri=/files/upload-urls status_code=200 ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009) user_id=1580559962386438
INFO[2115]request_logger.go:83 func1 incoming                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 query=defaultValue=false&key=recoveryKeyVerified req_body= req_id=997e13b8-2136-49e3-89fd-4a73512bfdae req_method=GET req_uri=/remote-store ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009)
INFO[2115]request_logger.go:100 func1 outgoing                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 h_latency=773.676µs latency_time=773.676µs query=defaultValue=false&key=recoveryKeyVerified req_body= req_id=997e13b8-2136-49e3-89fd-4a73512bfdae req_method=GET req_uri=/remote-store status_code=200 ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009) user_id=1580559962386438
INFO[2115]request_logger.go:83 func1 incoming                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 query=
 req_body= req_id=8ba4c810-c264-4b7f-9292-8567a3336708 req_method=GET req_uri=/users/sessions ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009)
INFO[2115]request_logger.go:100 func1 outgoing                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 h_latency=1.837853ms latency_time=1.837853ms query= req_body= req_id=8ba4c810-c264-4b7f-9292-8567a3336708 req_method=GET req_uri=/users/sessions status_code=200 ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009) user_id=1580559962386438
INFO[2115]request_logger.go:83 func1 incoming                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 query=count=4 req_body= req_id=f4ef585d-24b7-4045-bebd-1556987b2864 req_method=GET req_uri=/files/upload-urls ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009)
INFO[2115]file.go:253 GetUploadURLs Returning objectKeys: 1580559962386438/d1f76bc8-6f38-47b2-8b59-d401bac8ce0e, 1580559962386438/d212dd99-bb22-4fee-8ae1-400efcd8f921, 1580559962386438/5253b43c-0192-4fd1-b358-99e7c8f48a45, 1580559962386438/339194f2-a202-41af-9e66-4afaa5bb5068
INFO[2115]request_logger.go:100 func1 outgoing                                      client_ip=10.89.7.4 client_pkg=io.ente.photos.independent client_version=0.8.64 h_latency=17.290592ms latency_time=17.290592ms query=count=4 req_body= req_id=f4ef585d-24b7-4045-bebd-1556987b2864 req_method=GET req_uri=/files/upload-urls status_code=200 ua=Dalvik/2.1.0 (Linux; U; Android 14; Pixel 8 Pro Build/AP21.240119.009) user_id=1580559962386438

I am allowing all the ports I could dig up: 2112, 3200-3201, 5432, 8080.

Anyone have any ideas?

[mnvr]

I guess the mobile app currently doesn't support disabling the Cloudflare workers for file uploads. For context, these workers were introduced as a way of make file uploads faster (more details are in this blog post).

On the web apps, this can be disabled by setting the NEXT_PUBLIC_ENTE_DIRECT_UPLOAD=true environment variable - this is handy when running locally. But it seems there isn't a corresponding environment variable to do the same for the mobile apps.

Meanwhile, what you can do is - hardcode return true; from the disableCFWorker function in feature_flag_service.dart. It should work after that.

[mnvr]

Hey, I think I might've gotten confused here. The mobile app should already be bypassing these workers, what I was saying was applicable to the web. So I'm not sure what's going on here (I haven't yet had time to dig into the code, just preemptively wanted to call out the possibly incorrect info I gave yesterday).

That said, your question got me thinking and made me realise that we don't need an extra environment variable on the web client. NEXT_PUBLIC_ENTE_DIRECT_UPLOAD can be deduced to be true (and is thus not needed) whenever a custom endpoint is set by using NEXT_PUBLIC_ENTE_ENDPOINT. So I've made that change on the web client code - #688 - and thank you for triggering that chain of thought!

Maybe you can do run the web app to test things out first. Fetch the latest main, then

NEXT_PUBLIC_ENTE_ENDPOINT=http://localhost:8080 yarn dev:photos

(replace localhost:8080 with wherever your server is running).

This should work. If this is not working, that means there is some other issue not related to the mobile app itself.

[deepgosalia1]

Hi,
On the change pertaining to the Web client[self-hosting], i believe the upload is still failing.

I do not see any logs for cloudflare, is this something that is related to local file upload failure?

Thanks

[mnvr]

This should not be related to Cloudflare - we can see that it is trying to upload to localhost:3200, which is where the local MinIO server is running (inside the Docker compose cluster).

I'm not sure why this is happening. From the error it seems that localhost:3200 is not available from your machine. It might be some Docker networking issue.

This should be working AFAIK, so the best if you could try with a clean slate once using the steps in https://help.ente.io/self-hosting/#getting-started. Just make sure the simple localhost-only docker compose setup is working before trying to move the custom server to a different IP. Also, ensure you have the latest main.

[mnvr]

Also, keep an eye out in the logs from docker compose, just in case if something is failing there.

[mnvr]

Also (sorry for the multiple messages, I'm just adding things here as they keep occuring to me), if none of the above work, try going to Preferences > Advanced in the web app and turning faster uploads off:

This should not be needed when you're running locally (it is needed in other cases, though I'll try to soon update the code to just remove this option when trying to connect to a self hosted instance). But I'm just mentioning this anyway, in case it is somehow the thing causing the issue in your setup. Try this only if nothing else above helped though.

[mnvr]

We found an issue for folks running on Windows that'd explain why the uploads were failing: the minio-provision script was not being executed because its line endings were being converted on Windows to CRLF.

I've added gitattributes to prevent that from happening, now the uploads should work fine on the latest main even on Windows.

[alexelgt]

Hi, I’m having the same problem as @deepgosalia1 and I don’t use Windows. Disabling faster uploads does not work.

If I go to one one the "localhost:3200/b2-eu-cen/something" links the output is a xml with an "SignatureDoesNotMatch" error.

When I looked at the code it seemed like this is to do a backup of the file and after that the upload is done. If that’s the case, does it make sense to do a local backup on the self-hosting device?

[mnvr]

When I looked at the code it seemed like this is to do a backup of the file and after that the upload is done. If that’s the case, does it make sense to do a local backup on the self-hosting device?

I'm not sure what you mean by this. There is no extra backup on the self hosted instance by default (the replication flag in the server settings is false). But maybe I didn't understand your message fully.

If I go to one one the "localhost:3200/b2-eu-cen/something" links the output is a xml with an "SignatureDoesNotMatch" error.

If possible, can you

1. Verify that there are no errors in the logs for the minio and minio-provision docker containers (we want to make sure that the initialization was complete).
2. Paste the request and response of the failing network request from the browser developer tools (we want to see what issue the web app faced when it tried to upload, and the URL it was using)
3. See if there is activity in the logs for any of the docker containers when you try to upload the file (we want to see if the request is even reaching the self hosted server or if there is some other networking issue).

Thank you!

[alexelgt]

Sorry I misunderstood what the minio server did.

I can send logs later in the day but I think I've found the reason.

I've tried to do everything on the hosting device and go to localhost:3000 and the upload works.

It's when I go to 192.168.1.xxx:3000 that the upload does not work.

Maybe the reason is that the minio server url is always set to localhost:3200 and when tried on another device it does not find it (because is not on localhost)

So a solution could be to set an env variable to change localhost:3200 to another url?

[mnvr]

Great that you got it working locally first!

I think logs are not needed, since what is happening is expected. If you are using external S3 storage (that is, not storing using the bundled minio), then you'll need to change your server's bucket configuration to point to the wherever the S3 buckets are. The "localhost:3200" part is not hardcoded on the client, it is being returned by the server (depending on which S3 buckets it is configured to ask the client to upload to).

You can see an example setup using non-localhost S3 / minio documented in this guide - https://help.ente.io/self-hosting/guides/external-s3

[alexelgt]

But I want to use the bundled minio.

The problem is if I access the web client from another device in the local network with 192.168.1.xxx:3000, the upload does not work

[mnvr]

Sure. This is where you'll need to change the minio bucket URL - https://github.com/ente-io/ente/blob/main/server/scripts/compose/credentials.yaml#L13 (instead of changing that file directly it can be overridden in a separate file as outlined in the guide I linked to).

In your case you want this value to be something that can be accessed both in your local network, and from inside the Docker compose network. I can't give you the solution off the top of my head, but I feel something should be doable (e.g. similar to how we're using socat to forward ports) by tinkering with the networking configuration for the docker compose setup.

[alexelgt]

I got it working. Thank you so much

[Bleep5143]

@alexelgt - Could you share what change(s) you made got your uploads to finally work? I'm experiencing the same problem and am struggling to set the right values.

[alexelgt]

Hi, what I did to get it working locally was to edit this file:

https://github.com/ente-io/ente/blob/main/server/scripts/compose/credentials.yaml#L13

Change localhost:3200 with http://192.168.1.xxx:3200 (or whatever is your server’s IP)

[nicognaW]

If you changed the values in credentials.yaml, or somehow changed the s3 config and still get SignatureDoesNotMatch, you should check if your s3 endpoints are all lowercase. From someone who spent 4 hours troubleshooting, and turned out it was just accidentally hitting shift when filling out the endpoint.