Update dropboxlist.php

[ousekjarr]

Introduced an item count, partially because it is useful to know, but mainly because there are issues with processing more than ~1000 items at a time. When there are too many items, the collection and tag fields are not processed, with the result that all items are added with no collection and no tags.

I suspect this is due to the length limit of an HTTP POST request, and that the POST content is truncated. The list of files to be processed comes before the other fields, and hence they drop off the bottom of the request. A better approach may be to include a hidden field with a name of "upload complete" or similar, and if this is not received, throw an error that the POST data has been truncated.

Here's a (cropped to just the last few entries) sample of a POST payload:

&dropbox-files%5B%5D=IMG_7065.JPG&dropbox-files%5B%5D=IMG_7066.JPG&dropbox-files%5B%5D=IMG_7067.JPG&dropbox-files%5B%5D=IMG_7068.JPG&dropbox-files%5B%5D=IMG_7069.JPG&dropbox-files%5B%5D=IMG_7070.JPG&dropbox-files%5B%5D=IMG_7071.JPG&dropbox-files%5B%5D=IMG_7072.JPG&dropbox-files%5B%5D=IMG_7073.JPG&submit=Upload+Files+as+Items&dropbox-public=0&dropbox-public=1&dropbox-featured=0&dropbox-collection-id=375&dropbox-tags=2010

Chrome inspection provides a clearer format:

dropbox-files[]: IMG_7068.JPG
dropbox-files[]: IMG_7069.JPG
dropbox-files[]: IMG_7070.JPG
dropbox-files[]: IMG_7071.JPG
dropbox-files[]: IMG_7072.JPG
dropbox-files[]: IMG_7073.JPG
submit: Upload Files as Items
dropbox-public: 0
dropbox-public: 1
dropbox-featured: 0
dropbox-collection-id: 375
dropbox-tags: 2010

With ~550 files to upload, my payload is 19KB.

My php.,ini defines "post_max_size = 500M", and Apache has no limits applied.

[zerocrates]

The limit you're probably hitting at ~1000 is not post_max_size, but max_input_vars. The default for max_input_vars is 1000.

max_input_vars, unlike post_max_size, has the undesirable property that it truncates after hitting the limit, where exceeding post_max_size drops the whole POST instead.

Increasing max_input_vars in php.ini should fix your issue. As for Dropbox detecting it... a sentinel value at the end could be a good option, yes.

[ousekjarr]

Thanks, I stumbled on that possibility shortly after creating the PR, but I think that it is better to detect bad application behaviour and prevent something odd happening than to blame the configuration. Maybe it is also worth retrieving the PHP setting for max_input_vars and alerting if the count is higher than this? Best Wishes, Paul. From: John Flatness ***@***.*** Sent: 26 June 2023 21:25 To: omeka/plugin-Dropbox Cc: Paul Murphy; Author Subject: Re: [omeka/plugin-Dropbox] Update dropboxlist.php (PR #24) The limit you're probably hitting at ~1000 is not post_max_size, but max_input_vars. The default for max_input_vars is 1000. max_input_vars, unlike post_max_size, has the undesirable property that it truncates after hitting the limit, where exceeding post_max_size drops the whole POST instead. Increasing max_input_vars should fix your issue. As for Dropbox detecting it... a sentinel value at the end could be a good option, yes. — Reply to this email directly, view it on GitHub<#24 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3GQKELUEL2YN3TQJDDKDM3XNHV2HANCNFSM6AAAAAAZTGXS7A>. You are receiving this because you authored the thread.Message ID: ***@***.***> [ { ***@***.***": "http://schema.org", ***@***.***": "EmailMessage", "potentialAction": { ***@***.***": "ViewAction", "target": "#24 (comment)", "url": "#24 (comment)", "name": "View Pull Request" }, "description": "View this Pull Request on GitHub", "publisher": { ***@***.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[zerocrates]

We can try that (counting). For Dropbox just doing the bulk import to multiple items option it could work OK, though it's likely to be inaccurate at the margins, as other things besides files that get submitted will count against your limit so need to be accounted for. In the context of doing Dropbox when editing/creating an individual item, figuring out the exact number of submitted variables would be tricky.

Sadly this (max_input_vars) is a PHP feature where the way it's implemented isn't well-done in terms of letting applications detect and deal with the condition.