New archive system

[Tzahi12345]

Wanted to open a discussion regarding the new archive system in v4.3.1. Basically, archive files are no more, and archives are handled internally within the database.

I probably should've made this a discussion before implementing this, but we had file system problems when it came to managing large archive files (e.g. #777, #755)

There's a couple things you need to know about this change:

New archive manager

You can manage archives from the new archive manager by hitting the 3 dots in the top right corner. You can import and export archives from here. (I'll upload a screenshot here later)

Subscriptions temporary archive

No archive files are generated from one-off files, but we do generate a temporary archive for subscriptions during the "check subscription for any missing videos" stage. No videos get added to the archive because it's not downloading anything, but it speeds things up during the check as it knows to avoid extracting info (which is slow) for videos in the archive.

Delete forever option for everyone!

Even with archives disabled, you can now delete a video forever in a subscription. This is functionality everyone expects, but isn't possible without either leveraging the archive system or by implementing it internally.

Migration

We auto migrate your existing archives into the new system. Here's the relevant code for that:

YoutubeDL-Material/backend/archive.js

Lines 31 to 52 in e573f34

	exports.importArchiveFile = async (archive_text, type, user_uid = null, sub_id = null) => {
	let archive_import_count = 0;
	const lines = archive_text.split('\n');
	for (let line of lines) {
	const archive_line_parts = line.trim().split(' ');
	// should just be the extractor and the video ID
	if (archive_line_parts.length !== 2) {
	continue;
	}
	const extractor = archive_line_parts[0];
	const id = archive_line_parts[1];
	if (!extractor || !id) continue;
	// we can't do a bulk write because we need to avoid duplicate archive items existing in db
	const archive_item = createArchiveItem(extractor, id, type, null, user_uid, sub_id);
	await db_api.insertRecordIntoTable('archives', archive_item, {extractor: extractor, id: id, type: type, sub_id: sub_id, user_uid: user_uid});
	archive_import_count++;
	}
	return archive_import_count;
	}

Basically, any line that is mangled it will skip. Any line that has two strings separated by a space, it will interpret as <extractor_id> <video id>, for example youtube dQw4w9WgXcQ

Let me know if this doesn't work well for you.

---

That's pretty much it. Let me know y'alls thoughts and I'll try to include it in the next update