diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7fd16aa..2d1453a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,7 +8,7 @@ Before contributing ensure you've read and understood the [documentation](https: - If you're issuing a feature request, please mention why it will be useful or the use case it solves. - If you intend to implement the new feature request OR fix a bug yourself, then give a brief outline of the proposed solution. -* Fork the `main` branch for making your changes. +* Fork the `main` branch and make your changes. Ensure that the new code is formatted with [black](https://pypi.org/project/black/) * You can issue a new [pull request](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork) to the `main` branch that closes the issue *after* you've been assigned. - Any non-trivial pull request without having opened an earlier issue will be closed. diff --git a/README.md b/README.md index e5b7f9c..1a50be3 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ $ pip install elara ``` -* Latest - `v0.5.2` +* Latest - `v0.5.3` Go through the [release notes](#releases) for details on upgrades as breaking changes might happen between version upgrades while Elara is in beta. @@ -113,7 +113,7 @@ Using `exe_secure()` without a key file or without the correct key file correspo ```python import elara -db = elara.exe("new.db", "newdb.key") # commit=False +db = elara.exe_secure("new.db", "newdb.key") # commit=False db.set("num", 20) @@ -148,9 +148,15 @@ All the following operations are methods that can be applied to the instance ret * `clear()` - clears the database data currently stored in-memory. * `exists(key)` - returns `True` if the key exists. * `commit()` - write in-memory changes into the database file. +
+ * `getset(key, value)` - Sets the new value and returns the old value for that key or returns `False`. * `getkeys()` - returns the list of keys in the database with. The list is ordered with the `most recently accessed` keys starting from index 0. * `numkeys()` - returns the number of keys in the database. +* `getmatch(match)` - Takes the `match` argument and returns a Dictionary of key-value pairs of which the keys contain `match` as a sub string. + +
+ * `retkey()` - returns the Key used to encrypt/decrypt the db file; returns `None` if the file is unprotected. * `retmem()` - returns all the in-memory db contents. * `retdb()` - returns all the db file contents. @@ -179,6 +185,28 @@ Note - `retmem()` and `retdb()` will return the same value if `commit` is set to [Go back to the table of contents](#contents) + +Elara adds some syntax sugar for get(), set() and rem() : + +```python +import elara + +db = elara.exe("new.db") + +db["key"] = "value" + +print(db["key"]) +# value + +del self.db["key"] + +print(db.retmem()) +# {} + +``` + +[Go back to the table of contents](#contents) + ### Cache: @@ -318,7 +346,7 @@ print(cache.get("obj").num) * Elara uses checksums and a file version flag to verify database file integrity. -All database writes are atomic (uses the [safer](https://github.com/rec/safer) library). +All database writes are atomic (uses the [safer](https://github.com/rec/safer) library). Database writes are done in a separate thread along with a thread lock. [Go back to the table of contents](#contents) @@ -493,7 +521,8 @@ $ python -m unittest -v ## Release notes * Latest - `v0.5.x` - - `v0.5.2` - No breaking changes + - `v0.5.3` - No breaking changes + - `v0.5.2` - `v0.5.1` - `v0.5.0` @@ -507,7 +536,7 @@ $ python -m unittest -v To safeguard data, its better to **export all existing data** from any existing database file before upgrading Elara. (use `exportdb(export_path)`). -View Elara's [release history](https://github.com/saurabh0719/elara/releases/). +View Elara's detailed [release history](https://github.com/saurabh0719/elara/releases/). [Go back to the table of contents](#contents) diff --git a/README.rst b/README.rst index c437eef..221423a 100644 --- a/README.rst +++ b/README.rst @@ -96,7 +96,7 @@ pushed into an upstream repository. import elara - db = elara.exe("new.db", "newdb.key") # commit=False + db = elara.exe_secure("new.db", "newdb.key") # commit=False db.set("num", 20) @@ -141,6 +141,8 @@ file. Set the ``commit`` argument to ``True`` else manually use the - ``getkeys()`` - returns the list of keys in the database with. The list is ordered with the ``most recently accessed`` keys starting from index 0. +- ``getmatch(match)`` - Takes the ``match`` argument and returns a + Dictionary of key-value pairs of which the keys contain ``match`` as a sub string. - ``numkeys()`` - returns the number of keys in the database. - ``retkey()`` - returns the Key used to encrypt/decrypt the db file; returns ``None`` if the file is unprotected. @@ -170,6 +172,24 @@ Note - ``retmem()`` and ``retdb()`` will return the same value if ``commit`` is set to ``True`` or if the ``commit()`` method is used before calling ``retdb()`` +Elara adds some syntax sugar for get(), set() and rem() : + +.. code:: python + + import elara + + db = elara.exe("new.db") + + db["key"] = "value" + + print(db["key"]) + # value + + del self.db["key"] + + print(db.retmem()) + # {} + Cache: ~~~~~~ @@ -311,7 +331,7 @@ as long as they are ``in-memory`` and ``not persisted in the file``, as that wou - To persist a simple object as a dictionary, use the ``__dict__`` attribute. - Elara uses checksums and a file version flag to verify database file integrity. -All database writes are atomic (uses the safer library). +All database writes are atomic (uses the safer library). Database writes are done in a separate thread along with a thread lock. API reference ------------- @@ -522,7 +542,8 @@ Releases notes - Latest - ``v0.5.x`` - - ``v0.5.2`` - No breaking changes + - ``v0.5.3`` - No breaking changes + - ``v0.5.2`` - ``v0.5.1`` - ``v0.5.0`` @@ -538,7 +559,7 @@ instead. To safeguard data, its better to export all existing data from any existing database file before upgrading Elara. (using ``exportdb(export_path)``) -View Elara's release history +View Elara's detailed release history `here `__. diff --git a/elara.png b/elara.png index 9166f75..38e2d87 100644 Binary files a/elara.png and b/elara.png differ diff --git a/elara/db_thread.py b/elara/db_thread.py new file mode 100644 index 0000000..ac98675 --- /dev/null +++ b/elara/db_thread.py @@ -0,0 +1,26 @@ +""" +Copyright (c) 2021, Saurabh Pujari +All rights reserved. + +This source code is licensed under the BSD-style license found in the LICENSE file in the root directory of this source tree. +""" + + +import threading + + +class DB_Thread(threading.Thread): + def run(self): + self.exception = None + + try: + self.ret_val = self._target(*self._args) + except BaseException as e: + self.exception = e + + def join(self): + super(DB_Thread, self).join() + + if self.exception: + raise self.exception + return self.ret_val diff --git a/elara/elara.py b/elara/elara.py index c3e8ea7..a2af3b1 100644 --- a/elara/elara.py +++ b/elara/elara.py @@ -4,12 +4,14 @@ This source code is licensed under the BSD-style license found in the LICENSE file in the root directory of this source tree. """ +import threading +from .db_thread import DB_Thread import os import atexit from .elarautil import Util +from .exceptions import InvalidCacheParams from .lru import LRU, Cache_obj from .status import Status -from .exceptions import InvalidCacheParams def is_pos(val): @@ -18,38 +20,45 @@ def is_pos(val): class Elara: - from .strings import setnx, append, getset, mget, mset, msetnx, slen + from .hashtables import hadd, haddt, hexists, hget, hkeys, hmerge, hnew, hpop, hvals from .lists import ( - lnew, - lpush, + lappend, + lexists, lextend, lindex, + linsert, + llen, + lnew, + lpop, + lpush, lrange, lrem, - lpop, - llen, - lappend, - lexists, - linsert, ) - from .hashtables import hnew, hadd, haddt, hget, hpop, hkeys, hvals, hexists, hmerge from .shared import ( - retmem, - retdb, - retkey, commit, exportdb, exportkeys, exportmem, + retdb, + retkey, + retmem, securedb, updatekey, ) + from .strings import append, getset, mget, mset, msetnx, setnx, slen def __init__(self, path, commitdb, key_path=None, cache_param=None): self.path = os.path.expanduser(path) self.commitdb = commitdb atexit.register(self._autocommit) + # Thread to write into the database + self.db_thread = None + self.db_lock = threading.Lock() + + # Write data into the database on exit + atexit.register(self._autocommit) + if cache_param == None: self.lru = LRU() self.max_age = None @@ -112,10 +121,28 @@ def _load(self): self.lru._load(self.db, self.max_age) def _dump(self): + if self.key: - Util.encrypt_and_store(self) # Enclose in try-catch + if self.db_thread is not None: + self.db_thread.join() + + self.db_thread = DB_Thread( + target=Util.encrypt_and_store, args=(self, self.db_lock) + ) + self.db_thread.start() + self.db_thread.join() # Enclose in try-catch else: - Util.store_plain_db(self) + + if self.db_thread is not None: + self.db_thread.join() + + self.db_thread = DB_Thread( + target=Util.store_plain_db, args=(self, self.db_lock) + ) + self.db_thread.start() + self.db_thread.join() + + # Util.store_plain_db(self) def _autocommit(self): if self.commitdb: @@ -140,6 +167,19 @@ def __delitem__(self, key): def __contains__(self, key): return self.exists(key) + # syntax sugar for get, set, rem and exists + def __getitem__(self, key): + return self.get(key) + + def __setitem__(self, key, value): + return self.set(key, value) + + def __delitem__(self, key): + return self.rem(key) + + def __contains__(self, key): + return self.exists(key) + # Take max_age or self.max_age def set(self, key, value, max_age=None): if isinstance(key, str): @@ -255,6 +295,15 @@ def getmatch(self, match): return res + def getmatch(self, match): + deleted_keys, cache = self.lru.all() + self._remkeys_db_only(deleted_keys) + res = {} + for key, value in self.db.items(): + if match in key: + res[key] = value + return res + def incr(self, key, val=1): if self.exists(key): data = self.get(key) diff --git a/elara/elarautil.py b/elara/elarautil.py index 04ee896..a51b304 100644 --- a/elara/elarautil.py +++ b/elara/elarautil.py @@ -4,6 +4,7 @@ This source code is licensed under the BSD-style license found in the LICENSE file in the root directory of this source tree. """ +import multiprocessing import os from typing import Dict from zlib import crc32 @@ -57,17 +58,21 @@ def read_plain_db(obj) -> Dict: return curr_db @staticmethod - def store_plain_db(obj): - with safer.open(obj.path, "wb") as fctx: - try: - data = msgpack.packb(obj.db) - buffer = b"ELDB" - buffer += obj.db_format_version.to_bytes(2, "little") - buffer += (crc32(data)).to_bytes(4, "little") - buffer += data - fctx.write(buffer) - except FileExistsError: - raise FileAccessError("File already exists") + def store_plain_db(obj, lock): + data = msgpack.packb(obj.db) + buffer = b"ELDB" + buffer += obj.db_format_version.to_bytes(2, "little") + buffer += (crc32(data)).to_bytes(4, "little") + buffer += data + try: + lock.acquire() + with safer.open(obj.path, "wb") as file: + file.write(buffer) + lock.release() + return True + + except: + raise FileAccessError("File already exists") @staticmethod def read_and_decrypt(obj): @@ -97,7 +102,8 @@ def read_and_decrypt(obj): return None @staticmethod - def encrypt_and_store(obj): + def encrypt_and_store(obj, lock): + # pass lock maybe. if obj.key: fernet = Fernet(obj.key) db_snapshot = msgpack.packb(obj.db) @@ -108,11 +114,15 @@ def encrypt_and_store(obj): buffer += crc32(encrypted_data).to_bytes(4, "little") buffer += encrypted_data try: + + lock.acquire() with safer.open(obj.path, "wb") as file: file.write(buffer) + lock.release() return True + except FileExistsError: - raise FileAccessError("File exists") + raise FileAccessError("File already exists") else: return False diff --git a/elara/shared.py b/elara/shared.py index 00c13b2..db312c4 100644 --- a/elara/shared.py +++ b/elara/shared.py @@ -9,6 +9,7 @@ from .elarautil import Util import json import os +import safer from .exceptions import FileAccessError, FileKeyError @@ -41,7 +42,13 @@ def exportdb(self, export_path, sort=True): db = self.retdb() new_export_path = os.path.expanduser(export_path) try: - json.dump(db, open(new_export_path, "wt"), indent=4, sort_keys=sort) + json.dump( + db, + safer.open(new_export_path, "wt", encoding="utf8"), + ensure_ascii=False, + indent=4, + sort_keys=sort, + ) except Exception: raise FileAccessError("Store JSON error. File path might be wrong") @@ -50,7 +57,13 @@ def exportmem(self, export_path, sort=True): db = self.retmem() new_export_path = os.path.expanduser(export_path) try: - json.dump(db, open(new_export_path, "wt"), indent=4, sort_keys=sort) + json.dump( + db, + safer.open(new_export_path, "wt", encoding="utf8"), + ensure_ascii=False, + indent=4, + sort_keys=sort, + ) except Exception: raise FileAccessError("Store JSON error. File path might be wrong") @@ -63,7 +76,13 @@ def exportkeys(self, export_path, keys=[], sort=True): new_export_path = os.path.expanduser(export_path) try: - json.dump(db, open(new_export_path, "wt"), indent=4, sort_keys=sort) + json.dump( + db, + safer.open(new_export_path, "wt", encoding="utf8"), + ensure_ascii=False, + indent=4, + sort_keys=sort, + ) except Exception: raise FileAccessError("Store JSON error. File path might be wrong") @@ -82,7 +101,7 @@ def securedb(self, key_path=None): Util.keygen(new_key_path) else: # os.remove(new_key_path) - f = open(new_key_path, "r+") + f = safer.open(new_key_path, "r+") f.truncate(0) f.close() Util.keygen(new_key_path) @@ -90,7 +109,8 @@ def securedb(self, key_path=None): Util.keygen(new_key_path) self.key = Util.readkey(new_key_path) - Util.encrypt_and_store(self) + # Util.encrypt_and_store(self) + self._dump() return True @@ -105,7 +125,7 @@ def updatekey(self, key_path=None): Util.keygen(new_key_path) else: # os.remove(new_key_path) - f = open(new_key_path, "r+") + f = safer.open(new_key_path, "r+") f.truncate(0) f.close() Util.keygen(new_key_path) @@ -114,11 +134,12 @@ def updatekey(self, key_path=None): Util.keygen(new_key_path) # clear db file and encrypt contents with new key - f = open(self.path, "r+") + f = safer.open(self.path, "r+") f.truncate(0) f.close self.key = Util.readkey(new_key_path) - Util.encrypt_and_store(self) + # Util.encrypt_and_store(self) + self._dump() else: raise FileKeyError("Update key Failed") diff --git a/setup.py b/setup.py index 6ecbc4f..9a5e814 100644 --- a/setup.py +++ b/setup.py @@ -6,7 +6,7 @@ setup( name="elara", packages=["elara"], - version="0.5.2", + version="0.5.3", license="three-clause BSD", description="Elara DB is an easy to use, lightweight NoSQL database written for python that can also be used as a fast in-memory cache for JSON-serializable data. Includes various methods and features to manipulate data structures in-memory, protect database files and export data.", long_description=long_description, @@ -38,4 +38,3 @@ "Programming Language :: Python", ], ) - diff --git a/test/test_1.py b/test/test_1.py index e6b07dc..14603af 100644 --- a/test/test_1.py +++ b/test/test_1.py @@ -69,7 +69,7 @@ def test_decr(self): self.assertEqual(self.db.get("one"), 0.35) self.db.decr("one") self.assertEqual(self.db.get("one"), -0.65) - + def test_sugar(self): self.db.clear() self.db["key"] = "value" @@ -78,10 +78,12 @@ def test_sugar(self): self.assertEqual(self.db.retmem(), {}) self.db["key"] = "value" assert "key" in self.db - + def test_getmatch(self): self.db.clear() - self.db.set('key-one', 'value') - self.db.set('key-two', 'value') - self.db.set('value', 'value') - self.assertEqual(self.db.getmatch('key'), {'key-one': 'value', 'key-two': 'value'}) \ No newline at end of file + self.db.set("key-one", "value") + self.db.set("key-two", "value") + self.db.set("value", "value") + self.assertEqual( + self.db.getmatch("key"), {"key-one": "value", "key-two": "value"} + )