diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 7fd16aa..2d1453a 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -8,7 +8,7 @@ Before contributing ensure you've read and understood the [documentation](https:
- If you're issuing a feature request, please mention why it will be useful or the use case it solves.
- If you intend to implement the new feature request OR fix a bug yourself, then give a brief outline of the proposed solution.
-* Fork the `main` branch for making your changes.
+* Fork the `main` branch and make your changes. Ensure that the new code is formatted with [black](https://pypi.org/project/black/)
* You can issue a new [pull request](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork) to the `main` branch that closes the issue *after* you've been assigned.
- Any non-trivial pull request without having opened an earlier issue will be closed.
diff --git a/README.md b/README.md
index e5b7f9c..1a50be3 100644
--- a/README.md
+++ b/README.md
@@ -14,7 +14,7 @@
$ pip install elara
```
-* Latest - `v0.5.2`
+* Latest - `v0.5.3`
Go through the [release notes](#releases) for details on upgrades as breaking changes might happen between version upgrades while Elara is in beta.
@@ -113,7 +113,7 @@ Using `exe_secure()` without a key file or without the correct key file correspo
```python
import elara
-db = elara.exe("new.db", "newdb.key") # commit=False
+db = elara.exe_secure("new.db", "newdb.key") # commit=False
db.set("num", 20)
@@ -148,9 +148,15 @@ All the following operations are methods that can be applied to the instance ret
* `clear()` - clears the database data currently stored in-memory.
* `exists(key)` - returns `True` if the key exists.
* `commit()` - write in-memory changes into the database file.
+
+
* `getset(key, value)` - Sets the new value and returns the old value for that key or returns `False`.
* `getkeys()` - returns the list of keys in the database with. The list is ordered with the `most recently accessed` keys starting from index 0.
* `numkeys()` - returns the number of keys in the database.
+* `getmatch(match)` - Takes the `match` argument and returns a Dictionary of key-value pairs of which the keys contain `match` as a sub string.
+
+
+
* `retkey()` - returns the Key used to encrypt/decrypt the db file; returns `None` if the file is unprotected.
* `retmem()` - returns all the in-memory db contents.
* `retdb()` - returns all the db file contents.
@@ -179,6 +185,28 @@ Note - `retmem()` and `retdb()` will return the same value if `commit` is set to
[Go back to the table of contents](#contents)
+
+Elara adds some syntax sugar for get(), set() and rem() :
+
+```python
+import elara
+
+db = elara.exe("new.db")
+
+db["key"] = "value"
+
+print(db["key"])
+# value
+
+del self.db["key"]
+
+print(db.retmem())
+# {}
+
+```
+
+[Go back to the table of contents](#contents)
+
### Cache:
@@ -318,7 +346,7 @@ print(cache.get("obj").num)
* Elara uses checksums and a file version flag to verify database file integrity.
-All database writes are atomic (uses the [safer](https://github.com/rec/safer) library).
+All database writes are atomic (uses the [safer](https://github.com/rec/safer) library). Database writes are done in a separate thread along with a thread lock.
[Go back to the table of contents](#contents)
@@ -493,7 +521,8 @@ $ python -m unittest -v
## Release notes
* Latest - `v0.5.x`
- - `v0.5.2` - No breaking changes
+ - `v0.5.3` - No breaking changes
+ - `v0.5.2`
- `v0.5.1`
- `v0.5.0`
@@ -507,7 +536,7 @@ $ python -m unittest -v
To safeguard data, its better to **export all existing data** from any existing database file before upgrading Elara. (use `exportdb(export_path)`).
-View Elara's [release history](https://github.com/saurabh0719/elara/releases/).
+View Elara's detailed [release history](https://github.com/saurabh0719/elara/releases/).
[Go back to the table of contents](#contents)
diff --git a/README.rst b/README.rst
index c437eef..221423a 100644
--- a/README.rst
+++ b/README.rst
@@ -96,7 +96,7 @@ pushed into an upstream repository.
import elara
- db = elara.exe("new.db", "newdb.key") # commit=False
+ db = elara.exe_secure("new.db", "newdb.key") # commit=False
db.set("num", 20)
@@ -141,6 +141,8 @@ file. Set the ``commit`` argument to ``True`` else manually use the
- ``getkeys()`` - returns the list of keys in the database with. The
list is ordered with the ``most recently accessed`` keys starting
from index 0.
+- ``getmatch(match)`` - Takes the ``match`` argument and returns a
+ Dictionary of key-value pairs of which the keys contain ``match`` as a sub string.
- ``numkeys()`` - returns the number of keys in the database.
- ``retkey()`` - returns the Key used to encrypt/decrypt the db file;
returns ``None`` if the file is unprotected.
@@ -170,6 +172,24 @@ Note - ``retmem()`` and ``retdb()`` will return the same value if
``commit`` is set to ``True`` or if the ``commit()`` method is used
before calling ``retdb()``
+Elara adds some syntax sugar for get(), set() and rem() :
+
+.. code:: python
+
+ import elara
+
+ db = elara.exe("new.db")
+
+ db["key"] = "value"
+
+ print(db["key"])
+ # value
+
+ del self.db["key"]
+
+ print(db.retmem())
+ # {}
+
Cache:
~~~~~~
@@ -311,7 +331,7 @@ as long as they are ``in-memory`` and ``not persisted in the file``, as that wou
- To persist a simple object as a dictionary, use the ``__dict__`` attribute.
- Elara uses checksums and a file version flag to verify database file integrity.
-All database writes are atomic (uses the safer library).
+All database writes are atomic (uses the safer library). Database writes are done in a separate thread along with a thread lock.
API reference
-------------
@@ -522,7 +542,8 @@ Releases notes
- Latest - ``v0.5.x``
- - ``v0.5.2`` - No breaking changes
+ - ``v0.5.3`` - No breaking changes
+ - ``v0.5.2``
- ``v0.5.1``
- ``v0.5.0``
@@ -538,7 +559,7 @@ instead.
To safeguard data, its better to export all existing data from any existing database file before upgrading Elara.
(using ``exportdb(export_path)``)
-View Elara's release history
+View Elara's detailed release history
`here `__.
diff --git a/elara.png b/elara.png
index 9166f75..38e2d87 100644
Binary files a/elara.png and b/elara.png differ
diff --git a/elara/db_thread.py b/elara/db_thread.py
new file mode 100644
index 0000000..ac98675
--- /dev/null
+++ b/elara/db_thread.py
@@ -0,0 +1,26 @@
+"""
+Copyright (c) 2021, Saurabh Pujari
+All rights reserved.
+
+This source code is licensed under the BSD-style license found in the LICENSE file in the root directory of this source tree.
+"""
+
+
+import threading
+
+
+class DB_Thread(threading.Thread):
+ def run(self):
+ self.exception = None
+
+ try:
+ self.ret_val = self._target(*self._args)
+ except BaseException as e:
+ self.exception = e
+
+ def join(self):
+ super(DB_Thread, self).join()
+
+ if self.exception:
+ raise self.exception
+ return self.ret_val
diff --git a/elara/elara.py b/elara/elara.py
index c3e8ea7..a2af3b1 100644
--- a/elara/elara.py
+++ b/elara/elara.py
@@ -4,12 +4,14 @@
This source code is licensed under the BSD-style license found in the LICENSE file in the root directory of this source tree.
"""
+import threading
+from .db_thread import DB_Thread
import os
import atexit
from .elarautil import Util
+from .exceptions import InvalidCacheParams
from .lru import LRU, Cache_obj
from .status import Status
-from .exceptions import InvalidCacheParams
def is_pos(val):
@@ -18,38 +20,45 @@ def is_pos(val):
class Elara:
- from .strings import setnx, append, getset, mget, mset, msetnx, slen
+ from .hashtables import hadd, haddt, hexists, hget, hkeys, hmerge, hnew, hpop, hvals
from .lists import (
- lnew,
- lpush,
+ lappend,
+ lexists,
lextend,
lindex,
+ linsert,
+ llen,
+ lnew,
+ lpop,
+ lpush,
lrange,
lrem,
- lpop,
- llen,
- lappend,
- lexists,
- linsert,
)
- from .hashtables import hnew, hadd, haddt, hget, hpop, hkeys, hvals, hexists, hmerge
from .shared import (
- retmem,
- retdb,
- retkey,
commit,
exportdb,
exportkeys,
exportmem,
+ retdb,
+ retkey,
+ retmem,
securedb,
updatekey,
)
+ from .strings import append, getset, mget, mset, msetnx, setnx, slen
def __init__(self, path, commitdb, key_path=None, cache_param=None):
self.path = os.path.expanduser(path)
self.commitdb = commitdb
atexit.register(self._autocommit)
+ # Thread to write into the database
+ self.db_thread = None
+ self.db_lock = threading.Lock()
+
+ # Write data into the database on exit
+ atexit.register(self._autocommit)
+
if cache_param == None:
self.lru = LRU()
self.max_age = None
@@ -112,10 +121,28 @@ def _load(self):
self.lru._load(self.db, self.max_age)
def _dump(self):
+
if self.key:
- Util.encrypt_and_store(self) # Enclose in try-catch
+ if self.db_thread is not None:
+ self.db_thread.join()
+
+ self.db_thread = DB_Thread(
+ target=Util.encrypt_and_store, args=(self, self.db_lock)
+ )
+ self.db_thread.start()
+ self.db_thread.join() # Enclose in try-catch
else:
- Util.store_plain_db(self)
+
+ if self.db_thread is not None:
+ self.db_thread.join()
+
+ self.db_thread = DB_Thread(
+ target=Util.store_plain_db, args=(self, self.db_lock)
+ )
+ self.db_thread.start()
+ self.db_thread.join()
+
+ # Util.store_plain_db(self)
def _autocommit(self):
if self.commitdb:
@@ -140,6 +167,19 @@ def __delitem__(self, key):
def __contains__(self, key):
return self.exists(key)
+ # syntax sugar for get, set, rem and exists
+ def __getitem__(self, key):
+ return self.get(key)
+
+ def __setitem__(self, key, value):
+ return self.set(key, value)
+
+ def __delitem__(self, key):
+ return self.rem(key)
+
+ def __contains__(self, key):
+ return self.exists(key)
+
# Take max_age or self.max_age
def set(self, key, value, max_age=None):
if isinstance(key, str):
@@ -255,6 +295,15 @@ def getmatch(self, match):
return res
+ def getmatch(self, match):
+ deleted_keys, cache = self.lru.all()
+ self._remkeys_db_only(deleted_keys)
+ res = {}
+ for key, value in self.db.items():
+ if match in key:
+ res[key] = value
+ return res
+
def incr(self, key, val=1):
if self.exists(key):
data = self.get(key)
diff --git a/elara/elarautil.py b/elara/elarautil.py
index 04ee896..a51b304 100644
--- a/elara/elarautil.py
+++ b/elara/elarautil.py
@@ -4,6 +4,7 @@
This source code is licensed under the BSD-style license found in the LICENSE file in the root directory of this source tree.
"""
+import multiprocessing
import os
from typing import Dict
from zlib import crc32
@@ -57,17 +58,21 @@ def read_plain_db(obj) -> Dict:
return curr_db
@staticmethod
- def store_plain_db(obj):
- with safer.open(obj.path, "wb") as fctx:
- try:
- data = msgpack.packb(obj.db)
- buffer = b"ELDB"
- buffer += obj.db_format_version.to_bytes(2, "little")
- buffer += (crc32(data)).to_bytes(4, "little")
- buffer += data
- fctx.write(buffer)
- except FileExistsError:
- raise FileAccessError("File already exists")
+ def store_plain_db(obj, lock):
+ data = msgpack.packb(obj.db)
+ buffer = b"ELDB"
+ buffer += obj.db_format_version.to_bytes(2, "little")
+ buffer += (crc32(data)).to_bytes(4, "little")
+ buffer += data
+ try:
+ lock.acquire()
+ with safer.open(obj.path, "wb") as file:
+ file.write(buffer)
+ lock.release()
+ return True
+
+ except:
+ raise FileAccessError("File already exists")
@staticmethod
def read_and_decrypt(obj):
@@ -97,7 +102,8 @@ def read_and_decrypt(obj):
return None
@staticmethod
- def encrypt_and_store(obj):
+ def encrypt_and_store(obj, lock):
+ # pass lock maybe.
if obj.key:
fernet = Fernet(obj.key)
db_snapshot = msgpack.packb(obj.db)
@@ -108,11 +114,15 @@ def encrypt_and_store(obj):
buffer += crc32(encrypted_data).to_bytes(4, "little")
buffer += encrypted_data
try:
+
+ lock.acquire()
with safer.open(obj.path, "wb") as file:
file.write(buffer)
+ lock.release()
return True
+
except FileExistsError:
- raise FileAccessError("File exists")
+ raise FileAccessError("File already exists")
else:
return False
diff --git a/elara/shared.py b/elara/shared.py
index 00c13b2..db312c4 100644
--- a/elara/shared.py
+++ b/elara/shared.py
@@ -9,6 +9,7 @@
from .elarautil import Util
import json
import os
+import safer
from .exceptions import FileAccessError, FileKeyError
@@ -41,7 +42,13 @@ def exportdb(self, export_path, sort=True):
db = self.retdb()
new_export_path = os.path.expanduser(export_path)
try:
- json.dump(db, open(new_export_path, "wt"), indent=4, sort_keys=sort)
+ json.dump(
+ db,
+ safer.open(new_export_path, "wt", encoding="utf8"),
+ ensure_ascii=False,
+ indent=4,
+ sort_keys=sort,
+ )
except Exception:
raise FileAccessError("Store JSON error. File path might be wrong")
@@ -50,7 +57,13 @@ def exportmem(self, export_path, sort=True):
db = self.retmem()
new_export_path = os.path.expanduser(export_path)
try:
- json.dump(db, open(new_export_path, "wt"), indent=4, sort_keys=sort)
+ json.dump(
+ db,
+ safer.open(new_export_path, "wt", encoding="utf8"),
+ ensure_ascii=False,
+ indent=4,
+ sort_keys=sort,
+ )
except Exception:
raise FileAccessError("Store JSON error. File path might be wrong")
@@ -63,7 +76,13 @@ def exportkeys(self, export_path, keys=[], sort=True):
new_export_path = os.path.expanduser(export_path)
try:
- json.dump(db, open(new_export_path, "wt"), indent=4, sort_keys=sort)
+ json.dump(
+ db,
+ safer.open(new_export_path, "wt", encoding="utf8"),
+ ensure_ascii=False,
+ indent=4,
+ sort_keys=sort,
+ )
except Exception:
raise FileAccessError("Store JSON error. File path might be wrong")
@@ -82,7 +101,7 @@ def securedb(self, key_path=None):
Util.keygen(new_key_path)
else:
# os.remove(new_key_path)
- f = open(new_key_path, "r+")
+ f = safer.open(new_key_path, "r+")
f.truncate(0)
f.close()
Util.keygen(new_key_path)
@@ -90,7 +109,8 @@ def securedb(self, key_path=None):
Util.keygen(new_key_path)
self.key = Util.readkey(new_key_path)
- Util.encrypt_and_store(self)
+ # Util.encrypt_and_store(self)
+ self._dump()
return True
@@ -105,7 +125,7 @@ def updatekey(self, key_path=None):
Util.keygen(new_key_path)
else:
# os.remove(new_key_path)
- f = open(new_key_path, "r+")
+ f = safer.open(new_key_path, "r+")
f.truncate(0)
f.close()
Util.keygen(new_key_path)
@@ -114,11 +134,12 @@ def updatekey(self, key_path=None):
Util.keygen(new_key_path)
# clear db file and encrypt contents with new key
- f = open(self.path, "r+")
+ f = safer.open(self.path, "r+")
f.truncate(0)
f.close
self.key = Util.readkey(new_key_path)
- Util.encrypt_and_store(self)
+ # Util.encrypt_and_store(self)
+ self._dump()
else:
raise FileKeyError("Update key Failed")
diff --git a/setup.py b/setup.py
index 6ecbc4f..9a5e814 100644
--- a/setup.py
+++ b/setup.py
@@ -6,7 +6,7 @@
setup(
name="elara",
packages=["elara"],
- version="0.5.2",
+ version="0.5.3",
license="three-clause BSD",
description="Elara DB is an easy to use, lightweight NoSQL database written for python that can also be used as a fast in-memory cache for JSON-serializable data. Includes various methods and features to manipulate data structures in-memory, protect database files and export data.",
long_description=long_description,
@@ -38,4 +38,3 @@
"Programming Language :: Python",
],
)
-
diff --git a/test/test_1.py b/test/test_1.py
index e6b07dc..14603af 100644
--- a/test/test_1.py
+++ b/test/test_1.py
@@ -69,7 +69,7 @@ def test_decr(self):
self.assertEqual(self.db.get("one"), 0.35)
self.db.decr("one")
self.assertEqual(self.db.get("one"), -0.65)
-
+
def test_sugar(self):
self.db.clear()
self.db["key"] = "value"
@@ -78,10 +78,12 @@ def test_sugar(self):
self.assertEqual(self.db.retmem(), {})
self.db["key"] = "value"
assert "key" in self.db
-
+
def test_getmatch(self):
self.db.clear()
- self.db.set('key-one', 'value')
- self.db.set('key-two', 'value')
- self.db.set('value', 'value')
- self.assertEqual(self.db.getmatch('key'), {'key-one': 'value', 'key-two': 'value'})
\ No newline at end of file
+ self.db.set("key-one", "value")
+ self.db.set("key-two", "value")
+ self.db.set("value", "value")
+ self.assertEqual(
+ self.db.getmatch("key"), {"key-one": "value", "key-two": "value"}
+ )