UnicodeDecodeError

[AlaaAlzahrani]

I tried to use "clean_data" from Tnkeeh and I kept getting this error (even when I changed the Arabic text):

File d:\python\2024\substilte_frequency\.venv\Lib\site-packages\tnkeeh\tnkeeh.py:261, in clean_data(file_path, save_path, segment, remove_special_chars, remove_english, normalize, remove_diacritics, excluded_chars, remove_tatweel, remove_html_elements, remove_links, remove_twitter_meta, remove_long_words, remove_repeated_chars, by_chunk, chunk_size, normalize_dots) 

259 text = ("").join(f.readlines(chunk_size)) 260 else: --> 261 text = f.read() 263 if len(text) == 0: 264 break File 

~\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py:23, in IncrementalDecoder.decode(self, input, final) 22 def decode(self, input, final=False): ---> 23 return codecs.charmap_decode(input,self.errors,decoding_table)[0] 

UnicodeDecodeError: 'charmap' codec can't decode byte

To solve this issue, I copied the "clean_data" function and modified it to include "encoding="utf8"" in all "open" commands in the function, like the following

with open(file_path, encoding="utf-8", mode="r") as f:

and this modification solved the issue for me.

I am not sure this solution would apply to others, but I wanted to let you know about the possibility of this issue

Thanks a lot for creating this much-needed tool