efactor(scraper): remove ParserManager and switch to mdformat for mar…

…kdown conversion Removed the `ParserManager` class and its associated file. Updated the `Scraper` class to use `mdformat` for converting HTML to Markdown directly, simplifying the process and reducing dependencies. Updated `requirements.txt` to include necessary `mdformat` packages.
obeone · Jun 13, 2024 · e888329 · e888329
1 parent a42aa30
commit e888329
Show file tree

Hide file tree

Showing 3 changed files with 10 additions and 381 deletions.
diff --git a/parser_manager.py b/parser_manager.py
diff --git a/requirements.txt b/requirements.txt
@@ -2,4 +2,9 @@ beautifulsoup4==4.12.3
 coloredlogs==15.0.1                                                                                                                               
 tqdm==4.66.4    
 requests==2.32.2  
-trafilatura==1.8.1
+trafilatura==1.10.0
+mdformat==0.7.17
+mdformat-gfm==0.3.6
+mdformat_footnote==0.1.1
+mdformat_frontmatter==2.0.8
+mdformat_tables==0.4.1