-
Notifications
You must be signed in to change notification settings - Fork 33
/
data_distribution.txt
84 lines (77 loc) · 3.12 KB
/
data_distribution.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
DATA DISTRIBUTION
DOMAIN WISE
Medical Publication 13,410
News 18,389
Religious texts 48,927
Miscellaneous 63,896
Software 79,132
Educational content 213,284
TED Talk 303,131
Fictional Books 473,306
Movies Subtitles 721,174
Wikipedia 1,094,248
Legal Document 1,134,813
-----------------
TOTAL 4,163,710
RESOURCE WISE
Data sources come from the following three categories:
1. Available parallel corpuses
OpenSubtitle,WikiMatrix,Bible-Uedin, Qed, KDE4, Gnome,Ubuntu,Tatoeba,TED2020v1,ELRC: https://opus.nlpl.eu/
2. Scrape from the internet:
Websites for Medical data:
https://yhoctphcm.ump.edu.vn
http://jmp.huemed-univ.edu.vn
http://tonghoiyhoc.vn
http://hoinhikhoavn.com
http://hoiyhoctphcm.org.vn
https://jns.vn
https://jprp.vn
http://hocvienquany.edu.vn
https://sinhlyhoc.com.vn
https://tapchinghiencuuyhoc.vn
http://tapchi.vienbongquocgia.vn
http://vienduoclieu.org.vn
https://vjpm.vn
http://vjfc.nifc.gov.vn
https://vjs.ac.vn
http://vutm.edu.vn
https://jcmhch.com
http://www.yhth.vn
https://sj.ctu.edu.vn
https://radiology.com.vn
https://vjol.info.vn
http://www.vjph.vn
Others websites:
https://vietanhsongngu.com
https://baosongngu.com
https://sachsongngu.top
https://thuvienphapluat.vn
http://vbpl.vn
http://automation.net
http://tapchixaydungbxd.vn
https://duytan.edu.vn
https://tapchikhcn.haui.edu.vn
https://tapchivatuyentap.tlu.edu.vn
http://tapchimoitruong.vn
https://translations.launchpad.net
https://translationproject.org
https://issuu.com/
https://lyricstranslate.com
https://www.wikihow.com
https://d2l.aivivn.com (for 2 books: 'Dive in to Deep Learning' & 'Machine Learning Yearning')
Subtitles from the following channels on Youtube:
- Khan Academy
- Ted Ed
- Asap Science
- Crash courses
- GCP Grey
- Veritasium
- Vsauce
3. Book downloads and aligned at paragraph-level from
https://www.dtv-ebook.com
https://www.pdfdrive.com
https://vn1lib.org
https://trinhdinhlinh.com
https://sachvui.vn
The paragraph-aligning algorithm is described here:
https://github.com/vietai/SAT/blob/main/dp_algorithm.md