fix sqrt issue when there is only one char various in traindata #154

hakehuang · 2017-02-28T07:42:22Z

this is a fix for below issue

lsi = ClassifierReborn::LSI.new
lsi.add_item 'log message Error: 1', :Error
lsi.add_item 'log message Error: 0', :passenger_ship: 
result  = lsi.classify 'log message Error: 1'

then the sqrt will have error when do the svd operation.

D:/projects/P_hobbit/AI/log_classifier/lib/classifier-reborn/extensions/vector.rb:58:in `sqrt': Numerical argument is out of domain - "sqrt" (Math::DomainError)
	from D:/projects/P_hobbit/AI/log_classifier/lib/classifier-reborn/extensions/vector.rb:58:in `block in SV_decomp'
	from D:/projects/P_hobbit/AI/log_classifier/lib/classifier-reborn/extensions/vector.rb:57:in `times'
	from D:/projects/P_hobbit/AI/log_classifier/lib/classifier-reborn/extensions/vector.rb:57:in `SV_decomp'
	from D:/projects/P_hobbit/AI/log_classifier/lib/classifier-reborn/lsi.rb:311:in `build_reduced_matrix'
	from D:/projects/P_hobbit/AI/log_classifier/lib/classifier-reborn/lsi.rb:143:in `build_index'
	from D:/projects/P_hobbit/AI/log_classifier/lib/classifier-reborn/lsi.rb:77:in `add_item'
	from D:/projects/P_hobbit/AI/log_classifier/pass_fail.rb:34:in `<main>'

Signed-off-by: Hake Huang hakehuang@gmail.com

Signed-off-by: Hake Huang <hake.huang@nxp.com>

ibnesayeed · 2017-03-06T21:37:06Z

lib/classifier-reborn/extensions/hasher.rb

@@ -27,7 +27,7 @@ def clean_word_hash(str, language = 'en', enable_stemmer = true)
    def word_hash_for_words(words, language = 'en', enable_stemmer = true)
      d = Hash.new(0)
      words.each do |word|
-        next unless word.length > 2 && !STOPWORDS[language].include?(word)
+        next unless word.length > 0 && !STOPWORDS[language].include?(word)


I am afraid that this will start white-listing tokens of one and two characters which is not the current behavior. This can be better dealt with the #131 proposal.

fix sqrt issue when there is only one char various in traindata

cbf7226

Signed-off-by: Hake Huang <hake.huang@nxp.com>

hakehuang mentioned this pull request Feb 28, 2017

the lsi meets `sqrt': Numerical argument is out of domain - "sqrt" (Math::DomainError) #153

Closed

update test cases for 1 byte fixing

ecfca6a

Signed-off-by: Hake Huang <hake.huang@nxp.com>

ibnesayeed reviewed Mar 6, 2017

View reviewed changes

Ch4s3 closed this Apr 13, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix sqrt issue when there is only one char various in traindata #154

fix sqrt issue when there is only one char various in traindata #154

hakehuang commented Feb 28, 2017

ibnesayeed Mar 6, 2017

Ch4s3 Mar 6, 2017

fix sqrt issue when there is only one char various in traindata #154

fix sqrt issue when there is only one char various in traindata #154

Conversation

hakehuang commented Feb 28, 2017

ibnesayeed Mar 6, 2017

Choose a reason for hiding this comment

Ch4s3 Mar 6, 2017

Choose a reason for hiding this comment