Questions about wink-distance

[paul-bell]

Hello Sanjaya,

I've just stumbled upon your wink.js and am very intrigued. But please note that I am a relative noob in the ML/NLP world; trying to learn.

I have a few questions about the use of wink-distance to detect similarities between raw, unstructured texts:

• Do you think wink-distance is suitable for such a task?
• Am I right that one must create a "bag of words" from the raw text as proper input to wink-distance?
• In wink-distance/test/bow-cosine-spec.js is { a: { the: 2, dog: 1, chased: 1, cat: 1 }, b: { the: 2, dog: 1, chased: 1, cat: 1 } } an example of a "bow?"
• If so, does wink.js offer a way to create a "bow?"

Thanks very much for your help.

Cordially,

Paul

[sanjayaksaxena]

Hello @paul-bell

Yes, wink-distance has APIs to determine similarity between texts. Cosine distance is one of the prominent way to determine similarity; other being jaccard and tversky. The cosine distance accepts bow, whereas jaccard and tversky accept set as the input parameters. Note, since it computes "distance", you will have to subtract from 1 to obtain the similarity!

You could use wink-nlp to create the bag-of-words (bow). Here is the code to do it:

// Require packages & models.
const distance = require( 'wink-distance' );
const winkNLP = require( 'wink-nlp' );
const its = require( 'wink-nlp/src/its.js' );
const as = require( 'wink-nlp/src/as.js' );
const model = require( 'wink-eng-lite-web-model' );
// Instantiate wink nlp using the model.
const nlp = winkNLP( model )
// Texts to be tested for similaity.
const text1 = `the dog chased the cat`;
const text2 = `the cat chased the mouse`;
// Compute bow using wink nlp.
const bow1 = nlp.readDoc( text1 ).tokens().out( its.normal, as.bow );
const bow2 = nlp.readDoc( text2 ).tokens().out( its.normal, as.bow );
// Use bag-of-words to compute cosine similarity .
const cosineSimilarity = 1 - distance.bow.cosine( bow1, bow2 );

The above code is available as a RunKit Notebook — you can play with it there.

The wink-nlp is a developer friendly natural language processing package; with winkNLP, you will find NLP as easy as manipulating DOM using Javascript! Check out its tutorial at https://winkjs.org/wink-nlp/getting-started.html and showcases at https://winkjs.org/showcase.html.

Best,
Sanjaya

[sanjayaksaxena]

Hi @paul-bell, I have transferred the issue here at wink-nlp as a discussion. Please feel free to discuss more details here. Thanks. — Sanjaya

[paul-bell]

Hi Sanjaya,

Most kind of you; thanks very much!

I can't wait to try this out. With any luck, I'll get to it before end of day. If not, then first thing in the AM.

Again, thank you.

Cordially,

Paul

[paul-bell]

Hello Sanjaya,

Here we are one month later and I am finally able to start playing with the BOW/cosine similarity code you've generously given me (thank you again).

Because the code I am working on is strictly backend, I chose to use wink-eng-lite-model rather than its 'web' variant.

Do you have such models in languages other than English?

Thank you.

Cordially,

Paul

[sanjayaksaxena]

Hello Paul,

The Similarity and BM25 Vectorizer are now part of winkNLP.

We have plans to develop more language models but for now there are only these two models are available.

What language models do you need? May be you can help us to get started!

Best,
Sanjaya

[paul-bell]

Hi Sanjaya,

Please forgive my ignorance, but do Similarity and BM25 Vectorizer include non-English models?

To your last question, I am open to the possibility of assisting in such an effort but would probably depend on some initial guidance from you as to what's required.

OTTOMH, French and German; and at least one non-Western language.

Cordially,

Paul

[sanjayaksaxena]

Similarity and BM25 Vectorizer are language agnostic.

[paul-bell]

Thank you. I will give them a spin.

I forgot to mention that Spanish is probably the most important of the western alphabet languages.

When you have the time, can you tell me what I'd need to do to create such a model?

-Paul