Cohen 1994 GRAIN model accuracy

[kmantel]

content is from Jeshua Tromp at Universiteit Leiden:

After some other project detours I'm back on coding with Psyneulink, and now in the end phase of a modeling project coded in Psyneulink. However, in that project, I wanted to compare my model with the GRAIN model (https://princetonuniversity.github.io/PsyNeuLink/Cohen_HustonModel.html).
In your documentation of that model, you show that you are able to reproduce the key results of the paper in a single run without noise. However, therefore you had to change several parameters of the model (compared to 1994 original) to get the desired results. I've run the model myself as well, but there I saw a potential mistake in the code which could have led to you having to make the changes from the original. The mistake is in the following piece of code:

for cond in range(conditions):
    response_color_weights = pnl.MappingProjection(
        matrix=np.array([
            [0.0, 0.0, 0.0],
            [0.0, 0.0, 0.0]
        ])
    )
    response_word_weights = pnl.MappingProjection(
        matrix=np.array([
            [0.0, 0.0, 0.0],
            [0.0, 0.0, 0.0]
        ])
    )
    Bidirectional_Stroop.run(inputs=Stimulus[cond][0], num_trials=settle_trials)
    response_color_weights = pnl.MappingProjection(
        matrix=np.array([
            [1.5, 0.0, 0.0],
            [0.0, 1.5, 0.0]
        ])
    )
    response_word_weights = pnl.MappingProjection(
        matrix=np.array([
            [2.5, 0.0, 0.0],
            [0.0, 2.5, 0.0]
        ])
    )
    Bidirectional_Stroop.run(inputs=Stimulus[cond][1], termination_processing=terminate_trial)

You set the response_color_weight/response_word_weight parameter WITHOUT specifying the context, which results effectively in that in both the initialization and the actual run, the model runs with response_color_weights and response_word_weights of value 0. I've checked this by actually removing those parts of the script, and the result is the same.
If I then run the model by setting the matrix parameter in the correct way, I am not able to reproduce the key effects of the 1994 paper (especially that interference > facilitation). However, when I input noise as in the 1994 paper, the result does look more alike. Even when setting all the parameters the same as in the original paper (intergration rate of 0.1, threshold of 0.55).
I hope I am wrong here. Otherwise, are you able to reproduce the 1994 results in a single noise-run when setting the weights in the correct manner? Perhaps the noise is necessary.

The "correct" way I do it is using the set function:

# Change color and word weights to 0 for the initialization run
response_color_weights.parameters.matrix.set(
    np.array([
        [0.0, 0.0, 0.0],
        [0.0, 0.0, 0.0]]),
    Bidirectional_Stroop)
response_word_weights.parameters.matrix.set(
    np.array([
        [0.0, 0.0, 0.0],
        [0.0, 0.0, 0.0]]),
    Bidirectional_Stroop)
color_response_weights.parameters.matrix.set(
    np.array([
        [0.0, 0.0],
        [0.0, 0.0],
        [0.0, 0.0]]),
    Bidirectional_Stroop)
word_response_weights.parameters.matrix.set(
    np.array([
        [0.0, 0.0],
        [0.0, 0.0],
        [0.0, 0.0]]),
    Bidirectional_Stroop)
# Execute initialization run
Bidirectional_Stroop.run(inputs=init_input, num_trials=settle)
# Change color and word weights to back for trial run
response_color_weights.parameters.matrix.set(
    np.array([
        [1.5, 0.0, 0.0],
        [0.0, 1.5, 0.0]]),
    Bidirectional_Stroop)
response_word_weights.parameters.matrix.set(
    np.array([
        [2.5, 0.0, 0.0],
        [0.0, 2.5, 0.0]]),
    Bidirectional_Stroop)
color_response_weights.parameters.matrix.set(
    np.array([
        [1.5, 0.0],
        [0.0, 1.5],
        [0.0, 0.0]]),
    Bidirectional_Stroop)
word_response_weights.parameters.matrix.set(
    np.array([
        [2.5, 0.0],
        [0.0, 2.5],
        [0.0, 0.0]]),
    Bidirectional_Stroop)
# Execute model run and stop when response threshold has been reached (terminate trial value)
Bidirectional_Stroop.run(inputs=type_input, termination_processing=terminate_trial)

[kmantel]

The weights should be set to replicate the model but are not in this script. Additionally, even without the weights set, the generated GRAIN results shown in the documentation have some discrepancies wit the original.

[JeshuaT]

Hi. The key findings in the 1994 paper were an average from 100 noise-induced trials for each condition. Is it possible that this is key for replicating the qualitative findings with the exact same settings as the 1994 paper?

I've also found a single-trial noise-less solution to match the qualitative patterns of the 1994 paper which requires minor tweaking. If you change the inhibition within layers from 2 to 2.5, you should be able to replicate the qualitative patterns.

[tylergiallanza]

I've also been working with this model and have noticed similar issues. Many of the fixes suggested by Jeshua were super helpful for me, but I would be careful when changing the inhibition parameter. Changing inhibition from -2 to -2.5 results in qualitative changes for the dynamics of processing in late stages of response (e.g., the activation of the WR unit at the end of the trial changes dramatically), which has major impacts for simulations building off of the model (such as conflict monitoring). These changes caused by -2.5 inhibition may actually be for the better for future modeling work (personally, I'll probably be using -2.5 inhibition going forward), but may not be good from the standpoint of replicating the findings from, e.g., Botvinick 2001.

I'll update with more information once I get the model working. I just wanted to note that changing the version in the psyneulink repo to use -2.5 inhibition may be premature given these concerns.

[JeshuaT]

Good point Tyler. I had located some minor bugs in the Cohen Psyneulink script, and now it seems to be working whilst keeping the inhibition on -2, and keeping all the other parameters on the same values as the original paper. This is the figure I get:

In the end, it seems to be an assignment issue, e.g.:

The correct way to change an already set parameter in this context:

    response_word_weights.parameters.matrix.set(
        np.array([
            [2.5, 0.0, 0.0],
            [0.0, 2.5, 0.0]
        ]), Bidirectional_Stroop
    )

The incorrect way to change parameter in this context:

    response_word_weights.parameters.matrix.set(
        np.array([
            [2.5, 0.0, 0.0],
            [0.0, 2.5, 0.0]
        ])
    )

or also wrong (how it was originally done):

    response_word_weights = pnl.MappingProjection(
        matrix=np.array([
            [0.0, 0.0, 0.0],
            [0.0, 0.0, 0.0]
        ])
    )

I'll send out a pull request soon implementing the revised script!

[tylergiallanza]

Thanks Jeshua for the insights on this - it's been very helpful for my project involving this model!

One other weird wrinkle of the current psyneulink version that I think is also a mistake is that there is no weight from the neutral color/word input to the corresponding neutral color/word hidden representation. This seems to be different than the original implementation - in the original, the only difference between real colors and neutral colors is that real colors have bidirectional connections with the response. Thus, a neutral color input in the original model would result in activity in the neutral color unit, and would also cause lateral inhibition to the other (red and green) color units. That is not the case with the current psyneulink model, where the neutral input has no weights to anything:

color_input_weights = pnl.MappingProjection(matrix=np.array([[1.0, 0.0, 0.0],
                                                             [0.0, 1.0, 0.0],
                                                             [0.0, 0.0, 0.0]]))

[JeshuaT]

I also noticed that indeed, and that for some reason the weights are set to 0 for word/color --> response only, but not for response --> word/color. This is then reversed for word reading for some reason.

Anyways, I've sent a pull request for the working script. Let me know if you have any questions/remarks on that updated script!

[jdcpni]

Thanks to both of you!! Let me know if any additional problems arise. jdc

…

Sent from my iPhone On Dec 3, 2021, at 8:27 AM, JeshuaT ***@***.***> wrote:  I also noticed that indeed, and that for some reason the weights are set to 0 for word/color --> response only, but not for response --> word/color. This is then reversed for word reading for some reason. Anyways, I've sent a pull request for the working script. Let me know if you have any questions/remarks on that updated script! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#2122 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACSHKGFDUFOJBBK6HKBSIFTUPDAVFANCNFSM5EGEJE7Q>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.