Local variables aren't parsed, or are parsed as kernel functions.

[Wonderer0]

Description

No local variables seem to be parsed as such, and those that have the same name as a Ruby Kernel method are parsed as kernel functions. The latter problem seems to be a consequence of the former, in that the parser assumes that they are kernel methods because they have the same name and haven't been parsed as local variables.

Steps to Reproduce

1. Create the following Ruby code in Atom editor:

p = 'Hello World'
p1 = p
test = p1
puts test

2. Open Atom's dev tools and inspect the HTML.

Expected behavior:
All local variables should be enclosed in HTML elements indicating what they are. e.g.:
<span class="syntax--variable syntax--other syntax--readwrite syntax--local syntax--ruby">p</span>

Actual behavior:
The variablesp and test have the following <span> element, wherever they are used:
<span class="syntax--support syntax--function syntax--kernel syntax--ruby">
The variable p1 is just bare text wherever it is used, so cannot easily be formatted by CSS.

Reproduces how often:
Every time.
All the local variables I have tested that have the same names as Kernel methods are parsed as kernel functions.
All others, including those with the names of methods of Object (e.g.clone) are not parsed.

Versions

Atom : 1.18.0
Electron: 1.3.15
Chrome : 52.0.2743.82
Node : 6.5.0
apm 1.18.1
npm 3.10.10
node 6.9.5 x64
python
git
visual studio
language-ruby : 0.71.0
Windows 7 Pro x64

Additional Information

[stevenpetryk]

It may be possible to highlight these correctly on the left side of the equals sign using a lookbehind. In all other cases, though, it is not possible to know when p is a local variable or a call to Kernel#p, because syntax highlighting packages are not interpreters or even parsers—they're lexers. Just a collection of naive regexes.

I ask you this: does overshadowing Kernel methods with local variables seem like a good idea? I would argue that, in general, this "bug" is useful behavior: it warns the user that they are treading dangerous waters.

[Wonderer0]

I think that dubious practices such as using the names of Kernel methods as variables should be detected and highlighted by a linter, but the syntax colouring should just reflect how the language is actually parsed. Appending the line p p to the script above does cause it to print "Hello World" as expected, so the different uses of p are being parsed correctly and Kernel#p isn't actually being shadowed by the variable p.

The example I gave is obviously contrived and unusual, but it does illustrate the underlying problem that local variables aren't being recognised as such at all. If they have the same name as a Kernel method then they are displayed as syntax--kernel elements, but usually they are just bare text in the enclosing <div>.

If Atom's syntax highlighting packages are limited to just using regexes you have an interesting challenge! If you could obtain or write a Ruby parser as a finite state machine/automaton you might then be able to convert that to regexes, as described on Stack Exchange and elsewhere. Just writing a parser to recognise local variables would be good enough for this particular problem.

It's a pity that Atom doesn't allow you to use existing parsers, such as the one that must be built into linter-ruby, rather than having to attempt to parse it yourself just using regexes. If it could accept a parse tree in some specified form all you would have to do would be to convert the output from a Ruby parser into that form if necessary.

I must also add that I am now using Visual Studio Code as my main code editor, mostly for it's IntelliSense, but it does also recognise and colour Ruby local variables in most contexts. So it's no longer urgent for me that this or the other problems I reported are fixed.

[winstliu]

We are working on an experimental new parser system at tree-sitter. But for the most part, Atom, GitHub, and VSCode all use TextMate-compatible grammars (with the latter two using many of Atom's languages), which are by definition only regexes.

[Wonderer0]

tree-sitter looks to be a good way forwards.
In the meantime perhaps Atom could use VSCode's regexes for Ruby local variables, if both editors use the same TextMate based mechanism.

[FranklinYu]

Please change the title. Local variable is different from instance variable. This issue has nothing to do with instance variable (like @foobar).