how to use the PEGTL parser with a separate lexer?

[Frank-Buss]

Here is a simple example:
https://stackoverflow.com/questions/65057971/how-to-use-the-pegtl-parser-with-a-separate-lexer
If I understand it correctly, I need to implement my own input type. I tried it like this:
https://pastebin.com/8rSkwCk1
But this results in 156 lines of compiler error. Is there an easy way to use a vector of my own type as an input for the parser, and specifically how would I do this for my example?

[ColinH]

The short answer is that the PEGTL is not currently in a shape to do this. I also couldn't say how deep the problems run since I haven't tried this myself ... yet. We could support you if you want to give it a try, it might only need small tweaks to the core library to make such an input possible, or it might be too much of a hassle and have to wait for the big input layer rewrite.

[Frank-Buss]

I think it should be possible. It is the first time I tried PEGTL, so maybe I don't understand what it all means, but I did some experiments, and looks like it just doesn't increment the iterator now, otherwise it should work:
https://pastebin.com/fuUSid3z

[Frank-Buss]

Oops, I missed an "&" after the "auto", now the last example works. I will try the rest now and report back.

[Frank-Buss]

@ColinH This is the full example:
https://pastebin.com/NakBHg3a
Parsing is ok, but looks like "apply" is not called, no idea why. And I guess this could be written nicer, maybe you can take a look at it?

[ColinH]

This looks reasonably promising, though there will be pitfalls around many corners.

The actions are not called because your token_input doesn't define an action_t as type for the (alternative) action input, hence the control class' apply() is disabled via SFINAE.

[ColinH]

Here's some minimal code to get the actions with inputs working, too: https://pastebin.com/axqyPxUS

[Frank-Buss]

Thanks!

[ColinH]

@d-frey We could check whether we want this in contrib or as an example; even though only partial it appears to be useful for some things.

[Frank-Buss]

I think this could be useful for parsing binary files, like PNG. A funny thing would be to define a token that encapsulates one bit. But probably more useful would be a higher level, like the header and chunks of PNG. Would be a pretty nice looking PNG reader.

[ColinH]

We already do binary parsing with the normal inputs using the rules for binary values or similar, see include/tao/pegtl/contrib/uint8.hpp and friends, and also the parsers for the binary file formats CBOR, MsgPack and UBJSON in our JSON library at https://github.com/taocpp/json, though of course things like counting line numbers doesn't make much sense.

[Frank-Buss]

That's nice, but looks too much coupled to bytes, at least for the cbor parser:
https://github.com/taocpp/json/blob/master/include/tao/json/cbor/internal/grammar.hpp
For parsing the major and minor parts, it assumes a uint8 input stream.

This concept would be difficult for true bitstream formats like SWF, see page 17 for an example:
https://www.adobe.com/content/dam/acom/en/devnet/pdf/swf-file-format-spec.pdf
or a zip file. With a "bit0" and "bit1" token, such a SWF parser could be implemented. But it would be probably too slow (but a test might be interesting), so higher level tokens might be useful.

[Frank-Buss]

I could test now the token_action_input with a more complex grammar, and it works. But there are 2 problems: When I use standard_trace<my_grammar>(in);, then the compiler says token_parse_input has no member named position. Not sure how to implement it right, but when I add this code I derived from other code in the library:

[[nodiscard]] TAO_PEGTL_NAMESPACE::position position() const
{
    internal::iterator c;
    return TAO_PEGTL_NAMESPACE::position( c, m_source );
}

then it works.

The other problem is when I try to create a tree and dot file like this:

const auto root = parse_tree::parse<my_grammar>(in);
parse_tree::print_dot(std::cout, *root);

I'll get an error about a missing type named subs_t in token_type. No idea how to fix this.

[ColinH]

The missing subs_t is easy to fix, it can be defined like for other atomic rules, i.e. using subs_t = empty_list; with the correct includes, see for example include/tao/pegtl/internal/any.hpp.

The position is more complicated to fix, that's where "the PEGTL is currently not well prepared for this" comes from. The contents of a position are hard-coded throughout the library although line and column numbers don't make sense when parsing tokens or binary data, at least not in the way they are currently managed. Doing this properly might require a lot of changes to the core library.

If you just want to get something running you can jeep your position function that returns some dummy values. Not pretty, but you can at least continue until you get to the next stumbling block - or too annoyed by the position information being wrong.

[Frank-Buss]

I added the using statement, but I get another compile error now. This is the code:
https://pastebin.com/RTb6tn5N

[ColinH]

Adding the using for subs_t to internal::token_type works for me, it compiles. What is the error?

[Frank-Buss]

When trying to compile the pastebin code, I get this error message, with g++ 8.3.0 on Linux:

In file included from parser.cc:7:
PEGTL/include/tao/pegtl/contrib/parse_tree.hpp: In instantiation of ‘void tao::pegtl::parse_tree::basic_node<T>::start(const ParseInput&, States&& ...) [with Rule = my_grammar; ParseInput = tao::pegtl::token_parse_input<my_token, std::__cxx11::basic_string<char> >; States = {}; T = tao::pegtl::parse_tree::node]’:
PEGTL/include/tao/pegtl/contrib/parse_tree.hpp:286:13:   required from ‘static void tao::pegtl::parse_tree::internal::make_control<Node, Selector, Control>::state_handler<Rule, true, B>::start(const ParseInput&, tao::pegtl::parse_tree::internal::state<Node>&, States&& ...) [with ParseInput = tao::pegtl::token_parse_input<my_token, std::__cxx11::basic_string<char> >; States = {}; Rule = my_grammar; bool B = false; Node = tao::pegtl::parse_tree::node; Selector = tao::pegtl::parse_tree::internal::store_all; Control = tao::pegtl::normal]’
PEGTL/include/tao/pegtl/contrib/shuffle_states.hpp:61:21:   required from ‘static void tao::pegtl::shuffle_states<Base, Shuffle>::start(const ParseInput&, State&&) [with ParseInput = tao::pegtl::token_parse_input<my_token, std::__cxx11::basic_string<char> >; State = tao::pegtl::parse_tree::internal::state<tao::pegtl::parse_tree::node>&; Base = tao::pegtl::parse_tree::internal::make_control<tao::pegtl::parse_tree::node, tao::pegtl::parse_tree::internal::store_all, tao::pegtl::normal>::state_handler<my_grammar, true, false>; Shuffle = tao::pegtl::internal::rotate_right<1>]’
PEGTL/include/tao/pegtl/match.hpp:136:32:   required from ‘auto tao::pegtl::match(ParseInput&, States&& ...) [with Rule = my_grammar; tao::pegtl::apply_mode A = (tao::pegtl::apply_mode)1; tao::pegtl::rewind_mode M = (tao::pegtl::rewind_mode)1; Action = tao::pegtl::nothing; Control = tao::pegtl::parse_tree::internal::make_control<tao::pegtl::parse_tree::node, tao::pegtl::parse_tree::internal::store_all, tao::pegtl::normal>::type; ParseInput = tao::pegtl::token_parse_input<my_token, std::__cxx11::basic_string<char> >; States = {tao::pegtl::parse_tree::internal::state<tao::pegtl::parse_tree::node>&}]’
PEGTL/include/tao/pegtl/normal.hpp:80:77:   required from ‘static bool tao::pegtl::normal<Rule>::match(ParseInput&, States&& ...) [with tao::pegtl::apply_mode A = (tao::pegtl::apply_mode)1; tao::pegtl::rewind_mode M = (tao::pegtl::rewind_mode)1; Action = tao::pegtl::nothing; Control = tao::pegtl::parse_tree::internal::make_control<tao::pegtl::parse_tree::node, tao::pegtl::parse_tree::internal::store_all, tao::pegtl::normal>::type; ParseInput = tao::pegtl::token_parse_input<my_token, std::__cxx11::basic_string<char> >; States = {tao::pegtl::parse_tree::internal::state<tao::pegtl::parse_tree::node>&}; Rule = my_grammar]’
PEGTL/include/tao/pegtl/parse.hpp:27:70:   required from ‘auto tao::pegtl::parse(ParseInput&&, States&& ...) [with Rule = my_grammar; Action = tao::pegtl::nothing; Control = tao::pegtl::parse_tree::internal::make_control<tao::pegtl::parse_tree::node, tao::pegtl::parse_tree::internal::store_all, tao::pegtl::normal>::type; tao::pegtl::apply_mode A = (tao::pegtl::apply_mode)1; tao::pegtl::rewind_mode M = (tao::pegtl::rewind_mode)1; ParseInput = tao::pegtl::token_parse_input<my_token, std::__cxx11::basic_string<char> >&; States = {tao::pegtl::parse_tree::internal::state<tao::pegtl::parse_tree::node>&}]’
PEGTL/include/tao/pegtl/contrib/parse_tree.hpp:422:120:   required from ‘std::unique_ptr<Node> tao::pegtl::parse_tree::parse(ParseInput&&, States&& ...) [with Rule = my_grammar; Node = tao::pegtl::parse_tree::node; Selector = tao::pegtl::parse_tree::internal::store_all; Action = tao::pegtl::nothing; Control = tao::pegtl::normal; ParseInput = tao::pegtl::token_parse_input<my_token, std::__cxx11::basic_string<char> >&; States = {}]’
PEGTL/include/tao/pegtl/contrib/parse_tree.hpp:437:60:   required from ‘std::unique_ptr<tao::pegtl::parse_tree::node, std::default_delete<tao::pegtl::parse_tree::node> > tao::pegtl::parse_tree::parse(ParseInput&&, States&& ...) [with Rule = my_grammar; Selector = tao::pegtl::parse_tree::internal::store_all; Action = tao::pegtl::nothing; Control = tao::pegtl::normal; ParseInput = tao::pegtl::token_parse_input<my_token, std::__cxx11::basic_string<char> >&; States = {}]’
parser.cc:208:55:   required from here
PEGTL/include/tao/pegtl/contrib/parse_tree.hpp:125:51: error: no matching function for call to ‘tao::pegtl::internal::iterator::iterator(const my_token* const&)’
          m_begin = TAO_PEGTL_NAMESPACE::internal::iterator( in.iterator() );
                                                   ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from PEGTL/include/tao/pegtl/position.hpp:15,
                 from PEGTL/include/tao/pegtl/parse_error.hpp:16,
                 from PEGTL/include/tao/pegtl/normal.hpp:15,
                 from PEGTL/include/tao/pegtl/parse.hpp:11,
                 from PEGTL/include/tao/pegtl.hpp:8,
                 from parser.cc:6:
PEGTL/include/tao/pegtl/internal/iterator.hpp:33:7: note: candidate: ‘constexpr tao::pegtl::internal::iterator::iterator(tao::pegtl::internal::iterator&&)’
       iterator( iterator&& ) = default;
       ^~~~~~~~
PEGTL/include/tao/pegtl/internal/iterator.hpp:33:7: note:   no known conversion for argument 1 from ‘const iterator_t’ {aka ‘const my_token* const’} to ‘tao::pegtl::internal::iterator&&’
PEGTL/include/tao/pegtl/internal/iterator.hpp:32:7: note: candidate: ‘constexpr tao::pegtl::internal::iterator::iterator(const tao::pegtl::internal::iterator&)’
       iterator( const iterator& ) = default;
       ^~~~~~~~
PEGTL/include/tao/pegtl/internal/iterator.hpp:32:7: note:   no known conversion for argument 1 from ‘const iterator_t’ {aka ‘const my_token* const’} to ‘const tao::pegtl::internal::iterator&’
In file included from PEGTL/include/tao/pegtl/position.hpp:15,
                 from PEGTL/include/tao/pegtl/parse_error.hpp:16,
                 from PEGTL/include/tao/pegtl/normal.hpp:15,
                 from PEGTL/include/tao/pegtl/parse.hpp:11,
                 from PEGTL/include/tao/pegtl.hpp:8,
                 from parser.cc:6:
PEGTL/include/tao/pegtl/internal/iterator.hpp:22:7: note: candidate: ‘tao::pegtl::internal::iterator::iterator(const char*, std::size_t, std::size_t, std::size_t)’
       iterator( const char* in_data, const std::size_t in_byte, const std::size_t in_line, const std::size_t in_column ) noexcept
       ^~~~~~~~
PEGTL/include/tao/pegtl/internal/iterator.hpp:22:7: note:   candidate expects 4 arguments, 1 provided
PEGTL/include/tao/pegtl/internal/iterator.hpp:18:16: note: candidate: ‘tao::pegtl::internal::iterator::iterator(const char*)’
       explicit iterator( const char* in_data ) noexcept
                ^~~~~~~~
PEGTL/include/tao/pegtl/internal/iterator.hpp:18:16: note:   no known conversion for argument 1 from ‘const iterator_t’ {aka ‘const my_token* const’} to ‘const char*’
PEGTL/include/tao/pegtl/internal/iterator.hpp:16:7: note: candidate: ‘constexpr tao::pegtl::internal::iterator::iterator()’
       iterator() = default;
       ^~~~~~~~
PEGTL/include/tao/pegtl/internal/iterator.hpp:16:7: note:   candidate expects 0 arguments, 1 provided

[Frank-Buss]

Compile command line:
g++ -std=c++17 -I PEGTL/include parser.cc

[d-frey]

The subs_t resolved one problem, but now you are running into the missing position problem again. (iterator is closely related to the position). And the position problem is much harder to solve as Colin already mentioned.

[Frank-Buss]

Ok, I thought I solved it with the lines 137-140. At least this solved the problem for standard_trace. Why does the parse_tree thing need more than standard_trace? Sorry, I don't know much of modern C++.

[d-frey]

While using a plain pointer to a token as an iterator is enough for other parts of the PEGTL, the parse tree does need more information right now. Specifically, it will try to access a member from the iterator (internal::iterator) called data. Therefore, even if I try and replace the iterator used in the parse tree's node with your iterator, it will still fail.