Correct approach for context-dependent rules

[samhocevar]

I wanted to know whether this is the correct approach to my problem. I am using your Lua grammar to parse a language close to Lua, i.e. mostly whitespace agnostic, with a few exceptions where carriage returns are disallowed. But the rest of the grammar perfectly fits the language so I have just modified it in the following ways:

• add an int nocrlf = 0 parse state

• add a disable_crlf<true|false> rule that always succeeds but changes the nocrlf value:

template<bool B> struct disable_crlf
{
    using subs_t = pegtl::type_list<>;

    template< pegtl::apply_mode, pegtl::rewind_mode,
              template< typename ... > class Action,
              template< typename ... > class Control,
              typename Input, typename... States >
    static bool match(Input &, int &nocrlf, States &&...)
    {
        nocrlf += B ? 1 : -1;
        return true;
    }
};

• in addition to the sep rule for separators, add sep_horiz (which only matches horizontal whitespace) and sep_normal (which matches all separators including carriage returns):

struct sep_normal : tao::pegtl::sor< tao::pegtl::ascii::space, comment > {};
struct sep_horiz : tao::pegtl::ascii::blank {};

• make the sep rule behave differently depending on the value of nocrlf:

struct sep
{
    using subs_t = pegtl::type_list< sep_horiz, sep_normal >;

    template< pegtl::apply_mode A, pegtl::rewind_mode R,
              template< typename ... > class Action,
              template< typename ... > class Control,
              typename Input, typename... States >
    static bool match(Input &in, int &disable_crlf, States &&...st)
    {
        if (disable_crlf > 0)
            return sep_horiz::match<A, R, Action, Control>(in, nocrlf, st...);
        return sep_normal::match<A, R, Action, Control>(in, nocrlf, st...);
    }
};

• add a new one_line_seq template rule that behaves like seq with carriage returns disabled:

template< typename... R >
struct one_line_seq :
    tao::pegtl::seq<
        disable_crlf< true >,
        tao::pegtl::sor< tao::pegtl::try_catch< tao::pegtl::seq< R... > >,
                         tao::pegtl::seq< disable_crlf< false >, tao::pegtl::failure > >,
        disable_crlf< false > > {};

This allows me to use all the other grammar rules inside one_line_seq without modifications. But I am not sure I am following the spirit of the PEGTL here.

• is the try_catch idiom the correct way to ensure nocrlf remains consistent with backtracking?
• is there a way to “hide” nocrlf somewhere in the grammar so that callers do not have to instantiate it? is this what tao::pegtl::state<> is for?

[ColinH]

Regarding try-catch and keeping the flag consistent, the grammar might throw exceptions, but it doesn't itself catch them; usually an exception aborts a parsing run which is why we call them "global errors", in which case what happens with the flag shouldn't matter, because you never backtrack after an exception. Unless you are doing something that makes the grammar continue after an exception you are done whenever one occurs.

[ColinH]

Regarding the additional state, you could indeed use something similar to tao::pegtl::state<> as a (near) top-level rule to your grammar so that the user doesn't need to supply the nocrlf state. I'm saying "similar", because that rule replaces all states with the new one, and I'm assuming that you want to add the state for your flag to the ones passed in by the user. You can simply copy the implementation from tao::pegtl::internal::state and adapt it to your needs.

[ColinH]

The way your sep rule calls sep_horiz and sep_normal bypasses the Control which might lead to unexpected results should a user attach actions to these rules or use some of the debugging facilities while parsing. You can fix this by using something like

Control< sep_horiz >::template match< A, R, Action, Control >( in, nocrlf, st... );

instead, i.e. routing the invocation of the sub-rules through the Control just like all built-in rules do.

[d-frey]

In the PEGTL, each rule should do the minimum of what is required and then combine those simple rules into more complicated constructs. This means:

struct is_crlf
{
    using subs_t = pegtl::empty_list;

    template< pegtl::apply_mode A, pegtl::rewind_mode R,
              template< typename ... > class Action,
              template< typename ... > class Control,
              typename Input, typename... States >
    static bool match(Input &in, int &disable_crlf, States &&...st)
    {
        return disable_crlf == 0;
    }
};

and then use pegtl::if_then_else< is_crlf, sep_normal, sep_horiz >. (Or the other way around? Anyways, you get the idea...)

I could also continue to talk about the recently added unwind() method of the control class which might help to track global errors and keep the state consistent in the presence of exceptions. But I think the idea of "hiding" the state can be taken one step further in your case.

It looks to me as if you and marking a whole sub-part of the grammar and these can potentially nest. Instead of having a run-time flag, one could also think about using a control class wrapper as a flag. This would automatically give you everything you need. Something along those (untested) lines: https://godbolt.org/z/RaAnM2

#include <type_traits>

#include <tao/pegtl.hpp>
using namespace tao::pegtl;

namespace demo
{
    namespace internal
    {
        template< template< typename... > class Control >
        struct disable_crlf_control
        {
            template< typename Rule >
            struct type
               : Control< Rule >
            {
                static constexpr bool crlf_disabled = true;
            };
        };

        template< typename Control, typename = void >
        struct is_crlf_disabled
           : std::false_type {};

        template< typename Control >
        struct is_crlf_disabled< Control, std::enable_if_t< Control::crlf_disabled > >
           : std::true_type {};

    }  // namespace internal

    template< typename Rule >
    struct disable_crlf
    {
        using subs_t = type_list< Rule >;

        template< apply_mode A,
                  rewind_mode M,
                  template< typename ... > class Action,
                  template< typename ... > class Control,
                  typename ParseInput,
                  typename... States >
        static bool match( ParseInput& in, States&&... st )
        {
            if constexpr( internal::is_crlf_disabled< Control< void > >::value ) {
                return Control< Rule >::template match< A, M, Action, Control >( in, st... );
            }
            else {
                return internal::disable_crlf_control< Control >::template type< Rule >::template match< A, M, Action, internal::disable_crlf_control< Control >::template type >( in, st... );
            }
        }
    };

    struct is_crlf
    {
        using subs_t = empty_list;

        template< apply_mode A,
                  rewind_mode M,
                  template< typename ... > class Action,
                  template< typename ... > class Control,
                  typename ParseInput,
                  typename... States >
        static bool match( ParseInput& in, States&&... st )
        {
            return !internal::is_crlf_disabled< Control< void > >::value;
        }
    };
}

[ColinH]

@samhocevar Did you find a good solution to this issue? Anything worth sharing?