RFC: Allow xpcall error handlers to yield

[ccuser44]

RFC: Allow xpcall error handlers to yield

Rendered

[bradsharp]

An additional drawback is that a number of users use xpcall to prevent yielding of code. That said, this can be reimplemented with:

local function noyield(f, ...)
  local t = coroutine.create(f)
  local r = table.pack(coroutine.resume(t, ...))
  if coroutine.status(t) == "suspended" then
    coroutine.close(t)
    error("Method is not allowed to yield")
  end
  return table.unpack(r)
end

[ccuser44]

An additional drawback is that a number of users use xpcall to prevent yielding of code. That said, this can be reimplemented with:

local function noyield(f, ...)
  local t = coroutine.create(f)
  local r = table.pack(coroutine.resume(t, ...))
  if coroutine.status(t) == "suspended" then
    coroutine.close(t)
    error("Method is not allowed to yield")
  end
  return table.unpack(r)
end

I see, that may be a problem. However I still absolutely think that the xpcall error handler should be able to yield.
First of all this behavior is totally undocumented. Relying on it is hacky and unreliable too as the yield behavior of both xpcall (in Luau) and pcall (before Luau when ypcall was deprecated).
Another is parity with vanilla Lua, the current behavior only partially mirrors that of vanilla Lua.
Also the debug semantics of not allowing the error handler to yield are quite un-user friendly.

[bradsharp]

I'd be in favor of this but we need to carefully consider how we'd roll this out without impacting users.

[vegorov-rbx]

I don't think it will be "Probably very minor though.", seems like a major increase in complexity of the VM where the whole Lua 5.2 implementation uses data structures incompatible with Luau VM.

Given limited use case and already existing yielding limitations, I don't think it will be worth adding this. Also don't want to accept an RFC to later revert as unimplementable.

Just because compatibility box in the docs in incorrect, doesn't mean we have to change Luau, we can just note the difference in the documentation correctly.