CIP-0132? | New Plutus Builtin DropList

[colll78]

We propose to introduce a new Plutus builtin dropList that drops a given number of elements from a BuiltInList. Currently, elemAt and drop must be implemented using fixed-point recursion and invoking many other Plutus builtins in each recursive call. These functions are critical components of many common smart contract design patterns (ie redeemer-indexing) so their poor performance has a large impact on the throughput of many protocols.

---

(latest version in branch)

[michaelpj]

The fundamental issue here is that we can't just make every function on builtin types a builtin (or even every recursive function). Today it's dropList, tomorrow takeList, then splitAt then who knows what. We don't have enough space for builtins, for one thing. The goal is generally to expose a minimal set of functions such that other functions can be implemented in Plutus Core, which is the general-purpose programming language after all.

If we can't write acceptably fast recursive functions in Plutus Core, that's a problem with Plutus Core.

It's also unclear to me whether the cost comes from doing the recursion, or from having to call tail many times? I suggest doing some profiling to find out. If it's the latter, you could consider converting to a SOP list which will be much faster to process.

[colll78]

We have done quite a bit of profiling. There is a drastic performance difference in the first elemAt (that recurses once for each call to tail) and the heuristic version that calls tail many times in each recursive call. I agree that generally we shouldn't have a builtin for everything, but given how widely (and extensively) used this function is across pretty much every major DApp, the benefit of a builtin in this specific case is quite significant.

Alternatively, O(1) index data-structures would address the vast majority of the use-cases for this builtin.

[michaelpj]

Right, but you can use the budget profiling to work out exactly where the budget is going in the naive version. If you're not using plutus-tx then just add some appropriate traces before and after the evaluations and you can see the difference. Alternatively if you've got a benchmark we could run it.

[rphair]

@colll78 @michaelpj @michele-nuzzi adding to next CIP meeting agenda (https://hackmd.io/@cip-editors/83) for introduction at least.

Please @colll78 remove the template comments from the markup, since they're no longer useful after the content is written & can get in the way of editing.

[rphair]

@michaelpj please let us know if this single checkbox item properly signifies all the requirements for Plutus Core changes as per CIP-0035, or if you think these need to be itemised further here & in similar proposals.

[rphair]

Suggested change

	- https://github.com/cardano-foundation/CIPs/pull/418/
	- https://github.com/cardano-foundation/CIPs/pull/418
	- https://github.com/cardano-foundation/CIPs/pull/767

I presume that the first of these links represents one of the "design patterns" for which this would be useful (please correct if the older link is unrelated).

[colll78]

Right, but you can use the budget profiling to work out exactly where the budget is going in the naive version. If you're not using plutus-tx then just add some appropriate traces before and after the evaluations and you can see the difference. Alternatively if you've got a benchmark we could run it.

pelemAt' :: PIsListLike l a => Term s (PInteger :--> l a :--> a)
pelemAt' = phoistAcyclic $
  pfix #$ plam $ \self n xs ->
    pif
      (n #== 0)
      (phead # xs)
      (self # (n - 1) #$ ptail # xs)

pelemAtFast :: PIsListLike l a => Term s (PInteger :--> l a :--> a)
pelemAtFast = phoistAcyclic $
  pfix #$ plam $ \self n xs ->
    pif
      (n #> 10)
      (self # (n - 10) #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail # xs)
      (pif
         (n #> 5) 
         (self # (n - 5) #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail # xs)
         (pif (n #== 0) (phead # xs) (pelemAt' # (n - 1) # (ptail # xs)))
      )
      
tail26 :: PIsListLike l a => Term s (l a :--> a)
tail26 = plam (\xs -> ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail #$ ptail # xs)


plotsOfHashes :: Term s (PBuiltinList PValidatorHash)
plotsOfHashes = pconstant [scred1, scred2, scred3, scred4, scred5, scred6, scred7, scred1, scred2, scred3, scred4, scred5, scred6, scred7, scred1, scred2, scred3, scred4, scred5, scred6, scred7, scred1, scred2, scred3, scred4, scred5, scred6, scred7, scred1, scred2, scred3, scred4, scred5, scred6, scred7, scred1, scred2, scred3, scred4, scred5, scred6, scred7]

canonicalOrdering = pconstant [26, 21, 6, 12, 13, 15]

main :: IO ()
main = do
  putStrLn "elemAt: naive"
  case evalT
    (  pelemAt' # 26 # plotsOfHashes
    ) of
    Right (result, budget, trc) -> print (unScript result) >> print trc >> print budget
    Left err -> print err

  putStrLn "\n"
  putStrLn "elemAt: fast "
  case evalT
    (  pelemAtFast # 26 # plotsOfHashes
    ) of
    Right (result, budget, trc) -> print (unScript result) >> print trc >> print budget
    Left err -> print err
    
  putStrLn "elemAt: canonical ordering naive"
  case evalT
    (  Plutarch.List.pmap # plam (\idx -> pelemAt' # idx # plotsOfHashes) # canonicalOrdering
    ) of
    Right (result, budget, trc) -> print (unScript result) >> print trc >> print budget
    Left err -> print err

  putStrLn "\n"
  putStrLn "elemAt: canonical ordering fast "
  case evalT
    (  Plutarch.List.pmap # plam (\idx -> pelemAtFast # idx # plotsOfHashes) # canonicalOrdering
    ) of
    Right (result, budget, trc) -> print (unScript result) >> print trc >> print budget
    Left err -> print err

elemAt 26: naive

• ExBudget {exBudgetCPU = ExCPU 35964686, exBudgetMemory = ExMemory 95270}

elemAtFast 26: heuristic optimized

• ExBudget {exBudgetCPU = ExCPU 10169745, exBudgetMemory = ExMemory 26688}

tail26: tail called 26 times no recursion

• ExBudget {exBudgetCPU = ExCPU 2381832, exBudgetMemory = ExMemory 6632}

elemAt: canonical ordering naive

• ExBudget {exBudgetCPU = ExCPU 136487314, exBudgetMemory = ExMemory 362752}

elemAt: canonical ordering fast

• ExBudget {exBudgetCPU = ExCPU 56393476, exBudgetMemory = ExMemory 149696}

tail called 26 times without recursion is 6632 mem, which blows everything else out of the water. It accounts for a very small portion of the budget. The difference in efficiency is quite dramatic.

[keyan-m]

The fundamental issue here is that we can't just make every function on builtin types a builtin (or even every recursive function). Today it's dropList, tomorrow takeList, then splitAt then who knows what.

I very much appreciate @michaelpj's stance here. On the other hand, imho lists are fundamental enough to warrant dedicated builtin functions.

[michaelpj]

I'd like to replicate the analysis to better understand what's going on here, but I haven't had time.

[rphair]

@colll78 I'm (belatedly?) marking this as Waiting for Author pending the issues noted above... as I understand it, to quantify the changes to efficiency through profiling. Editors can't validate Plutus proposals at that level, so we would have to wait for input from the enlisted Plutus representative(s) (@zliu41?) and perhaps need a new State: tag (as per the new effort #883) if it's them we're waiting for rather than you.

In any case please update this thread if anything has changed on your end or if you have any ideas how to push this forward in the Plutus world. Editors have not heard from Plutus official representatives in a while and we should consider it a top priority to confirm we have the right contacts & process for that (cc @Ryun1 @Crypto2099).

[colll78]

I'm not sure what this is waiting on. I believe the benchmarking above should be sufficient to demonstrate how critical this feature is. If anyone would like to reproduce the benchmarks locally you can git clone discovery follow the instructions in the README and then run nix develop and repl in and run the benchmarks in the file that I linked to.

[rphair]

OK @colll78 I do agree with what you are saying, and think this should move forward even without a response from the current Plutus representatives about MPJ's requested details about benchmarking.

As you say it should be independently verifiable and therefore we can work on merging this as Proposed and leave the eventual Active status dependent on the Plutus team's own implementation choices (which would eventually include their own benchmarking). cc @zliu41

So @Ryun1 @Crypto2099 if you agree let's assign a CIP number after tomorrow's meeting (I've put it on Review agenda: https://hackmd.io/@cip-editors/96) and leave aside the benchmarking question on the way to merge: unless the Plutus team wants to indicate any particular reservations.

[Ryun1]

Comments from CIP Editor meeting # 96
Good to move ahead and assign number, leaving this proposal in review.

[rphair]

On agenda for Review again next meeting (https://hackmd.io/@cip-editors/97) - in the meantime @colll78 please try to post here whatever you can find confirming an implementation plan & update whether a hard fork is expected to be required. I don't think any of these are required before merging but it would be nice to do that with the document as complete as possible.

Please don't forget to change the containing directory to CIP-0132 🎉

[rphair]

Suggested change

	CIP: ?
	CIP: 132

[effectfully]

I like this CIP and I propose that we merge it after @colll78 includes his benchmarking results in the CIP itself (at present it's quite vague on the benefits of adding the builtin). It would be even better if we could see the effect of adding the builtin on some actually used in practice smart contract rather than a microbenchmark.

There are however many things to discuss and investigate here. Why is the naive implementation so inefficient? Is it the cost of doing recursion in Plutus, is it the cost of calling 3 builtins per element, can there be something wrong with the UPLC of the benchmarks? And the most important question is whether this is an inherent limitation of Plutus or something that we can fix without introducing a tailored builtin (I highly doubt it's fixable without adding a builtin for any reasonable definition of "fixable").

And do we need to make the builtin tailored? Maybe we should add splitAt instead? Or sliceList similar to sliceByteString that we already have? Or some general paramorphism?

My conjecture is that paramorphism would be an overkill and splitAt a nice general enough solution that is still efficient.

So basically, show me that this meaningfully speeds up (by at least several percent) a popular smart contract and I'll be happy to champion this CIP. In the meantime, I've added the dropList builtin for experimentation purposes here.

[effectfully]

Just to nitpick, it's the Z combinator, not Y.

[effectfully]

The fact that it looks big does not mean that it's inefficient. We've benchmarked various ways to get recursion in Haskell (sic), the Z combinator is about on par with direct recursion. This doesn't translate to UPLC of course (since UPLC doesn't even have direct recursion), but my point is that it's very non-obvious how your Haskell code snippet illustrates any overhead.

The inefficiency comes from evaluating more things on the Plutus side than on the Haskell side. All those # are going to be evaluated in Plutus and if those were $ in Haskell instead, they'd be much more efficient -- and that is where the overhead arises from. Plus the fact that you currently need 3 builtin calls per element to implement dropList in Plutus and those are expensive.

[colll78]

I included that to illustrate how inefficient it is specifically in ex-unit terms, and as you said most of that cost comes from all those function applications that are invoked. Yes though, I agree that it only accounts for a small portion of the budget and most of the cost comes from the actual content of each recursive call.

[effectfully]

Why not just inline case_true and case_false instead of using f PTrue and f PFalse?

[colll78]

In Plutarch you can think of Haskell level work as a sort of pre-compilation step. The above would actually be inlined when compiled, because f or \case (a Haskell level function) is used on PTrue (a Haskell level argument), this would become:
pif b case_true case_false = pforce $ (pforce $ punsafeBuiltin PLC.IfThenElse) # b # pdelay case_true # pdelay case_false

after the pre-compilation step. I have edited it for clarity though.

[effectfully]

This loops on negative integers (I understand that this snippet is for illustration purposes only, but I'll still leave this comment).

[colll78]

Intended behavior is to error on negative integers anyway, in this case looping is a more efficient way to achieve that because looping results in a script failure from ex-unit budget exceeding and we don't need to waste a check for n > 0 in the happy path. The only case this isn't more efficient is if the script will fail anyway, and in that case we don't care.

[effectfully]

That's a really good point, thanks!