performance issue vertico+orderless when ripgrep/ripgrep-all processes running

[maxecharel]

Hi @minad,

I had to find specific matches in files buried in my filesystem. I started both a ripgrep and a ripgrep-all process, called from outside Emacs. Given that rg is multithreaded and that I asked it to basically span a substantial part of the filesystem, including files that first have to be converted to text, it was consuming a lot of resources. Anyway, I had no problem running --- at the same time --- Firefox along other instances of my terminal emulator. I tried to start other programs, no apparent performance issue. However, Emacs, which was already open when I started the ripgrep processes, became super slow, almost frozen. I quitted Emacs, took an eternity. Then I went for a good old emacs -Q, no problem, super fast.

I again restarted Emacs (no -Q this time), took forever to load. Once loaded, I quickly realized C-x C-f and C-x b (consult-buffer) were the slowest. As opposed to e.g. writing in the scratch buffer, or going for M-x and let autocompletion do its job, which worked just fine. So I went for the profiler. After having skimmed through the report, my primary conclusions are that vertico and the completion + filtering (orderless) framework are at the roots of the performance issue.

My system:

• Emacs 29.3 (built from source)
• Vertico 20240726.2131 (Melpa)
• Orderless 20240717.758 (Melpa)
• Consult 20240824.800 (Melpa)

Since I not sure of the accuracy of what I am claiming here, let me rely on your expertise @minad. Here is the last report, for which I tried to minimize the noise:

         900  87% - command-execute
         530  51%  - byte-code
         530  51%   - read-extended-command
         530  51%    - read-extended-command-1
         530  51%     - completing-read-default
         530  51%      - apply
         530  51%       - vertico--advice
         530  51%        - apply
         511  49%         - #<subr completing-read-default>
         487  47%          - vertico--exhibit
         471  45%           - vertico--update
         471  45%            - vertico--recompute
         463  45%             - vertico--filter-completions
         463  45%              - completion-all-completions
         463  45%               - apply
         463  45%                - #<subr completion-all-completions>
         463  45%                 - completion--nth-completion
         463  45%                  - completion--some
         463  45%                   - #<compiled -0x187af3ee9f3327bf>
         427  41%                    - orderless-all-completions
         427  41%                     - orderless--filter
         395  38%                      - #<compiled -0x9acd6943917ce03>
         395  38%                       - mapcar
         395  38%                        - #<compiled 0x1636a7b2db44c3c2>
         395  38%                         - all-completions
         391  38%                          - #<compiled 0x1d4955eebb2359dc>
         391  38%                           - apply
         391  38%                            - completion-table-with-predicate
         391  38%                             - complete-with-action
           4   0%                                all-completions
           4   0%                          + #<compiled 0x5b650fd04401257>
          32   3%                      + #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_55>
          36   3%                    + completion-basic-all-completions
           4   0%             + vertico-sort-history-length-alpha
           8   0%           + vertico--display-candidates
           4   0%             vertico--arrange-candidates
           4   0%          + redisplay_internal (C function)
         370  36%  - funcall-interactively
         359  34%   - consult-buffer
         359  34%    - consult--multi
         359  34%     - consult--read
         359  34%      - consult--read-1
         359  34%       - consult--with-preview-1
         359  34%        - #<compiled 0x13d7b3fabb54180d>
         359  34%         - completing-read
         359  34%          - completing-read-default
         359  34%           - apply
         359  34%            - vertico--advice
         359  34%             - apply
         339  33%              - #<subr completing-read-default>
         136  13%               + vertico--exhibit
          21   2%               + redisplay_internal (C function)
          12   1%               + consult--preview-post-command-hook
           3   0%                 timer-event-handler
           7   0%   + execute-extended-command
           4   0%   + outshine-self-insert-command
          65   6% + redisplay_internal (C function)
          47   4% + timer-event-handler
          14   1% + corfu--post-command
           0   0% + ...

Since the issue seems to come from vertico + orderless when ripgrep and ripgrep-all (basically a wrapper around ripgrep, if I am not mistaken) are subject to a high demand, is it possible that the former are conflicting with the latter, via e.g. process sharing?

Note that:

• as previously mentioned, rg and rga have not been started from Emacs
• I use the preview functionality of consult (not sure whether it is relevant, but ripgrep-all seems to use pandoc, maybe it is not completely orthogonal to the latent process implied by consult-preview?)

As usual, thank you for your work, which is amazing, and sorry for being basically clueless here.

Best,

M

[minad]

I suspect that rg and rga are starving Emacs such that it cannot run its IO operations. The profiles and the CPU load look normal.

[maxecharel]

So basically, the problem would 'just' originate from rg, and likely especially rga, intensively reading (and writing, as I was redirecting the output to files), causing Emacs to be starved of the IO bandwidth required for reading and then previewing via Consult.

Many thanks for the clarification. So nothing to do here, at least at a high level of abstraction, because clearly I do not see myself digging into IO scheduling.