-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optionally queue outgoing data #3844
base: master
Are you sure you want to change the base?
Conversation
Support queueing outgoing stanzas and stream management elements for up to a configurable number of milliseconds (with a configurable queue size limit). This allows for batching up multiple XML elements into a single TCP packet in order to reduce the TCP/IP overhead. The feature is supported by ejabberd_c2s, ejabberd_s2s_out, and ejabberd_service. It can be enabled by configuring the max. number of milliseconds to queue an element (default: 0), and optionally the max. number of elements to queue (default: 10). This can be done by using the following new ejabberd_c2s/ejabberd_service listener options: - max_send_queue_size - max_send_queue_delay For ejabberd_c2s, the following global options can be specified instead: - c2s_max_send_queue_size - c2s_max_send_queue_delay For ejabberd_s2s_out, the following global options can be specified: - s2s_max_send_queue_size - s2s_max_send_queue_delay
2da98d5
to
abfb62f
Compare
We are running this in large-scale production now and it works well |
@mzealey any experience you can share, eg. Values for those two settings ? |
50ms / batch size 10 works just fine for us to reduce number of packets flowing. Slight increase in latency obviously but we only need near-realtime and would prefer to save on packets |
I have applied those changes abfb62f But there is not improvement the message distribution still slow on a group of 1000 members.. plz help to advise.. |
@iPPLE two ideas, maybe this is not the right approach for you, maybe look at mucsub or disable presence in those groups But, you applied this PR and also used the corresponding xmpp lib PR processone/xmpp#63 yes? |
I have this error logs : 2023-11-15 15:34:25.267498+07:00 [error] <0.8281.0> ** Generic server <0.8281.0> terminating 2023-11-15 15:34:25.271062+07:00 [error] <0.8281.0>@proc_lib:crash_report/4:525 CRASH REPORT: |
I'm running ejabberd
%%%----------------------------------------------------------------------
%%%
%%% ejabberd, Copyright (C) 2002-2021 ProcessOne
%%%
%%% This program is free software; you can redistribute it and/or
%%% modify it under the terms of the GNU General Public License as
%%% published by the Free Software Foundation; either version 2 of the
%%% License, or (at your option) any later version.
%%%
%%% This program is distributed in the hope that it will be useful,
%%% but WITHOUT ANY WARRANTY; without even the implied warranty of
%%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
%%% General Public License for more details.
%%%
%%% You should have received a copy of the GNU General Public License along
%%% with this program; if not, write to the Free Software Foundation, Inc.,
%%% 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
%%%
%%%----------------------------------------------------------------------
{deps, [
{gun, ".*", {git, "https://github.com/ninenines/gun", {tag, "1.3.3"}}},
{jsx, ".*", {git, "https://github.com/talentdeficit/jsx", {tag, "2.9.0"}}},
{base64url, ".*", {git, "https://github.com/dvv/base64url", {tag, "1.0.1"}}},
{cache_tab, ".*", {git, "https://github.com/processone/cache_tab", {tag, "1.0.27"}}},
{eimp, ".*", {git, "https://github.com/processone/eimp", {tag, "1.0.19"}}},
{if_var_true, tools,
{ejabberd_po, ".*", {git, "https://github.com/processone/ejabberd-po", {branch, "main"}}}},
{if_var_true, elixir,
{elixir, ".*", {git, "https://github.com/elixir-lang/elixir", {tag, "v1.4.4"}}}},
{if_var_true, pam,
{epam, ".*", {git, "https://github.com/processone/epam", {tag, "1.0.10"}}}},
{if_var_true, redis,
{eredis, ".*", {git, "https://github.com/wooga/eredis", {tag, "v1.0.8"}}}},
{if_var_true, sip,
{esip, ".*", {git, "https://github.com/processone/esip", {tag, "1.0.41"}}}},
{if_var_true, zlib,
{ezlib, ".*", {git, "https://github.com/processone/ezlib", {tag, "1.0.9"}}}},
{fast_tls, ".*", {git, "https://github.com/processone/fast_tls", {tag, "1.1.11"}}},
{fast_xml, ".*", {git, "https://github.com/processone/fast_xml", {tag, "1.1.45"}}},
{fast_yaml, ".*", {git, "https://github.com/processone/fast_yaml", {tag, "1.0.30"}}},
{idna, ".*", {git, "https://github.com/benoitc/erlang-idna", {tag, "6.0.0"}}},
{jiffy, ".*", {git, "https://github.com/davisp/jiffy", {tag, "1.0.5"}}},
{jose, ".*", {git, "https://github.com/potatosalad/erlang-jose", {tag, "1.9.0"}}},
{lager, ".*", {git, "https://github.com/erlang-lager/lager", {tag, "3.6.10"}}},
{if_var_true, tools,
{luerl, ".*", {git, "https://github.com/rvirding/luerl", {tag, "v0.3"}}}},
{mqtree, ".*", {git, "https://github.com/processone/mqtree", {tag, "1.0.12"}}},
{p1_acme, ".*", {git, "https://github.com/processone/p1_acme", {tag, "1.0.11"}}},
{if_var_true, mysql,
{p1_mysql, ".*", {git, "https://github.com/processone/p1_mysql", {tag, "1.0.17"}}}},
{p1_oauth2, ".*", {git, "https://github.com/processone/p1_oauth2", {tag, "0.6.8"}}},
{if_var_true, pgsql,
{p1_pgsql, ".*", {git, "https://github.com/processone/p1_pgsql", {tag, "1.1.10"}}}},
{p1_utils, ".*", {git, "https://github.com/processone/p1_utils", {tag, "1.0.21"}}},
{pkix, ".*", {git, "https://github.com/processone/pkix", {tag, "1.0.7"}}},
{if_not_rebar3, %% Needed because modules are not fully migrated to new structure and mix
{if_var_true, elixir,
{rebar_elixir_plugin, ".*", {git, "https://github.com/processone/rebar_elixir_plugin", "0.1.0"}}}},
{if_var_true, sqlite,
{sqlite3, ".*", {git, "https://github.com/processone/erlang-sqlite3", {tag, "1.1.11"}}}},
{stringprep, ".*", {git, "https://github.com/processone/stringprep", {tag, "1.0.24"}}},
{if_var_true, stun,
{stun, ".*", {git, "https://github.com/processone/stun", {tag, "1.0.42"}}}},
{xmpp, ".*", {git, "https://github.com/weiss/xmpp", {branch, "feature/send-queue"}}},
{yconf, ".*", {git, "https://github.com/processone/yconf", {tag, "1.0.10"}}}
]}.
{gitonly_deps, [elixir, luerl]}.
{if_var_true, latest_deps,
{floating_deps, [cache_tab,
eimp,
epam,
esip,
ezlib,
fast_tls,
fast_xml,
fast_yaml,
mqtree,
p1_acme,
p1_mysql,
p1_oauth2,
p1_pgsql,
p1_utils,
pkix,
sqlite3,
stringprep,
stun,
xmpp,
yconf]}}.
{erl_first_files, ["src/ejabberd_sql_pt.erl", "src/ejabberd_config.erl",
"src/gen_mod.erl", "src/mod_muc_room.erl",
"src/mod_push.erl", "src/xmpp_socket.erl"]}.
{erl_opts, [nowarn_deprecated_function,
{i, "include"},
{if_version_above, "20", {d, 'DEPRECATED_GET_STACKTRACE'}},
{if_version_below, "21", {d, 'USE_OLD_HTTP_URI'}},
{if_version_below, "22", {d, 'LAGER'}},
{if_version_below, "23", {d, 'USE_OLD_CRYPTO_HMAC'}},
{if_version_below, "23", {d, 'USE_OLD_PG2'}},
{if_var_match, db_type, mssql, {d, 'mssql'}},
{if_var_false, debug, no_debug_info},
{if_var_true, debug, debug_info},
{if_var_true, elixir, {d, 'ELIXIR_ENABLED'}},
{if_var_true, hipe, native},
{if_var_true, new_sql_schema, {d, 'NEW_SQL_SCHEMA'}},
{if_var_true, roster_gateway_workaround, {d, 'ROSTER_GATWAY_WORKAROUND'}},
{if_var_true, sip, {d, 'SIP'}},
{if_var_true, stun, {d, 'STUN'}},
{if_have_fun, {erl_error, format_exception, 6}, {d, 'HAVE_ERL_ERROR'}},
{if_rebar3, {extra_src_dirs, [sql]}},
{src_dirs, [src,
{if_var_true, tools, tools},
{if_var_true, elixir, include}]}]}.
{deps_erl_opts, [{if_var_true, hipe, native}]}.
{if_rebar3, {plugins, [rebar3_hex, {provider_asn1, "0.2.0"}]}}.
{if_rebar3, {project_plugins, [configure_deps]}}.
{if_not_rebar3, {plugins, [
deps_erl_opts, override_deps_versions2, override_opts, configure_deps,
{if_var_true, elixir, rebar_elixir_compiler},
{if_var_true, elixir, rebar_exunit}
]}}.
{if_rebar3, {if_var_true, elixir,
{project_app_dirs, [".", "elixir/lib"]}}}.
{if_not_rebar3, {if_var_true, elixir,
{lib_dirs, ["deps/elixir/lib"]}}}.
{if_var_true, elixir,
{src_dirs, ["include"]}}.
{sub_dirs, ["rel"]}.
{keep_build_info, true}.
{xref_warnings, false}.
{xref_checks, [deprecated_function_calls]}.
{xref_exclusions, [
"(\"gen_transport\":_/_)",
"(\"eprof\":_/_)",
{if_var_false, elixir, "(\"Elixir.*\":_/_)"},
{if_var_false, http, "(\"lhttpc\":_/_)"},
{if_var_false, mysql, "(\".*mysql.*\":_/_)"},
{if_var_false, odbc, "(\"odbc\":_/_)"},
{if_var_false, pam, "(\"epam\":_/_)"},
{if_var_false, pgsql, "(\".*pgsql.*\":_/_)"},
{if_var_false, redis, "(\"eredis\":_/_)"},
{if_var_false, sqlite, "(\"sqlite3\":_/_)"},
{if_var_false, zlib, "(\"ezlib\":_/_)"}]}.
{eunit_compile_opts, [{i, "tools"},
{i, "include"}]}.
{cover_enabled, true}.
{cover_export_enabled, true}.
{recursive_cmds, ['configure-deps']}.
{overrides, [
{del, [{erl_opts, [warnings_as_errors]}]}]}.
{post_hook_configure, [{"eimp", []},
{if_var_true, pam, {"epam", []}},
{if_var_true, sip, {"esip", []}},
{if_var_true, zlib, {"ezlib", []}},
{"fast_tls", []},
{"fast_xml", [{if_var_true, full_xml, "--enable-full-xml"}]},
{"fast_yaml", []},
{"stringprep", []}]}.
%% Local Variables:
%% mode: erlang
%% End:
%% vim: set filetype=erlang tabstop=8:
%%%-------------------------------------------------------------------
%%% @author Evgeny Khramtsov <ekhramtsov@process-one.net>
%%% @doc
%%%
%%% @end
%%% Created : 18 Nov 2016 by Evgeny Khramtsov <ekhramtsov@process-one.net>
%%%
%%%
%%% Copyright (C) 2002-2022 ProcessOne, SARL. All Rights Reserved.
%%%
%%% Licensed under the Apache License, Version 2.0 (the "License");
%%% you may not use this file except in compliance with the License.
%%% You may obtain a copy of the License at
%%%
%%% http://www.apache.org/licenses/LICENSE-2.0
%%%
%%% Unless required by applicable law or agreed to in writing, software
%%% distributed under the License is distributed on an "AS IS" BASIS,
%%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
%%% See the License for the specific language governing permissions and
%%% limitations under the License.
%%%
%%%-------------------------------------------------------------------
{application, xmpp,
[{description, "Erlang/Elixir XMPP parsing and serialization library"},
{vsn, "1.5.8"},
{modules, []},
{registered, []},
{applications, [kernel, stdlib, ezlib, fast_tls, fast_xml, idna, p1_utils, stringprep]},
{mod, {xmpp, []}},
{env, []},
%% hex.pm packaging:
{files, ["src/", "specs/", "asn1/", "include/", "c_src/jid.c", "c_src/xmpp_uri.c", "c_src/xmpp_lang.c", "rebar.config", "rebar.config.script", "README.md", "LICENSE.txt"]},
{exclude_files, ["src/XmppAddr.erl", "src/XmppAddr.asn1db", "include/XmppAddr.hrl"]},
{licenses, ["Apache 2.0"]},
{links, [{"Github", "https://github.com/processone/xmpp"}]}]}.
%% Local Variables:
%% mode: erlang
%% End:
%% vim: set filetype=erlang tabstop=8: |
Yes. As I've submitted this PR, I'm notified on comments. Actually, I've enabled all notifications for the ejabberd repository, so there's no point in pinging me. I didn't respond to @iPPLE since @licaon-kter already mentioned that "this is not the right approach" for solving the issue of "message distribution" being "slow" in groupchats. I'd rather not spam the discussion of this PR with unrelated topics and would therefore suggest @iPPLE to back out this patch and open a new issue to describe the actual problem. As for the |
weiss & Neustradamus Thanks for the reply.. i thought this PR is somehow helping to improve the group message distributions of 1k members as mzealey claimed that it is working.. but actually it is not working for me.. i've been wondering if something i have missed in that PR... Because it seems the messages are not batched into one for that configuration... Anyway is there any approach for that big group messages distribution ? is it improved or fixed in the latest version of ejabberd ? because this PR is not included in the official latest version of ejabberd. Thanks. |
@weiss plz keep me posted when the patch will be done for WebSocket.. Thanks. |
Actually, I didn't plan to support WebSocket (as it's not super-trivial and I don't need it myself). The new options are documented to work for I might have a quick look into supporting WebSocket after all, but as I said above, this patch wouldn't help with the issues you mentioned. |
After all i don't find any improvement on the message distribution, I'm trying to check the code to see where the messages are batched but it seems everything is ended up here
and the server still route the message one by one i guess. i'm wondering what scenario that the processone/xmpp#63 is for ?
My scenario is that i managed to have a room of Please kindly help to explain and advice on the 2 points. |
Support queueing outgoing stanzas and stream management elements for up to a configurable number of milliseconds (with a configurable queue size limit). This allows for batching up multiple XML elements into a single TCP packet in order to reduce the TCP/IP overhead.
The feature is supported by
ejabberd_c2s
,ejabberd_s2s_out
, andejabberd_service
. It can be enabled by configuring the maximum number of milliseconds to queue an element (default:0
), and optionally the maximum number of elements to queue (default:10
). This can be done by using the following newejabberd_c2s
/ejabberd_service
listener options:max_send_queue_size
max_send_queue_delay
For
ejabberd_c2s
, the following global options can be specified instead:c2s_max_send_queue_size
c2s_max_send_queue_delay
For
ejabberd_s2s_out
, the following global options can be specified:s2s_max_send_queue_size
s2s_max_send_queue_delay
This PR depends on processone/xmpp#63.