-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FEAT(positional-audio): Plugin for Grounded #6491
base: master
Are you sure you want to change the base?
Conversation
plugins/grounded/CMakeLists.txt
Outdated
@@ -0,0 +1,18 @@ | |||
|
|||
# Copyright 2020-2023 The Mumble Developers. All rights reserved. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# Copyright 2020-2023 The Mumble Developers. All rights reserved. | |
# Copyright 2024 The Mumble Developers. All rights reserved. | |
plugins/grounded/grounded.cpp
Outdated
@@ -0,0 +1,177 @@ | |||
#include "ProcessWindows.h" | |||
#define MUMBLE_PLUGIN_NO_DEFAULT_FUNCTION_DEFINITIONS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why? Unless you are including the plugin header multiple times, having the default definitions saves you the trouble of having to implement some functions yourself 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I just copied this from elsewhere to try to be conventional, but I think you're right in that it doesn't seem appropriate here.
plugins/grounded/grounded.cpp
Outdated
uint8_t _unused1[4]; | ||
/* 950 */ float front[3]; | ||
uint8_t _unused2[4 * 40]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
uint8_t _unused1[4]; | |
/* 950 */ float front[3]; | |
uint8_t _unused2[4 * 40]; | |
std::uint8_t _unused1[4]; | |
/* 950 */ float front[3]; | |
std::uint8_t _unused2[4 * 40]; | |
plugins/grounded/grounded.cpp
Outdated
/* 9fc */ float pos[3]; | ||
}; | ||
|
||
#define unreal_to_mumble_units(unreal) (unreal / 100.f) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#define unreal_to_mumble_units(unreal) (unreal / 100.f) | |
constexpr float unreal_to_mumble_units(float unreal) { return unreal / 100.0f; } | |
plugins/grounded/grounded.cpp
Outdated
|
||
#define unreal_to_mumble_units(unreal) (unreal / 100.f) | ||
|
||
static_assert(sizeof(struct GroundedCam) == 200, ""); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
static_assert(sizeof(struct GroundedCam) == 200, ""); | |
static_assert(sizeof(struct GroundedCam) == 200, "GroundedCam struct has unexpected size"); | |
plugins/grounded/grounded.cpp
Outdated
|
||
uint8_t mumble_initPositionalData(const char *const *programNames, const uint64_t *programPIDs, size_t programCount) { | ||
for (size_t i = 0; i < programCount; ++i) { | ||
if (strcmp(programNames[i], "Maine-Win64-Shipping.exe") != 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would define Maine-Win...
as a std::string
constant at the beginning of the function. Then you can simply do a programNames[i] != myConstant
and also reuse the constant in other places where you need the process name (retyping it every time introduces the potential of typos)
plugins/grounded/grounded.cpp
Outdated
continue; | ||
} | ||
|
||
handle = std::make_unique< GroundedHandle >(GroundedHandle{ std::move(proc), p }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
handle = std::make_unique< GroundedHandle >(GroundedHandle{ std::move(proc), p }); | |
handle = std::make_unique< GroundedHandle >(std::move(proc), p); |
plugins/grounded/grounded.cpp
Outdated
avatarAxis[0] = cameraAxis[0] = -cam.top[0]; | ||
avatarAxis[1] = cameraAxis[1] = cam.top[2]; | ||
avatarAxis[2] = cameraAxis[2] = -cam.top[1]; | ||
|
||
avatarDir[0] = cameraDir[0] = -cam.front[0]; | ||
avatarDir[1] = cameraDir[1] = cam.front[2]; | ||
avatarDir[2] = cameraDir[2] = -cam.front[1]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why don't we need a unit conversion for these fields?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Trigonometry. The range of values goes from -1 to 1.
Let's take cam.front[2]
for example: if you look at the sky it will be ~1
, if you look at the ground it will be ~-1
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They're unit vectors. They have a length of one in both the game and in mumble.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. In that case adding an explicit assert
that checks their magnitude to not differ significantly from one would make the code more self-explanatory.
Also, a short comment could be added that these are unitless (direction only)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I can add comments that these should be unit vectors.
I couldn't find any other plugins to reference that do runtime assertions for these values. My understanding is a failed assertion typically causes the program to exit. If that happens, because the memory we read from the game isn't what we expect, we probably don't want mumble to quit. So I'm not sure an assert is the best choice here?
Instead, I could add a sanity check to initPositionalData that checks that the top and front vectors are of length zero (at the main menu) or one (in a game)?
Alternatively, in the general case, mumble could check that the vectors from fetch functions in positional audio plugins are of valid lengths and possibly respond by disabling the plugin if they appear to be misbehaving.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I couldn't find any other plugins to reference that do runtime assertions for these values. My understanding is a failed assertion typically causes the program to exit. If that happens, because the memory we read from the game isn't what we expect, we probably don't want mumble to quit. So I'm not sure an assert is the best choice here?
That's exactly what happens and that's also why it's a good thing (the crash happens right at the place where an assumption is violated - including information about where exactly the error occurred). Note however, that asserts are only active in debug builds.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I put in an assert.
But I think is worth mentioning that, depending on how memory is written and read, it seems maybe possible to me that mumble could read a copy of memory while these vectors are being updated by the game and partially written for a particular frame. My guess is that each dimension in the three-dimensional vectors are updated as separate instructions. And, though the timing would need to be very unfortunate, it's possible mumble could read a copy of this memory in the middle of an update. So one dimension might be accurate for the current frame and the others dimensions for the previous frame. In that case, the vector that mumble reads might not be a unit vector since it isn't accurate to one particular game state. Here, there isn't really anything mumble can do except maybe discard that fetch and/or try to read the memory again. I don't know that exiting is super useful, but ultimately that's not my decision.
On the other hand, it could make it very obvious if the plugin is attempting to be used but is out of date. So that could be nice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since it will only be an issue in debug builds that will be used by developers, I wouldn't consider this to be an issue. It's true that the assertion could produce a false-positive under very specific circumstances but I think it is still better to have the assumptions encoded into the code explicitly.
So now that you are using the move ctor, things work as expected? Making the
With Mumble 1.5 you can use the positional data viewer inside Mumble to see whether the coordinates are updated when you walk around in the game. If they seem correct, I don't see a reason why positional audio shouldn't work as expected.
No problem
Interesting 🤔 |
Agreed, I would do the same with
Please note that this is a problem we were already aware of: #5728 |
@davidebeatrici the issue being closed seems to imply that the underlying problem in Mumble has been fixed, no? But apparently that's not the case, unless OP is using an older version of Mumble 👀 |
The plugin for Left 4 Dead 2 was updated, but the underlying issue was not fixed. |
e8ed42d
to
b4b3122
Compare
@Krzmbrzl Unfortunately the issue is unacceptable, I would consider it a blocker. When the game is running and you're not in a game the entire client becomes laggy (audio included). |
How long exactly do calls to Looking at Mumble's implementation it seems like in mumble/src/mumble/PluginManager.cpp Lines 254 to 256 in b75fe54
we should probably just return false and set the active plugin to nullptr instead of continue with the looping. However, that shouldn't cause the lag as different processes should be filtered out by the plugin super fast (it should only need one string comparison).
I would expect the frequency of calls to mumble/src/mumble/PluginManager.cpp Lines 127 to 130 in b75fe54
mumble/src/mumble/PluginManager.h Lines 104 to 105 in b75fe54
|
On Linux the issue doesn't appear to be present, in fact:
|
On my machine in a debug build, the constructor to |
I had a suspicion that was the culprit, embarrassing for Microsoft to say the least. I wonder if this is a bug that was introduced in a recent build of Windows 11, since I'm pretty sure it was never encountered when we first wrote our interface. |
Ouch 👀 |
To be fair, And I'm not aware of another way to read the info we need... |
I'm not aware how this plugin needs the information from Nevertheless, it's probably not a bad idea to run this plugin initialization stuff in another thread instead of the main/GUI thread. But I haven't looked into how much work that would be. |
You're absolutely right. We simply never noticed the issue until the new plugin framework became a thing ( In my opinion plugin initialization should happen in a separate thread in any case, especially because the plugins that may be installed by a user are not guaranteed to be programmed correctly. As for the modules: only a few games I've encountered hold memory areas of interest in libraries rather than the (main) executable. Refactoring the interface so that it doesn't iterate through all modules by default would make fully sense. |
Just to recap. Previously, mumble would load all of the process modules and memory regions each time it tried to initialize the plugin. But the plugin would only succeed in initializing when a game world was loaded in. So mumble's UI thread would freeze every second as it tried and failed to initialize the Grounded plugin while Grounded was running on the computer but only at the main menu. I changed the plugin a bit so that should initialize and link even at the main menu. And it should work better for future versions of the game. It can still lag for about a second in system calls while initializing, but in my testing it succeed to link the first try. So arguably the problem preventing merging earlier might not be a problem anymore. Further down I explain the changes an what I found. It uses findPattern to find code that helps start the pointer chain. The pointer it finds points to valid memory at the main menu, but it encounters a null pointer until the game world loads in. If we see this in fetching, the plugin considers it okay and expects the pointer chain to resolve to useful positional data later on. I found the earlier implementation broke between minor game updates. This use of findPattern works across the most recent three patches. So I hope it is unlikely to break in the future. (It might be useful to clarify that in the plugin description somehow? The plugin isn't necessarily for a particular version but it is for the Steam release as opposed to the Microsoft XBox Game Pass release which has a different exe name last I checked.) I'll copy some details about the slowness from the matrix thread:
Since using findPattern now, using VirtualQueryEx is more useful than before as it can tell us which memory regions are executable and worth using findPattern in. But still it doesn't need to be loaded for every module. With this change, The only other concern is that if initialization fails, it will retry frequently and can cause performance issues. Even with the performance improvements planned, findPattern in another process's memory might not be a quick enough for the UI thread to do every second. I was wondering how other plugins do this and I noticed a TODO about this in the GTA 5 plugin. One option is for the plugin to track how many times it has failed and return at some point with MUMBLE_PDEC_ERROR_PERM. On the other hand, this disables positional audio for the plugin entirely rather than just for the process that it saw. So it seems a bit heavy handed. I'm not sure if anybody else agrees with that, but, if we do, an option is for the plugin to record the pid along with its failure count and just skip the pid if it has failed to many times in the initialization. (I am not sure if pid re-use is a concern on windows.)
edit: I had profiled findPattern at taking about 300ms and I must have been testing a debug build, a release build with optimizations takes under 30ms. |
Thank you very much for the extensive explanation and testing! Also, I really appreciate the use of the pattern matching technique as it makes the plugin much more robust to game updates (when done properly).
Perfectly fine, this matches the behavior of other plugins.
I think specifying the minimum version that is known to be compatible is fine, for example:
That way if the plugin ever breaks the description will still be in part true.
In theory the only difference, aside from the exe name, is the missing Steam integration. Without checking we cannot be sure though and unfortunately I don't know anyone with that variant of the game. Please rebase against Feel free to open an issue for Thank you again! |
/home/vsts/work/1/s/plugins/grounded/grounded.cpp:16:13: error: ‘unique_ptr’ in namespace ‘std’ does not name a template type
16 | static std::unique_ptr< GroundedHandle > handle;
| ^~~~~~~~~~
/home/vsts/work/1/s/plugins/grounded/grounded.cpp:13:1: note: ‘std::unique_ptr’ is defined in header ‘<memory>’; did you forget to ‘#include <memory>’?
12 | #include <cstring>
+++ |+#include <memory>
13 |
/home/vsts/work/1/s/plugins/grounded/grounded.cpp: In function ‘uint8_t mumble_initPositionalData(const char* const*, const uint64_t*, size_t)’:
/home/vsts/work/1/s/plugins/grounded/grounded.cpp:173:17: error: ‘handle’ was not declared in this scope
173 | handle = std::make_unique< GroundedHandle >(std::move(proc), addr);
| ^~~~~~
/home/vsts/work/1/s/plugins/grounded/grounded.cpp:173:31: error: ‘make_unique’ is not a member of ‘std’
173 | handle = std::make_unique< GroundedHandle >(std::move(proc), addr);
| ^~~~~~~~~~~
/home/vsts/work/1/s/plugins/grounded/grounded.cpp:173:31: note: ‘std::make_unique’ is defined in header ‘<memory>’; did you forget to ‘#include <memory>’?
/home/vsts/work/1/s/plugins/grounded/grounded.cpp:173:59: error: expected primary-expression before ‘>’ token
173 | handle = std::make_unique< GroundedHandle >(std::move(proc), addr);
| ^
/home/vsts/work/1/s/plugins/grounded/grounded.cpp:173:70: warning: ignoring return value of ‘constexpr typename std::remove_reference<_Tp>::type&& std::move(_Tp&&) [with _Tp = ProcessWindows&; typename std::remove_reference<_Tp>::type = ProcessWindows]’, declared with attribute ‘nodiscard’ [-Wunused-result]
173 | handle = std::make_unique< GroundedHandle >(std::move(proc), addr);
| ~~~~~~~~~^~~~~~
In file included from /usr/include/c++/11/bits/stl_pair.h:59,
from /usr/include/c++/11/bits/stl_algobase.h:64,
from /usr/include/c++/11/bits/stl_tree.h:63,
from /usr/include/c++/11/set:60,
from /home/vsts/work/1/s/plugins/Module.h:10,
from /home/vsts/work/1/s/plugins/HostLinux.h:9,
from /home/vsts/work/1/s/plugins/ProcessBase.h:13,
from /home/vsts/work/1/s/plugins/ProcessWindows.h:9,
from /home/vsts/work/1/s/plugins/grounded/grounded.cpp:6,
from /home/vsts/work/1/b/plugins/grounded/CMakeFiles/grounded.dir/Unity/unity_0_cxx.cxx:4:
/usr/include/c++/11/bits/move.h:104:5: note: declared here
104 | move(_Tp&& __t) noexcept
| ^~~~
In file included from /home/vsts/work/1/b/plugins/grounded/CMakeFiles/grounded.dir/Unity/unity_0_cxx.cxx:4:
/home/vsts/work/1/s/plugins/grounded/grounded.cpp: In function ‘void mumble_shutdownPositionalData()’:
/home/vsts/work/1/s/plugins/grounded/grounded.cpp:182:9: error: ‘handle’ was not declared in this scope
182 | handle.reset();
| ^~~~~~
/home/vsts/work/1/s/plugins/grounded/grounded.cpp: In function ‘bool mumble_fetchPositionalData(float*, float*, float*, float*, float*, float*, const char**, const char**)’:
/home/vsts/work/1/s/plugins/grounded/grounded.cpp:227:53: error: ‘handle’ was not declared in this scope
227 | const ProcessWindows &proc = std::get< 0 >(*handle);
| ^~~~~~ Could you also make the first letter of "plugin" uppercase in the commit's title? Otherwise everything looks good to me. |
Fixed commit title and that compiler error should be gone. I closed this on accident, sorry for the churn. I'm bad at windows and using git from windows is half broken and using anything in wsl is half broken so my setup is a recipe for mistakes but I did check that I could compile it in on fedora 40 in wsl. |
No worries, it's fine. |
|
Positional audio support for the Steam release of Grounded. Tested on 1.4.3.4578, 1.4.4.4634, and 1.4.5.4679.
oops my bad, I thought I had configured git right but apparently not. windows really does keep things surprising and interesting |
Positional audio support for the Steam release of Grounded 1.4.3.4578.
I don't really know what I'm doing but I gave it a shot. I'm open to feedback here or on matrix.
One thing I struggled with was moving
ProcessWindows
into auniqe_ptr
without closing the handle (which happens in the destructor). The default copy constructor forHostWindows
doesn't really work with the RAII it's trying to do. I've deleted it and added a move constructor instead that should be valid. It's possible that what I'm doing with std::move is not a good approach, but I feel strongly thatHostWindows
's default copy constructor is invalid and deleting it is an improvement.Also, I haven't tested this version with another person. I did test a similar program last year that used the Link plugin and that worked with another player. The memory layout has changed a bit since then; but I think the structures and xyz-mapping from the game to mumble are the same. So this should work.
I haven't built mumble on Linux yet, I can't promise that this doesn't break Linux builds. I thought I installed the dependencies for tumbleweed but CMake said something about not being able to find Qt5 Core. I'm really bad at CMake and debugging it always takes so much of my time so I don't want to go down that road unless it's necessary.
Caveats
No context or identity support.
The game's camera is used for both the avatar and camera position in mumble. The game is mostly first person. There are a few circumstances when the game temporarily switches to third person, like if the player opens their inventory menu. In that case, their voice will come from their camera behind their character model. I haven't found separate positional information for the character nearby, though I haven't looked very hard either. If this is a noticable problem I am open to resolving this. But when I played this the effect wasn't obvious.
Mumble's UI is a bit laggy and unresponsive while the game is open at the main menu, before loading in to a world.
The lookup this does to read positional information fails until after the user loads a world and selects their character. This implementation of
mumble_initPositionalData
will return MUMBLE_PDEC_ERROR_TEMP until that lookup is successful once. So, while the game is open, but before loading in,mumble_initPositionalData
will open a handle to the process and attempt the lookup and fail. It feels like this is happening in the main/UI thread and opening a handle to the process or something takes a long enough time that the interface feels unresponsive as mumble is calling it repeatedly.Alternatively, when the process is detected in
mumble_initPositionalData
, it could open a handle and return MUMBLE_PDEC_OK. In that case, the fetch function could attempt the lookup instead. The reason I didn't start with that approach was that if the lookup will never succeed, as a result a game update or something, it seemed preferable that the initialization fails rather than mumble reporting to the user that the game has linked.Looking forward to your feedback.