[Experimental] Adding interfaces for Feature "Enhanced UX Notification for Video and Audio Call"

[satkh]

The Enhanced UI for Calling Notifications introduces a series of design improvements aimed at making video call notifications clearer, more interactive, and user-friendly. This feature is designed to increase the likelihood of users responding to calls promptly and efficiently by optimizing how notifications are presented and interacted with. We are adding improvements below to video calling UI/UX.  

 
Interactive Elements: 
 
Quick Actions: Integrated buttons for mic and camera to allow users to manage their audio device and camera directly from the notification without needing to switch apps or windows. 
 
Live camera feed preview: Users can view a live feed of themselves when they make/receive a video call. This will help users be better prepared for video calls and also potentially avoid embarrassing situations.

A screenshot of a video call

[jonwis]

When we do API design specs, it's useful to show examples of how an app would call the new code. Please add here - or to the pull request - some speculative code on various ways "Contoso Calling" might want to use this feature.

[anupriya13]

User Story: Enhanced UI for Calling Notifications

As a user of "Contoso Calling" I want to receive clear and interactive notifications for both voice and video calls so that I can easily manage calls without needing to open the app, ensuring I can respond promptly and have control over my devices.

Feature Overview: The Enhanced UI for Calling Notifications introduces a series of design improvements aimed at making video call notifications more interactive, user-friendly, and informative. These updates are designed to increase the likelihood of users responding quickly to incoming calls by providing them with useful features directly within the notification.

Interactive Elements:

Live Camera Feed Preview: When receiving a video call, users will see a live preview of their camera feed directly within the notification. This helps users feel more prepared and confident for the video call, allowing them to adjust their appearance or environment before answering, reducing the risk of accidental or embarrassing situations.

Device Management: Integrated device controls in the notification enable users to easily switch between audio and video devices. For example, they can choose between different cameras, microphones, or speakers directly from the notification, improving flexibility when answering calls.

User Benefits:

Convenience: By providing essential call controls within the notification, users can manage their calls without the need to open the app, making the entire process more streamlined and user-friendly.

Preparedness: With the live camera feed preview, users can ensure they are ready for video calls before accepting, which minimizes awkward or unprepared moments.

Improved Responsiveness: The clear layout and easy-to-access buttons allow users to answer, reject, or manage their calls more quickly, improving overall responsiveness and efficiency during incoming calls.

Acceptance Criteria:

When receiving a video call, the notification should show a live camera preview.

The notification should allow users to select different audio/video devices from a dropdown menu.

The notification should present buttons for accepting, rejecting, or interacting with the call.

Users should be able to manage call-related actions without needing to open the app.

This enhancement empowers users by offering more control over their calls, ensuring they can respond effectively and efficiently without needing to leave the notification interface.

[JanRajnoha]

I think this is great idea and I really love it, but I have a few qustions:

What about Notification center when I did not interact with notification and it will hide there?
Will be camera preview switched on even if Notification center (with that notification) is closed?
And If I open Notification center, will notification with preview collapsed and preview switched off?

And what about I get notification during another video call (for example Teams and WhatsApp)?

I have set in Teams custom background, will it be available in notification or I will have only raw camera preview?

What about privacy? Will be available interfaec for hiding preview if I don't want it?

[anupriya13]

@JanRajnoha Thanks for the inputs!

What about Notification center when I did not interact with notification and it will hide there? Will be camera preview switched on even if Notification center (with that notification) is closed? And If I open Notification center, will notification with preview collapsed and preview switched off?

For Notification Center, we would avoid the camera preview box to be visible at all and camera shouldn't be launch then. It should be only during the toast pop-up on bottom right. However, if there's any change we will update here since we are in very initial stage of this feature.

And what about I get notification during another video call (for example Teams and WhatsApp)?

This will be handled on the UI if another app is using the camera and user will be notified about that on the toast pop-up.

I have set in Teams custom background; will it be available in notification or I will have only raw camera preview?

For now, we are starting with a very basic experimental feature displaying only simple camera preview and devices control option. However, in future release versions we would definitely plan to include backgrounds, filters, and more options. :)

What about privacy? Will be available interface for hiding preview if I don't want it?

Yes, user will be provided option to enable/disable camera permissions for notifications as well. If both notifications and app have the permission enabled in settings only then the camera preview would be displayed.

We are still working on the final design and will update more soon once finalised.

[JanRajnoha]

@anupriya13 Thank you for your quick answers. I know this is just experimental, but sometimes is good to think about future. Sometimes it could help with better architecture of whole thing.

To question about Notification center and hide camera preview: Right now, I don't know what Windows supports with Notification center, but do you know how to hide/switch off camera preview in Notification center? Because how I said, this is great idea, but i'm afraid there will be needed support from Windows team

[anupriya13]

@JanRajnoha Agreed, we would work on Windows OS level changes as well for this experiment.

[anupriya13]

@jonwis Please review:

Existing Toast Schema: Toast schema - Windows UWP applications | Microsoft Learn

1 new XML tag to be introduced: < cameraPreview / >

• It will be under <visual> and will display only the camera preview
• If API is not called then system default to be displayed
• If camera load fails then it should not be displayed
• It should be linked to below new property in case user changes the device id it should display the preview of updated camera device id.
• Min Occurance = 0
• Max Occurance = 1
• Properties: None

1 new property to be introduced in existing <action/> tag: settingType="VideDevices" and settingType="AudioDevices"

• It is under <actions> and will include settings button
• If no devices available then it should not be visible
• Selected device ids will be the ids received from API if API is not invoked then system default values will be selected
• Existing Properties: https://learn.microsoft.com/en-us/uwp/schemas/tiles/toastschema/element-action
• Actions: https://learn.microsoft.com/en-us/uwp/schemas/tiles/toastschema/element-actions
• Additional new Properties:

  • settingType: optional, string

    When set to "videoDevices" or “audioDevices” the action becomes a setting action added to the toast notification with drop down options on devices list rather than a traditional toast button.
    Devices list includes either [camera, microphone and speakers] or [microphone and speakers].
    settingType = “videoDevices” will show camera, mic and speaker list
    settingType= “audioDevices” will show mic and speaker list
    If placement="contextMenu" is also present, then this button won't be visible and will be ignored

Example of sending Video Call Notification using Toast XML (Contoso Calling App initiates a video call from user Jill Bender to user B sharing User B default App's device ids on the API):

void SendVideoCallNotification()  
{  
    winrt::hstring payload = LR"( 
<toast scenario="incomingCall" useButtonStyle="true">  
    <visual>  
        <binding template="ToastGeneric">  
            <text hint-maxLines="1">Jill Bender</text>  
            <text hint-maxLines="1">Incoming Video Call</text>  
           <cameraPreview/>  // NEW TAG!!
      </binding> 
    </visual> 
    <actions>  
        <action  
          content=""  
          imageUri="Assets/Icons/Setting.png"  
          settingType="videoDevices”   // NEW PROPERTY!!!
         arguments=""/>   
        <action  
          content=""  
          imageUri="Assets/Icons/Accept.png"  
         hint-buttonStyle="Success"  
          activationType="background"  
          arguments="action=reply&amp;threadId=92187"/>  
        <action  
          content=""  
          imageUri="Assets/Icons/Decline.png"  
          activationType="background"  
          hint-buttonStyle="Critical"  
          arguments="action=reply&amp;threadId=92187"/>  
         <action  
          content=""  
          imageUri="Assets/Icons/Message.png"  
          activationType="background"  
          arguments="action=reply&amp;threadId=92187"/>  
    </actions>  
</toast> 
)"; 

    winrt::AppNotification notification(payload);  
 
    if(winrt::AppNotificationDevicesData::IsVideoOrAudioCallingSupported()) 
    {     
       // Assign Devices Data values for the video call notification [New APIs Below]
       winrt::AppNotificationDevicesData devicesData; 

       devicesData.VideoDeviceId(L"\\?\USB#VID_045E&PID_0990&MI_00#6&db32c28&0&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\GLOBAL");  
 
       devicesData.AudioInputDeviceId(L"\\?\SWD#MMDEVAPI#{0.0.1.00000000}.{a19be0b4-e6e9-404b-b3ae-e98dc182e850}#{2eef81be-33fa-4800-9670-1cd474972c3f}");  
 
       devicesData.AudioOutputDeviceId(L"\\?\SWD#MMDEVAPI#{0.0.1.00000000}.{a19be0b4-e6e9-404b-b3ae-e98dc182e850}#{2eef81be-33fa-4800-9670-1cd474972c3f}");  

       notification.DevicesData(devicesData); 
    } 
    winrt::AppNotificationManager::Default().Show(notification);  
} 

Example of sending Video Call Notification using AppNotificationBuilder (Contoso Calling App initiates a video call from user Jill Bender to user B sharing User B default App's device ids on the API):

void SendVideoCallNotification()  
{  
winrt::AppNotification notification =  
AppNotificationBuilder() 
    .SetScenario(AppNotificationScenario::IncomingCall) 
    .AddText(L"Jill Bender", AppNotificationTextProperties().SetMaxLines(1)) 
    .AddText(L"Incoming Video Call", AppNotificationTextProperties().SetMaxLines(1)) 
    .AddCameraPreview() // New API
    .AddButton(AppNotificationButton() 
        .SetIcon(winrt::Windows::Foundation::Uri(LR"(ms-appx://Assets/Icons/Setting.png)")) 
        .SetSettingType(AppNotificationSettingTypeButton::Devices)) // New API
    .AddButton(AppNotificationButton() 
        .AddArgument(L"action", L"acceptCall") 
        .AddArgument(L"threadId", L"92187") 
        .SetIcon(winrt::Windows::Foundation::Uri(LR"(ms-appx://Assets/Icons/Accept.png)")) 
        .SetButtonStyle(AppNotificationButtonStyle::Success)) 
    .AddButton(AppNotificationButton() 
        .AddArgument(L"action", L"declineCall") 
        .AddArgument(L"threadId", L"92187") 
        .SetIcon(winrt::Windows::Foundation::Uri(LR"(ms-appx://Assets/Icons/Decline.png)")) 
        .SetButtonStyle(AppNotificationButtonStyle::Critical)) 
    .AddButton(AppNotificationButton() 
        .AddArgument(L"action", L"message") 
        .AddArgument(L"threadId", L"92187") 
        .SetIcon(winrt::Windows::Foundation::Uri(LR"(ms-appx://Assets/Icons/Message.png)"))) 
    .BuildNotification(); 
    if(winrt::AppNotificationDevicesData::IsVideoOrAudioCallingSupported()) 
    {     
       // Assign Devices Data values for the video call notification  
       winrt::AppNotificationDevicesData devicesData; 

       devicesData.VideoDeviceId(L"\\?\USB#VID_045E&PID_0990&MI_00#6&db32c28&0&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\GLOBAL");  
 
       devicesData.AudioInputDeviceId(L"\\?\SWD#MMDEVAPI#{0.0.1.00000000}.{a19be0b4-e6e9-404b-b3ae-e98dc182e850}#{2eef81be-33fa-4800-9670-1cd474972c3f}");  
 
       devicesData.AudioOutputDeviceId(L"\\?\SWD#MMDEVAPI#{0.0.1.00000000}.{a19be0b4-e6e9-404b-b3ae-e98dc182e850}#{2eef81be-33fa-4800-9670-1cd474972c3f}");  

       notification.DevicesData(devicesData); 
    } 
    winrt::AppNotificationManager::Default().Show(notification);  
} 

Example of Invoke Callback Existing API with additional User Inputs key-values (Contoso Calling App will get the selected device ids from User B from below invoke API on any button clicked accept/ decline/ message):

//<toast >
//  <visual>
//      <binding template="ToastGeneric">
//          <text hint-maxLines="1">Jill Bender</text>
//      </binding>
//  </visual>
//  <actions>
//        <action  
//          content=""  
//          imageUri="Assets/Icons/Setting.png"  
//          settingType="VideoDevices”  // New Property
//          arguments=""/>   
 //       <action  
 //         content=""  
  //        imageUri="Assets/Icons/Accept.png"  
 //        hint-buttonStyle="Success"  
 //         activationType="background"  
 //         arguments="action=accept"/> 
//  </actions>
//</toast>

void ProcessNotificationArgs(const winrt::AppNotificationActivatedEventArgs& notificationActivatedEventArgs)
{
    // If the user clicks on a toast, the code will need to launch the chat thread window
    if (std::wstring(notificationActivatedEventArgs.Argument().c_str()).find(L"openThread") != std::wstring::npos)
    {
        GenerateChatThreadWindow();
    }
    else // If the user responds to a notification by clicking a accept button in the toast, we will need to initiate call to the other user
    if (std::wstring(notificationActivatedEventArgs.Argument().c_str()).find(L"accept") != std::wstring::npos)
    {
        auto input = notificationActivatedEventArgs.UserInput(); // Below 3 User Input key-values would be provided from OS 
        auto selectedCameraDeviceId  = input.Lookup(L"videoDeviceId");
        auto selectedMicrophoneDeviceId  = input.Lookup(L"audioInputDeviceId");
        auto selectedSpeakerDeviceId  = input.Lookup(L"audioOutputDeviceId");

        // Process the selected device ids and launch video / audio call 
        ActivateCallToUser(selectedCameraDeviceId  , selectedMicrophoneDeviceId  , selectedSpeakerDeviceId  );
    }
}