Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MPS restarts due to IDER session failure #1560

Open
a-fredcheng opened this issue Jul 26, 2024 · 7 comments
Open

MPS restarts due to IDER session failure #1560

a-fredcheng opened this issue Jul 26, 2024 · 7 comments

Comments

@a-fredcheng
Copy link

a-fredcheng commented Jul 26, 2024

Describe the bug 🪲
I made a KVM connection to a connected device, and then clicked the IDE-Redirection function, but the upload of the .iso file failed, KVM would interrupt the connection, and then the Dashboard and Devices list could not be displayed normally. Finally, MPS It will restart, and after the device is reconnected again, the MPS will return to normal.

To Reproduce 🪜
Steps to reproduce the behavior:

  1. Go to the Devices list and click on the device
  2. Start the KVM connection and enter user consent
  3. Click attach image file and select the mounted iso file
  4. Confirm whether the volume on the device is mounted
  5. Volume does not appear and KVM connection is lost
  6. MPS Dashboard and Devices list functions are invalid
  7. MPS restarted
  8. MPS Dashboard and Devices list functions restored
  9. Device reconnected

Expected behavior
The expected behavior should be that a newly mounted disk volume will appear on the device for subsequent related operations.

Screenshots 🖼️

  1. Start KVM and click attach image file.
    ider_ui_1

  2. Volume does not appear and KVM connection is lost
    ider_ui_2_mps_crashing

  3. MPS Dashboard and Devices list functions are invalid
    mps_devices_list_request_failed

  4. MPS shows restarted 2 minutes ago
    mps_restart_after_crash

AMT Device (please complete the following information): 🖥️

  • OS: Windows 10
  • AMT Version: 14.0.47
  • AMT Configuration Mode: Client Control Mode / Admin Control Mode
  • Network Configuration Dynamic IP

Service Deployment (please complete the following information): ⛈️

  • Deployment Type: Docker
  • Node Version:
  • Component & Version: MPS v2.13.13

Additional context
Because the ider function is still being tested, the iso file used is the sample iso file used when the ider function was previously released.
If it is a local deployment, ider will succeed, but the error will occur in the virtual machine deployed on the cloud.

@a-fredcheng a-fredcheng changed the title MPS restarts due to IDER function failure MPS restarts due to IDER session failure Jul 26, 2024
@graikhel-intel
Copy link
Contributor

graikhel-intel commented Jul 26, 2024

I couldn't reproduce this issue using the latest mps image (July release v2.13.15) on my local dev machine. I deployed the Toolkit using docker.

Regarding "If it is a local deployment, ider will succeed, but the error will occur in the virtual machine deployed on the cloud."

Just so I understand the scenario right, MPS crashes only when the Toolkit is deployed using Docker on a Virtual Machine that is hosted on cloud? You do not see the issue when the Toolkit is deployed on a local machine using Docker.

@a-fredcheng
Copy link
Author

Just so I understand the scenario right, MPS crashes only when the Toolkit is deployed using Docker on a Virtual Machine that is hosted on cloud? You do not see the issue when the Toolkit is deployed on a local machine using Docker.

I am confused about this situation. Does IDE-Redirection only apply to local network environment? But KVM does not seem to have this problem.

@a-fredcheng a-fredcheng closed this as not planned Won't fix, can't repro, duplicate, stale Jul 29, 2024
@a-fredcheng a-fredcheng reopened this Jul 29, 2024
@bwendlandt-intel bwendlandt-intel moved this from Done to Backlog in Sprint Planning Aug 1, 2024
@bwendlandt-intel
Copy link
Contributor

Hi @a-fredcheng , can you give some more detail on how your environment is set up? Is it correct that you are running VMs in the cloud and then running Docker within the VM? Or are you running the services natively in the VMs?

IDE-R does not only apply to local network. IDE-R is just like KVM (and SOL) and will function in both local or cloud-based environments.

@a-fredcheng
Copy link
Author

a-fredcheng commented Aug 20, 2024

Hi @a-fredcheng , can you give some more detail on how your environment is set up? Is it correct that you are running VMs in the cloud and then running Docker within the VM? Or are you running the services natively in the VMs?

IDE-R does not only apply to local network. IDE-R is just like KVM (and SOL) and will function in both local or cloud-based environments.

I do execute a virtual machine in the cloud and then execute docker inside it.
Then lock the following component versions to the toolkit version in the v2.22 state through docker-compose.yaml.
webui: intel/oact-webui:v3.6.0
rps: intel/oact-rps:v2.22.6
mps: intel/oact-mps:v2.13.13
mpsrouter: intel/oact-mpsrouter:v2.3.6

@bwendlandt-intel
Copy link
Contributor

Just to clarify, KVM is working in this environment for this device?

Can I ask why you made the decision to run Docker within a VM on the cloud rather than just deploying the containers directly using Azure ACI/AWS ECS/Alibaba ECI/etc? Would love to know more. Usually, that ends up being more cost effective and removes an extra layer.

@a-fredcheng
Copy link
Author

Just to clarify, KVM is working in this environment for this device?

Can I ask why you made the decision to run Docker within a VM on the cloud rather than just deploying the containers directly using Azure ACI/AWS ECS/Alibaba ECI/etc? Would love to know more. Usually, that ends up being more cost effective and removes an extra layer.

Yes, the KVM functionality of this device will work in this build environment.
As for the reason why I chose not to implement the container solution of related cloud services, I am no longer very impressed. I need to ask other colleagues about this part.
However, when I first used open-amt-cloud-toolkit to build related services, I used Docker to build them in the local VM.
Afterwards, the VM deployed to the cloud does not change the original architecture.

@a-fredcheng
Copy link
Author

I tried the Ider function again earlier, and found this error message in the mps log. I wonder if it is useful in troubleshooting the issue.

2024-08-30T02:09:52.235Z error: uncaughtException: Cannot set properties of undefined (setting 'iderConnect')
TypeError: Cannot set properties of undefined (setting 'iderConnect')
    at WsRedirect.handleClose (file:///mps/dist/utils/wsRedirect.js:68:54)
    at callListener (/mps/node_modules/ws/lib/event-target.js:290:14)
    at WebSocket.onClose (/mps/node_modules/ws/lib/event-target.js:220:9)
    at WebSocket.emit (node:events:530:35)
    at WebSocket.emitClose (/mps/node_modules/ws/lib/websocket.js:265:10)
    at Socket.socketOnClose (/mps/node_modules/ws/lib/websocket.js:1289:15)
    at Socket.emit (node:events:518:28)
    at TCP.<anonymous> (node:net:337:12)
2024-08-30T02:09:52.237Z error: uncaughtException: Cannot set properties of undefined (setting 'kvmConnect')
TypeError: Cannot set properties of undefined (setting 'kvmConnect')
    at WsRedirect.handleClose (file:///mps/dist/utils/wsRedirect.js:65:53)
    at callListener (/mps/node_modules/ws/lib/event-target.js:290:14)
    at WebSocket.onClose (/mps/node_modules/ws/lib/event-target.js:220:9)
    at WebSocket.emit (node:events:530:35)
    at WebSocket.emitClose (/mps/node_modules/ws/lib/websocket.js:265:10)
    at Socket.socketOnClose (/mps/node_modules/ws/lib/websocket.js:1289:15)
    at Socket.emit (node:events:518:28)
    at TCP.<anonymous> (node:net:337:12)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Backlog
Development

No branches or pull requests

3 participants