Frame Skipping Causes Label Misalignment in Video Annotation

[Buckler89]

Description

I created a video with OpenCV at 10 FPS and labeled its first 10 frames in Label Studio. When I refresh the page and open the task, the label aligns perfectly on the first frame. However, when I advance to the second frame, the video quickly skips the second frame and displays the third frame, but still shows the bounding box intended for the second frame (see the example below).

To reproduce:

Configuration

UI Labeling Configuration::

<View>
   <Labels name="videoLabels" toName="video" allowEmpty="true">
     <Label value="Man" background="blue"/>
     <Label value="Woman" background="red"/>
     <Label value="Other" background="green"/>
   </Labels>
   
   <!-- Please specify FPS carefully, it will be used for all project videos -->
   <Video name="video" value="$video" framerate="$fps"/>
   <VideoRectangle name="box" toName="video"/>
</View>

Task with Annotation

{
  "id": 15,
  "data": {
    "video": "/data/upload/1/2d96aee1-20241025-145032-937345_day.mp4",
    "fps": 10
  },
  "annotations": [
    {
      "id": 4,
      "result": [
        {
          "value": {
            "framesCount": 43,
            "duration": 4.3,
            "sequence": [
              {"enabled": true, "frame": 1, "height": 1.39, "rotation": 0, "time": 0.1, "width": 3.41, "x": 51.42, "y": 50.35},
              {"enabled": true, "frame": 2, "height": 1.39, "rotation": 0, "time": 0.2, "width": 3.27, "x": 50.43, "y": 50.35},
              {"enabled": true, "frame": 3, "height": 1.56, "rotation": 0, "time": 0.3, "width": 3.41, "x": 49.57, "y": 50.17},
              {"enabled": true, "frame": 4, "height": 1.56, "rotation": 0, "time": 0.4, "width": 3.41, "x": 48.58, "y": 50.17},
              {"enabled": true, "frame": 5, "height": 1.39, "rotation": 0, "time": 0.5, "width": 3.27, "x": 47.73, "y": 50.17},
              {"enabled": true, "frame": 6, "height": 1.39, "rotation": 0, "time": 0.6, "width": 3.27, "x": 46.88, "y": 50.17},
              {"enabled": true, "frame": 7, "height": 1.56, "rotation": 0, "time": 0.7, "width": 3.27, "x": 45.88, "y": 50},
              {"enabled": true, "frame": 8, "height": 1.56, "rotation": 0, "time": 0.8, "width": 3.41, "x": 45.03, "y": 50},
              {"enabled": true, "frame": 9, "height": 1.56, "rotation": 0, "time": 0.9, "width": 3.27, "x": 44.18, "y": 50},
              {"enabled": true, "frame": 10, "height": 1.56, "rotation": 0, "time": 1, "width": 3.27, "x": 43.18, "y": 50}
            ]
          },
          "id": "xNNNVLwq5Y",
          "from_name": "box",
          "to_name": "video",
          "type": "videorectangle",
          "origin": "manual"
        }
      ],
      "created_username": "di@mail.com",
      "created_ago": "3 hours, 56 minutes",
      "completed_by": {
        "id": 1,
        "email": "di@mail.com",
        "initials": "di"
      },
      "was_cancelled": false,
      "ground_truth": false,
      "created_at": "2024-11-01T18:05:06.388864Z",
      "updated_at": "2024-11-01T18:45:51.407308Z",
      "lead_time": 2214.572,
      "task": 15,
      "project": 1
    }
  ],
  "predictions": []
}

Expected Behavior:

Navigating the video back and forth, the labels should always remain aligned with the images.

Actual Behavior

1. After refreshing the page on the first frame: The label aligns correctly with the displayed image.
   

2. When advancing to the second frame: The second frame is quickly skipped, and the third frame is displayed, but it shows the label intended for the second frame. Note that the timeline is poinintg on the 2nd frame here
   

3. When returning to the first frame: The video displays the second frame, but with the label from the first frame. From this point onward, labels are consistently one frame behind the displayed image, regardless of the selected frame. This misalignment continues if I advance further, and the actual first frame is never shown again until I refresh the page.

After this misalignment starts, the labels stay one frame behind the images until I refresh the page. However, after refreshing, the issue recurs as soon as I advance by one frame, following the same pattern.

The root of the problem seems to be in the video display, which skips a frame as described, while the label display remains correctly sequenced.

Video used

2d96aee1-20241025-145032-937345_day.mp4

Summary

This appears to be a bug that causes video frames and labels to become misaligned, making accurate video labeling challenging. This misalignment significantly impacts labeling precision, so addressing it would improve usability and reliability for video annotation tasks.

Environment (please complete the following information):

• OS: ubuntu 22.04
• Label Studio Version: docker 1.13.1

[Buckler89]

I believe I have found a similar, though slightly different, issue. Based on the comment by @hlomzik in issue #3315, where it is stated that:

Video is played well anyway; the only thing affected is the timeline. We detect the current frame as (time passed / given frame rate).

I noticed that in my Label Studio instance, the computed video length is incorrect, as shown in the screenshot.

Details:

• Video FPS: 10
• Total Frames: 43

Given these details, the total length should be 4.3 seconds (since ( \frac{43 \text{ frames}}{10 \text{ FPS}} = 4.3 \text{ seconds} )). However, Label Studio displays the total length as 4.02 seconds (which I suppose might actually mean 4.2 seconds, but that's another issue).

In my opinion, this could be related to the problem of frame skipping due to time approximation in the calculations mentioned earlier. This is just an idea.

Could someone please confirm if this is a known issue or if there's a recommended workaround? Any assistance or guidance would be greatly appreciated.

[Buckler89]

To help reproduce the issue, I created a dummy video with a frame number overlay. Here’s the video that you can use to reproduce the issue. The frame count starts at 1 and goes up to 15, for a total of 15 frames. The video has an FPS of 10.

36ab8fd2-20241104-100249-868294_day.mp4

The media player UI should always match the frame counter displayed in the video. Start from the first frame and proceed one frame at a time. Moving back and forth frame by frame will clearly show the misalignment with the video timeline. Refresh the page to ensure the frame counter starts from 1 again.

Unfortunately, I do not have the ability to investigate the frontend code on my own at the moment. Any help in resolving this issue would be greatly appreciated, as this is a blocking problem for me: I have many short videos that need to be labeled frame by frame.

I’m confident this is a bug in the video media player.

[mandulaj]

I agree, we are running into the same issue. We have to manually offset the frames in post-processing. Look like a typical off-by-one error.

[Buckler89]

I'm afraid it's not quite as simple. Please keep in mind that when you try to label a frame, that label will be assigned to the previous frame, and there’s no certainty that the label will remain accurate if you shift it afterward. Objects don’t always move at a predictable speed between frames, so shifting labels can result in inaccuracies.

To maintain precision, it’s important that video playback and labeling tools remain synchronized to avoid any frame skipping during annotation. Using frame-by-frame navigation and confirming that each label aligns with the intended frame would help. Additionally, interpolation methods might be necessary for cases where objects don’t move consistently between frames, enhancing annotation reliability.

[mandulaj]

I think the issues seems to be related to this line where the frame number is calculated based on the currentTime and framerate:

label-studio/web/libs/editor/src/components/VideoCanvas/VideoCanvas.tsx

Line 170 in 4a6cefe

const frameNumber = Math.round(currentTime * framerate);

Current time is starting at 0 yet the frame indexes start at 1. And I am also a little bit uncertain with the rounding. I tried to do a simple:
const frameNumber = Math.round(currentTime * framerate) + 1; which seems to cause the displayed frames to align with the frame number, yet near the start and end there is still a discrepancy plus the frame-by-frame skip buttons are now very buggy! They sometimes skip one frame, sometimes multiple. I am not very familiar with the React code base but I have a strong feeling that the issue is around here. Frames being indexed starting at 1 but time starting at 0.

[Buckler89]

@mandulaj Did you manage to resolve the issue?
I also posted this issue on the Slack channel. Here’s the link. If you’d like, you can raise it there as well to help reinforce the request for a fix.

I’d like to try solving it, but I’m currently stuck because I’m experiencing the same problem that many others have reported #6567.
Could you also share how you set up the development environment?

[mandulaj]

@Buckler89 Unfortunately not 😢 I have been building the docker environment with the modified line and it seems to fix the visual issue, however now the timeline controls are broken and I don't have time to figure out how all the React components interact together.

[heidi-humansignal]

Hello,

Sorry for the delay. We do have bug ticket to fix this issue. We will keep you updated as eng team investigate this.

Thank you,
Abu

Comment by Abubakar Saad
Workflow Run

[Buckler89]

Hi Thomas,

The error you're encountering might actually be expected behaviour. For instance, when you open a task, Label Studio sends a request to the label-studio-ml-backend with the available data. At that point, "drafts" might not yet exist. If it's the first time you're opening the task, there are no drafts yet.

Since SAM (Segment Anything Model) works with prompts (like boxes or points) as input, it can't function in this situation. The fact that it raises an error shouldn't be a concern. From what I understand, this is the intended behaviour—it was already happening before this PR. I've observed similar behaviour in other label-studio-ml-backend integrations, so I adapted to it. That said, it might be clearer if the system handled this situation differently. For now, if it raises an error, it simply means nothing happens.

To get the backend to work properly (if the error is due to what I think), you can safely ignore the error, draw a box, and assign it a label. At that point, everything should work as expected.

As for debugging, you need to run the _wsgi.py file in debug mode with the correct environment variables set.

Here's an example of my PyCharm configuration:

And here's an example of my .env file configuration:

DEVICE=cuda
LABEL_STUDIO_API_KEY=<your-api-key>
LABEL_STUDIO_URL=http://0.0.0.0:8080
MAX_FRAMES_TO_TRACK=60
MODEL_CHECKPOINT=sam2.1_hiera_tiny.pt
MODEL_CONFIG=./configs/sam2.1/sam2.1_hiera_t.yaml
SEGMENT_ANYTHING_2_REPO_PATH=<path-to-sam2-repo>/sam2/segment-anything-2/
DEBUG=true

I hope this helps! Let me know if you need further clarification.