Issue with appending SampleBuffer

[abhinandan-chakraborty]

Hello, I am trying to append modified frames into stream (v2.0.0):

func captureOutput(
  _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer,
  from connection: AVCaptureConnection
) {
  guard let pixelBufferWide = CMSampleBufferGetImageBuffer(sampleBuffer), let stream = stream else {
    return
  }
  if isPublishingStream {
    Task {
      //modify the pixelBuffer to CIImage and create a new sample buffer using same `CMSampleTimingInfo` as the original SampleBuffer
      await stream.append(sampleBuffer)  //<-- append to the stream
    }
  }
}

this works great but after sometime, I start getting this warning error continously:

2024-16-11 16:44:57.439 [Warn] [com.haishinkit.HaishinKit] [RTMPStream.swift:753] append(_:) > invalidSequence
2024-16-11 16:45:13.054 [Warn] [com.haishinkit.HaishinKit] [RTMPStream.swift:753] append(_:) > invalidSequence
2024-16-11 16:45:25.102 [Warn] [com.haishinkit.HaishinKit] [RTMPStream.swift:753] append(_:) > invalidSequence
2024-16-11 16:45:36.273 [Warn] [com.haishinkit.HaishinKit] [RTMPStream.swift:753] append(_:) > invalidSequence
2024-16-11 16:45:37.091 [Warn] [com.haishinkit.HaishinKit] [RTMPStream.swift:753] append(_:) > invalidSequence
2024-16-11 16:45:48.130 [Warn] [com.haishinkit.HaishinKit] [RTMPStream.swift:753] append(_:) > invalidSequence

which is from this function

HaishinKit.swift/HaishinKit/Sources/RTMP/RTMPStream.swift

Line 738 in b03a9da

public func append(_ sampleBuffer: CMSampleBuffer) {

In my backend I am running ffmpeg to convert incoming RTMP stream to HLS stops writing whenever this warning starts to appear:

ffmpeg -i rtsp://127.0.0.1:xxxx/stream/stream_id -hls_time 1 -hls_list_size 5 -hls_flags delete_segments+independent_segments -hls_segment_type fmp4 -f hls -c:v libx264 -c:a aac -preset ultrafast master.m3u8

I don't know what's going wrong, it happens anytime after 5-10 minutes of running the stream. I also tried with new CMSampleTimingInfo but then ffmpeg is unable to write file in correct timestamp:

// Get the current timestamp
        let currentTime = CMClockGetTime(CMClockGetHostTimeClock())

        // Create CMTime instances
        let duration = CMTimeMake(value: 0, timescale: 0)
        let presentationTimeStamp = CMTimeMake(value: currentTime.value, timescale: currentTime.timescale)
        let decodeTimeStamp = CMTimeMake(value: 0, timescale: 0)

        // Create CMSampleTimingInfo instance
        let sampleTimingInfo = CMSampleTimingInfo(
            duration: duration,
            presentationTimeStamp: presentationTimeStamp,
            decodeTimeStamp: decodeTimeStamp
        )
        
        if !CMTIME_IS_VALID(timingInfo.presentationTimeStamp) {
            print("invalid pts")
            return
        }

[shogo4405]

The appendSampleBuffer method is designed under the assumption that it processes output from MediaMixer. I can't determine its behavior in the current use case.

Either solve it yourself or give up!

[abhinandan-chakraborty]

I see, is there a way to append sampleBuffer (either from cameraOutput or Reading from file) and push it to server directly without integrating with MediaMixer ? For example:

private var stream: RTMPStream
private let connection = RTMPConnection()

//later in init:
stream = RTMPStream(connection: connection)
await stream?.setVideoSettings(videoSettings)

//start stream somewhere:
let _ = try await connection.connect("rtmp_url")
let _ = try await stream.publish("stream_id")

//append to stream from either cameraOutput or AVAssetReaderTrackOutput
Task {
  await stream.append(sampleBuffer)
}

[shogo4405]

It is not prepared.

[abhinandan-chakraborty]

Thank you, It was an issue with how CMSampleTimingInfo was created and not with HaishinKit or FFmpeg.