I am not sure this Mediapipe API to get image from packet can be work or not.

Hello, @eknight7 and @jiuqiant .

I want to retrieve the input video stream data packet and inference with my own tflite models.

My expected input is RGB image.

I am not sure what is problem for my first attempt code.

Thanks and regards!

Hi @momo1986,

addPacketCallback will only work on output streams as documented here.

What is the source of your images?

Note that FrameProcessor.java implements onNewFrame methods which are called (depending on the source) when a new input frame is available for processing. These methods make a corresponding packet wrapping that frame and send it to the MediaPipe graph for processing.

But I am unsure why you need a PacketGetter call for the RGB bitmap, so let's step back and understand your question:

I want to inference the hand pose with Mediapipe model and my own model.
I have my own tf-lite models, it can work on the RGB bitmap.
I try to query the RGB bitmap from input frame with data packet.

You have a model that can run inference on RGB bitmap images, and it seems you want to convert camera frame images into RGB bitmap images. Did I understand that correctly?

Note that Bitmap images are processed on the CPU, see FrameProcessor.onNewFrame(Bitmap, long) for details.

Here is an idea you could try:

1. Convert the TextureFrame into a Packet containing GpuBuffer.
2. Send that packet to the graph, the graph should have a node, GpuBufferToImageFrame that converts the GpuBuffer into an ImageFrame object on the CPU. Let the output stream of this node in your graph be input_video_cpu.
3. Add a addPacketCallback method like above to graph frames from the input_video_cpu, convert them into Bitmap objects, then send them to the graph via FrameProcessor.onNewFrame(Bitmap, long).

Hello, @eknight7 .

Thanks for your proposal.

I will try it.

You are correct, my tf-lite model is on CPU.

Maybe new graph is needed.

Thanks for your suggestion.

I wish you are all good.

Regards!
Momo

Hello, @momo1986 .

Did you figure out this issue? I also face this question.

Hello, @eknight7,

I followed these steps you suggested.

1. Convert the TextureFrame into a Packet containing GpuBuffer.
2. Send that packet to the graph, the graph should have a node, GpuBufferToImageFrame that converts the GpuBuffer into an ImageFrame object on the CPU. Let the output stream of this node in your graph be input_video_cpu.
3. Add a addPacketCallback method like above to graph frames from the input_video_cpu, convert them into Bitmap objects, then send them to the graph via FrameProcessor.onNewFrame(Bitmap, long).

But it threw this exception:

native: E0727 09:14:42.380656 27956 graph.cc:407] ; Input Stream "input_video_cpu" for node with sorted index 1 does not have a corresponding output stream.

Is there anything i need to do else?

much appreciated your help.