🐛 Camera preview blank screen when using Skia Frame Processors

[zexueteh]

What's happening?

Hi @mrousavy, thank you for the great work with this library.

However, I have been having issues with applying filters using useSkiaFrameProcessors, even when using the boilerplate provided in the docs.
I've tried two implementations

1. My own app using a grain filter (right)
2. Example app with an inverted colour filter (left)

The camera preview ends up being just the filter, without any image. Please refer to the code and logs from both cases below.

I'm wondering whether this might be a hardware issue? I'm currently developing on an old Huawei P10 (released in 2017, Android 9). Ive had a friend run the app on a Samsung S23 Ultra and he did not encounter this issue.

Reproduceable Code

// RECIPICS Code
  const grainFilter = Skia.RuntimeEffect.Make(`
  uniform shader image;
  half4 main(vec2 pos) {
    vec4 color = image.eval(pos);
    float grain = fract(sin(dot(pos, vec2(12.9898, 78.233))) * 43758.5453);
    color.rgb += grain * ${color};
    return color;  }`)
  const shaderBuilder = Skia.RuntimeShaderBuilder(grainFilter)
  const imageFilter = Skia.ImageFilter.MakeRuntimeShader(shaderBuilder, null, null)
  const paint = Skia.Paint()
  paint.setImageFilter(imageFilter)
  const frameProcessor = useSkiaFrameProcessor((frame) => {
    'worklet'
    frame.render(paint)
  }, [paint])
        <Camera
          style={{
            flex: 1
          }}
          ref={camera}
          style={{ width: 400, height: 400 }}
          device={device}
          isActive={true}
          frameProcessor={frameProcessor}
          pixelFormat="yuv"
          photo={false}
          format={format}
        />
      }

// Example App Code
  const invertColorsFilter = Skia.RuntimeEffect.Make(`
  uniform shader image;
  half4 main(vec2 pos) {
    vec4 color = image.eval(pos);
    return vec4((1.0 - color).rgb, 1.0);
  }
`)
  const shaderBuilder = Skia.RuntimeShaderBuilder(invertColorsFilter);

  const imageFilter = Skia.ImageFilter.MakeRuntimeShader(shaderBuilder, null, null)
  const paint = Skia.Paint()
  paint.setImageFilter(imageFilter)

  const frameProcessor = useSkiaFrameProcessor((frame) => {
    'worklet'
    frame.render(paint)
  }, [paint])

Relevant log output

2024-06-04 17:21:01.877 30369-30668 CameraView              com.recipics                         I  invokeOnAverageFpsChanged(11.42263759086189)
2024-06-04 17:21:01.921 30369-30680 GRALLOC                 com.recipics                         I  LockFlexLayout: baseFormat: 11, yStride: 640, ySize: 307200, uOffset: 307200,  uStride: 640
2024-06-04 17:21:01.952 30369-30369 GLConsumer              com.recipics                         E  [SurfaceTexture-0-30369-2] checkAndUpdateEglState: invalid current EGLContext
2024-06-04 17:21:01.952 30369-30369 surfaceTexture          com.recipics                         E  Exception updateTexImage
                                                                                                    java.lang.IllegalStateException: Unable to update texture contents (see logcat for details)
                                                                                                    	at android.graphics.SurfaceTexture.nativeUpdateTexImage(Native Method)
                                                                                                    	at android.graphics.SurfaceTexture.updateTexImage(SurfaceTexture.java:249)
                                                                                                    	at com.shopify.reactnative.skia.PlatformContext.notifyDrawLoop(Native Method)
                                                                                                    	at com.shopify.reactnative.skia.PlatformContext.-$$Nest$mnotifyDrawLoop(Unknown Source:0)
                                                                                                    	at com.shopify.reactnative.skia.PlatformContext$1.doFrame(PlatformContext.java:61)
                                                                                                    	at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1090)
                                                                                                    	at android.view.Choreographer.doCallbacks(Choreographer.java:893)
                                                                                                    	at android.view.Choreographer.doFrame(Choreographer.java:809)
                                                                                                    	at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1078)
                                                                                                    	at android.os.Handler.handleCallback(Handler.java:907)
                                                                                                    	at android.os.Handler.dispatchMessage(Handler.java:105)
                                                                                                    	at android.os.Looper.loop(Looper.java:216)
                                                                                                    	at android.app.ActivityThread.main(ActivityThread.java:7625)
                                                                                                    	at java.lang.reflect.Method.invoke(Native Method)
                                                                                                    	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:524)
                                                                                                    	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:987)


2024-06-04 16:49:16.970  8524-8776  CameraView              com.mrousavy.camera.example          I  invokeOnAverageFpsChanged(4.060913705583756)
2024-06-04 16:49:17.091  8524-8863  GRALLOC                 com.mrousavy.camera.example          I  LockFlexLayout: baseFormat: 11, yStride: 3840, ySize: 8294400, uOffset: 8294400,  uStride: 3840
2024-06-04 16:49:17.123  8524-8609  <no-tag>                com.mrousavy.camera.example          E  [ZeroHung]zrhung_get_config: Get config failed for wp[0x0102]
2024-06-04 16:49:17.184  8524-8524  GLConsumer              com.mrousavy.camera.example          E  [SurfaceTexture-0-8524-1] checkAndUpdateEglState: invalid current EGLContext
2024-06-04 16:49:17.185  8524-8524  surfaceTexture          com.mrousavy.camera.example          E  Exception updateTexImage
                                                                                                    java.lang.IllegalStateException: Unable to update texture contents (see logcat for details)
                                                                                                    	at android.graphics.SurfaceTexture.nativeUpdateTexImage(Native Method)
                                                                                                    	at android.graphics.SurfaceTexture.updateTexImage(SurfaceTexture.java:249)
                                                                                                    	at com.shopify.reactnative.skia.PlatformContext.notifyDrawLoop(Native Method)
                                                                                                    	at com.shopify.reactnative.skia.PlatformContext.-$$Nest$mnotifyDrawLoop(Unknown Source:0)
                                                                                                    	at com.shopify.reactnative.skia.PlatformContext$1.doFrame(PlatformContext.java:61)
                                                                                                    	at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1090)
                                                                                                    	at android.view.Choreographer.doCallbacks(Choreographer.java:893)
                                                                                                    	at android.view.Choreographer.doFrame(Choreographer.java:809)
                                                                                                    	at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1078)
                                                                                                    	at android.os.Handler.handleCallback(Handler.java:907)
                                                                                                    	at android.os.Handler.dispatchMessage(Handler.java:105)
                                                                                                    	at android.os.Looper.loop(Looper.java:216)
                                                                                                    	at android.app.ActivityThread.main(ActivityThread.java:7625)
                                                                                                    	at java.lang.reflect.Method.invoke(Native Method)
                                                                                                    	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:524)
                                                                                                    	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:987)

Camera Device

[
  {
    "sensorOrientation": "landscape-left",
    "hardwareLevel": "limited",
    "maxZoom": 6,
    "minZoom": 1,
    "maxExposure": 4,
    "supportsLowLightBoost": false,
    "neutralZoom": 1,
    "physicalDevices": [
      "wide-angle-camera"
    ],
    "supportsFocus": true,
    "supportsRawCapture": false,
    "isMultiCam": false,
    "minFocusDistance": 10,
    "minExposure": -4,
    "name": "0 (BACK) androidx.camera.camera2",
    "hasFlash": true,
    "hasTorch": true,
    "position": "back",
    "id": "0"
  },
  {
    "sensorOrientation": "landscape-right",
    "hardwareLevel": "limited",
    "maxZoom": 6,
    "minZoom": 1,
    "maxExposure": 4,
    "supportsLowLightBoost": false,
    "neutralZoom": 1,
    "physicalDevices": [
      "wide-angle-camera"
    ],
    "supportsFocus": true,
    "supportsRawCapture": false,
    "isMultiCam": false,
    "minFocusDistance": 0,
    "minExposure": -4,
    "name": "1 (FRONT) androidx.camera.camera2",
    "hasFlash": false,
    "hasTorch": false,
    "position": "front",
    "id": "1"
  }
]

Device

Huawei P10 (Android 9.0)

VisionCamera Version

4.0.5

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

• I am using Expo
• I have enabled Frame Processors (react-native-worklets-core)
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[mrousavy]

Can you try setting your pixelFormat to rgb or yuv to see if there's a difference?

[zexueteh]

Trying in the Example App pixelFormat to yuv produces similar results

however rgb turns the filter grey, as seen here

[mrousavy]

Does it render if you do not pass a paint to the render(..) function?

[zexueteh]

Yup, it does render, but theres definitely some overhead due to the frameprocessor.

[mrousavy]

Then it surely is the paint (or the shader) that's wrong. I think this is more of a question than a VisionCamera bug report.

Overhead; yea I can see that it only runs at 3 FPS on this Android phone - is that an old phone? @wcandillon I think this is something we might need to investigate - it's either the makeNonTextureImage(), or our makeImageFromPlatformBuffer(...) function that's really slow here... 🤔

[wcandillon]

I would like to check the performance of the makeImageFromPlatformBuffer function on Android. I would be interesting to know about the phone being used here as well as parameters that would allow me to reproduce the issue (e.g a video file that has the same properties).

[zexueteh]

Hi @mrousavy I dont think its an issue with the paint or shader, as

1. the Example App uses the inverted colour shader from the docs
2. These shaders work well on newer hardware (Samsung S23 Ultra)

Granted, the device I'm testing on is very old (Huawei P10 was released 2017). I was wondering if it could be due to these 2 runtime errors as seen in the logs, and would there be any ways to catch them.
2024-06-04 16:49:17.184 8524-8524 GLConsumer com.mrousavy.camera.example E [SurfaceTexture-0-8524-1] checkAndUpdateEglState: invalid current EGLContext

2024-06-04 16:49:17.185 8524-8524 surfaceTexture com.mrousavy.camera.example E Exception updateTexImage java.lang.IllegalStateException: Unable to update texture contents (see logcat for details)

@wcandillon What are some information specifically that i could provide about the phone? Also, how would i be able to check the performance of makeImageFromPlatformBuffer?

[mrousavy]

Ah interesting - i didnt read those logs my bad. It looks like an error, yep. Not sure if it's RN Skia related or VisionCamera related tho

[mrousavy]

@wcandillon there's quite a few places in the codebase where RN Skia just swallows errors and logs them to the console (and returns null, or doesnt do anything) - I think throwing an Error (which will be propagated to JS) is better, especially in this case here.

[lucksp]

I too get a blank screen, maybe different from OP...when I start my Android device - after running a full build on EAS - I get an error on my device:

Invariant Violation: TurboModuleRegistry.getEnforcing(...): 'RNSkiaModule' could not be found. Verify that a module by this name is registered in the native binary.Bridgeless mode: false. TurboModule interop: false. Modules loaded: {"NativeModules":["PlatformConstants","LogBox","SourceCode","Timing","AppState","BlobModule","WebSocketModule","DevSettings","DevToolsSettingsManager","Networking","Appearance","DevLoadingView","HeadlessJsTaskSupport","DeviceInfo","UIManager","DeviceEventManager","RNCSafeAreaContext","IntentAndroid","NativeAnimatedModule","SoundManager","I18nManager","ImageLoader","RNCAsyncStorage","StatusBarManager","StatusBarManager"],"TurboModules":[],"NotFound":["NativePerformanceCxx","NativePerformanceObserverCxx","RedBox","BugReporting","LinkingManager","PlatformLocalStorage","RNC_AsyncSQLiteDBStorage","FrameRateLogger","KeyboardObserver","ModalManager","Worklets","RNSkiaModule"]}, js engine: hermes

I thought to try my emulator and it complains about minSdkVersion on Android needing to be on 26.

All I get is a white screen for my camera view.

    "react-native": "0.73.6",
    "react-native-vision-camera": "^4.0.1",
    "react-native-worklets-core": "^1.3.0",
    "react-native-fast-tflite": "^1.2.0",
    "vision-camera-resize-plugin": "^3.1.0"

I updated all the packages related to camera/frameprocessor/worklets, and now camera is working.

EDIT: Actually, I am unable to use useSkiaFrameProcessor on my Android device.

ERROR [session/recoverable-error: An unknown error occurred while creating the Camera Session, but the Camera can recover from it.]

[meleffendi]

I'm having a somewhat similar issue on OnePlus5 on 4.0.1. In my case the frame is partially displayed

The camera preview is displayed correctly without any issues if I'm using useFrameProcessor:

but once I switch to useSkiaFrameProcessor I get this: (notice the white space below and the L shaped black space