Cannot get Audio in recordings or live

[patrickacollier]

Describe the problem you are having

Cameras are Dahua cameras (shop & pool), stream is set at the camera to MP4 264 with AAC audio 16khz but I can't seem to get the audio in the live or recordings. I've changed configs a few times so please help with any recommendations.

Version

0.13.2-6476F8A

Frigate config file

mqtt:
  host: homeassistant.local
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: mqtt
  password: password
  stats_interval: 60
birdseye:
  enabled: True
  mode: motion
  inactivity_threshold: 150
##Hardware accelleration 
ffmpeg:
  input_args:
    - -avoid_negative_ts
    - make_zero
  output_args:
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
  hwaccel_args: preset-intel-qsv-h264
##Default recording parameters
record:
  enabled: True
  retain:
    days: 14
    mode: motion
  events:
    retain:
      default: 30
      mode: active_objects
      objects:
        dog: 2
        car: 14

go2rtc:
    streams:
      reolink: 
        - "ffmpeg:https://192.168.68.78/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=HA&password=password#video=copy#audio=copy#audio=opus"
        - "rtsp://username:[email protected]:554/h264Preview_01_main"
      reolink2: rtsp://username:[email protected]:554/h264Preview_01_sub
      RiataFront: rtsp://ddns:37777/cam/realmonitor?channel=1&subtype=1
      pool: 
        - "rtsp://username:[email protected]:554/cam/realmonitor?channel=1&subtype=2"
        - "ffmpeg:pool#video=copy#audio=copy"
      shop: 
        - "rtsp://username:[email protected]:554/cam/realmonitor?channel=1&subtype=2"
        - "ffmpeg:shop#video=copy#audio=copy"     
detect:
  enabled: True
  width: 640
  height: 480
  fps: 5
motion:
  # Optional: The threshold passed to cv2.threshold to determine if a pixel is different enough to be counted as motion. (default: shown below)
  # Increasing this value will make motion detection less sensitive and decreasing it will make motion detection more sensitive.
  # The value should be between 1 and 255.
  threshold: 40

snapshots:
  enabled: True
  retain:
    default: 30
    
cameras:
  reolink:
    detect:
      width: 1280
      height: 720
      fps: 5
    enabled: true
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/reolink?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/reolink2
          input_args: preset-rtsp-restream
          roles:
            - detect
    live:
      stream_name: reolink
  pool:
    enabled: true
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/pool?video=copy&audio=copy
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
      output_args:
        record: preset-record-generic-audio-copy
    live:
      stream_name: pool
  shop:
    enabled: true
    detect:
      width: 1280
      height: 720
      fps: 5
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/shop?video=copy&audio=copy
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
      output_args:
        record: preset-record-generic-audio-copy            
    live:
      stream_name: shop        
  RiataFront:
    enabled: true
    birdseye:
      enabled: False
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/RiataFront
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
    live:
      stream_name: RiataFront
    motion:
      mask:
        - 142,219,0,480,0,149
    zones:
      Entrance:
        coordinates: 208,97,540,196,640,333,483,398,160,385,73,139
        objects:
          - person
          - car
          - bus
    objects:
      track:
        - person
        - car
        - bus
      filters:
        person:
          # Optional: minimum width*height of the bounding box for the detected object (default: 0)
          min_area: 8000
          # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
          #max_area: 100000
          mask:
            - 142,219,0,480,0,149
            - 640,0,640,184,413,153,412,0
        car:
          # Optional: minimum width*height of the bounding box for the detected object (default: 0)
          min_area: 50000
          # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
          #max_area: 200000
          threshold: .5
          min_score: .4

          mask:
            - 142,219,0,480,0,149
        bus:
          # Optional: minimum width*height of the bounding box for the detected object (default: 0)
          min_area: 100000
          min_score: 0.1
          threshold: 0.1
          # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
          #max_area: 200000
          mask:
            - 142,219,0,480,0,149              
            
detectors: # <-------- Add this section to specify the detectors you want to use
  coral: # <---------- Name your detector (you can choose any name)
    type: edgetpu # <-- This specifies that you want to use the Coral USB Accelerator
    device: usb # <-- This tells Frigate to use the USB connected Coral device

Relevant log output

no logs

Frigate stats

unsure where this is https://hass.pacollier.com/ccab4aaf_frigate/api/stats did

Operating system

HassOS

Install method

HassOS Addon

Object Detector

Coral

Any other information that may be helpful

No response

[NickM-27]

there are a number of issues with the config here

1. manual input_args, in most cases a preset should be used here.
2. manual recording args, presets should be used here as well.
3. the secondary stream listed for go2rtc copies both things which is redundant. You will want audio transcoding here. What is your camera current set to for audio encoding?

[patrickacollier]

@NickM-27 Thanks for looking. I'll try to remove everything custom and work backwards as I originally started with default presets. i think the "number of issues" is my using pieces from multiple other "resolved" issues but it creating other issue. the only reason i had the manual input-args was to try and resolve the mp4 timestamp error. The docs don't specifically call out what to actually set configs to, and requires some deeper understanding of the individual components (FFMPEG, go2rtc etc)

with go2rtc i just want to limit network traffic, and I assumed I set this up correctly, but the docs don't go into any detail on h264/aac (standard rtsp cameras) so i'm left guessing.

        - "rtsp://username:[email protected]:554/cam/realmonitor?channel=1&subtype=2"
        - "ffmpeg:pool#video=copy#audio=copy" ```
     
Given the above you're saying I don't need the secondary stream "ffmpeg" as its redundant correct?

[patrickacollier]

although i'm not considering Frigate's WebRTC may only support opus vs the generic WebRTC supporting the 3. so that may just stop this discussion all together, I'll likely just transcode to opus. but now I'll need to determine the right encoding to limit bandwidth

[patrickacollier]

Here is the simplified config after removing the substream and using the presets.
This may have been the issue from the beginning. I do get audio on MSE but not on WebRTC (recordings work fine). I generally used WebRTC not even sure why, maybe it was low latency. But any reason why this would happen?

ffmpeg:
    hwaccel_args: preset-intel-qsv-h264

go2rtc:
    streams:
      pool: "rtsp://admin:[email protected]:554/cam/realmonitor?channel=1&subtype=2"
      shop: "rtsp://admin:[email protected]:554/cam/realmonitor?channel=1&subtype=2"       
detect:
  enabled: True
  width: 640
  height: 480
  fps: 5
motion:
  # Optional: The threshold passed to cv2.threshold to determine if a pixel is different enough to be counted as motion. (default: shown below)
  # Increasing this value will make motion detection less sensitive and decreasing it will make motion detection more sensitive.
  # The value should be between 1 and 255.
  threshold: 40

snapshots:
  enabled: True
  retain:
    default: 30
    
cameras:
  pool:
    detect:
      width: 1280
      height: 720
      fps: 5
    enabled: true
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/pool
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
            - snapshot
      output_args:
        record: preset-record-generic-audio-copy            
    live:
      stream_name: pool
  shop:
    enabled: true
    detect:
      width: 1280
      height: 720
      fps: 5
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/shop
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
      output_args:
        record: preset-record-generic-audio-copy               
    live:
      stream_name: shop

here is the FFPROBE (both cameras are the same)

Video:

Codec: H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
Resolution: 1280x720
FPS: 10/1

Audio:

Codec: AAC (Advanced Audio Coding)

[patrickacollier]

After determining the WebRTC issue, it appears WebRTC only supports specific audio encodings (Opus, G711, G722) the camera can do G711, G722, AAC would you recommend changing the audio to any of the G encodings or let FFMPEG transcode the audio to opus?

[hawkeye217]

Yep, if you want to use WebRTC, you need to transcode to opus. See the docs here: https://docs.frigate.video/configuration/live/#audio-support

[patrickacollier]

@hawkeye217 thanks, My question was referring to WebRTC supporting Opus, G711, G722 and my cameras native support for G711. If there is no reason not to use it, I could use G711 and have smaller bandwidth and not transcode. But i'm unsure of the tradeoffs. Is G711 (telco) ok for the limited audio use of ip cameras?

[hawkeye217]

My guess is that transcoding is not very cpu intensive, but I'd say give it a try and see what works best for you! Though if you change audio on the camera, you may go back to your original issue with audio in recordings. AAC support in browsers is pretty ubiquitous, not sure about any of the G encodings though.

[hawkeye217]

Also, you likely won't see much of a difference between MSE and WebRTC. If you're trying to do two-way talk with a doorbell, you'll need WebRTC though.

[patrickacollier]

yeah it was around 500-700ms latency difference with webRTC being faster. I think with live view, that latency may not make much difference but it is distinguishable.