Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: camera pass-through mode #313

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

Schmetzler
Copy link

I added a pass-through mode for ALVR... recently only tested in the lobby, but it should work while streaming (or should it?).

You can trigger the pass-through by double tapping on the side of your headset... this is a little experimental and needs some practice you should not hit too hard and also not to soft and you must start in a resting position.
Basically it detects 2 maxima of the derivative of the accelerator sensor signal on a predefined delay.
The algorithm is somewhat described in the comments of onSensorChanged
Or you can switch to passthrough in the settings menu (which is not really usable when the device is in the headset). Maybe it can be set to a predefined button (any suggestions?)

I also added a preference Activity, where you can tinker with the values for the pass-through or disable auto detection if it interferes while playing.

There are 2 more options in the Settings menu, the Passthrough fraction (the fraction of the view that should be filled with the camera image ... this is needed as some cameras hava a really narrow FOV so the passthrough would look kinda too close) and the set Recording Hint, if set to false this can increase the FOV but usually also decreases the FPS which is kinda hard to look at, but maybe this is a no issue on newer smartphones.

@ShootingKing-AM
Copy link
Member

seems like there are some lint issues, you can auto fix them by going though the documentation https://github.com/alvr-org/PhoneVR?tab=readme-ov-file#development

@Schmetzler
Copy link
Author

Ah I see... Sorry... the spotless apply thing was checked always before android studio build the apk, but it seems it didn't check the c++ part.

Ok I will update I guess tomorrow or the day after tomorrow.

@ShootingKing-AM
Copy link
Member

Ah I see... Sorry... the spotless apply thing was checked always before android studio build the apk, but it seems it didn't check the c++ part.

Ok I will update I guess tomorrow or the day after tomorrow.

Yeah, sure. I tried this and the passthough works really well, really good work !!

  • The only experience related issue is the side double tap. Firstly, brilliant idea :D I got it down via practice too, but can there be a little intimation to user somehow that he is not tapping hardenough or fast enough, like maybe a small Accelerometer graph over the glView, or maybe someother way ?
  • All GL functions returns errors, so we need a little more error handling. You can use this https://github.com/alvr-org/ALVR/blob/master/alvr/client_core/cpp/utils.h from ALVR. And then enclose all Gl* Functions in Gl() like in eg. https://github.com/alvr-org/ALVR/blob/28dc9205fde94e1834c7a8c5f28882f75dffb391/alvr/client_core/cpp/graphics.cpp#L337, that will take care of error handling of GL* functions. Also do check for errors in other functions if you missed :)
  • Do change the log level of acclSense values log.d to verbose or trace since they are its log is very frequent
  • Also use camelcase variable naming thoughout the code, its somewhere snake_case and sometimes camelCase. If the variable is member of the current class it has to be appended with "m" like mLowerBound, if its a variable only in the present context then simply lowerBound is good
  • If possible you can seprate the passthough code from main files (separate from alvr_main.cpp and from ALVRActivity.java), because they are huge no of lines in their own right
  • A little more documentation and some meaningful variable naming would be awesome, its hard to understand the code logic espicially in onSensorChanged() (for double tap detection)

Rest everything looks good

@ShootingKing-AM ShootingKing-AM changed the title Added pass-through mode feat: camera pass-through mode Apr 1, 2024
@Schmetzler
Copy link
Author

Schmetzler commented Apr 5, 2024

Ok so, lets see what the steps are:

  • User feedback (why it is not working) ... I guess this one is really difficult as I do not know in advance that someone want to do the double tapping.. but maybe the graph thing can be done (but don't know... this is kinda hard I think)... Maybe we could make a "training mode" for the correct thresholds.
  • Wrap GL functions
  • Change Log-Level (using Log.v now)
  • camelCase for all variables, and mVariable for member variables
  • separate passthrough code from ALVRActivity.java
  • separate passthrough code from alvr_main.cpp
  • more documentation (and meaningful variable names) (well for the double tap detection I guess an image would make things a little clearer... I will prepare something.

Another question... should I squash the commits together to one when I am finished here?

@ShootingKing-AM
Copy link
Member

ShootingKing-AM commented Apr 5, 2024

  • Training mode sounds awesome and easier.
  • Forget about squashing, i will take care of that :)

Edit: Also if you want more immediate response you can join Discord and communicate in #dev channel :3

@ShootingKing-AM
Copy link
Member

ShootingKing-AM commented Apr 12, 2024

Just to make myself clear, Training mode is nothing fancy, just an Activty showing realtime graphs of 3axis of accelSensor so that they(users) get used to it. You can use an existing libary for graphs like https://github.com/PhilJay/MPAndroidChart-Realm (other libs here https://github.com/lucasrafagnin/android-charts)

If you are looking for some button, there is a suggestion here: #77

@ShootingKing-AM ShootingKing-AM added the feature New feature label Apr 12, 2024
@Schmetzler
Copy link
Author

SumMean

So here some points to understand the algorithm:

  • First we gather accelerometer data from at least 600ms (this is around 9-10 datapoints)
  • We calculate the absolute derivative for each axis ($|x_i - x_{i+1}|$)
  • We calculate the sumMean on current position of the derivative:
    • mean for each axis
    • sum of these means

For each axis:

  • we split the accelerometer derivative into 2 windows
  • we calculate the maximum on each window
  • All of the following must be true to set it as a possible trigger:
    • are the maxima minus the respective window edges higher than lower bound
    • are the maxima and the sumMean between lower and upper bound
    • was sumMean on first position of window below max(lower bound - 0.3, 0.5)
    • is the maximum of second half not on the edge of second half

(Beware It is not important on which axis this applies... but that is also the reason why I use sumMean to prevent it to trigger when something like this happens in one axis, but the other axis have more "movement")

  • After possible trigger found, check whether the sumMean drops below lower bound during another 600ms

The current implementation needs a steady phase before the double tap and a steady phase after the double tap

@ShootingKing-AM
Copy link
Member

Nice explanation. Do add this documentation into the code :3

@ShootingKing-AM
Copy link
Member

ps: in #334 i added utils.h and modified it according to phonevr, i handled errors in existing alvr_main.cpp GL* calls which i forgot earlier. You may want to use that, it will be merged when tests are done.

@Schmetzler
Copy link
Author

Well it seems like I have to cleanup my git history somehow... as rebasing needs to many merges for unneeded commits. I am not really well versed in this kind of stuff... so I think I will make a branch, rebase there (if this is possible) do some commit cherry picking and hope everything works as suspected... and after that I think I must rewrite history of the master branch so that it can be pulled from here.

Sounds that reasonable or is my thinking wrong?

@ShootingKing-AM
Copy link
Member

ShootingKing-AM commented May 6, 2024

just update/merge the master into your branch if rebasing your is hard, if you are using Github Desktop it will prompt you to resolve conflicts while merging.

Select Update from Master and it will show the below panel to fix conflicts
image

You can resolve those conflits easily there. No need to rewrite history on this pr, you can make it as messy as it can be dont worry about it.

Or should i rebase it for you ?

Dominik Wetzel added 2 commits May 6, 2024 23:25
Combine 7 commits to one:

* Initial Passthrough implementation with menu to select passthrough mode.
* Made passthrough available via inflateMenu. Fixed crashes during change to passthrough mode.
* Added recordingHint to Settings. Open Camera only on passthrough mode.
* Added option to trigger passthrough by double tapping the headset on the side.
* run make format
* Moved passthrough code into separate file
* split passthrough stuff into separate file, made variables camelCase
@Schmetzler
Copy link
Author

Well I did some cherry-picking, rebasing force-pushing and so on...^^
On another branch I started to implement a visualisation, which currently works, but uses HelloChart (which is not available in the repos). So currently I use a downloaded jar file... Maybe I will use the thing you suggested, but I was a little worried about apk size, if this chart library just blows up the size for something that is not so important. (But maybe the mentioned chart library is also neglectable small... I do not know)

@ShootingKing-AM
Copy link
Member

ShootingKing-AM commented May 7, 2024

Quantify its size, i dont think android libraries would be that heavy. Still some linting errors :3

@ShootingKing-AM
Copy link
Member

/ok-to-test sha=16e98a2

@Schmetzler
Copy link
Author

I have a question... What am I missing? I don't get the last CI Test... Do I need to add some tests?

@ShootingKing-AM
Copy link
Member

Just reopened and closed the PR so that the tests will get retriggered, now.

If i remember there were some errors (maybe linting errors). Were you able to test at local properly ?

@Schmetzler
Copy link
Author

I don't get the Test APK error as it fails on Screenshot uploading... but I haven't even implemented a test (well... I know... it is not good design)... Maybe something with the CI Action is wrong?

@ShootingKing-AM
Copy link
Member

/ok-to-test sha=a6ebf88

Copy link

github-actions bot commented Jul 5, 2024

📷 Screenshots of tests:

📱 Pixel_2(AVD) - 11

🔧 ALVRActivityTest_saveDeviceScreenBitmap

🔧 ALVRActivityTest_saveDeviceScreenBitmap_LookingDown

🔧 ALVRActivityTest_saveDeviceScreenBitmap_LookingFrontDown

🔧 ALVRActivityTest_saveDeviceScreenBitmap_Popup

🔧 InitActivityTest_saveDeviceScreenBitmap

🔧 MainActivityTest_saveDeviceScreenBitmap

🔧 SettingsActivityTest_saveDeviceScreenBitmap

🔧 view-op-error-1

For commit a6ebf88

@ShootingKing-AM
Copy link
Member

I don't get the Test APK error as it fails on Screenshot uploading... but I haven't even implemented a test (well... I know... it is not good design)... Maybe something with the CI Action is wrong?

CI Actions are atlast fixed.

@ShootingKing-AM
Copy link
Member

@Schmetzler Anything left on your end to do here ?

@Schmetzler
Copy link
Author

If I find the time... I would like to look into the variable names, add some comments and maybe add some test cases... But I am not very well versed in android development (and also testing) anyway... I can imagine to add a test case where I put some predefined data into the algorithm and check if it triggers the passthrough mode... But I think it would take me at least a week (I need to find time, and I resetted my OS so I need to setup the development environment again)

@ShootingKing-AM
Copy link
Member

its okay, take your time. Let me know if you are stuck somewhere : )

@Schmetzler
Copy link
Author

Sorry I cannot afford to work on it anymore. Get everything to build is just too much of a hassle for me and stresses me... You may merge it the way it is (which is working and functional) but I won't change any naming or add comments without the possibility to check if everything works. And even for the testing it seems that there is no possibility to mock the sensordata.

If the code is too complicated I am willing to explain.

Hope you understand and find the code usefull nevertheless.

Thanks for your patience

@ShootingKing-AM
Copy link
Member

Its okay, no need to worry too much, i will look into this : )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants