Make integration tests more enjoyable to use and evolve #18676
Assignees
Labels
Comments
7 tasks
ilmotta
added a commit
that referenced
this issue
Mar 5, 2024
This commit brings numerous improvements to integration tests. The next step will be to apply the same improvements to contract tests. Fixes #18676 Improvements: - Setting up the application and logged account per test is now done with an async test fixture, which is a very idiomatic way to solve this problem. No need anymore to write macros to wrap day8.re-frame.test/wait-for. The macros in test-helpers.integration will be removed once we apply the same improvements to contract tests. - Integration test timeouts can be controlled per test, with a configurable, global default (60s). - Now the integration test suite will fail-fast by default, i.e. a test failure short-circuits the entire suite immediately. This option can be overridden on a test-by-test basis. This improvement is very useful when investigating failures because the error will be shown on the spot, with no need to search backwards across lots of logs. - Noisy messages from re-frame can be silenced with a test fixture. We can silence even more in the future if we remove the hardcoded printf call from C++ on every signal and control it with Clojure. We can disable most logs as well with the more direct (status-im.common.log/setup "ERROR") at the top of tests.integration-test.core. We can make verbosity even more convenient to control, but I think this should be designed outside this PR. - Removed dependency on lib day8.re-frame/test for integration tests (see detailed explanation below). - Each call to (our) wait-for can customize the timeout to process re-frame event IDs passed to it. - Syntax is now flat, instead of being nested on every call to wait-for. You can now compose other async operations anywhere in a test. Notes: - promesa.core/do is essential in the integration test suite, as it makes sync & async operations play nice with each other without the developer having to promisify anything manually. - There are lots of logs related to async storage ("unexpected token u in JSON at position..."). This isn't fixed yet. Are we not going to use day8.re-frame.test? We don't need this library in integration tests and we won't need it in contract tests. Whether it will be useful after we remove it from integration and contract tests is yet to be seen (probably not). A few reasons: - The async model of promises is well understood and battle tested. The one devised in the lib is poorly understood and rigid. - There's basically no way to correctly integrate other async operations in the test, unless they can be fully controlled via re-frame events. For instance, how would you control timeouts? How would you retry in a test? How would forcefully delay an operation by a few seconds? These things are useful (to me at least) when developing integration/contract tests. - Every call to day8.re-frame.test/wait-for forces you to nest code one more level. Code readability suffers from that choice. - Have you ever looked up the implementation of wait-for? It's quite convoluted. One could say the source code is not that important, but many times I had to look it up because I couldn't understand the async model they built with their macro approach. The de facto primitive in JS for asynchronicity is promises, and we fully leverage it in this PR. - The lib has an interesting macro run-test-sync, but we have no usage for it. I used it in status-mobile for a while. At one point, all event unit tests for the Activity Center used it (e.g. commit 08fb0de), but I replaced them with the simpler pure function style.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem
At the moment, integration tests in
src/status_im/integration_test/are not the easiest to write or read.day8.re-frame.test/wait-foris a macro.legacy.status-im.multiaccounts.logout.core/logout-method. We had a case where we forgot about this and the developer wasted a long time trying to find the reason. This also increases the verbosity of each integration test.src/status_im/integration_test/) to explain what they are and how to write them.Motivation for this issue
Reduce the effort to read & write integration tests. This will be more and more important as we write more integration tests, as we can see from all the issues opened to increase coverage through these tests. See https://github.com/status-im/status-mobile/issues?q=is%3Aopen+is%3Aissue+label%3A%22E%3AMobile+Integration+Tests%22.
cc @J-Son89
The text was updated successfully, but these errors were encountered: