Testing
As our software project grows, it will become harder and harder to change without breaking something. To mitigate this limitation, we write tests to enable us to confidently change the code in future. Tests give us more confidence that we are not breaking old behaviour when modifying code.
Tip
Whilst testing might seem boring/pointless in the beginning, they start to pay off later. The value of tests grow with time and the larger your project becomes.
Methodology
The testing methodology used for the development of Epic Fantasy Forge is to write all tests for a feature or bug fix in advance. More specifically the procedure followed is:
- Write E2E tests to cover all happy paths and all error paths that are feasible to manually test.
- Write integration tests to cover all happy paths and all error paths.
- Write unit tests to cover all happy paths and all error paths.
This method is similar to TDD (Test Driven Development) in the sense that tests are written before production code. However the main difference is that in TDD only a single test is written at a time and you iteratively write more tests as you write production code at the same time to make your new tests pass.
This is just an opinion, but I prefer to write as many tests in advance as possible. This allows you to plan a bit more in advance and think more about the architecture.
Starting with the E2E tests allows you to focus on the bigger picture first. Then as you gain more insights you drill deeper into the plan and architecture by writing integration and unit tests.
If whilst writing production code you think of a new test case, add it immediately to your tests.
Unit Tests
On Epic Fantasy Forge, we use three primary programming languages, Elixir, Rust and TypeScript. We will need a unit testing framework for each one.
The below table lists which unit testing frameworks we will use for which programming language on Epic Fantasy Forge:
| Unit Testing Framework | Programming Language |
|---|---|
| ExUnit | Elixir |
| Rust | Rust |
| Jest | TypeScript |
Rust has a built-in unit testing framework. To use TypeScript with the Jest unit testing framework, we will additionally need the ts-jest package since Jest on its own can only test JavaScript code but not TypeScript code.
Integration Tests
The below table lists which integration testing frameworks we will use for which programming language on Epic Fantasy Forge:
| Integration Testing Framework | Programming Language |
|---|---|
| Phoenix.ConnTest | Elixir |
| Tauri Testing Library | Rust |
| Jest | TypeScript |
The Phoenix.ConnTest module is built into the Phoenix framework. Jest can be used for both unit testing and integration testing. We will use it for both. For integration testing our Rust code we will use the Tauri testing library.
E2E Tests
For E2E tests we will rely on manual testing. It is possible to automate E2E tests, however writing them can be a lot of work and they may not be reliable. Furthermore, getting E2E tests to work for every platform we support could be very time-consuming.
During our manual testing, we will vary on what specific device we are testing and what web browser we are using for the web tests. It is naturally not feasible to test on all devices, however we can for every release use a different variety of devices or web browsers for testing.
As the E2E tests are time consuming to execute, we will only run all of them during release testing and then only on the platform being released in the current week. When developing a new feature, we create new E2E tests and run them only on our development machine. The other platforms will be tested during release testing.
Web
For E2E testing our web app, we will use BrowserStack. BrowserStack allows us to test our web app on various web browsers.
Desktop
Windows
For E2E testing on Windows, we will use our development machine. The development machine I run is dual-booting Fedora Linux and Windows 11. For more information about dual-booting, see the Multi-booting section of the Operating System page in this guide.
macOS
For E2E testing on macOS, we will use Scaleway. On Scaleway we can temporarily rent an Apple Mac mini M4 whenever we do release testing for macOS.
Linux
For E2E testing on Linux, we will use Instant Workstation. Instant Workstation allows us to test our Tauri app on various Linux distributions, including Ubuntu, Arch Linux, Fedora and Debian. Furthermore we can test both x86 and ARM versions of our Tauri App.
Mobile
Android
For E2E testing our Android app, we will use BrowserStack. BrowserStack allows us to test our Android app on various Android devices.
iOS
For E2E testing our iOS app, we will use BrowserStack. BrowserStack allows us to test our iOS app on various iOS devices.
Testing Tool
To create manual test cases and track their execution we will use Qase. Qase is offered in multiple pricing tiers, including a free version. This guide recommends using the free version since it includes everything we need for a small project. Start by creating an account on Qase. You might be automatically put on a free trial of the Qase business version, however this is fine since after your free trial ends you will be automatically downgraded to the free version unless you explicitly upgrade.
Once you have an account on Qase and completed the onboarding, you might already have an automatically generated demo project. You can delete this project.
Create a new project. Your your project name and code. Set the project access type depending on your needs. Since Epic Fantasy Forge is an open source project I set the project access type to public. Once done, click "Create project":
We will now add a test suite to our project. Click on "+ Suite" and set the "Suite name" to "Common". Then click "Create":
Now repeat the above step to create another suite named "Web Exclusive" and "App Exclusive":
You should now have three test suites in total. As the names suggest, the common suite will contain test cases common to both the web app and platform-specific apps. The exclusive suites contain test cases exclusive to the web app or exclusive to the platform-specific apps.
Later we will fill these test cases with test cases. We will only use Qase to record manual E2E test cases, not automated test cases. Note that we will write the test cases before implementing the functionality. For more details about this approach see the Methodology section above.