Testing @Murena

Types of Testing

At /e/OS we have three types of testing

• Sprint testing
• Upgrade testing
• Porting or Applications testing

Sprint Testing This is the regular testing for our builds. This testing is done for all devices in our supported list We follow the agile model of development where we have regular sprints. 2 weeks of development and 1 week of testing. Read more

Upgrade Testing When the OS on a device is to be upgraded - from nougat to Oreo or Pie version of /e/OS we do an upgrade testing. We will announce which devices are going to be upgraded in the posts in this category on our forum.For the duration of the upgrade testing we would create a testing team comprising of volunteer users. This team gets disbanded once the builds go live.

Porting or Applications Testing Occasionally we port /e/OS on devices which are not supported by other custom ROM’s. This can be smartphones, laptops or tablets. For e.g. we recently ported /e/OS to the FP3 or we came out with Olimex and Pinebook builds. The easy-installer was an application we created which required detailed testing during the development phase.

For the duration of the testing we would create a testing team comprising of volunteer users. This team gets disbanded once the builds go live.

What is expected from a Tester

As a tester you would be required to

• Flash the test build on your device
• This will be a manual flash the first time . Subsequent builds will or should be available OTA
• Flashing your device will wipe out existing data on your phone. So please take a backup before proceeding
• Only those who have experience in flashing ROM’s need join
• Testing will only be done for devices in our supported list.

If your device is not in the supported list you can request a device to be added. This also means you should not join the testing team.

Can I opt to leave the testing team?

Yes you can. Participation as a tester is a voluntary activity. You are free to leave the team any time you wish.

• Please note to return to the normal stable builds you will have to manually flash the stable builds on your device. This is because the test builds and development or stable builds are made with different keys.

How do I join the testing community?

• Join the Telegram testing community
• Find the current test session and documentation about /e/OS on Gitlab

Test session with Kiwi

From the /e/foundation we provide a smartphone OS for a huge number of devices. Unfortunately, we cannot check and evaluate a new /e/OS release on all supported devices by the project. First of all, our /e/team focuses on stable devices. Then the team and the testing community can also test their own devices with the last release. In order to centralize test sessions and reportings we use a tool named « Kiwi » to manage our tests.

Kiwi is an open source tool with a web interface for managing your tests. It is the best friend for QA teams as it supports good practices by design. It leads you to create a test plan for your products with a set of test cases. On each new release you can assign easily test runs for your team members and the community.

We add in each test case some meta-data like application and hardware information (Settings, Notes, GPS, Wifi…). It helps to get a first overview about what the test will do, then we are thinking to use these tags in the future to generate a full test report per device. Sharing this information with the community will be really useful before proceeding to the installation, the user will be able to show what is working or not on a particular device.

Kiwi is also extensible we can connect our bug tracker to open directly an issue in our backlog when a bug is detected. We had customized the issue description for helping its resolution with the Kiwi plugin system.

Testing process FAQ

• What happens if a build fails or bugs are detected in your testing?

If a build fails in the testing process we hold back the release of the build and it goes back to the developers. Similarly if a bug is reported in the testing we revert that fix to the previous working version and the bug goes back to the developer to be resolved.

• How do you handle such reverts in an agile development scenario?

As you may recall we have 2 week development cycles for a sprint. The bugs which are detected during the testing are moved to the next sprint cycle and only those fixes that are successful get into the build.

• Are only bugs fixed in these sprints what about enhancements?

That is a good question. We combine both enhancement as well as bug fixes in the sprints. If the enhancement related coding do not finish in the given timeline it is moved to the next sprint cycle.

• I do not have any experience of flashing ROM’s neither is my device in the supported list can I join the testing team?

We appreciate your enthusiasm but Please do not join the testing team if this is the case . As we mentioned above the testing team is working against a strict timeline. It will not be possible to handhold a user to understand how to flash the ROM. For that level of assistance you can go to our forum.