Turn a Discerning Eye to Your Apps with SUITcase
Introduction
Have you ever wondered how to add appearance verifications to your automated tests? The Devexperts team is happy to introduce its secret weapon – SUITCase. This is our internal tool for snapshot testing on iOS, iPadOS, and tvOS apps, and we’ve shared it on GitHub.((Devexperts team. “devexperts/suitcase” GitHub Website, GitHub, Inc., 2020, https://github.com/devexperts/suitcase))
In this article, we’ll tell you all about SUITCase and its features. It doesn’t matter whether you test iOS, iPadOS, or tvOS apps – this article is useful for all Quality Assurance engineers.
Problem
We use robust frameworks like XCTest and EarlGrey to test iOS apps.((Roman Zakharov. “Thirty Seconds to Pass: Performance Comparison of iOS UI Testing Frameworks” Devexperts Blog, Devexperts LLC, 2020, https://devexperts.com/blog/ios-ui-testing-frameworks-performance-comparison)) Why would we want to enhance their capabilities?
These two screenshots represent the final step in a test case that verifies the cropping feature in Apple’s Photos app. If we press the SQUARE button on the bottom toolbar, we get a cropped image.
Which assertions are necessary to verify this behavior? Well, we definitely want to check the resulting image. The problem is, there’s no way to check the images with XCTest and EarlGrey APIs.
What can we verify, then? The next images represent the data accessible using the built-in XCTest APIs. We call these images “T-800 Vision”. We can verify the elements’ existence, type, label, location, size, hittability, and if they’re selected.((Apple Developers Tools team. “XCUIElementAttributes | Apple Developer Documentation” Apple Developer Website, Apple Inc., 2020, developer.apple.com/documentation/xctest/xcuielementattributes)) For example, we can verify that the RESET button on the top has appeared and the crop handles have moved. But we cannot verify what’s happened to the image.
When we test our iPadOS and tvOS apps, we face the same problem. For the sake of concision, we’ll only discuss iOS app testing; but as we mentioned, SUITCase supports iOS, iPadOS, and tvOS apps.
Snapshot testing can help you verify the visual appearance of UI elements. It’scommonly used to check images and text edits. Snapshot Testing can also be helpful with charts, complex UI, and appearance changes.
iOS snapshot testing
Let’s briefly explore popular screenshot testing solutions.
The most popular tool for snapshot testing on iOS apps is iOSSnapshotTestCase((Alan Zeino, Dustin Barker, Adam Ernst, Daniel Doubrovkine, Reid Main and other contributors. “Snapshot View Unit Tests for iOS” GitHub Website, GitHub, Inc., 2020, github.com/uber/ios-snapshot-test-case)), previously known as FBSnapshotTestCase. This framework was an open-source project by Facebook; now Uber maintains it. You can use iOSSnapshotTestCase within your unit tests and verify UIViews and CALayers. It’s possible to configure an acceptance threshold for each screenshot. Although iOSSnapshotTestCase is designed for unit tests, you can use this tool within your UI tests.((Steve Moser. “How to Set Up Xcode UI Testing for Today’s Extensions” Resource Library, Levvel, 2020, levvel.io/resource-library/xcode-ui-testing-today-extensions))
We’ll also notice some interesting features if we take a look at the forks of this tool.((GitHub users. “Forks · uber/ios-snapshot-test-case” GitHub Website, GitHub, Inc., 2020, github.com/uber/ios-snapshot-test-case/network/members)) Some of the features can add color difference calculations to stabilize flaky tests. Some forks enable macOS and XCUIElement support. Other useful features include better screenshot naming and image scaling.
Snap.swift is another tool for snapshot testing.((Oscar Antonio, Duran Grillo and other contributors. “Snapshot testing in a Snap” GitHub Website, GitHub, Inc., 2020, github.com/skyweb07/Snap.swift)) In addition to iOSSnapshotTestCase, it was also designed for unit testing and verifies UIViews. It records reference screenshots super fast and performs well with FastLane. This tool also generates much more detailed logs than iOSSnapshotTestCase and adds the actual screenshot, reference, and difference presentation as attachments.
Now let’s highlight the interesting features of one more framework. Unreal Engine has built-in Screenshot Comparison Tools with automatic delays and retries if image verification fails.((Unreal Engine team. “Screenshot Comparison Tool | Unreal Engine Documentation” Unreal Engine Documentation, Epic Games, Inc., 2020, docs.unrealengine.com/en-US/Programming/Automation/ScreenShotComparison/index.html)) It also uses different thresholds for local and global differences. And you can compare the light intensity while ignoring colors altogether.
We packed the most interesting features into SUITCase – Snapshot UI Test Case. This tool is designed specifically for UI Testing in Xcode, keeping XCUIElement support in mind. It provides convenient recording and generates informative logs. It saves disk space by scaling screenshots and has multiple comparison methods to help you balance accuracy and stability. And we continue enhancing SUITCase on GitHub.((Devexperts team. “devexperts/suitcase” GitHub Website, GitHub, Inc., 2020, https://github.com/devexperts/suitcase))
Features
XCUIElement Support
SUITCase supports XCUIElements and allows many configurable ways to improve snapshot testing on iOS apps. Since this tool is designed for UI testing, you can run more test cases compared to the previously mentioned unit tests frameworks. With SUITCase it’s easy to verify the appearance of the whole device screen and particular elements while ignoring some of the other elements. This feature is great for stability and implements the single-responsibility principle.
Recording
SUITCase automatically saves reference, suggested, and unexpected images. If we enable record mode, SUITCase will save the reference images. If there is no reference screenshot, SUITCase will save the collected screenshot in a separate folder in case you want to use it as a reference later. You can set a threshold, which allows some of the collected pixels to differ from the reference ones. If the difference is lower than the threshold value, verification is passed. If the difference is higher than the threshold you set, SUITCase will save the unexpected screenshot and attach it to a test log, as well as to the reference screenshot and graphical representation of the difference.
Scaling
For accuracy, it’s best to keep high-resolution images and compare all pixels. The downside is we have to use significant storage space. But if we scale down screenshots, we can save storage space. Twenty two different iPhones support iOS 11 or higher. But thanks to Apple’s famous small fragmentation, only eight different rendered pixels resolutions exist.((PaintCode team. “The Ultimate Guide To iPhone Resolutions” PaintCode Website, PixelCut LLC, 2020, paintcodeapp.com/news/ultimate-guide-to-iphone-resolutions)) The rendered pixels resolution numbers look chaotic until we divide them by the screen scales to get the logical resolutions in points. Here we can see that iPhone 11 and iPhone 11 Pro Max (and their predecessors) use the same logical resolution, so we can use shared reference for a full-screen comparison. iOS devices also use only five widths – 428, 414, 390, 375, and 320. So if we want to compare medium-size elements such as navigation bars, toolbars, and tab bars, we only need to keep five different screenshots. If we compare fixed-size elements like buttons, we will only use one reference screenshot.
Five comparison methods
To help you balance accuracy and stability, SUITCase provides five different comparison methods. Strict method compares unchanged original images and is the only method that saves high-resolution references. Using the With Tolerance method, we can allow small color changes. You can read how we tuned this method in a separate article.((Roman Zakharov. “Getting to the Bottom of the RGB Pixels. Part 1: Color Difference” Devexperts Blog, Devexperts LLC, 2020, https://devexperts.com/blog/getting-to-the-bottom-of-the-rgb-pixels-part-1-color-difference)) Another comparison method called Greyscale only compares light intensity, ignoring the colors. DNA is inspired by Microsoft’s PhotoDNA algorithm designed to identify illegal images.((Microsoft team. “PhotoDNA | Microsoft” Microsoft Website, Microsoft Corporation, 2020, microsoft.com/en-us/photodna)) This method significantly downscales images, freeing up a lot of disk space. The last method Average Color is the least accurate, but it does come in handy in some cases.
Let’s see how these methods work. The next two images show Apple’s Shortcuts app localized in English and German. Aside from the different texts, we have four shortcuts with mismatched colors and icons.
If collected and reference images don’t match, SUITCase will attach both expected and actual colors for Average Color method. For the Strict, With Tolerance, Greyscale, and DNA methods, SUITCase will attach a difference visualization.
The next images show the visualizations. The Strict method highlights all non-matching pixels, and the With Tolerance method ignores the color change in the first shortcut. The Greyscale method highlights texts and icons ignoring the colors of both shortcuts, while DNA only shows the major changes. For the Greyscale and DNA methods, intermediate image processing steps are shown.
Results
Snapshot testing with SUITCase improved our test cases because now we’re able to perform check ups that were previously unavailable.
Thanks to features like XCUIElement Support, Scaling, and the With Tolerance method, we can verify the cropped image from the beginning of this article, reusing the reference image while testing many different devices. By the way, the picture was shot at our office in Porto.
Conclusion
SUITCase is a great way to improve your iOS, iPadOS, and tvOS tests. It has features like XCUIElement support, convenient recording, scaling, and five comparison methods to cover various test cases.
Even if you automate tests for other platforms, you can implement several SUITCase features into your tests. Our tool is open-source software and available on GitHub.((Devexperts team. “devexperts/suitcase” GitHub Website, GitHub, Inc., 2020, https://github.com/devexperts/suitcase)) We would be happy to receive contributors’ issues and pull-requests. SUITCase is one more tool you can use to improve your test automation. Pick your favorite tools, frameworks, and utilities and make apps better than ever before!
Which tools do you use to automate screenshot testing? How do you balance accuracy and stability?