# 圖片對比功能 本篇描述了 Appium 里圖片對比的一系列功能。圖片對比在所有的driver中都可以使用，這些功能依賴 OpenCV3 原生庫。并且，每個功能都可以可視化展示對比結果，所以你可以通過不斷地調參，得到最好的對比結果。 ## 前置條件 - OpenCV 3+ 的原生庫文件 - 安裝npm 模塊[opencv4nodejs](https://github.com/justadudewhohacks/opencv4nodejs) : `npm i -g opencv4nodejs`。 安裝 opencv4nodejs 默認會從源下載所需的OpenCV有關的庫文件，但是需要本地安裝了開發工具。 - Appium Server 1.8.0+ ## 目的 在許多自動化的任務中，圖片對比會更加方便，比如： - 判定給出的圖片當前是否在屏幕上 - 計算事先定義好的屏幕對象的坐標值 - 判定當前屏幕對象的狀態是否是期望的狀態 ## 基于特征的對比 通過模板來執行圖片對比，能夠在整圖中找出部分圖片出現的可能性。關于該主題，可參考 https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_feature2d/py_matcher/py_matcher.html 。這類對比通常在判斷原圖是否被旋轉/縮放的場景下很有用處。 ### 代碼示例 ```java // java byte[] screenshot = Base64.encodeBase64(driver.getScreenshotAs(OutputType.BYTES)); FeaturesMatchingResult result = driver .matchImagesFeatures(screenshot, originalImg, new FeaturesMatchingOptions() .withDetectorName(FeatureDetector.ORB) .withGoodMatchesFactor(40) .withMatchFunc(MatchingFunction.BRUTE_FORCE_HAMMING) .withEnabledVisualization()); assertThat(result.getVisualization().length, is(greaterThan(0))); assertThat(result.getCount(), is(greaterThan(0))); assertThat(result.getTotalCount(), is(greaterThan(0))); assertFalse(result.getPoints1().isEmpty()); assertNotNull(result.getRect1()); assertFalse(result.getPoints2().isEmpty()); assertNotNull(result.getRect2()); ``` 示例代碼中`FeaturesMatchingOptions`類方法的使用具體細節包含在源碼的doc描述當中（譯者注：下載源碼即可查看）。 ```ruby # Ruby image1 = File.read 'first/image/path.png' image2 = File.read 'second/image/path.png' match_result = @driver.match_images_features first_image: image1, second_image: image2 assert_equal %w(points1 rect1 points2 rect2 totalCount count), match_result.keys match_result_visual = @driver.match_images_features first_image: image1, second_image: image2, visualize: true assert_equal %w(points1 rect1 points2 rect2 totalCount count visualization), match_result_visual.keys File.open('match_result_visual.png', 'wb') { |f| f<< Base64.decode64(match_result_visual['visualization']) } assert File.size? 'match_result_visual.png' ``` ### 可視化示例  ## 圖片存在查詢 通過模板執行整圖匹配，查找局部圖片存在概率相關的主題的細節，可參考閱讀 https://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html 的內容。如果部分圖像是完整圖像一部分，這類比較就非常適用。 查詢圖片存在和基于特征的圖片對比之間存在細微的差別。當要查找的圖像是目標/屏幕截圖的子集時，使用前者。當要找到的圖像與目標基本相同，但旋轉或縮放時，使用后者。 ### 代碼示例 ```java // java byte[] screenshot = Base64.encodeBase64(driver.getScreenshotAs(OutputType.BYTES)); OccurrenceMatchingResult result = driver .findImageOccurrence(screenshot, partialImage, new OccurrenceMatchingOptions() .withEnabledVisualization()); assertThat(result.getVisualization().length, is(greaterThan(0))); assertNotNull(result.getRect()); ``` 示例代碼中`OccurrenceMatchingOptions`類方法的使用具體細節包含在源碼的doc描述當中（譯者注：下載源碼即可查看）。 ```ruby # Ruby image1 = File.read 'first/image/path.png' image2 = File.read 'partial/image/path.png' find_result = @driver.find_image_occurrence full_image: image1, partial_image: image2 assert_equal({ 'rect' => { 'x' => 0, 'y' => 0, 'width' => 750, 'height' => 1334 } }, find_result) find_result_visual = @driver.find_image_occurrence full_image: image1, partial_image: image2, visualize: true assert_equal %w(rect visualization), find_result_visual.keys File.open('find_result_visual.png', 'wb') { |f| f<< Base64.decode64(find_result_visual['visualization']) } assert File.size? 'find_result_visual.png' ``` ```javascript // Typescript / Javascript /* Typescsript code for occurrence comparison using the template matching algorithm. It detects if an image is contained in another image (called the template). The image must have the same scale and look the same. However, you can add a scaling transformation beforehand. official doc: https://github.com/appium/appium/blob/master../../writing-running-appium/image-comparison.md OpenCV algorithm doc: https://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html official sample code: https://github.com/justadudewhohacks/opencv4nodejs/blob/master/examples/templateMatching.js You must install opencv4nodejs using the -g option. The Javascript client driver webdriverio does not support (in January 2020) the "-image" strategy implemented in the Appium server. You will have more power and understanding while using openCV directly. Since the appium server is in Javascript, you can do all it does with opencv in your test suite. The testing framework mocha can be run with typescript to have async/await. You need to run mocha with those options in the right order and with the associated packages installed: NODE_PATH=/path/to/nodejs/lig/node_modules TS_NODE_PROJECT=config/tsconfig_test.json --require ts-node/register --require tsconfig-paths/register You will also need to make a basic config/tsconfig_test.json Note that paths in tsconfig.json does not support absolute paths. Hence, you cannot move the NODE_PATH there. */ import * as path from 'path'; const cv = require(path.join(process.env.NODE_PATH, 'opencv4nodejs')); const isImagePresent = async () => { /// Take screenshot and read the image const screenImagePath = './appium_screenshot1.png'; await driver.saveScreenshot(screenImagePath) const likedImagePath = './occurrence1.png'; // Load images const originalMatPromise = cv.imreadAsync(screenImagePath); const waldoMatPromise = cv.imreadAsync(likedImagePath); const [originalMat, waldoMat] = await Promise.all([originalMatPromise, waldoMatPromise]); // Match template (the brightest locations indicate the highest match) // In the OpenCV doc, the option 5 refers to the algorithm called CV_TM_CCOEFF_NORMED const matched = originalMat.matchTemplate(waldoMat, 5); // Use minMaxLoc to locate the highest value (or lower, depending of the type of matching method) const minMax = matched.minMaxLoc(); const { maxLoc: { x, y } } = minMax; // Draw bounding rectangle originalMat.drawRectangle( new cv.Rect(x, y, waldoMat.cols, waldoMat.rows), new cv.Vec(0, 255, 0), 2, cv.LINE_8 ); // Open result in new window // If the image is too big for your screen, you need to write to a file instead. // Check the source of opencv4nodejs for writing an image to a file. cv.imshow('We\'ve found Waldo!', originalMat); await cv.waitKey(); // then you know if the image was found by comparing the rectangle with a reference rectangle. // the structure minMax contains the property maxVal that gives the quality of the match // 1 is prefect match, but you may get .999. If you extract an image from the screenshot manually, // you will get an image that matches. }; ``` ### 可視化示例  左下角突出顯示的圖片是查找的結果匹配。 ## 相似度計算 圖片相似度是通過計算圖片之間相似性的分數來執行的。對比過程類似于圖片存在查詢中用到的`findImageOccurrence`，但是必須保證的是兩個圖像的大小要相等。如果原始圖像是原始圖像的副本，但內容發生了更改，這類比較就非常適用。 ### 代碼示例 ```java // java byte[] screenshot1 = Base64.encodeBase64(driver.getScreenshotAs(OutputType.BYTES)); byte[] screenshot2 = Base64.encodeBase64(driver.getScreenshotAs(OutputType.BYTES)); SimilarityMatchingResult result = driver .getImagesSimilarity(screenshot1, screenshot2, new SimilarityMatchingOptions() .withEnabledVisualization()); assertThat(result.getVisualization().length, is(greaterThan(0))); assertThat(result.getScore(), is(greaterThan(0.0))); ``` 示例代碼中`SimilarityMatchingOptions`類方法的使用具體細節包含在源碼的doc描述當中（譯者注：下載源碼即可查看）。 ```ruby # Ruby image1 = File.read 'first/image/path.png' image2 = File.read 'second/image/path.png' get_images_result = @driver.get_images_similarity first_image: image1, second_image: image2 assert_equal({ 'score' => 0.891606867313385 }, get_images_result) get_images_result_visual = @driver.get_images_similarity first_image: image1, second_image: image2, visualize: true assert_equal %w(score visualization), get_images_result_visual.keys File.open('get_images_result_visual.png', 'wb') { |f| f<< Base64.decode64(get_images_result_visual['visualization']) } assert File.size? 'get_images_result_visual.png' ``` ### Visualization Example ### 可視化示例  兩張圖片的相似度得分在0.98以上。