Flaky tests · Gitlab 中文文檔

# Flaky tests > 原文：[https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html](https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html) * [What’s a flaky test?](#whats-a-flaky-test) * [Quarantined tests](#quarantined-tests) * [Quarantine tests on the CI](#quarantine-tests-on-the-ci) * [Automatic retries and flaky tests detection](#automatic-retries-and-flaky-tests-detection) * [Problems we had in the past at GitLab](#problems-we-had-in-the-past-at-gitlab) * [Time-sensitive flaky tests](#time-sensitive-flaky-tests) * [Array order expectation](#array-order-expectation) * [Feature tests](#feature-tests) * [Capybara viewport size related issues](#capybara-viewport-size-related-issues) * [Capybara JS driver related issues](#capybara-js-driver-related-issues) * [PhantomJS / WebKit related issues](#phantomjs--webkit-related-issues) * [Capybara expectation times out](#capybara-expectation-times-out) * [Resources](#resources) # Flaky tests[](#flaky-tests "Permalink") ## What’s a flaky test?[](#whats-a-flaky-test "Permalink") 這個測試有時會失敗，但是如果您重試了多次，它最終會通過. ## Quarantined tests[](#quarantined-tests "Permalink") 當`master`的測試頻繁失敗時，應創建[?" master：broken"問題](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) . 如果無法及時修復測試，則會影響所有開發人員的工作效率，因此應通過分配`:quarantine`元數據將其放入隔離`:quarantine` . 這意味著除非使用`--tag quarantine`運行，否則它將被跳過： ``` bin/rspec --tag quarantine ``` **在對隔離進行測試之前，應確保存在?" master：broken"問題，這樣它才不會永遠留在隔離區.** Once a test is in quarantine, there are 3 choices: * 測試應該固定（即擺脫其脆弱性）嗎？ * 是否應將測試移至較低級別的測試？ * 是否應該完全刪除該測試（例如，因為已經有一個較低級別的測試，或者它正在復制另一個相同級別的測試，或者它測試的太多等等）？ ### Quarantine tests on the CI[](#quarantine-tests-on-the-ci "Permalink") 隔離測試在允許失敗的專用作業中在 CI 上運行： * `rspec-pg-quarantine` （CE 和 EE） * `rspec-pg-quarantine-ee` （僅適用于 EE） ## Automatic retries and flaky tests detection[](#automatic-retries-and-flaky-tests-detection "Permalink") 在 CI 上，我們使用[RSpec :: Retry](https://github.com/NoRedInk/rspec-retry)自動重試失敗的示例幾次（有關準確的重試次數，請參閱[`spec/spec_helper.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/spec_helper.rb) ）. 我們還使用自制的`RspecFlaky::Listener`偵聽`RspecFlaky::Listener` ，該`RspecFlaky::Listener`將`RspecFlaky::Listener`示例記錄在`master`上的 JSON 報告文件中（ `retrieve-tests-metadata`和`update-tests-metadata`作業）. 這最初是在以下[網址中](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021)實現的： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021) : [//gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021) . 如果要在本地啟用重試，則可以使用`RETRIES`環境變量. 例如`RETRIES=1 bin/rspec ...`將重試失敗的示例一次. ## Problems we had in the past at GitLab[](#problems-we-had-in-the-past-at-gitlab "Permalink") * [`rspec-retry` is biting us when some API specs fail](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29242): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9825](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9825) * [Sporadic RSpec failures due to `PG::UniqueViolation`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28307#note_24958837): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9846](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9846) * Follow-up: [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10688](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10688) * [Capybara.reset_session! should be called before requests are blocked](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33779): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12224](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12224) * FFaker 生成測試尚未準備就緒的時髦數據（測試應該是可預測的，所以很糟糕！）： * [使`spec/mailers/notify_spec.rb`更加健壯](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20121) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10015) : [`spec/mailers/notify_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20121) * [`spec/requests/api/commits_spec.rb`暫時失敗](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/27988#note_25342521) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9944) : [`spec/requests/api/commits_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/27988#note_25342521) * [用序列替換 FFaker 工廠數據](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184) : [//gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643) * [spec / finders / issues_finder_spec.rb 中的暫時失敗](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30211#note_26707685) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10404) : [//gitlab.com/gitlab-org/gitlab-foss/-//merge_requests/10404](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30211#note_26707685) ### Time-sensitive flaky tests[](#time-sensitive-flaky-tests "Permalink") * [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046) * [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10306](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10306) ### Array order expectation[](#array-order-expectation "Permalink") * [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10148](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10148) ### Feature tests[](#feature-tests "Permalink") * [Be sure to create all the data the test need before starting exercise](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/32622#note_31128195): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12059](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12059) * [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34609#note_34048715): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12604](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12604) * [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34698#note_34276286): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12664](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12664) * [Assert against the underlying database state instead of against a page’s content](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/31437): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10934](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10934) * 在 JS 測試中，當元素在 Capybara 發送點擊的確切時間移動時，移動元素會導致 Capybara 誤點擊 * [由于窗口大小和滾動位置，下拉菜單向上或向下渲染](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17660) * [延遲加載的圖像可能導致水豚誤點擊](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18713) * [Triggering JS events before the event handlers are set up](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18742) * [Wait for the image to be lazy-loaded when asserting on a Markdown image’s `src` attribute](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25408) #### Capybara viewport size related issues[](#capybara-viewport-size-related-issues "Permalink") * [Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29241#note_26743936): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10411](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10411) #### Capybara JS driver related issues[](#capybara-js-driver-related-issues "Permalink") * [Don’t wait for AJAX when no AJAX request is fired](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30461): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10454](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10454) * [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34647): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12626](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12626) #### PhantomJS / WebKit related issues[](#phantomjs--webkit-related-issues "Permalink") * 記憶在屋頂！（TL; DR：加載圖像但阻止圖像請求！）： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003) : [//gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003) #### Capybara expectation times out[](#capybara-expectation-times-out "Permalink") * [Test imports a project (via Sidekiq) that is growing over time, leading to timeouts when the import takes longer than 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22599) ## Resources[](#resources "Permalink") * [Flaky Tests: Are You Sure You Want to Rerun Them?](https://semaphoreci.com/blog/2017/04/20/flaky-tests.html) * [How to Deal With and Eliminate Flaky Tests](https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests) * [Tips on Treating Flakiness in your Rails Test Suite](https://semaphoreci.com/blog/2017/08/03/tips-on-treating-flakiness-in-your-test-suite.html) * [‘Flaky’ tests: a short story](https://www.ombulabs.com/blog/rspec/continuous-integration/how-to-track-down-a-flaky-test.html) * [Using Insights to Discover Flaky, Slow, and Failed Tests](https://s0circleci0com.icopy.site/blog/using-insights-to-discover-flaky-slow-and-failed-tests/) * * * [Return to Testing documentation](index.html)