# Geo Troubleshooting > 原文：[https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html](https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html) * [Basic troubleshooting](#basic-troubleshooting) * [Check the health of the **secondary** node](#check-the-health-of-the-secondary-node) * [Check if PostgreSQL replication is working](#check-if-postgresql-replication-is-working) * [Are nodes pointing to the correct database instance?](#are-nodes-pointing-to-the-correct-database-instance) * [Can Geo detect the current node correctly?](#can-geo-detect-the-current-node-correctly) * [Fixing errors found when running the Geo check Rake task](#fixing-errors-found-when-running-the-geo-check-rake-task) * [Fixing replication errors](#fixing-replication-errors) * [Message: `ERROR: replication slots can only be used if max_replication_slots > 0`?](#message-error--replication-slots-can-only-be-used-if-max_replication_slots--0) * [Message: `FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist`?](#message-fatal--could-not-start-wal-streaming-error--replication-slot-geo_secondary_my_domain_com-does-not-exist) * [Message: “Command exceeded allowed execution time” when setting up replication?](#message-command-exceeded-allowed-execution-time-when-setting-up-replication) * [Message: “PANIC: could not write to file `pg_xlog/xlogtemp.123`: No space left on device”](#message-panic-could-not-write-to-file-pg_xlogxlogtemp123-no-space-left-on-device) * [Message: “ERROR: canceling statement due to conflict with recovery”](#message-error-canceling-statement-due-to-conflict-with-recovery) * [Message: `LOG: invalid CIDR mask in address`](#message-log--invalid-cidr-mask-in-address) * [Message: `LOG: invalid IP mask "md5": Name or service not known`](#message-log--invalid-ip-mask-md5-name-or-service-not-known) * [Very large repositories never successfully synchronize on the **secondary** node](#very-large-repositories-never-successfully-synchronize-on-the-secondary-node) * [New LFS objects are never replicated](#new-lfs-objects-are-never-replicated) * [Resetting Geo **secondary** node replication](#resetting-geo-secondary-node-replication) * [Fixing errors during a failover or when promoting a secondary to a primary node](#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node) * [Message: ActiveRecord::RecordInvalid: Validation failed: Name has already been taken](#message-activerecordrecordinvalid-validation-failed-name-has-already-been-taken) * [Message: `NoMethodError: undefined method `secondary?' for nil:NilClass`](#message-nomethoderror-undefined-method-secondary-for-nilnilclass) * [Message: `sudo: gitlab-pg-ctl: command not found`](#message-sudo-gitlab-pg-ctl-command-not-found) * [Fixing Foreign Data Wrapper errors](#fixing-foreign-data-wrapper-errors) * [“Foreign Data Wrapper (FDW) is not configured” error](#foreign-data-wrapper-fdw-is-not-configured-error) * [Checking configuration](#checking-configuration) * [Manual reload of FDW schema](#manual-reload-of-fdw-schema) * [“Geo database has an outdated FDW remote schema” error](#geo-database-has-an-outdated-fdw-remote-schema-error) * [Expired artifacts](#expired-artifacts) * [Fixing sign in errors](#fixing-sign-in-errors) * [Message: The redirect URI included is not valid](#message-the-redirect-uri-included-is-not-valid) * [Fixing common errors](#fixing-common-errors) * [Geo database configuration file is missing](#geo-database-configuration-file-is-missing) * [An existing tracking database cannot be reused](#an-existing-tracking-database-cannot-be-reused) * [Geo node has a database that is writable which is an indication it is not configured for replication with the primary node](#geo-node-has-a-database-that-is-writable-which-is-an-indication-it-is-not-configured-for-replication-with-the-primary-node) * [Geo node does not appear to be replicating the database from the primary node](#geo-node-does-not-appear-to-be-replicating-the-database-from-the-primary-node) * [Geo database version (…) does not match latest migration (…)](#geo-database-version--does-not-match-latest-migration-) * [Geo database is not configured to use Foreign Data Wrapper](#geo-database-is-not-configured-to-use-foreign-data-wrapper) * [GitLab indicates that more than 100% of repositories were synced](#gitlab-indicates-that-more-than-100-of-repositories-were-synced) * [Geo Admin Area returns 404 error for a secondary node](#geo-admin-area-returns-404-error-for-a-secondary-node) # Geo Troubleshooting[](#geo-troubleshooting-premium-only "Permalink") 設置地理位置需要仔細注意細節，有時很容易錯過一個步驟. 這是您嘗試解決問題應采取的步驟的列表： * Perform [basic troubleshooting](#basic-troubleshooting). * 修復所有[復制錯誤](#fixing-replication-errors) . * 修復任何[外部數據包裝程序](#fixing-foreign-data-wrapper-errors)錯誤. * 修復所有[常見](#fixing-common-errors)錯誤. ## Basic troubleshooting[](#basic-troubleshooting "Permalink") 在嘗試更高級的故障排除之前： * Check [the health of the **secondary** node](#check-the-health-of-the-secondary-node). * Check [if PostgreSQL replication is working](#check-if-postgresql-replication-is-working). ### Check the health of the **secondary** node[](#check-the-health-of-the-secondary-node "Permalink") 訪問**主**節點的 **管理區>** 瀏覽器中的**地理位置** （ `/admin/geo/nodes` ）. 我們在每個**輔助**節點上執行以下運行狀況檢查，以幫助識別是否存在問題： * 節點是否在運行？ * 節點的輔助數據庫是否已配置用于流復制？ * 是否已配置節點的輔助跟蹤數據庫？ * 節點的輔助跟蹤數據庫是否已連接？ * 節點的輔助跟蹤數據庫是否是最新的？ [](img/geo_node_healthcheck.png) 有關如何解決從 UI 報告的常見錯誤的信息，請參閱" [修復常見錯誤"](#fixing-common-errors) . 如果用戶界面無法正常工作，或者您無法登錄，則可以手動運行地理健康檢查以獲取此信息以及更多詳細信息. 此 Rake 任務可以在**主要**或**輔助** Geo 節點中的應用程序節點上運行： ``` sudo gitlab-rake gitlab:geo:check ``` 輸出示例： ``` Checking Geo ... GitLab Geo is available ... yes GitLab Geo is enabled ... yes This machine's Geo node name matches a database record ... yes, found a secondary node named "Shanghai" GitLab Geo secondary database is correctly configured ... yes Database replication enabled? ... yes Database replication working? ... yes GitLab Geo tracking database is configured to use Foreign Data Wrapper? ... yes GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... yes GitLab Geo HTTP(S) connectivity ... * Can connect to the primary node ... yes HTTP/HTTPS repository cloning is enabled ... yes Machine clock is synchronized ... yes Git user has default SSH configuration? ... yes OpenSSH configured to use AuthorizedKeysCommand ... yes GitLab configured to disable writing to authorized_keys file ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Checking Geo ... Finished ``` 通過在任何**輔助**應用程序節點上運行以下 Rake 任務，可以手動找到當前同步信息： ``` sudo gitlab-rake geo:status ``` 輸出示例： ``` http://secondary.example.com/ ----------------------------------------------------- GitLab Version: 11.10.4-ee Geo Role: Secondary Health Status: Healthy Repositories: 289/289 (100%) Verified Repositories: 289/289 (100%) Wikis: 289/289 (100%) Verified Wikis: 289/289 (100%) LFS Objects: 8/8 (100%) Attachments: 5/5 (100%) CI job artifacts: 0/0 (0%) Repositories Checked: 0/289 (0%) Sync Settings: Full Database replication lag: 0 seconds Last event ID seen from primary: 10215 (about 2 minutes ago) Last event ID processed by cursor: 10215 (about 2 minutes ago) Last status report was: 2 minutes ago ``` ### Check if PostgreSQL replication is working[](#check-if-postgresql-replication-is-working "Permalink") 要檢查 PostgreSQL 復制是否正常，請檢查： * [Nodes are pointing to the correct database instance](#are-nodes-pointing-to-the-correct-database-instance). * [Geo can detect the current node correctly](#can-geo-detect-the-current-node-correctly). #### Are nodes pointing to the correct database instance?[](#are-nodes-pointing-to-the-correct-database-instance "Permalink") 您應確保**主**地理節點指向具有寫入權限的實例. 任何**輔助**節點都應僅指向只讀實例. #### Can Geo detect the current node correctly?[](#can-geo-detect-the-current-node-correctly "Permalink") Geo 通過以下方式在`/etc/gitlab/gitlab.rb`找到當前計算機的 Geo 節點名稱： * 使用`gitlab_rails['geo_node_name']`設置. * 如果未定義，請使用`external_url`設置. 該名稱用于在其中查找具有相同**名稱**的節點 **管理區>** **地理位置** . 要檢查當前計算機的節點名稱是否與數據庫中的節點匹配，請運行檢查任務： ``` sudo gitlab-rake gitlab:geo:check ``` 它顯示當前計算機的節點名稱以及匹配的數據庫記錄是**主**節點還是**輔助**節點. ``` This machine's Geo node name matches a database record ... yes, found a secondary node named "Shanghai" ``` ``` This machine's Geo node name matches a database record ... no Try fixing it: You could add or update a Geo node database record, setting the name to "https://example.com/". Or you could set this machine's Geo node name to match the name of an existing database record: "London", "Shanghai" For more information see: doc/administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-node-correctly ``` ## Fixing errors found when running the Geo check Rake task[](#fixing-errors-found-when-running-the-geo-check-rake-task "Permalink") 運行此 Rake 任務時，如果未正確配置節點，則可能會看到錯誤： ``` sudo gitlab-rake gitlab:geo:check ``` 1. 連接到數據庫時，Rails 沒有提供密碼 ``` Checking Geo ... GitLab Geo is available ... Exception: fe_sendauth: no password supplied GitLab Geo is enabled ... Exception: fe_sendauth: no password supplied ... Checking Geo ... Finished ``` * 確保將`gitlab_rails['db_password']`設置為為`postgresql['sql_user_password']`創建哈希時使用的純文本密碼. 2. Rails 無法連接到數據庫 ``` Checking Geo ... GitLab Geo is available ... Exception: FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL on FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL off GitLab Geo is enabled ... Exception: FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL on FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL off ... Checking Geo ... Finished ``` * 確保您具有`postgresql['md5_auth_cidr_addresses']`包含的 rails 節點的 IP 地址. * 確保在 IP 地址上包括子網掩碼： `postgresql['md5_auth_cidr_addresses'] = ['1.1.1.1/32']` . 3. Rails 提供了錯誤的密碼 ``` Checking Geo ... GitLab Geo is available ... Exception: FATAL: password authentication failed for user "gitlab" FATAL: password authentication failed for user "gitlab" GitLab Geo is enabled ... Exception: FATAL: password authentication failed for user "gitlab" FATAL: password authentication failed for user "gitlab" ... Checking Geo ... Finished ``` * 驗證正確的密碼設置為`gitlab_rails['db_password']`創建中的散列時所使用`postgresql['sql_user_password']`通過運行`gitlab-ctl pg-password-md5 gitlab`并輸入密碼. 4. 檢查返回的不是輔助節點 ``` Checking Geo ... GitLab Geo is available ... yes GitLab Geo is enabled ... yes GitLab Geo secondary database is correctly configured ... not a secondary node Database replication enabled? ... not a secondary node ... Checking Geo ... Finished ``` * 確保您已在**主**節點的管理區添加輔助節點. * 在管理節點中添加輔助節點屬于**主**節點時，請確保輸入了`external_url`或`gitlab_rails['geo_node_name']` . * 之前 GitLab 12.4，編輯中的**主**節點的管理區的二次節點，并確保有一個尾隨`/`在`Name`字段中. 5. 檢查返回`Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist` ``` Checking Geo ... GitLab Geo is available ... no Try fixing it: Upload a new license that includes the GitLab Geo feature For more information see: https://about.gitlab.com/features/gitlab-geo/ GitLab Geo is enabled ... Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist LINE 8: WHERE a.attrelid = '"geo_nodes"'::regclass ^ : SELECT a.attname, format_type(a.atttypid, a.atttypmod), pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod, c.collname, col_description(a.attrelid, a.attnum) AS comment FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum LEFT JOIN pg_type t ON a.atttypid = t.oid LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation WHERE a.attrelid = '"geo_nodes"'::regclass AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum ... Checking Geo ... Finished ``` 在執行 PostgreSQL 主版本（9> 10）時，這是預期的. 跟隨： * [initiate-the-replication-process](database.html#step-3-initiate-the-replication-process) * [Geo database has an outdated FDW remote schema](troubleshooting.html#geo-database-has-an-outdated-fdw-remote-schema-error) ## Fixing replication errors[](#fixing-replication-errors "Permalink") 以下各節概述了解決復制錯誤的疑難解答步驟. ### Message: `ERROR: replication slots can only be used if max_replication_slots > 0`?[](#message-error--replication-slots-can-only-be-used-if-max_replication_slots--0 "Permalink") 這意味著需要在**主**數據庫上設置`max_replication_slots` PostgreSQL 變量. 在 GitLab 9.4 中，我們將此設置默認設置為 1.如果您有更多**輔助**節點，則可能需要增加該值. 確保重新啟動 PostgreSQL 才能生效. 有關更多詳細信息，請參見《 [PostgreSQL 復制設置](database.html#postgresql-replication)指南》. ### Message: `FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist`?[](#message-fatal--could-not-start-wal-streaming-error--replication-slot-geo_secondary_my_domain_com-does-not-exist "Permalink") 當 PostgreSQL 沒有該名稱的**輔助**節點的復制插槽時，會發生這種情況. 您可能需要在**輔助**節點上重新運行[復制過程](database.html) . ### Message: “Command exceeded allowed execution time” when setting up replication?[](#message-command-exceeded-allowed-execution-time-when-setting-up-replication "Permalink") 在**輔助**節點上[啟動復制過程](database.html#step-3-initiate-the-replication-process)時可能會發生這種情況，并表明您的初始數據集太大，無法在默認超時（30 分鐘）內進行復制. 重新運行`gitlab-ctl replicate-geo-database` ，但是為`--backup-timeout`包含一個更大的值： ``` sudo gitlab-ctl \ replicate-geo-database \ --host=<primary_node_hostname> \ --slot-name=<secondary_slot_name> \ --backup-timeout=21600 ``` 這將使初始復制最多需要六個小時才能完成，而不是默認的三十分鐘. 根據安裝要求進行調整. ### Message: “PANIC: could not write to file `pg_xlog/xlogtemp.123`: No space left on device”[](#message-panic-could-not-write-to-file-pg_xlogxlogtemp123-no-space-left-on-device "Permalink") 確定**主**數據庫中是否有未使用的復制插槽. 這可能導致在`pg_xlog`建立大量的日志數據. 刪除未使用的插槽可以減少`pg_xlog`使用的空間. 1. 啟動 PostgreSQL 控制臺會話： ``` sudo gitlab-psql ``` 注意： **注意：**使用`gitlab-rails dbconsole`無效，因為管理復制插槽需要超級用戶權限. 2. 使用以下方法查看您的復制插槽： ``` SELECT * FROM pg_replication_slots; ``` `active`為`f`插槽不活動. * 當該插槽應處于活動狀態時，因為您已使用該插槽配置了**輔助**節點，請登錄到該**輔助**節點，并檢查 PostgreSQL 日志為什么復制未運行. * 如果您不再使用該插槽（例如，您不再啟用地理位置），則可以在 PostgreSQL 控制臺會話中將其刪除： ``` SELECT pg_drop_replication_slot('<name_of_extra_slot>'); ``` ### Message: “ERROR: canceling statement due to conflict with recovery”[](#message-error-canceling-statement-due-to-conflict-with-recovery "Permalink") 在正常使用情況下，此錯誤很少會發生，并且系統具有足夠的恢復能力. 但是，在某些情況下，輔助數據庫上的某些數據庫查詢可能運行時間過長，從而增加了此錯誤的發生頻率. 在某些時候，其中某些查詢由于每次都會被取消而永遠無法完成. 這些長期運行的查詢[計劃在將來刪除](https://gitlab.com/gitlab-org/gitlab/-/issues/34269) ，但作為解決方法，我們建議啟用[hot_standby_feedback](https://s0www0postgresql0org.icopy.site/docs/10/hot-standby.html) . 這增加了**主**節點上發生膨脹的可能性，因為它阻止了`VACUUM`刪除最近失效的行. 但是，它已在 GitLab.com 上成功用于生產中. 要啟用`hot_standby_feedback` ，請將以下內容添加到**輔助**節點上的`/etc/gitlab/gitlab.rb` ： ``` postgresql['hot_standby_feedback'] = 'on' ``` 然后重新配置 GitLab： ``` sudo gitlab-ctl reconfigure ``` 為了幫助我們解決這個問題，可以考慮在評論[這個問題](https://gitlab.com/gitlab-org/gitlab/-/issues/4489) . ### Message: `LOG: invalid CIDR mask in address`[](#message-log--invalid-cidr-mask-in-address "Permalink") 這發生在`postgresql['md5_auth_cidr_addresses']`格式錯誤的地址上. ``` 2020-03-20_23:59:57.60499 LOG: invalid CIDR mask in address "***" 2020-03-20_23:59:57.60501 CONTEXT: line 74 of configuration file "/var/opt/gitlab/postgresql/data/pg_hba.conf" ``` 要解決此問題，請更新`postgresql['md5_auth_cidr_addresses']`下`/etc/gitlab/gitlab.rb`的 IP 地址，以遵守 CIDR 格式（即`1.2.3.4/32` ）. ### Message: `LOG: invalid IP mask "md5": Name or service not known`[](#message-log--invalid-ip-mask-md5-name-or-service-not-known "Permalink") 當您在`postgresql['md5_auth_cidr_addresses']`添加了沒有子網掩碼的 IP 地址時，就會發生這種情況. ``` 2020-03-21_00:23:01.97353 LOG: invalid IP mask "md5": Name or service not known 2020-03-21_00:23:01.97354 CONTEXT: line 75 of configuration file "/var/opt/gitlab/postgresql/data/pg_hba.conf" ``` 要解決此問題， `/etc/gitlab/gitlab.rb`在`postgresql['md5_auth_cidr_addresses']`下`postgresql['md5_auth_cidr_addresses']`添加子網掩碼，以遵守 CIDR 格式（即`1.2.3.4/32` ）. ### Very large repositories never successfully synchronize on the **secondary** node[](#very-large-repositories-never-successfully-synchronize-on-the-secondary-node "Permalink") GitLab 對所有存儲庫克隆都設置了超時，包括項目導入和地理同步操作. 如果**主數據庫**上的存儲庫的新`git clone`花費幾分鐘以上，則您可能會受到此影響. 要增加超時，請將以下行添加到**輔助**節點上的`/etc/gitlab/gitlab.rb` ： ``` gitlab_rails['gitlab_shell_git_timeout'] = 10800 ``` 然后重新配置 GitLab： ``` sudo gitlab-ctl reconfigure ``` 這會將超時增加到三個小時（10800 秒）. 選擇足夠長的時間來容納您最大的存儲庫的完整克隆. ### New LFS objects are never replicated[](#new-lfs-objects-are-never-replicated "Permalink") 如果新的 LFS 對象永遠不會復制到輔助 Geo 節點，請檢查您正在運行的 GitLab 版本. GitLab 版本 11.11.x 或 12.0.x 受[錯誤](https://gitlab.com/gitlab-org/gitlab/-/issues/32696)影響， [該錯誤導致新的 LFS 對象無法復制到 Geo 輔助節點](https://gitlab.com/gitlab-org/gitlab/-/issues/32696) . 要解決此問題，請升級到 GitLab 12.1 或更高版本. ### Resetting Geo **secondary** node replication[](#resetting-geo-secondary-node-replication "Permalink") 如果您使**輔助**節點處于損壞狀態，并且想要重置復制狀態（從頭開始），那么可以采取以下步驟來幫助您： 1. 停止 Sidekiq 和 Geo LogCursor 可以使 Sidekiq 正常停止，但可以使其停止獲取新作業，并等到當前作業完成處理為止. 您需要在第一階段發送**SIGTSTP 終止**信號，并在所有作業完成后向它們發送**SIGTERM** . 否則，只需使用`gitlab-ctl stop`命令. ``` gitlab-ctl status sidekiq # run: sidekiq: (pid 10180) <- this is the PID you will use kill -TSTP 10180 # change to the correct PID gitlab-ctl stop sidekiq gitlab-ctl stop geo-logcursor ``` 您可以查看 Sidekiq 日志以了解 Sidekiq 作業處理何時完成： ``` gitlab-ctl tail sidekiq ``` 2. 重命名存儲庫存儲文件夾并創建新的. 如果您不擔心可能的孤立目錄和文件，則只需跳過此步驟. ``` mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old mkdir -p /var/opt/gitlab/git-data/repositories chown git:git /var/opt/gitlab/git-data/repositories ``` **提示**您可能希望將來在確認不再需要`/var/opt/gitlab/git-data/repositories.old`時將其刪除，以節省磁盤空間. 3. *（可選）*重命名其他數據文件夾并創建新的 **警告** ： **輔助**節點上可能仍有文件已從**主**節點中刪除，但未反映出刪除情況. 如果您跳過此步驟，它們將永遠不會從此 Geo 節點中刪除. 任何上載的內容（如文件附件，化身或 LFS 對象）都存儲在以下兩個路徑之一的子文件夾中： * `/var/opt/gitlab/gitlab-rails/shared` * `/var/opt/gitlab/gitlab-rails/uploads` 要重命名它們： ``` gitlab-ctl stop mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old mkdir -p /var/opt/gitlab/gitlab-rails/shared mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old mkdir -p /var/opt/gitlab/gitlab-rails/uploads gitlab-ctl start geo-postgresql ``` 重新配置以便重新創建文件夾并確保權限和所有權正確 ``` gitlab-ctl reconfigure ``` 4. 重置跟蹤數據庫 ``` gitlab-rake geo:db:drop # on a secondary app node gitlab-ctl reconfigure # on the tracking database node gitlab-rake geo:db:setup # on a secondary app node ``` 5. 重新啟動先前停止的服務 ``` gitlab-ctl start ``` 6. 刷新外部數據包裝器表 ``` gitlab-rake geo:db:refresh_foreign_tables ``` ## Fixing errors during a failover or when promoting a secondary to a primary node[](#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node "Permalink") 以下是故障轉移期間或通過解決策略將輔助節點提升為主節點時可能遇到的錯誤. ### Message: ActiveRecord::RecordInvalid: Validation failed: Name has already been taken[](#message-activerecordrecordinvalid-validation-failed-name-has-already-been-taken "Permalink") [升級**輔助**節點時](../disaster_recovery/index.html#step-3-promoting-a-secondary-node) ，您可能會遇到以下錯誤： ``` Running gitlab-rake geo:set_secondary_as_primary... rake aborted! ActiveRecord::RecordInvalid: Validation failed: Name has already been taken /opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:236:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:221:in `block (2 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Tasks: TOP => geo:set_secondary_as_primary (See full trace by running task with --trace) You successfully promoted this node! ``` 如果您在運行`gitlab-rake geo:set_secondary_as_primary`或`gitlab-ctl promote-to-primary-node`時遇到此消息，請執行`gitlab-rake geo:set_secondary_as_primary`任一操作： * 輸入 Rails 控制臺并運行： ``` Rails.application.load_tasks; nil Gitlab::Geo.expire_cache! Rake::Task['geo:set_secondary_as_primary'].invoke ``` * 如果安全的話，升級到 GitLab 12.6.3 或更高版本. 例如，如果故障轉移只是一個測試. [緩存相關的錯誤](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021)是固定的. ### Message: `NoMethodError: undefined method `secondary?' for nil:NilClass`[](#message-nomethoderror-undefined-method-secondary-for-nilnilclass "Permalink") [升級**輔助**節點時](../disaster_recovery/index.html#step-3-promoting-a-secondary-node) ，您可能會遇到以下錯誤： ``` sudo gitlab-rake geo:set_secondary_as_primary rake aborted! NoMethodError: undefined method `secondary?' for nil:NilClass /opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:232:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:221:in `block (2 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Tasks: TOP => geo:set_secondary_as_primary (See full trace by running task with --trace) ``` 該命令僅應在輔助節點上執行，如果嘗試在主節點上運行此命令，則會顯示此錯誤. ### Message: `sudo: gitlab-pg-ctl: command not found`[](#message-sudo-gitlab-pg-ctl-command-not-found "Permalink") When [promoting a **secondary** node with multiple servers](../disaster_recovery/index.html#promoting-a-secondary-node-with-multiple-servers), you need to run the `gitlab-pg-ctl` command to promote the PostgreSQL read-replica database. 在 GitLab 12.8 和更早版本中，此命令將失敗，并顯示以下消息： ``` sudo: gitlab-pg-ctl: command not found ``` 在這種情況下，解決方法是使用二進制文件的完整路徑，例如： ``` sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote ``` GitLab 12.9 及更高版本[不受此錯誤影響](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5147) . ## Fixing Foreign Data Wrapper errors[](#fixing-foreign-data-wrapper-errors "Permalink") 本節介紹解決潛在的外部數據包裝器錯誤的方法. ### “Foreign Data Wrapper (FDW) is not configured” error[](#foreign-data-wrapper-fdw-is-not-configured-error "Permalink") When setting up Geo, you might see this warning in the `gitlab-rake gitlab:geo:check` output: ``` GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... foreign data wrapper is not configured ``` 有幾點要記住： 1. FDW 設置在地理**跟蹤**數據庫上配置. 2. 配置的外部服務器允許登錄到 Geo **二級**只讀數據庫. 默認情況下，Geo 輔助數據庫和跟蹤數據庫在不同端口上的同一主機上運行. 即分別是 5432 和 5431. #### Checking configuration[](#checking-configuration "Permalink") **注意：**以下步驟僅適用于 Omnibus 安裝. 在 GitLab 11.5 中**已不建議**在基于源的安裝中使用 Geo. 要檢查配置： 1. SSH 進入**輔助**節點中的應用程序節點： ``` sudo -i ``` 注意：應用程序節點是指至少運行以下服務之一的任何計算機： * `puma` * `unicorn` * `sidekiq` * `geo-logcursor` 2. 輸入數據庫控制臺： 如果跟蹤數據庫在同一節點上運行： ``` gitlab-geo-psql ``` 或者，如果跟蹤數據庫在其他節點上運行，則在進入數據庫控制臺時必須指定用戶和主機： ``` gitlab-geo-psql -U gitlab_geo -h <IP of tracking database> ``` 系統將提示您輸入`gitlab_geo`用戶的密碼. 您可以在`/etc/gitlab/gitlab.rb`以純文本`/etc/gitlab/gitlab.rb`找到它： ``` geo_secondary['db_password'] = '<geo_tracking_db_password>' ``` 通常在[步驟 3：在輔助節點上配置跟蹤數據庫時在跟蹤數據庫上](multiple_servers.html#step-3-configure-the-tracking-database-on-the-secondary-node)設置此密碼，而在[步驟 4：在輔助節點上配置前端應用程序服務器](multiple_servers.html#step-4-configure-the-frontend-application-servers-on-the-secondary-node)期間在應用程序節點上設置該密碼. 3. 使用以下語句檢查是否存在任何表： ``` SELECT * from information_schema.foreign_tables; ``` 如果一切正常，您應該看到類似以下內容： ``` gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_name -------------------------+----------------------+-------------------------------------------------+-------------------------+--------------------- gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary <snip> ``` 但是，如果查詢返回`0 rows` ，則繼續進行下一步. 4. 通過`\des+`檢查外部服務器映射是否正確. 結果應如下所示： ``` gitlabhq_geo_production=# \des+ List of foreign servers -[ RECORD 1 ]--------+------------------------------------------------------------ Name | gitlab_secondary Owner | gitlab-psql Foreign-data wrapper | postgres_fdw Access privileges | "gitlab-psql"=U/"gitlab-psql" + | gitlab_geo=U/"gitlab-psql" Type | Version | FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') Description | ``` **注意：請**特別注意 FDW 選項下的主機和端口. 該配置應指向地理輔助數據庫. 如果需要嘗試更改主機或密碼，則以下查詢演示了如何： ``` ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>'); ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); ``` 如果更改主機和/或端口，則還必須在`/etc/gitlab/gitlab.rb`調整以下設置并運行`gitlab-ctl reconfigure` ： * `gitlab_rails['db_host']` * `gitlab_rails['db_port']` 5. 檢查通過`\deu+`是否正確配置了用戶映射： ``` gitlabhq_geo_production=# \deu+ List of user mappings Server | User name | FDW Options ------------------+------------+-------------------------------------------------------------------------------- gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') (1 row) ``` 確保密碼正確. 您可以通過運行`psql`來測試登錄是否有效： ``` # Connect to the tracking database as the `gitlab_geo` user sudo \ -u git /opt/gitlab/embedded/bin/psql \ -h /var/opt/gitlab/geo-postgresql \ -p 5431 \ -U gitlab_geo \ -W \ -d gitlabhq_geo_production ``` 如果需要更正密碼，以下查詢將顯示如何： ``` ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>'); ``` 如果更改用戶或密碼，則還必須在`/etc/gitlab/gitlab.rb`調整以下設置并運行`gitlab-ctl reconfigure` ： * `gitlab_rails['db_username']` * `gitlab_rails['db_password']` 如果要[在輔助數據庫前面](database.html#pgbouncer-support-optional)使用[PgBouncer](database.html#pgbouncer-support-optional) ，請確保更新以下設置： * `geo_postgresql['fdw_external_user']` * `geo_postgresql['fdw_external_password']` #### Manual reload of FDW schema[](#manual-reload-of-fdw-schema "Permalink") 如果仍然無法使 FDW 正常工作，則可能需要嘗試手動重新加載 FDW 模式. 手動重新加載 FDW 模式： 1. 在運行 Geo 跟蹤數據庫的節點上，通過`gitlab_geo`用戶進入 PostgreSQL 控制臺： ``` sudo \ -u git /opt/gitlab/embedded/bin/psql \ -h /var/opt/gitlab/geo-postgresql \ -p 5431 \ -U gitlab_geo \ -W \ -d gitlabhq_geo_production ``` 確保為您的配置調整端口和主機名. 可能會要求您輸入密碼. 2. 通過以下方式重新加載架構： ``` DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; CREATE SCHEMA gitlab_secondary; GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; ``` 3. 測試查詢是否有效： ``` SELECT * from information_schema.foreign_tables; SELECT * FROM gitlab_secondary.projects limit 1; ``` ### “Geo database has an outdated FDW remote schema” error[](#geo-database-has-an-outdated-fdw-remote-schema-error "Permalink") GitLab 可能會因`Geo database has an outdated FDW remote schema`錯誤`Geo database has an outdated FDW remote schema`消息而出錯. 例如： ``` Geo database has an outdated FDW remote schema. It contains 229 of 236 expected tables. Please refer to Geo Troubleshooting. ``` 要解決此問題，請在**輔助服務器**上運行以下命令： ``` sudo gitlab-rake geo:db:refresh_foreign_tables ``` ## Expired artifacts[](#expired-artifacts "Permalink") 如果您出于某種原因注意到，地理輔助節點上的工件比地理主節點上的工件更多，則可以使用 Rake 任務來[清理孤立的工件文件](../../../raketasks/cleanup.html#remove-orphan-artifact-files) . 在 Geo **輔助**節點上，此命令還將清除與磁盤上的孤立文件有關的所有 Geo 注冊表記錄. ## Fixing sign in errors[](#fixing-sign-in-errors "Permalink") ### Message: The redirect URI included is not valid[](#message-the-redirect-uri-included-is-not-valid "Permalink") 如果您能夠登錄到**主**節點，但是在嘗試登錄到**輔助**節點時收到此錯誤，則應檢查 Geo 節點的 URL 是否與其外部 URL 匹配. 1. 首先，訪問 **管理區>** **地理位置** . 2. 找到受影響的**輔助節點** ，然后單擊" **編輯"** . 3. 確保**URL**字段與在**輔助**節點的前端服務器上`external_url "https://gitlab.example.com"`中的`/etc/gitlab/gitlab.rb`中找到的值匹配. ## Fixing common errors[](#fixing-common-errors "Permalink") 本部分介紹了管理界面中報告的常見錯誤以及如何修復它們. ### Geo database configuration file is missing[](#geo-database-configuration-file-is-missing "Permalink") GitLab 找不到或沒有訪問`database_geo.yml`配置文件的權限. 在 Omnibus GitLab 安裝中，該文件應位于`/var/opt/gitlab/gitlab-rails/etc` . 如果它不存在或對其進行了無意的更改，請運行`sudo gitlab-ctl reconfigure`將其恢復到正確的狀態. 如果此路徑安裝在遠程卷上，請檢查您的卷配置，并具有正確的權限. ### An existing tracking database cannot be reused[](#an-existing-tracking-database-cannot-be-reused "Permalink") Geo cannot reuse an existing tracking database. 最安全的方式是使用全新的[輔助節點](#resetting-geo-secondary-node-replication) ，或通過遵循[重置地理輔助節點復制來重置](#resetting-geo-secondary-node-replication)整個[輔助節點](#resetting-geo-secondary-node-replication) . ### Geo node has a database that is writable which is an indication it is not configured for replication with the primary node[](#geo-node-has-a-database-that-is-writable-which-is-an-indication-it-is-not-configured-for-replication-with-the-primary-node "Permalink") 此錯誤是指 Geo 希望**輔助**節點上的數據庫副本存在問題. 它通常意味著： * 使用了不受支持的復制方法（例如，邏輯復制）. * 設置[地理數據庫復制](database.html)的說明未正確遵循. * 您的數據庫連接詳細信息不正確，即您在`/etc/gitlab/gitlab.rb`文件中指定了錯誤的用戶. 與**輔助**節點混淆的一個常見原因是它需要兩個單獨的 PostgreSQL 實例： * **主**節點的只讀副本. * 包含復制元數據的常規可寫實例. 即，地理位置跟蹤數據庫. ### Geo node does not appear to be replicating the database from the primary node[](#geo-node-does-not-appear-to-be-replicating-the-database-from-the-primary-node "Permalink") 導致數據庫無法正確復制的最常見問題是： * **Secondary** nodes cannot reach the **primary** node. Check credentials, firewall rules, etc. * SSL 證書問題. 確保從**主**節點復制了`/etc/gitlab/gitlab-secrets.json` . * 數據庫存儲磁盤已滿. * 數據庫復制插槽配置錯誤. * 數據庫未使用復制插槽或其他替代方法，由于已清除 WAL 文件，因此無法追趕. 確保按照[地理數據庫復制](database.html)說明進行支持的配置. ### Geo database version (…) does not match latest migration (…)[](#geo-database-version--does-not-match-latest-migration- "Permalink") 如果您使用的是 Omnibus GitLab 安裝，則升級期間可能會失敗. 您可以： * Run `sudo gitlab-ctl reconfigure`. * 通過運行以下`sudo gitlab-rake geo:db:migrate`手動觸發數據庫遷移： `sudo gitlab-rake geo:db:migrate`作為**輔助**節點上的 root 用戶. ### Geo database is not configured to use Foreign Data Wrapper[](#geo-database-is-not-configured-to-use-foreign-data-wrapper "Permalink") 此錯誤表示地理位置跟蹤數據庫未配置 FDW 服務器和憑據. See [“Foreign Data Wrapper (FDW) is not configured” error?](#foreign-data-wrapper-fdw-is-not-configured-error). ### GitLab indicates that more than 100% of repositories were synced[](#gitlab-indicates-that-more-than-100-of-repositories-were-synced "Permalink") 這可能是由項目注冊表中的孤立記錄引起的. 您可以[使用 Rake 任務](../../../administration/raketasks/geo.html#remove-orphaned-project-registries)清除它們. ### Geo Admin Area returns 404 error for a secondary node[](#geo-admin-area-returns-404-error-for-a-secondary-node "Permalink") 有時`sudo gitlab-rake gitlab:geo:check`指示**輔助**節點運行`sudo gitlab-rake gitlab:geo:check` ，但是在**主**節點的地理管理區域中返回了**輔助**節點 404 錯誤. 解決此問題的方法： * 嘗試使用`sudo gitlab-ctl restart` **輔助** `sudo gitlab-ctl restart` . * 檢查`/var/log/gitlab/gitlab-rails/geo.log`以查看**輔助**節點是否正在使用 IPv6 將其狀態發送到**主要**節點. 如果是這樣，請使用`/etc/hosts`文件中的 IPv4 將條目添加到**主**節點. 或者，您應該[在**主**節點上啟用 IPv6](https://docs.gitlab.com/omnibus/settings/nginx.html) .