Geo self-service framework (alpha) · Gitlab 中文文檔

# Geo self-service framework (alpha) > 原文：[https://docs.gitlab.com/ee/development/geo/framework.html](https://docs.gitlab.com/ee/development/geo/framework.html) * [Nomenclature](#nomenclature) * [Geo Domain-Specific Language](#geo-domain-specific-language) * [The replicator](#the-replicator) * [Linking to a model](#linking-to-a-model) * [API](#api) * [Library](#library) * [Existing Replicator Strategies](#existing-replicator-strategies) * [Blob Replicator Strategy](#blob-replicator-strategy) * [Replication](#replication) * [Verification](#verification) * [Metrics](#metrics) * [GraphQL API](#graphql-api) * [Admin UI](#admin-ui) # Geo self-service framework (alpha)[](#geo-self-service-framework-alpha "Permalink") **注意：**本文檔可能會隨時更改. 這是我們正在研究的建議，一旦實施完成，此文檔將得到更新. 跟隨[史詩般的](https://gitlab.com/groups/gitlab-org/-/epics/2161)進度.**注意：** Geo 自助服務框架當前處于 Alpha 狀態. 如果您需要復制新的數據類型，請與 Geo 小組聯系以討論選項. 您可以在 Slack 的`#g_geo`與他們聯系，或在問題或合并請求中提及`@geo-team` . Geo 提供了一個 API，使跨 Geo 節點輕松復制數據類型成為可能. 該 API 以 Ruby 域特定語言（DSL）的形式呈現，旨在使創建數據類型的工程師只需花費很少的精力即可復制數據. ## Nomenclature[](#nomenclature "Permalink") 在深入研究 API 之前，開發人員需要了解一些特定于地理位置的命名約定. Model 模型是活動模型，在整個 Rails 代碼庫中都是如此. 它通常與數據庫表綁定. 從地理角度來看，模型可以具有一個或多個資源. Resource 資源是屬于模型的一條數據，由 GitLab 功能生成. 使用存儲機制將其持久化. 默認情況下，資源不可復制. Data type Data type is how a resource is stored. Each resource should fit in one of the data types Geo supports: :- Git repository :- Blob :- Database 有關更多詳細信息，請參見[數據類型](../../administration/geo/replication/datatypes.html) . Geo Replicable 可復制資源是 Geo 希望在 Geo 節點之間同步的資源. 受支持的可復制數據類型有限. 實現屬于已知數據類型之一的資源的復制所需的工作量很小. Geo Replicator 地理復制器是知道如何復制可復制對象的對象. 它負責：：-觸發事件（生產者）：-消費事件（消費者）它與 Geo Replicable 數據類型相關. 所有復制器都有一個公共接口，可用于處理（即產生和使用）事件. 它負責主節點（產生事件的地方）和次節點（消耗事件的地方）之間的通信. 想要將 Geo 納入其功能的工程師將使用復制器的 API 來實現這一目標. Geo Domain-Specific Language 語法糖使工程師可以輕松指定應復制哪些資源以及如何復制. ## Geo Domain-Specific Language[](#geo-domain-specific-language "Permalink") ### The replicator[](#the-replicator "Permalink") 首先，您需要編寫一個復制器. 復制器位于[`ee/app/replicators/geo`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/app/replicators/geo) . 對于每個需要復制的資源，即使多個資源綁定到同一模型，也應指定一個單獨的復制器. 例如，以下復制器復制軟件包文件： ``` module Geo class PackageFileReplicator < Gitlab::Geo::Replicator # Include one of the strategies your resource needs include ::Geo::BlobReplicatorStrategy # Specify the CarrierWave uploader needed by the used strategy def carrierwave_uploader model_record.file end # Specify the model this replicator belongs to def self.model ::Packages::PackageFile end end end ``` 類名應該是唯一的. 它還與注冊表的表名緊密相關，因此在此示例中，注冊表表將為`package_file_registry` . 對于不同的數據類型，Geo 支持包括不同的策略. 選擇一個適合您的需求. ### Linking to a model[](#linking-to-a-model "Permalink") 要將此復制器綁定到模型，需要在模型代碼中添加以下內容： ``` class Packages::PackageFile < ApplicationRecord include ::Gitlab::Geo::ReplicableModel with_replicator Geo::PackageFileReplicator end ``` ### API[](#api "Permalink") 設置好后，可以通過模型輕松訪問復制器： ``` package_file = Packages::PackageFile.find(4) # just a random id as example replicator = package_file.replicator ``` 或者從復制器取回模型： ``` replicator.model_record => <Packages::PackageFile id:4> ``` 復制器可用于生成事件，例如在 ActiveRecord 掛鉤中： ``` after_create_commit -> { replicator.publish_created_event } ``` #### Library[](#library "Permalink") 所有這些背后的框架位于[`ee/lib/gitlab/geo/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/geo) . ## Existing Replicator Strategies[](#existing-replicator-strategies "Permalink") 在編寫一種新的復制器策略之前，請檢查以下內容，以查看現有策略之一是否已經可以處理您的資源. 如果不確定，請咨詢地理團隊. ### Blob Replicator Strategy[](#blob-replicator-strategy "Permalink") 使用`Geo::BlobReplicatorStrategy`模塊，Geo 可以輕松支持使用[CarrierWave 的](https://github.com/carrierwaveuploader/carrierwave) `Uploader::Base`模型. 首先，每個文件應具有其自己的主要 ID 和模型. Geo 強烈建議將*每個文件都*視為頭等公民，因為根據我們的經驗，這大大簡化了跟蹤復制和驗證狀態. 例如，要添加對具有`Widget` `widgets`表的`Widget`模型引用的文件的支持，您將執行以下步驟： #### Replication[](#replication "Permalink") 1. 在`Widget`類中包含`Gitlab::Geo::ReplicableModel` ，并使用`with_replicator Geo::WidgetReplicator`指定 Replicator 類. 此時， `Widget`類應如下所示： ``` # frozen_string_literal: true class Widget < ApplicationRecord include ::Gitlab::Geo::ReplicableModel with_replicator Geo::WidgetReplicator mount_uploader :file, WidgetUploader def self.replicables_for_geo_node # Should be implemented. The idea of the method is to restrict # the set of synced items depending on synchronization settings end ... end ``` 2. 創建`ee/app/replicators/geo/widget_replicator.rb` . 實現`#carrierwave_uploader`方法，該方法應返回`CarrierWave::Uploader` . 并實現類方法`.model`以返回`Widget`類. ``` # frozen_string_literal: true module Geo class WidgetReplicator < Gitlab::Geo::Replicator include ::Geo::BlobReplicatorStrategy def self.model ::Widget end def carrierwave_uploader model_record.file end end end ``` 3. 創建`ee/spec/replicators/geo/widget_replicator_spec.rb`并執行必要的設置，以定義共享示例的`model_record`變量. ``` # frozen_string_literal: true require 'spec_helper' RSpec.describe Geo::WidgetReplicator do let(:model_record) { build(:widget) } it_behaves_like 'a blob replicator' end ``` 4. 創建`widget_registry`表，以便 Geo 次要對象可以跟蹤每個 Widget 文件的同步和驗證狀態： ``` # frozen_string_literal: true class CreateWidgetRegistry < ActiveRecord::Migration[6.0] DOWNTIME = false disable_ddl_transaction! def up unless table_exists?(:widget_registry) ActiveRecord::Base.transaction do create_table :widget_registry, id: :bigserial, force: :cascade do |t| t.integer :widget_id, null: false t.integer :state, default: 0, null: false, limit: 2 t.integer :retry_count, default: 0, limit: 2 t.text :last_sync_failure t.datetime_with_timezone :retry_at t.datetime_with_timezone :last_synced_at t.datetime_with_timezone :created_at, null: false t.index :widget_id t.index :retry_at t.index :state end end end add_text_limit :widget_registry, :last_sync_failure, 255 end def down drop_table :widget_registry end end ``` 5. Create `ee/app/models/geo/widget_registry.rb`: ``` # frozen_string_literal: true class Geo::WidgetRegistry < Geo::BaseRegistry include Geo::ReplicableRegistry MODEL_CLASS = ::Widget MODEL_FOREIGN_KEY = :widget_id belongs_to :widget, class_name: 'Widget' end ``` 方法`has_create_events?` 在大多數情況下應該返回`true` . 但是，如果您添加的實體沒有創建事件，則根本不要添加該方法. 6. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`. 7. Create `ee/spec/factories/geo/widget_registry.rb`: ``` # frozen_string_literal: true FactoryBot.define do factory :geo_widget_registry, class: 'Geo::WidgetRegistry' do widget state { Geo::WidgetRegistry.state_value(:pending) } trait :synced do state { Geo::WidgetRegistry.state_value(:synced) } last_synced_at { 5.days.ago } end trait :failed do state { Geo::WidgetRegistry.state_value(:failed) } last_synced_at { 1.day.ago } retry_count { 2 } last_sync_failure { 'Random error' } end trait :started do state { Geo::WidgetRegistry.state_value(:started) } last_synced_at { 1.day.ago } retry_count { 0 } end end end ``` 8. Create `ee/spec/models/geo/widget_registry_spec.rb`: ``` # frozen_string_literal: true require 'spec_helper' RSpec.describe Geo::WidgetRegistry, :geo, type: :model do let_it_be(:registry) { create(:geo_widget_registry) } specify 'factory is valid' do expect(registry).to be_valid end include_examples 'a Geo framework registry' describe '.find_registry_differences' do ... # To be implemented end end ``` 小部件現在應該由 Geo 復制！ #### Verification[](#verification "Permalink") 1. 將驗證狀態字段添加到`widgets`表中，以便 Geo 主數據庫可以跟蹤驗證狀態： ``` # frozen_string_literal: true class AddVerificationStateToWidgets < ActiveRecord::Migration[6.0] DOWNTIME = false def change add_column :widgets, :verification_retry_at, :datetime_with_timezone add_column :widgets, :verified_at, :datetime_with_timezone add_column :widgets, :verification_checksum, :binary, using: 'verification_checksum::bytea' add_column :widgets, :verification_failure, :string add_column :widgets, :verification_retry_count, :integer end end ``` 2. 在`verification_failure`和`verification_checksum`上添加部分索引，以確保可以高效執行重新驗證： ``` # frozen_string_literal: true class AddVerificationFailureIndexToWidgets < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers DOWNTIME = false disable_ddl_transaction! def up add_concurrent_index :widgets, :verification_failure, where: "(verification_failure IS NOT NULL)", name: "widgets_verification_failure_partial" add_concurrent_index :widgets, :verification_checksum, where: "(verification_checksum IS NOT NULL)", name: "widgets_verification_checksum_partial" end def down remove_concurrent_index :widgets, :verification_failure remove_concurrent_index :widgets, :verification_checksum end end ``` 要做的事情：在二級服務器上添加驗證. 這應作為以下內容的一部分完成[：Geo：自助服務框架-包文件驗證的首次實現](https://gitlab.com/groups/gitlab-org/-/epics/1817) 小部件現在應由 Geo 驗證！ #### Metrics[](#metrics "Permalink") 指標由`Geo::MetricsUpdateWorker`收集，保存在`GeoNodeStatus`以顯示在 UI 中，然后發送給 Prometheus. 1. 將字段`widget_count` ， `widget_checksummed_count` ， `widget_checksum_failed_count` ， `widget_synced_count` ， `widget_failed_count`和`widget_registry_count`到`ee/app/models/geo_node_status.rb` `GeoNodeStatus#RESOURCE_STATUS_FIELDS`數組中. 2. 將相同的字段添加到`ee/app/models/geo_node_status.rb` `GeoNodeStatus#PROMETHEUS_METRICS`哈希中. 3. 將相同字段添加到`doc/administration/monitoring/prometheus/gitlab_metrics.md` `Sidekiq metrics`表中. 4. 將相同的字段添加到`doc/api/geo_nodes.md` `GET /geo_nodes/status`示例響應中. 5. 將相同的字段添加到`ee/spec/models/geo_node_status_spec.rb`和`ee/spec/factories/geo_node_statuses.rb` . 6. Set `widget_count` in `GeoNodeStatus#load_data_from_current_node`: ``` self.widget_count = Geo::WidgetReplicator.primary_total_count ``` 7. 添加`GeoNodeStatus#load_widgets_data`來設置`widget_synced_count` ， `widget_failed_count`和`widget_registry_count` ： ``` def load_widget_data self.widget_synced_count = Geo::WidgetReplicator.synced_count self.widget_failed_count = Geo::WidgetReplicator.failed_count self.widget_registry_count = Geo::WidgetReplicator.registry_count end ``` 8. Call `GeoNodeStatus#load_widgets_data` in `GeoNodeStatus#load_secondary_data`. 9. Set `widget_checksummed_count` and `widget_checksum_failed_count` in `GeoNodeStatus#load_verification_data`: ``` self.widget_checksummed_count = Geo::WidgetReplicator.checksummed_count self.widget_checksum_failed_count = Geo::WidgetReplicator.checksum_failed_count ``` 小部件復制和驗證指標現在應該可以在 API，管理區域 UI 和 Prometheus 中使用！ #### GraphQL API[](#graphql-api "Permalink") 1. 在`ee/app/graphql/types/geo/geo_node_type.rb`向`GeoNodeType`添加一個新字段： ``` field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type, null: true, resolver: ::Resolvers::Geo::WidgetRegistriesResolver, description: 'Find widget registries on this Geo node', feature_flag: :geo_self_service_framework ``` 2. 新添加`widget_registries`字段名的`expected_fields`在陣列`ee/spec/graphql/types/geo/geo_node_type_spec.rb` . 3. Create `ee/app/graphql/resolvers/geo/widget_registries_resolver.rb`: ``` # frozen_string_literal: true module Resolvers module Geo class WidgetRegistriesResolver < BaseResolver include RegistriesResolver end end end ``` 4. Create `ee/spec/graphql/resolvers/geo/widget_registries_resolver_spec.rb`: ``` # frozen_string_literal: true require 'spec_helper' RSpec.describe Resolvers::Geo::WidgetRegistriesResolver do it_behaves_like 'a Geo registries resolver', :geo_widget_registry end ``` 5. Create `ee/app/finders/geo/widget_registry_finder.rb`: ``` # frozen_string_literal: true module Geo class WidgetRegistryFinder include FrameworkRegistryFinder end end ``` 6. Create `ee/spec/finders/geo/widget_registry_finder_spec.rb`: ``` # frozen_string_literal: true require 'spec_helper' RSpec.describe Geo::WidgetRegistryFinder do it_behaves_like 'a framework registry finder', :geo_widget_registry end ``` 7. Create `ee/app/graphql/types/geo/widget_registry_type.rb`: ``` # frozen_string_literal: true module Types module Geo # rubocop:disable Graphql/AuthorizeTypes because it is included class WidgetRegistryType < BaseObject include ::Types::Geo::RegistryType graphql_name 'WidgetRegistry' description 'Represents the sync and verification state of a widget' field :widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Widget' end end end ``` 8. Create `ee/spec/graphql/types/geo/widget_registry_type_spec.rb`: ``` # frozen_string_literal: true require 'spec_helper' RSpec.describe GitlabSchema.types['WidgetRegistry'] do it_behaves_like 'a Geo registry type' it 'has the expected fields (other than those included in RegistryType)' do expected_fields = %i[widget_id] expect(described_class).to have_graphql_fields(*expected_fields).at_least end end ``` 9. Add integration tests for providing Widget registry data to the frontend via the GraphQL API, by duplicating and modifying the following shared examples in `ee/spec/requests/api/graphql/geo/registries_spec.rb`: ``` it_behaves_like 'gets registries for', { field_name: 'widgetRegistries', registry_class_name: 'WidgetRegistry', registry_factory: :geo_widget_registry, registry_foreign_key_field_name: 'widgetId' } ``` 現在應該可以通過 GraphQL API 獲得各個小部件同步和驗證數據！ 1. 注意復制"更新"事件. Geo Framework 目前不支持復制"更新"事件，因為此時添加到框架的所有實體都是不可變的. 如果您要添加的實體屬于這種情況，請遵循[https://gitlab.com/gitlab-org/gitlab/-/issues/118743](https://gitlab.com/gitlab-org/gitlab/-/issues/118743)和[https://gitlab.com/gitlab-org/gitlab /// issues / 118745](https://gitlab.com/gitlab-org/gitlab/-/issues/118745)作為添加新事件類型的示例. 添加通知后，請同時刪除它. #### Admin UI[](#admin-ui "Permalink") 要做的事情：這應該作為《 [地理手冊》的](https://gitlab.com/groups/gitlab-org/-/epics/2525)一部分完成[：實現自助服務框架可復制的前端](https://gitlab.com/groups/gitlab-org/-/epics/2525) 窗口小部件同步和驗證數據（總計和個人）現在應該在管理界面中可用！