9.6 使用Python CFFI混合C，C++，Fortran和Python · cmake-cookbook

# 9.6 使用Python CFFI混合C，C++，Fortran和Python **NOTE**:*此示例代碼可以在 https://github.com/dev-cafe/cmake-cookbook/tree/v1.0/chapter-9/recipe-06 中找到，其中有一個C++示例和一個Fortran示例。該示例在CMake 3.11版(或更高版本)中是有效的，并且已經在GNU/Linux、macOS和Windows上進行過測試。* 前面的三個示例中，我們使用Cython、Boost.Python和pybind11作為連接Python和C++的工具。之前的示例中，主要連接的是C++接口。然而，可能會遇到這樣的情況：將Python與Fortran或其他語言進行接口。本示例中，我們將使用Python C的外部函數接口(CFFI，參見https://cffi.readthedocs.io)。由于C是通用語言，大多數編程語言(包括Fortran)都能夠與C接口進行通信，所以Python CFFI是將Python與大量語言結合在一起的工具。Python CFFI的特性是，生成簡單且非侵入性的C接口，這意味著它既不限制語言特性中的Python層，也不會對C層以下的代碼有任何限制。本示例中，將使用前面示例的銀行帳戶示例，通過C接口將Python CFFI應用于Python和C++。我們的目標是實現一個上下文感知的接口。接口中，我們可以實例化幾個銀行帳戶，每個帳戶都帶有其內部狀態。我們將通過討論如何使用Python CFFI來連接Python和Fortran來結束本教程。第11章第3節中，通過PyPI分發一個用CMake/CFFI構建的C/Fortran/Python項目，屆時我們將重新討論這個例子，并展示如何打包它，使它可以用`pip`安裝。 ## 準備工作我們從C++實現和接口開始，把它們放在名為`account/implementation`的子目錄中。實現文件(`cpp_implementation.cpp`)類似于之前的示例，但是包含有斷言，因為我們將對象的狀態保持在一個不透明的句柄中，所以必須確保對象在訪問時已經創建： ```c++ #include "cpp_implementation.hpp" #include <cassert> Account::Account() { balance = 0.0; is_initialized = true; } Account::~Account() { assert(is_initialized); is_initialized = false; } void Account::deposit(const double amount) { assert(is_initialized); balance += amount; } void Account::withdraw(const double amount) { assert(is_initialized); balance -= amount; } double Account::get_balance() const { assert(is_initialized); return balance; } ``` 接口文件(` cpp_implementation.hpp `)包含如下內容： ```c++ #pragma once class Account { public: Account(); ~Account(); void deposit(const double amount); void withdraw(const double amount); double get_balance() const; private: double balance; bool is_initialized; }; ``` 此外，我們隔離了C-C++接口(`c_cpp_interface.cpp`)。這將是我們與Python CFFI連接的接口： ```c++ #include "account.h" #include "cpp_implementation.hpp" #define AS_TYPE(Type, Obj) reinterpret_cast<Type *>(Obj) #define AS_CTYPE(Type, Obj) reinterpret_cast<const Type *>(Obj) account_context_t *account_new() { return AS_TYPE(account_context_t, new Account()); } void account_free(account_context_t *context) { delete AS_TYPE(Account, context); } void account_deposit(account_context_t *context, const double amount) { return AS_TYPE(Account, context)->deposit(amount); } void account_withdraw(account_context_t *context, const double amount) { return AS_TYPE(Account, context)->withdraw(amount); } double account_get_balance(const account_context_t *context) { return AS_CTYPE(Account, context)->get_balance(); } ``` `account`目錄下，我們聲明了C接口(`account.h`): ```c++ #ifndef ACCOUNT_API #include "account_export.h" #define ACCOUNT_API ACCOUNT_EXPORT #endif #ifdef __cplusplus extern "C" { #endif struct account_context; typedef struct account_context account_context_t; ACCOUNT_API account_context_t *account_new(); ACCOUNT_API void account_free(account_context_t *context); ACCOUNT_API void account_deposit(account_context_t *context, const double amount); ACCOUNT_API void account_withdraw(account_context_t *context, const double amount); ACCOUNT_API double account_get_balance(const account_context_t *context); #ifdef __cplusplus } #endif #endif /* ACCOUNT_H_INCLUDED */ ``` 我們還描述了Python接口，將在稍后對此進行討論(`__init_ _.py`): ```python from subprocess import check_output from cffi import FFI import os import sys from configparser import ConfigParser from pathlib import Path def get_lib_handle(definitions, header_file, library_file): ffi = FFI() command = ['cc', '-E'] + definitions + [header_file] interface = check_output(command).decode('utf-8') # remove possible \r characters on windows which # would confuse cdef _interface = [l.strip('\r') for l in interface.split('\n')] ffi.cdef('\n'.join(_interface)) lib = ffi.dlopen(library_file) return lib # this interface requires the header file and library file # and these can be either provided by interface_file_names.cfg # in the same path as this file # or if this is not found then using environment variables _this_path = Path(os.path.dirname(os.path.realpath(__file__))) _cfg_file = _this_path / 'interface_file_names.cfg' if _cfg_file.exists(): config = ConfigParser() config.read(_cfg_file) header_file_name = config.get('configuration', 'header_file_name') _header_file = _this_path / 'include' / header_file_name _header_file = str(_header_file) library_file_name = config.get('configuration', 'library_file_name') _library_file = _this_path / 'lib' / library_file_name _library_file = str(_library_file) else: _header_file = os.getenv('ACCOUNT_HEADER_FILE') assert _header_file is not None _library_file = os.getenv('ACCOUNT_LIBRARY_FILE') assert _library_file is not None _lib = get_lib_handle(definitions=['-DACCOUNT_API=', '-DACCOUNT_NOINCLUDE'], header_file=_header_file, library_file=_library_file) # we change names to obtain a more pythonic API new = _lib.account_new free = _lib.account_free deposit = _lib.account_deposit withdraw = _lib.account_withdraw get_balance = _lib.account_get_balance __all__ = [ '__version__', 'new', 'free', 'deposit', 'withdraw', 'get_balance', ] ``` 我們看到，這個接口的大部分工作是通用的和可重用的，實際的接口相當薄。項目的布局為： ```shell . ├── account │ ├── account.h │ ├── CMakeLists.txt │ ├── implementation │ │ ├── c_cpp_interface.cpp │ │ ├── cpp_implementation.cpp │ │ └── cpp_implementation.hpp │ ├── __init__.py │ └── test.py └── CMakeLists.txt ``` ## 具體實施現在使用CMake來組合這些文件，形成一個Python模塊: 1. 主`CMakeLists.txt`文件包含一個頭文件。此外，根據GNU標準，設置編譯庫的位置: ```cmake # define minimum cmake version cmake_minimum_required(VERSION 3.5 FATAL_ERROR) # project name and supported language project(recipe-06 LANGUAGES CXX) # require C++11 set(CMAKE_CXX_STANDARD 11) set(CMAKE_CXX_EXTENSIONS OFF) set(CMAKE_CXX_STANDARD_REQUIRED ON) # specify where to place libraries include(GNUInstallDirs) set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR}) ``` 2. 第二步，是在`account`子目錄下包含接口和實現的定義： ```cmake # interface and sources add_subdirectory(account) ``` 3. 主`CMakeLists.txt`文件以測試定義(需要Python解釋器)結束： ```cmake # turn on testing enable_testing() # require python find_package(PythonInterp REQUIRED) # define test add_test( NAME python_test COMMAND ${CMAKE_COMMAND} -E env ACCOUNT_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR} ACCOUNT_HEADER_FILE=${CMAKE_CURRENT_SOURCE_DIR}/account/account.h ACCOUNT_LIBRARY_FILE=$<TARGET_FILE:account> ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/account/test.py ) ``` 4. ` account/CMakeLists.txt`中定義了動態庫目標： ```cmake add_library(account SHARED plementation/c_cpp_interface.cpp implementation/cpp_implementation.cpp ) target_include_directories(account PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR} ) ``` 5. 導出一個可移植的頭文件: ```cmake include(GenerateExportHeader) generate_export_header(account BASE_NAME account ) ``` 6. 使用Python-C接口進行對接: ```shell $ mkdir -p build $ cd build $ cmake .. $ cmake --build . $ ctest Start 1: python_test 1/1 Test #1: python_test ...................... Passed 0.14 sec 100% tests passed, 0 tests failed out of 1 ``` ## 工作原理雖然，之前的示例要求我們顯式地聲明Python-C接口，并將Python名稱映射到C(++)符號，但Python CFFI從C頭文件(示例中是`account.h`)推斷出這種映射。我們只需要向Python CFFI層提供描述C接口的頭文件和包含符號的動態庫。在主`CMakeLists.txt`文件中使用了環境變量集來實現這一點，這些環境變量可以在`__init__.py`中找到: ```python # ... def get_lib_handle(definitions, header_file, library_file): ffi = FFI() command = ['cc', '-E'] + definitions + [header_file] interface = check_output(command).decode('utf-8') # remove possible \r characters on windows which # would confuse cdef _interface = [l.strip('\r') for l in interface.split('\n')] ffi.cdef('\n'.join(_interface)) lib = ffi.dlopen(library_file) return lib # ... _this_path = Path(os.path.dirname(os.path.realpath(__file__))) _cfg_file = _this_path / 'interface_file_names.cfg' if _cfg_file.exists(): # we will discuss this section in chapter 11, recipe 3 else: _header_file = os.getenv('ACCOUNT_HEADER_FILE') assert _header_file is not None _library_file = os.getenv('ACCOUNT_LIBRARY_FILE') assert _library_file is not None _lib = get_lib_handle(definitions=['-DACCOUNT_API=', '-DACCOUNT_NOINCLUDE'], header_file=_header_file, library_file=_library_file) # ... ``` `get_lib_handle`函數打開頭文件(使用`ffi.cdef `)并解析加載庫(使用` ffi.dlopen`)。并返回庫對象。前面的文件是通用的，可以在不進行修改的情況下重用，用于與Python和C或使用Python CFFI的其他語言進行接口的其他項目。 `_lib`庫對象可以直接導出，這里有一個額外的步驟，使Python接口在使用時，感覺更像Python: ```python # we change names to obtain a more pythonic API new = _lib.account_new free = _lib.account_free deposit = _lib.account_deposit withdraw = _lib.account_withdraw get_balance = _lib.account_get_balance __all__ = [ '__version__', 'new', 'free', 'deposit', 'withdraw', 'get_balance', ] ``` 有了這個變化，可以將例子寫成下面的方式: ```python import account account1 = account.new() account.deposit(account1, 100.0) ``` 另一種選擇則不那么直觀: ```python from account import lib account1 = lib.account_new() lib.account_deposit(account1, 100.0) ``` 需要注意的是，如何使用API來實例化和跟蹤上下文: ```python account1 = account.new() account.deposit(account1, 10.0) account2 = account.new() account.withdraw(account1, 5.0) account.deposit(account2, 5.0) ``` 為了導入`account`的Python模塊，需要提供`ACCOUNT_HEADER_FILE`和`ACCOUNT_LIBRARY_FILE`環境變量，就像測試中那樣： ```cmake add_test( NAME python_test COMMAND ${CMAKE_COMMAND} -E env ACCOUNT_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR} ACCOUNT_HEADER_FILE=${CMAKE_CURRENT_SOURCE_DIR}/account/account.h ACCOUNT_LIBRARY_FILE=$<TARGET_FILE:account> ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/account/test.py ) ``` 第11章中，將討論如何創建一個可以用`pip`安裝的Python包，其中頭文件和庫文件將安裝在定義良好的位置，這樣就不必定義任何使用Python模塊的環境變量。討論了Python方面的接口之后，現在看下C的接口。` account.h`內容為： ```c++ struct account_context; typedef struct account_context account_context_t; ACCOUNT_API account_context_t *account_new(); ACCOUNT_API void account_free(account_context_t *context); ACCOUNT_API void account_deposit(account_context_t *context, const double amount); ACCOUNT_API void account_withdraw(account_context_t *context, const double amount); ACCOUNT_API double account_get_balance(const account_context_t *context); ``` 黑盒句柄`account_context`會保存對象的狀態。`ACCOUNT_API`定義在`account_export.h`中，由`account/interface/CMakeLists.txt`生成： ```cmake include(GenerateExportHeader) generate_export_header(account BASE_NAME account ) ``` ` account_export.h `頭文件定義了接口函數的可見性，并確保這是以一種可移植的方式完成的，實現可以在`cpp_implementation.cpp`中找到。它包含`is_initialized`布爾變量，可以檢查這個布爾值確保API函數按照預期的順序調用：上下文在創建之前或釋放之后都不應該被訪問。 ## 更多信息設計Python-C接口時，必須仔細考慮在哪一端分配數組：數組可以在Python端分配并傳遞給C(++)實現，也可以在返回指針的C(++)實現上分配。后一種方法適用于緩沖區大小事先未知的情況。但返回到分配給C(++)端的數組指針可能會有問題，因為這可能導致Python垃圾收集導致內存泄漏，而Python垃圾收集不會“查看”分配給它的數組。我們建議設計C API，使數組可以在外部分配并傳遞給C實現。然后，可以在`__init__.py`中分配這些數組，如下例所示： ```python from cffi import FFI import numpy as np _ffi = FFI() def return_array(context, array_len): # create numpy array array_np = np.zeros(array_len, dtype=np.float64) # cast a pointer to its data array_p = _ffi.cast("double *", array_np.ctypes.data) # pass the pointer _lib.mylib_myfunction(context, array_len, array_p) # return the array as a list return array_np.tolist() ``` `return_array`函數返回一個Python列表。因為在Python端完成了所有的分配工作，所以不必擔心內存泄漏，可以將清理工作留給垃圾收集。對于Fortran示例，讀者可以參考以下Git庫:https://github.com/dev-cafe/cmake-cookbook/tree/v1.0/chapter09/recipe06/Fortran-example 。與C++實現的主要區別在于，account庫是由Fortran 90源文件編譯而成的，我們在`account/CMakeLists.txt`中使用了Fortran 90源文件: ```cmake add_library(account SHARED implementation/fortran_implementation.f90 ) ``` 上下文保存在用戶定義的類型中: ```fortran type :: account private real(c_double) :: balance logical :: is_initialized = .false. end type ``` Fortran實現可以使用`iso_c_binding`模塊解析`account.h`中定義的符號和方法: ```fortran module account_implementation use, intrinsic :: iso_c_binding, only: c_double, c_ptr implicit none private public account_new public account_free public account_deposit public account_withdraw public account_get_balance type :: account private real(c_double) :: balance logical :: is_initialized = .false. end type contains type(c_ptr) function account_new() bind (c) use, intrinsic :: iso_c_binding, only: c_loc type(account), pointer :: f_context type(c_ptr) :: context allocate(f_context) context = c_loc(f_context) account_new = context f_context%balance = 0.0d0 f_context%is_initialized = .true. end function subroutine account_free(context) bind (c) use, intrinsic :: iso_c_binding, only: c_f_pointer type(c_ptr), value :: context type(account), pointer :: f_context call c_f_pointer(context, f_context) call check_valid_context(f_context) f_context%balance = 0.0d0 f_context%is_initialized = .false. deallocate(f_context) end subroutine subroutine check_valid_context(f_context) type(account), pointer, intent(in) :: f_context if (.not. associated(f_context)) then print *, 'ERROR: context is not associated' stop 1 end if if (.not. f_context%is_initialized) then print *, 'ERROR: context is not initialized' stop 1 end if end subroutine subroutine account_withdraw(context, amount) bind (c) use, intrinsic :: iso_c_binding, only: c_f_pointer type(c_ptr), value :: context real(c_double), value :: amount type(account), pointer :: f_context call c_f_pointer(context, f_context) call check_valid_context(f_context) f_context%balance = f_context%balance - amount end subroutine subroutine account_deposit(context, amount) bind (c) use, intrinsic :: iso_c_binding, only: c_f_pointer type(c_ptr), value :: context real(c_double), value :: amount type(account), pointer :: f_context call c_f_pointer(context, f_context) call check_valid_context(f_context) f_context%balance = f_context%balance + amount end subroutine real(c_double) function account_get_balance(context) bind (c) use, intrinsic :: iso_c_binding, only: c_f_pointer type(c_ptr), value, intent(in) :: context type(account), pointer :: f_context call c_f_pointer(context, f_context) call check_valid_context(f_context) account_get_balance = f_context%balance end function end module ``` 這個示例和解決方案的靈感來自Armin Ronacher的帖子“Beautiful Native Libraries”： http://lucumr.pocoo.org/2013/8/18/beautiful-native-libraries/