概述
分三部分，加解密框架（crypto framework），加解密设备（crypto dev），安全协议（Security Framework）
× API，设计思路等，都在加解密框架里：见文档：http://doc.dpdk.org/guides-18.11/prog_guide/cryptodev_lib.html
× 设备层的事情，加解密设备的分类，调度，主备等，见文档：http://doc.dpdk.org/guides-18.11/cryptodevs/index.html
× ESP，PDCP等网络加密协议相关的事情，主要功能是整合NIC pmd和Crypto pmd，见文档：http://doc.dpdk.org/guides-18.11/prog_guide/rte_security.html
另外还有一个追加的部分，ipsec （rte_ipsec）
* 这一部分主要是对前面三个部分的整合封装调用，用来专门处理ipsec报文。见文档：http://doc.dpdk.org/guides/prog_guide/ipsec_lib.html
画个图:
![[dev][dpdk][crypto] dpdk加解密设备与IPSEC](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwybHRaekl3TVRndVkyNWliRzluY3k1amIyMHZZbXh2Wnk4eE1ESTROVFF3THpJd01Ua3dNeTh4TURJNE5UUXdMVEl3TVRrd016QTFNVFV3TVRBMk1qUTFMVE0zTkRrNU1ETTFPQzV3Ym1jPS5qcGc%3D.jpg?w=700&webp=1 "[dev][dpdk][crypto] dpdk加解密设备与IPSEC")
如果，以上三个文档，你都读完了，那么请不要再往下读了。内容重复。且本人懒惰，多以摘要为主，写的也不好 ：）
一般编程模型
画了一张图，还没完全整理好，后续有可能会再更新，也有可能不更新了。
是参考dpdk的这个例子里的流程画的：http://doc.dpdk.org/guides/sample_app_ug/ipsec_secgw.html
初始化加密设备的一般流程。
![[dev][dpdk][crypto] dpdk加解密设备与IPSEC](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwybHRaekl3TVRndVkyNWliRzluY3k1amIyMHZZbXh2Wnk4eE1ESTROVFF3THpJd01Ua3dNeTh4TURJNE5UUXdMVEl3TVRrd016RTJNVGN5TmpJd01UUTNMVFl4T0RrMU1qazNOUzV3Ym1jPS5qcGc%3D.jpg?w=700&webp=1 "[dev][dpdk][crypto] dpdk加解密设备与IPSEC")
设备就绪后的一般使用流程。（需要关注的是：在分析rte_ipsec的源码过程中，并没有发现对API rte_crypto_op_attach_sym_session的调用。奇怪。。。）
![[dev][dpdk][crypto] dpdk加解密设备与IPSEC](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwybHRaekl3TVRndVkyNWliRzluY3k1amIyMHZZbXh2Wnk4eE1ESTROVFF3THpJd01Ua3dNeTh4TURJNE5UUXdMVEl3TVRrd016SXlNakl4TkRJd01qRTNMVEUyTlRBNE5qWTRNREF1Y0c1bi5qcGc%3D.jpg?w=700&webp=1 "[dev][dpdk][crypto] dpdk加解密设备与IPSEC")
// TODO init dpdk crypto things.
// 1. init sa->xforms like function sa_add_rules()
// 2. init two parameter ipsec_xform/crypto_xform
// 2. ipsec_sa_init()
// 2.1 rte_ipsec_sa_init()
// 3. create_session
// 3.1 rte_security_session_create()
// 3.2 rte_cryptodev_sym_session_create()
// 3.3 rte_cryptodev_sym_session_init()
// 3.4 rte_ipsec_session_prepare()
// 3.4.1 sa->ips->pkt_func = ipsec_sa_pkt_func_select()
// 3.5 rte_ipsec_pkt_process()
// 3.5.1 sa->ips->pkt_func.process()
// 3.6 rte_ipsec_pkt_crypto_prepare()
// 3.6.1 sa->ips->pkt_func.prepare()
// 7 rte_cryptodev_enqueue_burst()
// 8.rte_cryptodev_dequeue_burst()
// 9 rte_ipsec_pkt_crypto_group()
// 10 rte_ipsec_pkt_process()
// 3.5.1 sa->ips->pkt_func.process()
// outbound
// 4 rte_security_attach_session()
// 5 rte_crypto_op_attach_sym_session()
// 6 rte_security_set_pkt_metadata()
// 7.rte_cryptodev_enqueue_burst()
// 8 rte_cryptodev_dequeue_burst()
第二部分，设备（crypto dev）
所有的加解密设备大概分为以下几种：openssl，null，硬件架构相关的设备。
null为纯软件的最小实现，可以用来调试等。
硬件相关的主要包括，Intel，arm，NXP，AMD，QAT卡等。
还有一种，基于VIRTIO的PMD设备。
不同的设备，所支持的加解密算法也各有不同，有一个详细的对比列表，见：
http://doc.dpdk.org/guides-18.11/cryptodevs/overview.html
设备调度
在各设备的上层，还有一个调度设备，用于管理和协调多个加解密设备直接的数据流。叫做 cryptodev scheduler PMD
http://doc.dpdk.org/guides-18.11/cryptodevs/scheduler.html
![[dev][dpdk][crypto] dpdk加解密设备与IPSEC](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwybHRaekl3TVRndVkyNWliRzluY3k1amIyMHZZbXh2Wnk4eE1ESTROVFF3THpJd01Ua3dNeTh4TURJNE5UUXdMVEl3TVRrd016QTBNVEUwT0RJNE56a3hMVEV4TmpnMk1qUXhOemt1Y0c1bi5qcGc%3D.jpg?w=700&webp=1 "[dev][dpdk][crypto] dpdk加解密设备与IPSEC")
接口API
加解密设备的调度API（schduler）
http://doc.dpdk.org/api-18.11/rte__cryptodev__scheduler_8h.html
加解密设备的使用API （cryptodev）
http://doc.dpdk.org/api-18.11/rte__cryptodev_8h.html
第一部分 框架（crypto framework）
架构设计文档：http://doc.dpdk.org/guides-18.11/prog_guide/cryptodev_lib.html
设计理念
The cryptodev library follows the same basic principles as those used in DPDKs Ethernet Device framework.
The Crypto framework provides a generic Crypto device framework which supports both physical (hardware)
and virtual (software) Crypto devices as well as a generic Crypto API which allows Crypto devices to be
managed and configured and supports Crypto operations to be provisioned on Crypto poll mode driver.
设备初始化方法
A。 初始化设备
实体设备：与PCI网卡相同。
虚拟设备：1. 可以使用 --vdev参数在dpdk程序启动时使用。2. 在运行时使用api：
rte_vdev_init("crypto_aesni_mb",
"max_nb_queue_pairs=2,socket_id=0")
B。配置设备
设备识别：用Id或name
设备配置：使用api：
int rte_cryptodev_configure(uint8_t dev_id,
struct rte_cryptodev_config *config) struct rte_cryptodev_config {
int socket_id;
/**< Socket to allocate resources on */
uint16_t nb_queue_pairs;
/**< Number of queue pairs to configure on device */
};
C。配置queue
int rte_cryptodev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
const struct rte_cryptodev_qp_conf *qp_conf,
int socket_id) struct rte_cryptodev_qp_conf {
uint32_t nb_descriptors; /**< Number of descriptors per queue pair */
};
D。数据流程
1. 在queue pair 上burst 提取数据。
uint16_t rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
struct rte_crypto_op **ops, uint16_t nb_ops)
uint16_t rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
struct rte_crypto_op **ops, uint16_t nb_ops)
2. 私有数据的存储。分基于session的和不基于session的两种情况。
3. 关键的用于加解密操作的数据结构 rte_crypto_op, 可以把op理解为mbuf，op是mempool管理的。
![[dev][dpdk][crypto] dpdk加解密设备与IPSEC](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwybHRaekl3TVRndVkyNWliRzluY3k1amIyMHZZbXh2Wnk4eE1ESTROVFF3THpJd01Ua3dNeTh4TURJNE5UUXdMVEl3TVRrd016QTBNVFl3TURRMk1qZzNMVFkzTXpNeE9EazRPUzV3Ym1jPS5qcGc%3D.jpg?w=700&webp=1 "[dev][dpdk][crypto] dpdk加解密设备与IPSEC")
4. op的mempool的API
extern struct rte_mempool *
rte_crypto_op_pool_create(const char *name, enum rte_crypto_op_type type,
unsigned nb_elts, unsigned cache_size, uint16_t priv_size,
int socket_id);
struct rte_crypto_op *rte_crypto_op_alloc(struct rte_mempool *mempool,
enum rte_crypto_op_type type) unsigned rte_crypto_op_bulk_alloc(struct rte_mempool *mempool,
enum rte_crypto_op_type type,
struct rte_crypto_op **ops, uint16_t nb_ops)
void rte_crypto_op_free(struct rte_crypto_op *op)
5. session
session用来存储加密过程中的key，以及分组信息等。
用来存储这些的叫做private session data
![[dev][dpdk][crypto] dpdk加解密设备与IPSEC](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwybHRaekl3TVRndVkyNWliRzluY3k1amIyMHZZbXh2Wnk4eE1ESTROVFF3THpJd01Ua3dNeTh4TURJNE5UUXdMVEl3TVRrd016QTBNVFl4TWpVek9EVTNMVEl4TVRnM01EY3pORFF1Y0c1bi5qcGc%3D.jpg?w=700&webp=1 "[dev][dpdk][crypto] dpdk加解密设备与IPSEC")
api：
rte_cryptodev_sym_session_create()
rte_cryptodev_sym_session_init()
rte_cryptodev_sym_session_clear()
rte_cryptodev_sym_session_free（）
6. transforms
Currently there are three transforms types cipher, authentication and AEAD.
Also it is important to note that the order in which the transforms are passed indicates the order of the chaining.
struct rte_crypto_sym_xform {
struct rte_crypto_sym_xform *next;
/**< next xform in chain */
enum rte_crypto_sym_xform_type type;
/**< xform type */
union {
struct rte_crypto_auth_xform auth;
/**< Authentication / hash xform */
struct rte_crypto_cipher_xform cipher;
/**< Cipher xform */
struct rte_crypto_aead_xform aead;
/**< AEAD xform */
};
};
7. 运行状态下的最小配置
As a minimum the symmetric operation must have a source data buffer (m_src), a valid session (or transform chain if in session-less mode) and the minimum authentication/ cipher/ AEAD parameters
required depending on the type of operation specified in the session or the transform chain.
示例程序：
/*
* Simple example to encrypt several buffers with AES-CBC using
* the Cryptodev APIs.
*/ #define MAX_SESSIONS 1024
#define NUM_MBUFS 1024
#define POOL_CACHE_SIZE 128
#define BURST_SIZE 32
#define BUFFER_SIZE 1024
#define AES_CBC_IV_LENGTH 16
#define AES_CBC_KEY_LENGTH 16
#define IV_OFFSET (sizeof(struct rte_crypto_op) + \
sizeof(struct rte_crypto_sym_op)) struct rte_mempool *mbuf_pool, *crypto_op_pool, *session_pool;
unsigned int session_size;
int ret; /* Initialize EAL. */
ret = rte_eal_init(argc, argv);
if (ret < )
rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n"); uint8_t socket_id = rte_socket_id(); /* Create the mbuf pool. */
mbuf_pool = rte_pktmbuf_pool_create("mbuf_pool",
NUM_MBUFS,
POOL_CACHE_SIZE,
,
RTE_MBUF_DEFAULT_BUF_SIZE,
socket_id);
if (mbuf_pool == NULL)
rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n"); /*
* The IV is always placed after the crypto operation,
* so some private data is required to be reserved.
*/
unsigned int crypto_op_private_data = AES_CBC_IV_LENGTH; /* Create crypto operation pool. */
crypto_op_pool = rte_crypto_op_pool_create("crypto_op_pool",
RTE_CRYPTO_OP_TYPE_SYMMETRIC,
NUM_MBUFS,
POOL_CACHE_SIZE,
crypto_op_private_data,
socket_id);
if (crypto_op_pool == NULL)
rte_exit(EXIT_FAILURE, "Cannot create crypto op pool\n"); /* Create the virtual crypto device. */
char args[];
const char *crypto_name = "crypto_aesni_mb0";
snprintf(args, sizeof(args), "socket_id=%d", socket_id);
ret = rte_vdev_init(crypto_name, args);
if (ret != )
rte_exit(EXIT_FAILURE, "Cannot create virtual device"); uint8_t cdev_id = rte_cryptodev_get_dev_id(crypto_name); /* Get private session data size. */
session_size = rte_cryptodev_sym_get_private_session_size(cdev_id); /*
* Create session mempool, with two objects per session,
* one for the session header and another one for the
* private session data for the crypto device.
*/
session_pool = rte_mempool_create("session_pool",
MAX_SESSIONS * ,
session_size,
POOL_CACHE_SIZE,
, NULL, NULL, NULL,
NULL, socket_id,
); /* Configure the crypto device. */
struct rte_cryptodev_config conf = {
.nb_queue_pairs = ,
.socket_id = socket_id
};
struct rte_cryptodev_qp_conf qp_conf = {
.nb_descriptors =
}; if (rte_cryptodev_configure(cdev_id, &conf) < )
rte_exit(EXIT_FAILURE, "Failed to configure cryptodev %u", cdev_id); if (rte_cryptodev_queue_pair_setup(cdev_id, , &qp_conf,
socket_id, session_pool) < )
rte_exit(EXIT_FAILURE, "Failed to setup queue pair\n"); if (rte_cryptodev_start(cdev_id) < )
rte_exit(EXIT_FAILURE, "Failed to start device\n"); /* Create the crypto transform. */
uint8_t cipher_key[] = {};
struct rte_crypto_sym_xform cipher_xform = {
.next = NULL,
.type = RTE_CRYPTO_SYM_XFORM_CIPHER,
.cipher = {
.op = RTE_CRYPTO_CIPHER_OP_ENCRYPT,
.algo = RTE_CRYPTO_CIPHER_AES_CBC,
.key = {
.data = cipher_key,
.length = AES_CBC_KEY_LENGTH
},
.iv = {
.offset = IV_OFFSET,
.length = AES_CBC_IV_LENGTH
}
}
}; /* Create crypto session and initialize it for the crypto device. */
struct rte_cryptodev_sym_session *session;
session = rte_cryptodev_sym_session_create(session_pool);
if (session == NULL)
rte_exit(EXIT_FAILURE, "Session could not be created\n"); if (rte_cryptodev_sym_session_init(cdev_id, session,
&cipher_xform, session_pool) < )
rte_exit(EXIT_FAILURE, "Session could not be initialized "
"for the crypto device\n"); /* Get a burst of crypto operations. */
struct rte_crypto_op *crypto_ops[BURST_SIZE];
if (rte_crypto_op_bulk_alloc(crypto_op_pool,
RTE_CRYPTO_OP_TYPE_SYMMETRIC,
crypto_ops, BURST_SIZE) == )
rte_exit(EXIT_FAILURE, "Not enough crypto operations available\n"); /* Get a burst of mbufs. */
struct rte_mbuf *mbufs[BURST_SIZE];
if (rte_pktmbuf_alloc_bulk(mbuf_pool, mbufs, BURST_SIZE) < )
rte_exit(EXIT_FAILURE, "Not enough mbufs available"); /* Initialize the mbufs and append them to the crypto operations. */
unsigned int i;
for (i = ; i < BURST_SIZE; i++) {
if (rte_pktmbuf_append(mbufs[i], BUFFER_SIZE) == NULL)
rte_exit(EXIT_FAILURE, "Not enough room in the mbuf\n");
crypto_ops[i]->sym->m_src = mbufs[i];
} /* Set up the crypto operations. */
for (i = ; i < BURST_SIZE; i++) {
struct rte_crypto_op *op = crypto_ops[i];
/* Modify bytes of the IV at the end of the crypto operation */
uint8_t *iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *,
IV_OFFSET); generate_random_bytes(iv_ptr, AES_CBC_IV_LENGTH); op->sym->cipher.data.offset = ;
op->sym->cipher.data.length = BUFFER_SIZE; /* Attach the crypto session to the operation */
rte_crypto_op_attach_sym_session(op, session);
} /* Enqueue the crypto operations in the crypto device. */
uint16_t num_enqueued_ops = rte_cryptodev_enqueue_burst(cdev_id, ,
crypto_ops, BURST_SIZE); /*
* Dequeue the crypto operations until all the operations
* are proccessed in the crypto device.
*/
uint16_t num_dequeued_ops, total_num_dequeued_ops = ;
do {
struct rte_crypto_op *dequeued_ops[BURST_SIZE];
num_dequeued_ops = rte_cryptodev_dequeue_burst(cdev_id, ,
dequeued_ops, BURST_SIZE);
total_num_dequeued_ops += num_dequeued_ops; /* Check if operation was processed successfully */
for (i = ; i < num_dequeued_ops; i++) {
if (dequeued_ops[i]->status != RTE_CRYPTO_OP_STATUS_SUCCESS)
rte_exit(EXIT_FAILURE,
"Some operations were not processed correctly");
} rte_mempool_put_bulk(crypto_op_pool, (void **)dequeued_ops,
num_dequeued_ops);
} while (total_num_dequeued_ops < num_enqueued_ops);
第三部分，Security Framework
前边的内容，都在讲加解密。但是实际上呢，我是想要处理IPSEC的报文。下面这个库，将搞定这个事情：
http://doc.dpdk.org/guides-18.11/prog_guide/rte_security.html
既然是ipsec，所以，自然用到nic和crypto两部分的pmd
+---------------+
| rte_security |
+---------------+
\ /
+-----------+ +--------------+
| NIC PMD | | CRYPTO PMD |
+-----------+ +--------------+
硬件offload
加解密offload
就是说收到的包就解好了密，发包之前不需要进行包加密的意思。这个是硬件offload支持的，不支持的话，还是需要软件去把包送给crypto dev 处理。
协议offload
区别于加解密offload，ESP的包头之类的，也可以offload的给硬件。
lookaside offload
处理前边提到那两个，就连sa的管理，也可以直接offload。
设备能力
ipsec的封解包，是由device决定的。不同的device可能不一定支持，需要通过下面的方法具体查看。这里的device即包括了eth也包括了crypto
The device (crypto or ethernet) capabilities which support security operations, are defined by the security action type, security protocol,
protocol capabilities and corresponding crypto capabilities for security. For the full scope of the Security capability see definition of
rte_security_capability structure in the DPDK API Reference.
API：
const struct rte_security_capability *rte_security_capabilities_get(uint16_t id);
安全会话（security session）
Security Sessions are created to store the immutable fields of a particular Security Association for a particular protocol which is defined by a security session
configuration structure which is used in the operation processing of a packet flow. Sessions are used to manage protocol specific information as well as crypto
parameters. Security sessions cache this immutable data in a optimal way for the underlying PMD and this allows further acceleration of the offload of Crypto workloads.
session也是mempool里的内存
sessions are mempool objects.
API：
http://doc.dpdk.org/api/rte__security_8h.html
rte_security_session_create()
rte_cryptodev_get_sec_ctx() or rte_eth_dev_get_sec_ctx（）
rte_security_session_destroy()
关键结构体
rte_security_session_conf
struct rte_security_session_conf {
enum rte_security_session_action_type action_type;
/**< Type of action to be performed on the session */
enum rte_security_session_protocol protocol;
/**< Security protocol to be configured */
union {
struct rte_security_ipsec_xform ipsec;
struct rte_security_macsec_xform macsec;
struct rte_security_pdcp_xform pdcp;
};
/**< Configuration parameters for security session */
struct rte_crypto_sym_xform *crypto_xform;
/**< Security Session Crypto Transformations */
void *userdata;
/**< Application specific userdata to be saved with session */
};
enum rte_security_session_action_type {
RTE_SECURITY_ACTION_TYPE_NONE,
/**< No security actions */
RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO,
/**< Crypto processing for security protocol is processed inline
* during transmission */
RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL,
/**< All security protocol processing is performed inline during
* transmission */
RTE_SECURITY_ACTION_TYPE_LOOKASIDE_PROTOCOL
/**< All security protocol processing including crypto is performed
* on a lookaside accelerator */
};
其他
如果是NIC设备的offload，必须注意flow和加解密序列和port直接的对应关系。
In the case of NIC based offloads, the security session specified in the ‘rte_flow_action_security’
must be created on the same port as the flow action that is being specified.
第四部分 RTE_IPSEC
文档： http://doc.dpdk.org/guides/prog_guide/ipsec_lib.html
一般使用流程
/* enqueue for processing by crypto-device */
rte_ipsec_pkt_crypto_prepare(...);
rte_cryptodev_enqueue_burst(...);
/* dequeue from crypto-device and do final processing (if any) */
rte_cryptodev_dequeue_burst(...);
rte_ipsec_pkt_crypto_group(...); /* optional */
rte_ipsec_pkt_process(...);
api：
这个还没发布。。。 在master分支上，模块名字叫librte_ipsec
详细的用法可以见头文件： http://git.dpdk.org/dpdk/tree/lib/librte_ipsec/rte_ipsec.h?h=releases