Cni: Difference between revisions
Martindenico (talk | contribs) No edit summary |
Martindenico (talk | contribs) (activeActiveL3 changed to activePassiveL3 as per approved contribution NFVSOL(22)000333) |
||
(11 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
[[https://nfvwiki.etsi.org/index.php?title=NFVI_Platform_Capability_Registry back to NFVI Platform Capability registry]] | |||
== '''1 Registration information''' == | |||
{| class="wikitable" | {| class="wikitable" | ||
|+ | |+ | ||
|- | |- | ||
|Name | |Name | ||
Line 23: | Line 25: | ||
|- | |- | ||
|Description | |Description | ||
|This capability represents the realization of a secondary container cluster network interface. | |This capability represents the realization of a secondary container cluster network interface. The expression of the related requirements allows the selection of a suitable CNI<sup>TM</sup> and configuration of the Network Attachment Definition (NAD) or NADs accordingly. | ||
|} | |||
== '''2 JSON schema''' == | |||
{| class="wikitable" | |||
|+ | |||
!cni | |||
|- | |||
|{ | |||
"$schema": "<nowiki>http://json-schema.org/draft-07/schema#</nowiki>", | |||
"title": "cni", | |||
"description": "Realization of a secondary container cluster network interface", | |||
"additionalProperties": false, | |||
"required": [ | |||
"interfaceType" | |||
], | |||
"type": "object", | |||
"properties": { | |||
"interfaceType": { | |||
"type": "string", | |||
"enum": [ | |||
"kernel.netdev", | |||
"direct.userspace", | |||
"direct.kernel", | |||
"bond", | |||
"userspace" | |||
], | |||
"description": "Type of network interface" | |||
}, | |||
"nicOptions": { | |||
"type": "array", | |||
"items": { | |||
"type": "object", | |||
"properties": { | |||
"nicType": { | |||
"type": "string", | |||
"enum": [ "i710", "e810", "mlx-cx5", "mlx-cx6"], | |||
"description": "Nic types the driver is verified to work with in a direct.userspace interface." | |||
}, | |||
"speeds": { | |||
"type": "array", | |||
"items": { | |||
"type": "integer" | |||
}, | |||
"description": "List of speeds in bps the driver is verified to work with in a direct.userspace, direct.kernel or bond interface. If nicType is present, the speeds are applicable to that nicType." | |||
} | |||
}, | |||
"minProperties": 1 | |||
}, | |||
"description": "For direct.userdriver interface nic types and/or speeds the driver is verified to work with. For direct.kernel and bond speeds the driver is verified to work with" | |||
}, | |||
"interfaceOptions": { | |||
"type": "array", | |||
"items": { | |||
"type": "string", | |||
"enum": [ "memif", "virtio"] | |||
}, | |||
"description": "For userspace vnic configurations the network interface is verified to work with " | |||
}, | |||
"interfaceRedundancy": { | |||
"type": "string", | |||
"enum": [ "infraProvided", "left", "right", "activeActiveBond", "activePassiveBond", "activePassiveL3" ], | |||
"description": " Switchplane redundancy management method " | |||
} | |||
}, | |||
"allOf": [ | |||
{ | |||
"if": { | |||
"properties": { | |||
"interfaceType": { | |||
"enum": [ "direct.userspace", "direct.kernel" ] | |||
} | |||
} | |||
}, | |||
"then": { | |||
"required": ["interfaceRedundancy" ], | |||
"properties": { | |||
"interfaceRedundancy": { | |||
"enum": [ "activePassiveBond", "activeActiveBond", "activePassiveL3", "Left", "Right", "infraProvided" ] | |||
} | |||
} | |||
} | |||
}, | |||
{ | |||
"if": { | |||
"properties": { | |||
"interfaceType": { | |||
"enum": ["kernel.netdev", "userspace" ] | |||
} | |||
} | |||
}, | |||
"then": { | |||
"properties": { | |||
"interfaceRedundancy": { | |||
"enum": ["infraProvided" ], | |||
"default": "infraProvided" | |||
} | |||
} | |||
} | |||
}, | |||
{ | |||
"if": { | |||
"properties": { | |||
"interfaceType": { | |||
"enum": ["bond"] | |||
} | |||
} | |||
}, | |||
"then": { | |||
"required": ["interfaceRedundancy" ], | |||
"properties": { | |||
"interfaceRedundancy": { | |||
"enum": ["activePassiveBond", "activeActiveBond" ] | |||
} | |||
} | |||
} | |||
} | |||
] | |||
} | |||
|} | |} | ||
== '''3 Detailed description''' == | |||
The interfaceType can take one of the following values: | |||
{| class="wikitable" | |||
|+ | |||
|kernel.netdev | |||
|Linux kernel or vSwitch based. It is supported by CNIs<sup>TM</sup> like: OVS, MACVLAN, IPVLAN, BRIDGE, PTP, etc. | |||
|- | |||
|direct.userspace | |||
|Provides direct access to network PCIe, typically an SR-IOV VF, but could also be a PF. Uses application specific driver contained in the image. The device is bound to a driver by the cloud provider which supports user space control by the application (e.g. vfio_pci, igb_uio). It is supported by the following CNIs<sup>TM</sup>: SRIOV, HOST-DEVICE. | |||
|- | |||
|direct.kernel | |||
|Similar to direct.userspace but device is bound to a kernel driver by the cloud provider and can be consumed in the same way as a kernel netdev. It is supported by the following CNIs<sup>TM</sup>: SRIOV, HOST-DEVICE. | |||
|- | |||
|bond | |||
|Based on the BOND CNI<sup>TM</sup>. Requires the existence of a mated pair of direct kernel network attachments prior to the creation of the bond. Thus, when this interface type is indicated, three attachments definitions to the secondary container cluster network are expected: two providing the direct kerneldriver functionality and one with the BOND CNI<sup>TM</sup> | |||
|- | |||
|userspace | |||
|Based on the userspace CNI<sup>TM.</sup> requires infrastructure to support a DPDK-OVS or VPP based vSwitch. | |||
|} | |||
The nicOptions can take one of the following values: | |||
{| class="wikitable" | |||
|+ | |||
|e710 | |||
|Intel<sup>®</sup> Corporation Ethernet Controller 710 product family. | |||
|- | |||
|e810 | |||
|Intel<sup>®</sup> Corporation Ethernet Controller E810 product family. | |||
|- | |||
|mlx-cx5 | |||
|NVIDIA<sup>®</sup> Mellanox<sup>®</sup> ConnectX<sup>®</sup>-5 adapters product family. | |||
|- | |||
|mlx-cx6 | |||
|NVIDIA<sup>®</sup> Mellanox<sup>®</sup> ConnectX<sup>®</sup>-6 adapters product family. | |||
|} | |||
The interfaceRedundancy can take one of the following values: | |||
{| class="wikitable" | |||
|+ | |||
|infraProvided | |||
|The application sees one vNIC but the infrastructure provides redundant access to the network via both switch planes. For interface of type kernel.netdev the redundancy is provided by the vSwitch or Linux bonding. For interfaces of type direct, it requires that the physical NIC is connected to both switch planes (a.k.a. smartNIC). Thus it imposes hardware requirements on the infrastructure. For interface of type userspace redundancy handled by a DPDK-OVS or VPP vSwitch | |||
|- | |||
|left | |||
|A vNIC connected non-redundantly to the network via one specific (left) switchplane. Valid for interfaces of type direct.userdrive or direct.kernel. Redundancy is implemented by having two connection points, indicating different values (left/right) in the descriptor. | |||
|- | |||
|right | |||
|A vNIC connected non-redundantly to the network via one specific (right) switchplane. Valid for interfaces of type direct.userdrive or direct.kernel. Redundancy is implemented by having two connection points, indicating different values (left/right) in the descriptor. | |||
NOTE: left and right switch planes are logical terms that refer to physically disjoint switching resources. They can for example be implemented as two independent sets of ethernet switches or a set of switches connected so that no single switch failure would cause end to end connectivity over the switching infrastructure to fail. | |||
|- | |||
| colspan="2" |For the following cases the infrastructure provides for the single VduCp a mated pair of vNICs connected to the same network, one via left switchplane, the other via the right switchplane: | |||
|- | |||
|activeActiveBond | |||
|Requires a static multi-chassis LAG in active-active mode from the infrastructure. Typically corresponds to bonding mode-2 “balance-xor” | |||
|- | |||
|activePassiveBond | |||
|The interfaces are bonded in active-passive mode in the application with move of bond MAC address. No specific requirements on DC fabric. Typically corresponds to bonding mode-1 “active-backup” | |||
|- | |||
|activePassiveL3 | |||
|Application moves its IP address by attaching it to one of the two MACs by use of GARP messages. | |||
|} | |||
The valid combinations of values for the interfaceType and the interfaceRedundancy properties as well the applicability of the nicOptions and interfaceOptions are shown in the following table: | |||
{| class="wikitable" | |||
|+ | |||
!'''interfaceType''' | |||
!'''nicOptions''' | |||
!'''interfaceOptions''' | |||
!'''interfaceRedundancy''' | |||
|- | |||
|kernel.netdev | |||
|Not applicable | |||
|Not applicable | |||
|infraProvided | |||
(default, no need to indicate) | |||
|- | |||
|direct.userspace | |||
|Applicable | |||
|Not applicable | |||
|left, right, activeActiveBond, activePassiveBond, activePassiveL3, infraProvided | |||
|- | |||
|direct.kernel | |||
|Applicable | |||
(only speeds) | |||
|Not applicable | |||
|left, right, activeActiveBond, activePassiveBond, activePassiveL3, infraProvided | |||
|- | |||
|bond | |||
|Applicable | |||
(only speeds) | |||
|Not applicable | |||
|activeActiveBond, activePassiveBond | |||
|- | |||
|userspace | |||
|Not applicable | |||
|Applicable | |||
|InfraProvided | |||
(default, no need to indicate) | |||
|} | |||
An invalid combination of interfaceType and interfaceRedundancy shall be rejected. | |||
An invalid use of nicOptions or interfaceOptions can be ignored. | |||
== '''4 Examples of requirements on the cni capability for SOL001''' == | |||
1. The following example shows the expression of a requirement for this capability for an interface of type kernel.netdev. The requirement can be fulfilled e.g. with the OVS CNI<sup>TM</sup> and vSwitch redundancy. One single NAD is required. | |||
Note: it is not necessary to indicate the interfaceRedundancy for this interfaceType. | |||
{| class="wikitable" | |||
| | |||
network_interface_requirements: | |||
cni: | | |||
version: 1.0 | |||
mandatory: true | |||
configuration_value: | |||
interfaceType: kernel.netdev | |||
interfaceRedundancy: infraProvided | |||
|} | |||
2. The following example shows the expression of a requirement for this capability for an interface of type direct.kernel and interface redundancy ActivePassiveBond. The requirement can be fulfilled with the SRIOV CNI<sup>TM</sup> for a cluster on bare metal or with a HOST-DEVICE CNI<sup>TM</sup> for a VM based cluster. The redundancy is based on a mated pair of vNICs provided by the infrastructure one connected to the left switch plane, the other to the right switch plane. Thus, two NADs are required. | |||
{| class="wikitable" | |||
| | |||
network_interface_requirements: | |||
cni: | | |||
version: 1.0 | |||
mandatory: true | |||
configuration_value: | |||
interfaceType: direct.kernel | |||
interfaceRedundancy: ActivePassiveBond | |||
|} | |||
3. The following example shows the expression of a requirement for this capability for an interface of type direct.kernel and interface redundancy infraProvided. The requirement can be fulfilled with the SRIOV CNI<sup>TM</sup> for a cluster on bare metal or with a HOST-DEVICE CNI<sup>TM</sup> for a VM based cluster. There is a single vNIC and the redundancy is based on a physical NIC connected to both switchplanes and providing the bonding (a.k.a. smartNIC). One single NAD is required. | |||
{| class="wikitable" | |||
| | |||
network_interface_requirements: | |||
cni: | | |||
version: 1.0 | |||
mandatory: true | |||
configuration_value: | |||
interfaceType: direct.kernel | |||
interfaceRedundancy: infraProvided | |||
|} | |||
4. The following example shows the expression of a requirement for this capability for an interface of type bond and interface redundancy ActivePassiveBond. This is also a direct mode that provides direct access to the network PCI. The requirement is fulfilled with the BOND CNI<sup>TM</sup>. The redundancy is based on a mated pair of vNICs provided by the infrastructure one connected to the left switch plane, the other to the right switch plane and the BOND CNI<sup>TM</sup> provides the bonding in userspace. Thus, three NADs are required. | |||
{| class="wikitable" | |||
| | |||
network_interface_requirements: | |||
cni: | | |||
version: 1.0 | |||
mandatory: true | |||
configuration_value: | |||
interfaceType: bond | |||
interfaceRedundancy: ActivePassiveBond | |||
|} | |||
5. The following example shows the expression of a requirement for this capability for two independent interfaces of type direct.kernel. The application manages the redundancy with support of routing protocols. The requirement can be fulfilled with the SRIOV CNI<sup>TM</sup> for a cluster on bare metal or with a HOST-DEVICE CNI<sup>TM</sup> for a VM based cluster. One NADs is required in each of the VduCps. | |||
In VduCpd (left): | |||
{| class="wikitable" | |||
| | |||
network_interface_requirements: | |||
cni: | | |||
version: 1.0 | |||
mandatory: true | |||
configuration_value: | |||
interfaceType: direct.kernel | |||
interfaceRedundancy: left | |||
|} | |||
In VduCpd (right): | |||
{| class="wikitable" | |||
| | |||
network_interface_requirements: | |||
cni: | | |||
version: 1.0 | |||
mandatory: true | |||
configuration_value: | |||
interfaceType: direct.kernel | |||
interfaceRedundancy: right | |||
|} | |||
6. The following example shows the expression of a requirement for this capability for an interface of type direct.userspace and interface redundancy ActivePassiveBond. The requirement can be fulfilled with the SRIOV CNI<sup>TM</sup> for a cluster on bare metal or with a HOST-DEVICE CNI<sup>TM</sup> for a VM based cluster. The redundancy is based on a mated pair of vNICs provided by the infrastructure one connected to the left switch plane, the other to the right switch plane. Thus, two NADs are required. The application has been verified to work with the i710 NIC and 1 Gbps and 10 Gbps. | |||
{| class="wikitable" | |||
| | |||
network_interface_requirements: | |||
cni: | | |||
version: 1.0 | |||
mandatory: true | |||
configuration_value: | |||
interfaceType: direct.kernel | |||
interfaceRedundancy: ActivePassiveBond | |||
nicOptions: | |||
- nicType: i710 | |||
speeds: [ 1000000000, 10000000000 ] | |||
|} |
Latest revision as of 14:40, 7 July 2022
[back to NFVI Platform Capability registry]
1 Registration information
Name | cni |
Category | Network |
Permitted values | Per schema |
Version | 1.0 |
Schema | See below |
Type | Generic |
Description | This capability represents the realization of a secondary container cluster network interface. The expression of the related requirements allows the selection of a suitable CNITM and configuration of the Network Attachment Definition (NAD) or NADs accordingly. |
2 JSON schema
cni |
---|
{
"$schema": "http://json-schema.org/draft-07/schema#", "title": "cni", "description": "Realization of a secondary container cluster network interface", "additionalProperties": false, "required": [ "interfaceType" ], "type": "object", "properties": { "interfaceType": { "type": "string", "enum": [ "kernel.netdev", "direct.userspace", "direct.kernel", "bond", "userspace" ], "description": "Type of network interface" }, "nicOptions": { "type": "array", "items": { "type": "object", "properties": { "nicType": { "type": "string", "enum": [ "i710", "e810", "mlx-cx5", "mlx-cx6"], "description": "Nic types the driver is verified to work with in a direct.userspace interface." }, "speeds": { "type": "array", "items": { "type": "integer" }, "description": "List of speeds in bps the driver is verified to work with in a direct.userspace, direct.kernel or bond interface. If nicType is present, the speeds are applicable to that nicType." } }, "minProperties": 1 }, "description": "For direct.userdriver interface nic types and/or speeds the driver is verified to work with. For direct.kernel and bond speeds the driver is verified to work with" }, "interfaceOptions": { "type": "array", "items": { "type": "string", "enum": [ "memif", "virtio"] }, "description": "For userspace vnic configurations the network interface is verified to work with " }, "interfaceRedundancy": { "type": "string", "enum": [ "infraProvided", "left", "right", "activeActiveBond", "activePassiveBond", "activePassiveL3" ], "description": " Switchplane redundancy management method " } }, "allOf": [ { "if": { "properties": { "interfaceType": { "enum": [ "direct.userspace", "direct.kernel" ] } } }, "then": { "required": ["interfaceRedundancy" ], "properties": { "interfaceRedundancy": { "enum": [ "activePassiveBond", "activeActiveBond", "activePassiveL3", "Left", "Right", "infraProvided" ] } } } }, { "if": { "properties": { "interfaceType": { "enum": ["kernel.netdev", "userspace" ] } } }, "then": { "properties": { "interfaceRedundancy": { "enum": ["infraProvided" ], "default": "infraProvided" } } } }, { "if": { "properties": { "interfaceType": { "enum": ["bond"] } } }, "then": { "required": ["interfaceRedundancy" ], "properties": { "interfaceRedundancy": { "enum": ["activePassiveBond", "activeActiveBond" ] } } } } ] } |
3 Detailed description
The interfaceType can take one of the following values:
kernel.netdev | Linux kernel or vSwitch based. It is supported by CNIsTM like: OVS, MACVLAN, IPVLAN, BRIDGE, PTP, etc. |
direct.userspace | Provides direct access to network PCIe, typically an SR-IOV VF, but could also be a PF. Uses application specific driver contained in the image. The device is bound to a driver by the cloud provider which supports user space control by the application (e.g. vfio_pci, igb_uio). It is supported by the following CNIsTM: SRIOV, HOST-DEVICE. |
direct.kernel | Similar to direct.userspace but device is bound to a kernel driver by the cloud provider and can be consumed in the same way as a kernel netdev. It is supported by the following CNIsTM: SRIOV, HOST-DEVICE. |
bond | Based on the BOND CNITM. Requires the existence of a mated pair of direct kernel network attachments prior to the creation of the bond. Thus, when this interface type is indicated, three attachments definitions to the secondary container cluster network are expected: two providing the direct kerneldriver functionality and one with the BOND CNITM |
userspace | Based on the userspace CNITM. requires infrastructure to support a DPDK-OVS or VPP based vSwitch. |
The nicOptions can take one of the following values:
e710 | Intel® Corporation Ethernet Controller 710 product family. |
e810 | Intel® Corporation Ethernet Controller E810 product family. |
mlx-cx5 | NVIDIA® Mellanox® ConnectX®-5 adapters product family. |
mlx-cx6 | NVIDIA® Mellanox® ConnectX®-6 adapters product family. |
The interfaceRedundancy can take one of the following values:
infraProvided | The application sees one vNIC but the infrastructure provides redundant access to the network via both switch planes. For interface of type kernel.netdev the redundancy is provided by the vSwitch or Linux bonding. For interfaces of type direct, it requires that the physical NIC is connected to both switch planes (a.k.a. smartNIC). Thus it imposes hardware requirements on the infrastructure. For interface of type userspace redundancy handled by a DPDK-OVS or VPP vSwitch |
left | A vNIC connected non-redundantly to the network via one specific (left) switchplane. Valid for interfaces of type direct.userdrive or direct.kernel. Redundancy is implemented by having two connection points, indicating different values (left/right) in the descriptor. |
right | A vNIC connected non-redundantly to the network via one specific (right) switchplane. Valid for interfaces of type direct.userdrive or direct.kernel. Redundancy is implemented by having two connection points, indicating different values (left/right) in the descriptor.
NOTE: left and right switch planes are logical terms that refer to physically disjoint switching resources. They can for example be implemented as two independent sets of ethernet switches or a set of switches connected so that no single switch failure would cause end to end connectivity over the switching infrastructure to fail. |
For the following cases the infrastructure provides for the single VduCp a mated pair of vNICs connected to the same network, one via left switchplane, the other via the right switchplane: | |
activeActiveBond | Requires a static multi-chassis LAG in active-active mode from the infrastructure. Typically corresponds to bonding mode-2 “balance-xor” |
activePassiveBond | The interfaces are bonded in active-passive mode in the application with move of bond MAC address. No specific requirements on DC fabric. Typically corresponds to bonding mode-1 “active-backup” |
activePassiveL3 | Application moves its IP address by attaching it to one of the two MACs by use of GARP messages. |
The valid combinations of values for the interfaceType and the interfaceRedundancy properties as well the applicability of the nicOptions and interfaceOptions are shown in the following table:
interfaceType | nicOptions | interfaceOptions | interfaceRedundancy |
---|---|---|---|
kernel.netdev | Not applicable | Not applicable | infraProvided
(default, no need to indicate) |
direct.userspace | Applicable | Not applicable | left, right, activeActiveBond, activePassiveBond, activePassiveL3, infraProvided |
direct.kernel | Applicable
(only speeds) |
Not applicable | left, right, activeActiveBond, activePassiveBond, activePassiveL3, infraProvided |
bond | Applicable
(only speeds) |
Not applicable | activeActiveBond, activePassiveBond |
userspace | Not applicable | Applicable | InfraProvided
(default, no need to indicate) |
An invalid combination of interfaceType and interfaceRedundancy shall be rejected.
An invalid use of nicOptions or interfaceOptions can be ignored.
4 Examples of requirements on the cni capability for SOL001
1. The following example shows the expression of a requirement for this capability for an interface of type kernel.netdev. The requirement can be fulfilled e.g. with the OVS CNITM and vSwitch redundancy. One single NAD is required.
Note: it is not necessary to indicate the interfaceRedundancy for this interfaceType.
network_interface_requirements: cni: | version: 1.0 mandatory: true configuration_value: interfaceType: kernel.netdev interfaceRedundancy: infraProvided |
2. The following example shows the expression of a requirement for this capability for an interface of type direct.kernel and interface redundancy ActivePassiveBond. The requirement can be fulfilled with the SRIOV CNITM for a cluster on bare metal or with a HOST-DEVICE CNITM for a VM based cluster. The redundancy is based on a mated pair of vNICs provided by the infrastructure one connected to the left switch plane, the other to the right switch plane. Thus, two NADs are required.
network_interface_requirements: cni: | version: 1.0 mandatory: true configuration_value: interfaceType: direct.kernel interfaceRedundancy: ActivePassiveBond |
3. The following example shows the expression of a requirement for this capability for an interface of type direct.kernel and interface redundancy infraProvided. The requirement can be fulfilled with the SRIOV CNITM for a cluster on bare metal or with a HOST-DEVICE CNITM for a VM based cluster. There is a single vNIC and the redundancy is based on a physical NIC connected to both switchplanes and providing the bonding (a.k.a. smartNIC). One single NAD is required.
network_interface_requirements: cni: | version: 1.0 mandatory: true configuration_value: interfaceType: direct.kernel interfaceRedundancy: infraProvided |
4. The following example shows the expression of a requirement for this capability for an interface of type bond and interface redundancy ActivePassiveBond. This is also a direct mode that provides direct access to the network PCI. The requirement is fulfilled with the BOND CNITM. The redundancy is based on a mated pair of vNICs provided by the infrastructure one connected to the left switch plane, the other to the right switch plane and the BOND CNITM provides the bonding in userspace. Thus, three NADs are required.
network_interface_requirements: cni: | version: 1.0 mandatory: true configuration_value: interfaceType: bond interfaceRedundancy: ActivePassiveBond |
5. The following example shows the expression of a requirement for this capability for two independent interfaces of type direct.kernel. The application manages the redundancy with support of routing protocols. The requirement can be fulfilled with the SRIOV CNITM for a cluster on bare metal or with a HOST-DEVICE CNITM for a VM based cluster. One NADs is required in each of the VduCps.
In VduCpd (left):
network_interface_requirements: cni: | version: 1.0 mandatory: true configuration_value: interfaceType: direct.kernel interfaceRedundancy: left |
In VduCpd (right):
network_interface_requirements: cni: | version: 1.0 mandatory: true configuration_value: interfaceType: direct.kernel interfaceRedundancy: right |
6. The following example shows the expression of a requirement for this capability for an interface of type direct.userspace and interface redundancy ActivePassiveBond. The requirement can be fulfilled with the SRIOV CNITM for a cluster on bare metal or with a HOST-DEVICE CNITM for a VM based cluster. The redundancy is based on a mated pair of vNICs provided by the infrastructure one connected to the left switch plane, the other to the right switch plane. Thus, two NADs are required. The application has been verified to work with the i710 NIC and 1 Gbps and 10 Gbps.
network_interface_requirements: cni: | version: 1.0 mandatory: true configuration_value: interfaceType: direct.kernel interfaceRedundancy: ActivePassiveBond nicOptions: - nicType: i710 speeds: [ 1000000000, 10000000000 ] |