i40iw Linux* Driver for Intel(R) Ethernet Connection X722 =============================================================================== December 11, 2017 =============================================================================== Contents -------- - Prerequisites - Building and Installation - Testing - Virtualization - Interoperability - RDMA Statistics - Known Issues ================================================================================ Prerequisites ------------- - A supported kernel configuration, choose from the following: 1) A Linux distribution supported by OFED 3.18-3 or OFED 4.8(recommended). Use OFED if it is required by software you wish to run. 2) An upstream kernel v4.8-v4.14, if you require fixes not in OFED. For example, NVMe over Fabrics (NVMeoF). 3) RHEL 7.4 or SLES 12 SP3 with infiniband support installed, if you do not want to install OFED or upstream kernel. - For OFED 3.18-3/OFED 4.8, it should be install with ./install.pl --all - For OFED 4.8 or Linux Kernel v4.8-v4.14, download and install the latest rdma_core from https://github.com/linux-rdma/rdma-core/releases NOTE: Internet Wide Area RDMA Protocol (iWARP) is not supported with the i40iwvf driver running on Microsoft* Hyper-V. Building and Installation ------------------------- OFED 3.18-3 ----------- 1. Untar i40iw-.tar.gz, i40iwvf-.tar.gz and libi40iw- .tar.gz. 2. Install the PF driver as follows: cd i40iw- directory ./build.sh 3 For example: ./build.sh /opt/i40e-2.3.3 3. Install the VF driver as follows: cd i40iwvf- directory ./build.sh 3 For example: ./build.sh /opt/i40evf-3.2.3 4. Install user-space library as follows: cd libi40iw- ./build.sh OFED 4.8 -------- 1. Untar i40iw-.tar.gz and i40iwvf-.tar.gz. 2. Install the PF driver as follows: cd i40iw- directory ./build.sh 4 For example: ./build.sh /opt/i40e-2.3.3 4 3. Install the VF driver as follows: cd i40iwvf- directory ./build.sh 4 For example: ./build.sh /opt/i40evf-3.2.3 4 4. OFED 4.8 comes with an older version of rdma_core user-space package Please download the latest from https://github.com/linux-rdma/rdma-core/ releases and follow its installation procedure. Linux Kernel v4.8-v4.14/RHEL 7.4/SLES 12 SP3 -------------------------------------------- 1. Untar i40iw-.tar.gz and i40iwvf-.tar.gz. 2. Install the PF driver as follows: cd i40iw- directory ./build.sh k For example: ./build.sh /opt/i40e-2.3.3 k 3. Install the VF driver as follows: cd i40iwvf- directory ./build.sh k For example: ./build.sh /opt/i40evf-3.2.3 k 4. Please download the latest rdma_core user-space package from https://github.com/linux-rdma/rdma-core/releases and follow its installation procedure. Adapter and Switch Flow Control Setting --------------------------------------- We recommend enabling link-level flow control (both TX and RX) on X722 and connected switch. To enable flow control on X722 use ethtool -A command. For example: ethtool -A p4p1 rx on tx on Confirm the setting with ethtool -a command. For example: ethtool -a p4p1 You should see this output: Pause parameters for p4p1: Autonegotiate: off X: on TX: on To enable link-level flow control on the switch, please consult your switch vendor's documentation. Look for flow-control and make sure both TX and RX are set. Here is an example for a generic switch to enable both TX and RX flow control on port 45: enable flow-control tx-pause ports 45 enable flow-control rx-pause ports 45 ================================================================================ Virtualization -------------- To enable SR-IOV support, load i40iw with the following parameters and then create VFs with i40e. resource_profile=2 max_rdma_vfs= For example: modprobe i40iw resource_profile=2 max_rdma_vfs=32 NOTE: Once the VFs are running, do not change the PF configuration. Interoperability ---------------- To interoperate with Chelsio iWARP devices with OFED 4.8 or Linux Kernels v4.8-v4.14: Load Chelsio T4/T5 RDMA driver (iw_cxgb4) with parameter dack_mode set to 0. modprobe iw_cxgb4 dack_mode=0 If iw_cxgb4 is loaded on system boot, create /etc/modprobe.d/iw_cxgb4.conf file with the following entry: options iw_cxgb4 dack_mode=0 Reload iw_cxgb4 for the new parameters to take effect. RDMA Statistics --------------- Use the following command to read RDMA Protocol statistics: cd /sys/class/infiniband/i40iw0/proto_stats; for f in *; do echo -n "$f: "; cat "$f"; done; cd The following counters will increment when RDMA applications are transferring data over the network: - ipInReceives - tcpInSegs - tcpOutSegs ================================================================================ Known Issues/Troubleshooting ---------------------------- * You may experience a kernel crash using OFED 3.18-3 under heavy load. It is fixed in upstream kernel with commit dafb558717. Incompatible Drivers in initramfs --------------------------------- There may be incompatible drivers in the initramfs image. You can either update the image or remove the drivers from initramfs. Specifically look for i40e, ib_addr, ib_cm, ib_core, ib_mad, ib_sa, ib_ucm, ib_uverbs, iw_cm, rdma_cm, rdma_ucm in the output of the following command: lsinitrd |less If you see any of those modules, rebuild initramfs with the following command and include the name of the module in the "" list. Below is an example: dracut --force --omit-drivers "i40e ib_addr ib_cm ib_core ib_mad ib_sa ib_ucm ib_uverbs iw_cm rdma_cm rdma_ucm" ================================================================================ Support ------- For general information, go to the Intel support website at: www.intel.com/support/ or the Intel Wired Networking project hosted by Sourceforge at: http://sourceforge.net/projects/e1000 If an issue is identified with the released source code on a supported kernel with a supported adapter, email the specific information related to the issue to e1000-rdma@lists.sourceforge.net ================================================================================ License ------- This software is available to you under a choice of one of two licenses. You may choose to be licensed under the terms of the GNU General Public License (GPL) Version 2, available from the file COPYING in the main directory of this source tree, or the OpenFabrics.org BSD license below: Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================================================ Trademarks ---------- Intel and Itanium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and/or other countries. * Other names and brands may be claimed as the property of others.