README for Intel(R) Ethernet Switch Host Interface Driver
===============================================================================

===============================================================================

February 23, 2017

===============================================================================

Contents
--------

- Overview
- Building and Installation
- Command Line Parameters
- Additional Features & Configurations
- Known Issues


================================================================================


Important Notes
---------------

Configuring SR-IOV for improved network security
------------------------------------------------
In a virtualized environment, on Intel(R) Ethernet Server Adapters that support
SR-IOV, the virtual function (VF) may be subject to malicious behavior.
Software-generated layer two frames, like IEEE 802.3x (link flow control), IEEE
802.1Qbb (priority based flow-control), and others of this type, are not
expected and can throttle traffic between the host and the virtual switch,
reducing performance. To resolve this issue, configure all SR-IOV enabled ports
for VLAN tagging. This configuration allows unexpected, and potentially
malicious, frames to be dropped.


Overview
--------
This driver supports kernel versions 2.6.32 and newer.

Driver information can be obtained using ethtool, lspci, and iproute2 ip.
Instructions on updating ethtool can be found in the section Additional
Configurations later in this document.


Intel(R) Ethernet SDI Adapter FM10420-100GbE-QDA2 Notes
-------------------------------------------------------

The Intel(R) Ethernet SDI Adapter FM10420-100GbE-QDA2 supports the following
cables:
Supplier		Part Number
FCI			10121178-2010LF
FCI			10121178-4020LF
FCI			10121178-4030LF
FCI			10121178-4040LF
FCI			10121178-4050LF

Note: The Intel(R) Ethernet SDI Adapter FM10420-100GbE-QDA2 is a Gen3 x16 card
that is electrically bifurcated into two x8 ports. Without a BIOS that supports
bifurcation, only 1 port is available.


================================================================================


Building and Installation
-------------------------
To build a binary RPM* package of this driver, run 'rpmbuild -tb
fm10k-<x.x.x>.tar.gz', where <x.x.x> is the version number for the driver tar
file.

Note: For the build to work properly, the currently running kernel MUST match
the version and configuration of the installed kernel sources. If you have just
recompiled the kernel reboot the system before building.

Note: RPM functionality has only been tested in Red Hat distributions.
_lbank_line_

1. Move the base driver tar file to the directory of your choice. For
   example, use '/home/username/fm10k' or '/usr/local/src/fm10k'.

2. Untar/unzip the archive, where <x.x.x> is the version number for the
   driver tar file:
   tar zxf fm10k-<x.x.x>.tar.gz

3. Change to the driver src directory, where <x.x.x> is the version number
   for the driver tar:
   cd fm10k-<x.x.x>/src/

4. Compile the driver module:
   # make install
   The binary will be installed as:
   /lib/modules/<KERNEL
VERSION>/updates/drivers/net/ethernet/intel/fm10k/fm10k.ko

   The install location listed above is the default location. This may differ
   for various Linux distributions.

5. Load the module using the modprobe command:
   modprobe <fm10k> [parameter=port1_value,port2_value]

   Make sure that any older fm10k drivers are removed from the kernel before
   loading the new module:
   rmmod fm10k; modprobe fm10k

6. Assign an IP address to the interface by entering the following,
   where ethX is the interface name that was shown in dmesg after modprobe:
  
   ip address add <IP_address>/<netmask bits> dev ethX

  NOTE: Before proceeding, ensure that netdev is enabled and that a
  switch manager is running. To enable netdev, use one of the following
  commands:
  #ifconfig <netdev> up
  or
  #ip link set <netdev> up

7. Verify that the interface works. Enter the following, where IP_address
   is the IP address for another machine on the same subnet as the interface
   that is being tested:
   ping <IP_address>

Note: For certain distributions like (but not limited to) RedHat Enterprise
Linux 7 and Ubuntu, once the driver is installed the initrd/initramfs file may
need to be updated to prevent the OS loading old versions of the fm10k driver.
The dracut utility may be used on RedHat distributions:
	# dracut --force
   For Ubuntu:
	# update-initramfs -u


================================================================================


Command Line Parameters
-----------------------
If the driver is built as a module, the following optional parameters are used
by entering them on the command line with the modprobe command using this
syntax:
modprobe fm10k [<option>=<VAL1>]

For example:
modprobe fm10k max_vfs=7

The default value for each parameter is generally the recommended setting,
unless otherwise noted.


RSS
---
Valid Range: 0-128
0 = Assign up to the lesser value of the number of CPUs or the number of queues
X = Assign X queues, where X is less than or equal to the maximum number of
queues (128 queues).


max_vfs
-------
This parameter adds support for SR-IOV. It causes the driver to spawn up to
max_vfs worth of virtual functions.
Valid Range:0-64

NOTE: This parameter is only used on kernel 3.7.x and below. On kernel 3.8.x
and above, use sysfs to enable VFs. Also, for Red Hat distributions, this
parameter is only used on version 6.6 and older. For version 6.7 and newer, use
sysfs. For example:
#echo $num_vf_enabled > /sys/class/net/$dev/device/sriov_numvfs	//enable
VFs
#echo 0 > /sys/class/net/$dev/device/sriov_numvfs	//disable VFs

The parameters for the driver are referenced by position. Thus, if you have a
dual port adapter, or more than one adapter in your system, and want N virtual
functions per port, you must specify a number for each port with each parameter
separated by a comma. For example:

  modprobe fm10k max_vfs=4

This will spawn 4 VFs on the first port.

  modprobe fm10k max_vfs=2,4

This will spawn 2 VFs on the first port and 4 VFs on the second port.

NOTE: Caution must be used in loading the driver with these parameters.
Depending on your system configuration, number of slots, etc., it is impossible
to predict in all cases where the positions would be on the command line.

NOTE: Neither the device nor the driver control how VFs are mapped into config
space. Bus layout will vary by operating system. On operating systems that
support it, you can check sysfs to find the mapping.

NOTE: When SR-IOV mode is enabled, hardware VLAN filtering and VLAN tag
stripping/insertion will remain enabled. Please remove the old VLAN filter
before the new VLAN filter is added. For example,
ip link set eth0 vf 0 vlan 100	// set vlan 100 for VF 0
ip link set eth0 vf 0 vlan 0	// Delete vlan 100
ip link set eth0 vf 0 vlan 200	// set a new vlan 200 for VF 0


Configuring SR-IOV for improved network security
------------------------------------------------
In a virtualized environment, on Intel(R) Ethernet Server Adapters that support
SR-IOV, the virtual function (VF) may be subject to malicious behavior.
Software-generated layer two frames, like IEEE 802.3x (link flow control), IEEE
802.1Qbb (priority based flow-control), and others of this type, are not
expected and can throttle traffic between the host and the virtual switch,
reducing performance. To resolve this issue, configure all SR-IOV enabled ports
for VLAN tagging. This configuration allows unexpected, and potentially
malicious, frames to be dropped.


Configuring VLAN tagging on SR-IOV enabled adapter ports
--------------------------------------------------------
To configure VLAN tagging for the ports on an SR-IOV enabled adapter, use the
following command. The VLAN configuration should be done before the VF driver
is loaded or the VM is booted.

$ ip link set dev <PF netdev id> vf <id> vlan <vlan id>

For example, the following instructions will configure PF eth0 and the first VF
on VLAN 10.
$ ip link set dev eth0 vf 0 vlan 10


================================================================================


Additional Features and Configurations
-------------------------------------------

Configuring the Driver on Different Distributions
-------------------------------------------------
Configuring a network driver to load properly when the system is started is
distribution dependent. Typically, the configuration process involves adding an
alias line to /etc/modules.conf or /etc/modprobe.conf as well as editing other
system startup scripts and/or configuration files. Many popular Linux
distributions ship with tools to make these changes for you. To learn the
proper way to configure a network device for your system, refer to your
distribution documentation. If during this process you are asked for the driver
or module name, the name for the Base Driver is fm10k.


Viewing Link Messages
---------------------
Link messages will not be displayed to the console if the distribution is
restricting system messages. In order to see network driver link messages on
your console, set dmesg to eight by entering the following:
dmesg -n 8

NOTE: This setting is not saved across reboots.


Jumbo Frames
------------
Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU)
to a value larger than the default value of 1500.

Use the ifconfig command to increase the MTU size. For example, enter the
following where <x> is the interface number:

   ifconfig eth<x> mtu 9000 up
Alternatively, you can use the ip command as follows:
   ip link set mtu 9000 dev eth<x>
   ip link set up dev eth<x>

This setting is not saved across reboots. The setting change can be made
permanent by adding 'MTU=9000' to the file:
/etc/sysconfig/network-scripts/ifcfg-eth<x> for RHEL or to the file
/etc/sysconfig/network/<config_file> for SLES.

NOTE: The maximum MTU setting for Jumbo Frames is 15342. This value coincides
with the maximum Jumbo Frames size of 15364 bytes.

NOTE: This driver will attempt to use multiple page sized buffers to receive
each jumbo packet. This should help to avoid buffer starvation issues when
allocating receive packets.


ethtool
-------
The driver utilizes the ethtool interface for driver configuration and
diagnostics, as well as displaying statistical information. The latest ethtool
version is required for this functionality. Download it at:
http://ftp.kernel.org/pub/software/network/ethtool/

Supported ethtool Commands and Options for Filtering
----------------------------------------------------
-n --show-nfc
  Retrieves the receive network flow classification configurations.

rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6
  Retrieves the hash options for the specified network traffic type.

-N --config-nfc
  Configures the receive network flow classification.

rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6
m|v|t|s|d|f|n|r...
  Configures the hash options for the specified network traffic type.

  udp4 UDP over IPv4
  udp6 UDP over IPv6

  f Hash on bytes 0 and 1 of the Layer 4 header of the rx packet.
  n Hash on bytes 2 and 3 of the Layer 4 header of the rx packet.


NAPI
----
NAPI (Rx polling mode) is supported in the fm10k driver.
For more information on NAPI, see
https://www.linuxfoundation.org/collaborate/workgroups/networking/napi


Flow Control
------------
The Intel(R) Ethernet Switch Host Interface Driver does not support Flow
Control. It will not send pause frames. This may result in dropped frames.


VXLAN Overlay HW Offloading
---------------------------

Virtual Extensible LAN (VXLAN) allows you to extend an L2 network over an L3
network, which may be useful in a virtualized or cloud environment. Some
Intel(R) Ethernet Network devices perform VXLAN processing, offloading it from
the operating system. This reduces CPU utilization.

VXLAN offloading is controlled by the tx and rx checksum offload options
provided by ethtool. That is, if tx checksum offload is enabled, and the
adapter has the capability, VXLAN offloading is also enabled.

If rx checksum offload is enabled, then the VXLAN packets rx checksum will be
offloaded, unless the command #ethtool -K $INTERFACE_NAME rx off was used to
specifically disable the VXLAN rx offload.

VXLAN Overlay HW Offloading is enabled by default. To view and configure VXLAN
offload on a VXLAN-overlay offload enabled device, use the following command:

  # ethtool -k ethX
   (This command displays the offloads and their current state.)

For more information on configuring your network for overlay HW offloading
support, refer to the Intel Technical Brief, "Creating Overlay Networks Using
Intel Ethernet Converged Network Adapters" (Intel Networking Division, August
2013):

http://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/
overlay-networks-using-converged-network-adapters-brief.pdf


================================================================================


Known Issues/Troubleshooting
----------------------------

FUM_BAD_VF_QACCESS error on port reset
--------------------------------------
A FUM_BAD_VF_QACCESS error may be written to the message buffer when a command
or application triggers a reset on the port's physical function (PF). When the
PF is reset, any bound virtual functions (VFs) can no longer access their
queues. This behavior is expected. No user intervention is required. After the
PF reset is complete, the VFs will be able to access their queues normally.


Traffic and pings fail to pass after switch reset command issued
----------------------------------------------------------------
Resetting the switch may cause a failure to pass traffic and pings, with a
"init_hw failed: -4" message. To fix this, reload the VF driver or unbind
and rebind the VF device.


Packets dropped when issuing ping with very large payload
---------------------------------------------------------

If you issue a ping with a very large payload, you may see packets dropped.
Very large frames can fragment into many descriptors with a long enough delay
that the CPU will go to sleep. If the CPU is in a deep C-state, it may not
wake fast enough to copy those packets out of the internal descriptor cache
and into host memory. If some descriptors are dropped, then the whole
fragmented frame will be discarded because the ping is not able to located
all required fragments.

To fix this, ensure the CPU does not enter a deep C-state by using one of
the following methods:
  - Change your BIOS to disable the lowest C-states
  - Change the CPU governor
  - Use /dev/cpu_dma_latency (see the Linux Kernel's Documentation folder)


Driver cannot bind to PF when VM assigned to PF is started
----------------------------------------------------------
In a virtualization setup, if the fm10k driver is not installed on the host OS
and a VM that has been assigned one of the PCI PF host interface devices is
started, the fm10k driver can no longer bind to the PCI PF host interface
devices on the host OS.

To correct this, manually unbind the bus driver from the PF host interface
device (or stop the VM) and then manually bind the fm10k driver to the PF host
interface device. Alternatively, you can add "fm10k" to the device's
driver_override entry in the /sys filesystem to prevent the bus driver from
binding to the PF host interface device in the first place.


================================================================================


Support
-------
For general information, go to the Intel support website at:
http://www.intel.com/support/

or the Intel Wired Networking project hosted by Sourceforge at:
http://sourceforge.net/projects/e1000
If an issue is identified with the released source code on a supported kernel
with a supported adapter, email the specific information related to the issue
to e1000-devel@lists.sf.net.


================================================================================


License
-------
This program is free software; you can redistribute it and/or modify it under
the terms and conditions of the GNU General Public License, version 2, as
published by the Free Software Foundation.

This program is distributed in the hope it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program; if not, write to the Free Software Foundation, Inc., 51 Franklin
St - Fifth Floor, Boston, MA 02110-1301 USA.

The full GNU General Public License is included in this distribution in the
file called "COPYING".

Copyright(c) 2015-2017 Intel Corporation.
================================================================================


Trademarks
----------
Intel and Itanium are trademarks or registered trademarks of Intel Corporation
or its subsidiaries in the United States and/or other countries.

* Other names and brands may be claimed as the property of others.