SmartBits & NIST Net Testing

NIST Net Calibration

Components

The NIST Net calibration test bed consists of a Redhat 7.3 Linux router (2.4.22) and a Spirent Communications SmartBits 6000B performance analysis system. The router is running NIST Net version 2.0.12, with IPv4 forwarding enabled and contains two NetGear GA622T 10/100/1000 Copper Gigabit Ethernet cards with addresses 10.0.0.1 [eth2] and 11.0.0.1 [eth3]. The NetGear cards have no separate on board memory chips since all needed memory is integrated into the MAC chip. The router hardware consists of a Tyan S2462UNG motherboard, AMD Athlon MP2000 (1.67 GHz) processor and 1-GB of PC2100 DDR SDRAM.

The SmartBits chassis (firmware 2.50.009) contains a LAN-3301A 10/100/1000 Two-port Copper Ethernet TeraMetrics module (firmware 4.10.002) with 256MB of memory. The module ports (chassis slot 2, ports 0 and 1) have assigned addresses 10.0.0.10 [Port 0] and 11.0.0.11 [Port 1]. All four network interface cards are running in Full Duplex mode at 1000Mbs with auto-negotiation disabled. The SmartLibrary (see below) functionality and NIST Net settings are being called through the use of TCL (8.4.2) code, which is running on a Redhat 8.0 (2.4.18) system connected to the SmartBits chassis and router through a 100Mbs Class B Ethernet/IP network.

Certain software, equipment, instruments, or materials are identified in order to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the materials or equipment identifed are necessarily the best available for the purpose. Product names mentioned are trademarks of their respective manufacturers.

Testing & Results

Using SmartLibrary (version 3.50), Spirent’s function library for controlling SmartCards, hardware parameters are configured and reserved, traffic streams are created and transmitted, and received packet information is captured into histograms. When multiple streams are used the fourth-octet transmit address is incremented and created; 10.0.0.[10...109]. To assure accuracy with the calibration results, all additional traffic was denied access to subnets 10 and 11 for the duration of the testing. Below is a segment of code:

    #--------------------------------------------------------
    # Send 60 Second Continual Burst; Create, Start Histogram
    #--------------------------------------------------------

    HGStop
    HTGap $gap $hub $slot $port0
    HTTransmitMode $CONTINUOUS_PACKET_MODE $hub $slot $port0
    HTRun $HTRUN $hub $slot $port0
    after 10000

        # Create Raw Tags Histogram
        HTSetCommand $L3_HIST_RAW_TAGS 0 0 0 "" $hub $slot $port1
        HTSetCommand $L3_HIST_START 0 0 0 "" $hub $slot $port1

after 50000
HTRun $HTSTOP $hub $slot $port0

#--------------------------------------------------------

A total of 4,000 calibration runs were completed and were broken down into 50 sets of 80 tests. We achieved the number 80 (4x5x4) by exhausting all possible combinations of:

4 Stream Sets (1, 10, 50, 100) [flows]
5 Offered Loads (10, 30 , 50, 70, 90) [percent]
4 Packet Sizes (128, 512, 1024, 1500) [bytes]

The 50 sets were a result of 48 (8x6) NIST Net settings plus two experimental control tests; 1) SmartBits chassis loopback and 2) the router with the emulater disabled. Also of note, configuring NIST Net with delays over 1000ms overran the network interface controller buffer in the router and crashed the system.

8 Delay Settings (0, 10, 20, 50, 100, 200, 500, 1000) [ms]
6 Variance Settings (0, 10, 50, 100, 200, 500) [ms]

Each of the 4,000 histograms contain over 200,000 lines of captured packet information with each line consisting of a stream number, sequence number and latency time. Over 800,000,000 total packets were captured resulting in ~2.5 billion data points. The tests took 10 days to complete and amassed over 14GB of information.

The files are organized in a hierarchy where Delay and Variance values are directories each containing 80 files in the Streams.OfferedLoad.PacketSize format. For example:

dragon [100]{.../Delay_100/Variance_50}-> pwd
/users/www/proj-nms/NN_Calibration/HISTOGRAMS/Delay_100/Variance_50

dragon [101]{.../Delay_100/Variance_50}-> ls -l 10.70.1024
-rw-r--r-- 1 santay nms 3455091 Feb 18 01:03 10.70.1024

    dragon [102]{.../Delay_100/Variance_50}-> head -3 10.70.1024
    5 26397 1570340
    8 26755 1135602
    3 26837 1036851

Presentation

Once the raw data had been collected histogram.stats files were generated, which contain Mean and Standard Deviation calculations of the latency values. An interface to explore that data has be created and generates Gnuplots of the specified search. An example of such a plot is below.

A few items of note about the calibration should be mentioned... In most cases test runs with 128-byte packets experienced up to 20 percent packet loss. Also, due to the limitations of the testing platform (namely 16-bit counters), and the data pattern disruption caused by NIST Net Delay and Variance, the raw data collected required modest interpretation to understand the effects of looping sequence numbers and unordered packet arrival.

For all inquires about these calibration efforts, results or methodoligies or to request the raw data please send email to nistnet-dev@antd.nist.gov.