Research

• Transmission Control Protocol
• Flowgrind
  • Motivation
  • Architecture
  • Output
  • Manpages
• Netgrind
• Autoconfiguration

 

NAME

flowgrind - network performance measurement tool  

SYNOPSIS

flowgrind [general options] [flow options]  

DESCRIPTION

flowgrind is a distributed network performance measurement tool. Using the flowgrind controller, tests can be setup between hosts running flowgrindd(1) , the flowgrind daemon.

Flowgrind performs extensive TCP tests. On systems running the Linux kernel flowgrind collects and reports the TCP metrics returned by the TCP_INFO socket option, such as the size of the congestion window or the RTT.

Each test connection is called a flow and flowgrindd has to be running on every host that supped to be a flow endpoint. The flowgrind controller only sets up the flows as well as gathers and prints the results. The controller does however not actually participate in the test. However, mixing 32bit and 64bit systems can potentially lead to unexpected results and is therefore not recommended.

 

OPTIONS

 

General information

Two important groups of options are the global options and flow options. Like the name suggests, global options apply globally and potentially affect all flows. Flow-specific options only apply to the subset of flows selected using the -F option.

 

Miscellaneous

-h

Show help and exit.

-h [s|g]

show additinal help for socket options and traffic generation.

-v

Print version information and exit.

 

General options

-b mean1,mean2,mean3

-b lwr_bound1,upr_bound1,lwr_bound2,upr_bound2,lwr_bound3,upr_bound3

Perform Anderson-Darling Test for exponential distribution OR lower and upper bounds for computing the test for uniform distribution with the given bounds. Test data always generated from first flow.

-c -begin,-end,-thrpt,-transac,+blocks,-rtt,-iat,-kernel,+status

Comma separated list of column groups to display in output. Prefix with either + to show column group, - to hide column group.

-d

Increase debugging verbosity. Add option multiple times to be even more verbose. Only available if compiled with ./configure --enable-debug set.

-e PREFIX

Prepend PREFIX to log and dump filename (default: "flowlog-").

-i #.#

Reporting interval in seconds (default: 0.05s).

-l NAME

Use log filename NAME instead. If not specified the current time is used for the filename.

-m

Report throughput in 2**20 bytes/second (MiB/s) (default: 10**6 bit/sec (Mbit/s)).

-n #

Number of test flows (default: 1).

-o

Overwrite existing log files (default: don't).

-p

don't print symbolic values (like INT_MAX) instead of numbers (default: off)

-q

Be quiet, do not log to screen (default: off).

-w

Write output to logfile (default: off).

 

Flow options

All flows have two endpoints, a source and a destination. The distinction between source and destination endpoints only affects connection establishment. When starting a flow the destination endpoint listens on a socket and the source endpoint connects to it. For the actual test this makes no difference, both endpoints have exactly the same capabilities. Data can be sent in either direction and many settings can be configured individually for each endpoint.

Some of these options take the flow endpoint as argument. Is is denoted by 'x' in the option syntax. 'x' needs to be replaced with either 's' for the source endpoint, 'd' for the destination endpoint or 'b' for both endpoints. To specify different values for both endpoints, separate them by comma.

Example: -T s=5,d=10

-A x

activate minimal request and response size for RTT calculation. (same as -G p=C,24)

-B x=#

Set requested sending buffer in bytes.

-C x

Stop flow if it is experiencing local congestion.

-D x=DSCP

DSCP value for type-of-service IP header byte.

-E x

Enumerate bytes in payload (default: don't).

-F #[,#]*

Comma-separated list of flows. Flow options following this option apply only to the specified flows. Useful in combination with -n to set specific options for certain flows. Numbering starts with 0, so -F 1 refers to the second flow. All flow options before the first -F apply to all flows.

-G x=[q|p|g],[C|E|P|N|U],#1,(#2)

Activate stochastic traffic generation and set parameters according to the used distribution.

-H x=HOST[/RPCADDRESS[:PORT]]

Test from/to HOST. Optional argument is the address and port of the RPC server. An endpoint that isn't specified is assumed to be 127.0.0.1.

-J #

Use random seed # (default: read /dev/urandom).

-L

Call connect() on test socket immediately before starting to send data (late connect). If not specified the test connection is established in the preparation phase before the test starts.

-N

Call shutdown() after test flow is scheduled to end.

-M x

dump traffic using libpcap.

-O x=OPT

Set specific socket options on test socket. For a list of supported socket options see -h s option.

-P x

Do not iterate through select() to continue sending in case block size did not suffice to fill sending queue (pushy).

-Q

Summarize only, skip interval reports (quiet).

-R x=#.#[z|k|M|G][b|B][p|P]

Rate limiting. Send data at specified rate per second, where: z=2**0,k=2**10,M=2**20,G=2**30. b=bitspersecond(default),y=bytespersecond,B=blockspersecond. p=periodic(default),P=Poissondistributed.

-T x=#.#

Set flow duration, in seconds (default: s=10,d=0).

-U#

Set application buffer size (default: 8192). truncates values if used with stochastic traffic generation.

-S x=#

Set block size, same as -G s=q,C,#.

-W x=#

Set requested receiver buffer (advertised window) in bytes.

-Y x=#.#

Set initial delay before the host starts to send data.

 

Traffic Generation Options

-G x=[q|p|g],[C|U|E|N|L|P|W],#1,(#2)

Activate stochastic traffic generation and set parameters for the chosen distribution.

use distribution for the following flow parameter:

q request size (in bytes)

p response size (in bytes)

g request interpacket gap (in s)
                possible distributions:

C constant (param 1: value, param 2: not used)

U uniform (param 1: min, param 2: max)

E exponential (param 1: lamba - lifetime, param 2: not used)

N normal (param 1: mu - mean value, param 2: sigma_square - variance)

P pareto (param 1: k - shape, x_min - scale)

W weibull (param 1: lambda - scale, param 2: k - shape)

L lognormal (param 1: zeta - mean value, param 2: sigma - std dev)

advanced distributions like weibull are only available if flowgrind is compiled with libgsl support.

-U # specify a cap for the calculated values for request and response sizes, needed because the advanced distributed values are unbounded, but we need to know the buffersize (it's not needed for constant values or uniform distribution). Values outside the bounds are recalculated until a valid result occurs but at most 10 times (then the bound value is used).

 

EXAMPLES

flowgrind

default settings, same as flowgrind -H b=127.0.0.1 -T s=10,d=0

flowgrind -H s=host1,d=host2

Start bulk TCP transfer with host1 as source and host2 as destination endpoint. Both endpoints need to be running the flowgrind daemon. The default flow options are used, with a flow of 10 seconds duration with data sent from the source to the destination endpoint.

flowgrind -H s=host1,d=host2 -T s=0,d=10

Same as the above but instead with a flow sending data for 10 seconds from the destination to the source endpoint.

flowgrind -n 2 -F 0 -H s=192.168.0.1,d=192.168.0.69 -F 1 -H s=10.0.0.1,d=10.0.0.2

Setup two flows, first flow between 192.168.0.1 and 192.168.0.69, second flow between 10.0.0.1 to 10.0.0.2

flowgrind -p -H s=10.0.0.100/192.168.1.100,d=10.0.0.101/192.168.1.101 -A s

Setup one flow between 10.0.0.100 and 10.0.0.101 and use 192.168.1.x IP addresses for configuration. Activate minimal response for RTT calculation and show numerical values.

flowgrind -G s=q,C,400 -G s=p,N,2000,50 -G s=g,U,0.005,0.01 -U 32000

q,C,400 use constant request size of 400 bytes p,N,2000,50 use normal distributed response size with mean 2000 bytes and variance 50 g,U,0.005,0.01 use uniform distributed interpacket gap with min 0.005s and and max 10ms -U 32000 truncate block sizes at 32 kbytes (needed for normal distribution)

 

Traffic Generation Scenarios

The following examples demonstrate how Traffic Generation can be used. These have been incorporated in different tests for flowgrind and have been proven meaningful. But as Internet Traffic is diverse, there is no guarantee that these are approicated in every situation.

Request Response Style (HTTP)

This scenario is based on the work in http://www.3gpp2.org/Public_html/specs/C.R1002-0_v1.0_041221.pdf

flowgrind -r 42 -M s -G s=q,C,350 -G s=p,L,9055,115.17 -U 100000

-r 42

Use random seed 42 to make measurements reproduceable

-M s

Dump traffic on sender side

-G s=q,C,350

Use constant requests size 350 bytes..TP -G s=p,L,9055,115 Use lognormal distribution with mean 9055 and variance 115 for response size

-U 100000

Truncate response at 100 kbytes

For this scenario we recommened to focus on RTT (lower values are better) and Network Transactions/s as metric (higher values are btter).

Interactive Session (Telnet)

This scenario emulates a telnet session.

flowgrind -G s=q,U,40,10000 -G s=q,U,40,10000 -O b=TCP_NODELAY

-G s=q,U,40,10000 -G s=q,U,40,10000

Use Uniform distributed request and response size between 40 bytes and 10 kilobytes

-O b=TCP_NODELAY

Set socket options TCP_NODELAY as used by telnet applications.

For this scenario RTT (lower is better) and Network Transactions/s are useful metrics (higher is better).

Rate Limited (Streaming Media)

This scenario emulates a video stream transfer with a bitrate of 800 kbit/s.

flowgrind -G s=q,C,800 -G s=g,N,0.008,0.01

Use normal distributed interpacket gap with mean 0.008 and a small variance (0.001). In conjuction with request size 800 bytes a average bitrate of approx 800 kbit/s is achieved. The variance is added to emulate a variable bitrate like it's used in todays video codecs.

For this scenario the IAT (lower is better) and minimal throughput (higher is better) are interesting metrics.

 

OUTPUT COLUMNS

#

The endpoint, either S for source or D for destination.

ID

The numerical flow identifier.

begin and end

The boundaries of the measuring interval in seconds. The time shown is the elapsed time since receiving the RPC message to start the test from the daemons point of view.

 

Application layer metrics

through

The transmitting goodput of the flow endpoint during this measurement interval, measured in Mbit/s (default) or MB/s (-m).

transac

The number of successfully received response blocks per second (we call it network transactions/s).

requ/resp

The number of request and response block sent during this measurement interval (column disabled by default)

IAT and RTT

The 1-way and 2-way block (application layer) delays respectively block IAT and block RTT. For both delays the minimum and maximum encountered values in that interval are displayed in addition to the arithmetic mean. If no block acknowledgement arrived during that report interval, inf is displayed (for example when no responses are send, if in doubt try -A s)

 

Kernel metrics (TCP_INFO)

cwnd (tcpi_cwnd)

Size of TCP congestion window in number of segments. All TCP specific metrics are obtained from the Linux kernel through the TCP_INFO socket option at the end of every reporting interval.

ssth (tcpi_snd_sshtresh)

The slowstart threshold of the sender in number of segments.

uack (tcpi_unacked) and sack (tcpi_sacked)

Statistics about the number of unacknowledged and selectively acknowledged segments.

lost (tcpi_lost)

Number of segments assumed lost at the end of the reporting interval.

retr (tcpi_retrans)

Number of unacknowledged retransmitted segments.

tret (tcpi_retransmits)

Number of retransmissions of the same segment due a retransmission timeout.

fack (tcpi_fackets)

Number of segments between SND.UND and the highest selectively acknowledged sequence number.

reor (tcpi_reordering)

Segment reordering metric. The Linux kernel can detect and cope with reordering without loss of performance if the distance a segment gets displaced does not exceed the reordering metric.

rtt (tcpi_rtt) and rttvar (tcpi_rttvar)

TCP round-trip time and its variance given in ms.

rto (tcpi_rto)

The retransmission timeout given in ms.

bkof (tcpi_backoff)

Number of backoffs.

ca state (tcpi_ca_state)

Internal state of congestion control state machine as implemented in the Linux kernel. Can be one of open, disorder, cwr, recovery or loss.

Open

is the normal state. It indicates that there are no issues with the connection.

Disorder

is similar to Open but is entered upon receiving duplicate ACKs or selective acknowledgements as special attention might be neded in the near future.

CWR

is entered when the size of the congestion window got lowered due to receiving an ICMP Source Quench message or a notification from Explicit Congestion Notification (ECN).

Recovery

indicates that the congestion window got lowered and a segment is fast-retransmitted.

Loss

is entered if the RTO expires. Again the size of the congestion window got lowered in this state.

smss and pmtu

Sender maximum segment size and path maximum transmission unit in bytes.

 

Internal flowgrind state (only enabled in debug builds)

status

The state of the flow inside flowgrind for diagnostic purposes. It is a tuple of two values, the first for sending and the second for receiving. Ideally the states of both the source and destination endpoints of a flow should be symmetrical but since they are not synchronized they may not change at the same time. The possible values are:

c

Direction completed sending/receiving.

d

Waiting for initial delay.

f

Fault state.

l

Active state, nothing yet transmitted or received.

n

Normal activity, some data got transmitted or received.

o

Flow has zero duration in that direction, no data is going to be exchanged.

 

SEE ALSO

flowgrindd(1), flowgrind-stop(1)

---

 

Index

NAME

SYNOPSIS

DESCRIPTION

OPTIONS

General information

Miscellaneous

General options

Flow options

Traffic Generation Options

EXAMPLES

Traffic Generation Scenarios

OUTPUT COLUMNS

Application layer metrics

Kernel metrics (TCP_INFO)

Internal flowgrind state (only enabled in debug builds)

SEE ALSO