Chapter 2 Tools used in the project


This chapter introduces the tools that will be used in this project and their source codes.


2.1 Blast Test Code (By Prof. Bob Russell, UDP version is by Chaoxing Lin)


Most of the performance experiments will use blast test tools. Here is how it works. Client side sends a number (iteration times, specified from command line) of packets of given request size (specified from command line) on end and close the connection. Server side keeps receiving packets and swallows it. Then when client closes connection it sends 1 byte back. For the complete blast test code, please refer to


2.2 Packet Dropper Code (By Glenn Herrin, Modified by Chaoxing Lin)


The first impairment we introduce to the router kernel is packet dropper module. This is a kernel module. It randomly drops packet with a specified destination address at a given rate. Code are inserted and executed on virtual DEVICE layer.


On receiving each packet, the Dropper checks the destination IP address. If the destination is the target IP, we get a 16-bit random number, if the random number is less than the dropping threshold value which is rate*65535, we drop this packet, otherwise, process it normally.




This is what dev_queue_xmit will call while this module is installed ****/


int packet_dropper(struct sk_buff *skb)


  unsigned short t;

  if (skb->nh.iph->daddr == target) {


  /* the following code is modified by lin: begin (*/

        t = getUnsignedShortRandom();

        if (t < cutoff)



        return 1;    /* drop this packet */


  /* modified by lin : end ) */



  return 0;             /* continue with normal routine */

}  /* packet_dropper */


The random number is generated in this way. This way is only good for Intel processor because the assembly instruction “rdtsc” is only in Intel processor.


inline unsigned short getUnsignedShortRandom()


        unsigned l, h;

        unsigned short low;

        unsigned char * lp;

        unsigned char * hp;

        unsigned char ldata;


     /* get a CPU cycle. Only good for Intel processor */

         __asm__ volatile("rdtsc": "=a" (l), "=d" (h));


         /* get the lower 16 bits */

         low = (unsigned short) l & 0xFFFF;


         /****Swap lower byte with the higher byte *******/

         hp=(unsigned char*)(&low);/*point to high byte */

         lp=hp+1;                   /* point to low byte */

         /* swap the higher byte with the lower byte */

         ldata = *lp;

         *lp = *hp;

         *hp = ldata;

         return low;



The packet dropper module also exports symbols that will be used in proc file system:


unsigned short cutoff;           /* drop threshold value */
float rate;                    /* drop percentage */
unsigned number_of_packet_dropped;/* #of packet dropped */
_u32 target = 0x0202A8C0;      /* address */

Kernel source changed:




at line: 942  Add:          int (*xmit_test_function)( struct sk_buff * ) = 0;

See the context:


    919 #ifdef CONFIG_HIGHMEM

    920 /* Actually, we should eliminate this check as soon as we know, that:

    921  * 1. IOMMU is present and allows to map all the memory.

    922  * 2. No high memory really exists on this machine.

    923  */


    925 static inline int

    926 illegal_highdma(struct net_device*dev,struct sk_buff *skb)

    927 {

    928     int i;


    930     if (dev->features&NETIF_F_HIGHDMA)

    931         return 0;


    933     for (i=0; i<skb_shinfo(skb)->nr_frags; i++)

    934         if (skb_shinfo(skb)->frags[i].page >=highmem_start_page)

    935             return 1;


    937     return 0;

    938 }

    939 #else

    940 #define illegal_highdma(dev, skb)   (0)

    941 #endif


    943 /* ADDED BY LIN to test packet-dropper:begin (   */


    945 int (*xmit_test_function)( struct sk_buff * ) = 0;

     946 /* ADDED BY LIN to test packet-dropper:end   )  */



 *  dev_queue_xmit - transmit a buffer

 *  @skb: buffer to transmit


 * Queue a buffer for transmission to a network device. The caller must

 *  have set the device and priority and built the buffer *before calling this  function. The function can be called *from an interrupt.

 *  A negative errno code is returned on a failure. A success does not

 *  guarantee the frame will be transmitted as it may be dropped due

 *  to congestion or traffic shaping.




In function: int dev_queue_xmit(struct sk_buff *skb)

At the very beginning, add:


if ( xmit_test_function && ( *xmit_test_function )(skb) )


    kfree_skb( skb );

    return 0;



See the context:



int dev_queue_xmit(struct sk_buff *skb)


    struct net_device *dev = skb->dev;

    struct Qdisc  *q;



if ( xmit_test_function && ( *xmit_test_function )(skb) )


    kfree_skb( skb );

    return 0;




    if (skb_shinfo(skb)->frag_list &&

        !(dev->features&NETIF_F_FRAGLIST) &&

        skb_linearize(skb, GFP_ATOMIC) != 0) {


        return -ENOMEM;





at line: 570  Add:


extern int ( *xmit_test_function ) ( struct sk_buff * );

EXPORT_SYMBOL_NOVERS( xmit_test_function );


After changing kernel source code, don’t forget to re-compile the kernel.


For the complete packet_dropper code, please refer to



2.3 Proc File System Operation Code


In order to check the exact number of packets dropped by the packet dropper and to easily change dropping rate and destination IP address, we develop a kernel module to do this job. It dynamically probes current dropping rate, destination packet, number of packet dropped. It can also set these values.


static struct proc_dir_entry *dropperInfo, *fileEntry;

static int proc_reset_num(struct file *file,const char *

                        buffer,unsigned long count,void *data);

static int proc_read_num(char *buf, char **start,

                off_t off, int count,int *eof, void *data);

static int proc_read_dropper(char *buf, char **start,

                off_t off, int count,int *eof, void *data);

static int proc_write_dropper(struct file *file,const char

                        *buffer,unsigned long count,void *data);


int init_module()


        int rv = 0;


        /* create directory */

        dropperInfo = proc_mkdir(“dropperInfo”, NULL);

        if(dropperInfo == NULL) {

                printk("<1> dropperInfo failed\n");

                rv = -ENOMEM;

                goto out;


        dropperInfo->owner = THIS_MODULE;


        /* create “dropper” and "numDropped" files */

        fileEntry = create_proc_entry("dropper", 0644, dropperInfo);

        if(fileEntry == NULL) {

                rv = -ENOMEM;

                goto error;


        fileEntry->read_proc = proc_read_dropper;

        fileEntry->write_proc = proc_write_dropper;

        fileEntry->owner = THIS_MODULE;


        fileEntry = create_proc_entry("numDropped", 0644, dropperInfo);

        if(fileEntry == NULL) {

                rv = -ENOMEM;

                goto error;


        fileEntry->read_proc = proc_read_num;

        fileEntry->write_proc = proc_reset_num;

        fileEntry->owner = THIS_MODULE;


        /* everything OK */

        printk(KERN_INFO "%s  initialized\n", MODULE_NAME );


        return 0;


        remove_proc_entry(MODULE_NAME, NULL);

        out:    return rv;



Inserting this module to kernel will create a directory /proc/dropperInfo. In this directory there will be 2 files:



         Use “cat  /proc/dropperInfo/dropper” to see the rate, ip, # of packet dropped.

         Use “input rate ip_addr” to reset rate and ip_addr dynamically.


         Use “cat   /proc/dropperInfo/numDropped” to see # of packet dropped since last


      Use “reset  to set “# of packet dropped” to 0


For the complete proc operation code,

please refer to


For the proc file system handling, see


2.4 Tool used to synchronize router and source host


For some the experiments, as far as the control is on the same machine, we can write a shell script to let the experiments run again and again.  But it’s almost impossible to write a script to control remote machines. In here, it’s really hard to write script to synchronize the router and the sending host. We need to run blast test for a few times at a given dropping rate. Then on router side, we need to set a new dropping rate and let experiments go on.

Our goal is to reduce the human interactions to the minimum when we do these experiments. The automation tool will also make it easy to repeat the experiments with the same settings sometime later.


Based on this, we develop a simple tool to synchronize the router and sending host. This tool contains 2 applications


dublin” is to be run as server on

madrid” is to be run as client on


Step 1. Dublin installs “packet dropper” module.


Step 2. Dublin installs “proc file system manipulation” module.


Step 3. Dublin opens server port 5678 and waits for connection. And after connection is

 created, it waits for control message from Madrid and do as instructed. Control

messages are:


SET_RATE      rate:     Dublin sets dropping rate to rate. It also creates a directory

with the name rate. Number of packet loss at this rate will be put under this



RECORD_LOSS:        Dublin checks the proc file system, get the number of

packets dropped since last reset. And then it resets the number_of_packet to 0.


SET_ITERATION times: (used only in regular dropping test with dropping rate

 1/6 ). Dublin creates a directory with the name times. Number_of_packet loss

will be put under this directory.


Step 4 Prague opens “blast test server”( using port 1026 because it’s hard coded in this

            tools )


Step 5.Madrid opens tool client. It sends control message to Dublin to set rate. After it

gets acknowledgement, it does experiment(blastclient 1026 100000

1448). After each experiment, Madrid sends control message to Dublin to record

 the packet loss. For each specific rate, Madrid does 10 times experiments and

then increment the rate and set it to Dublin.


Note: Don’t forget to redirect the result to a file. We will use this file to do statistics.


Step 6 On Dublin, run script “collectRawDropNumwhose content is:


cd  dirName

for d in `ls`;


        cd $d

        echo $d >> /home/lin/Tools/rawDrop

        for f in `ls`;


                cat $f >> /home/lin/Tools/rawDrop


        cd ..


cd ..

(Suppose that dirName is the directory created by “dublin”) we will get a file with the number of packets dropped for each experiment.


Run “getNum rawDrop dropStat”, we will get the statistical result of “number of packet dropped during each experiment”


Step 7. On Madrid, Suppose that we redirect the experiment result to a file “rawElapse“,

 rungetElapse rawElapse elapseStat” we can get the statistical result. By this

result we can run gnuplot to draw the graph.


For the complete AutoTools code,

please refer to