Plot of Bandwidth vs Block Size for memcpy between two 1600 MB buffers

Here’s data from another machine of mine, with ~2G physical RAM. I had to limit myself to using 2 x 800 MB buffers here.

Plot of Bandwidth vs Block Size for memcpy between two 800 MB buffers

From these two it seems that the maximum bandwidth is achieved by using a block size which is around half, though not exactly but definitely not less than half (see the huge drop in bandwidth to it’s left). I guess this behaviour is due to the implementation of memcpy in glibc (v2.11.1-1.x86_64). I haven’t looked at the sources yet, so I don’t know how to explain this, yet.

I’m including the program source here, please do leave a comment!

/* Dumb memory bandwidth measurer
   Best run in runlevel 1
*/

#include <stdio.h>
#include <sys/time.h>

#define MB * (1024 * 1024)
#define KB * (1024)
int main()
{
 int i, j;
 struct timeval t1, t2, t3;
 char *a, *b;
 int nsteps, max;
 int mem = 1500;

 a = malloc(mem MB);
 b = malloc(mem MB);

 mlock(a, mem MB);
 mlock(b, mem MB);

 nsteps = 160;
 max = mem;

 for (j = 1; j <= max; j += max / nsteps) {
   gettimeofday(&t1, NULL);

   for (i = 0; i < mem / j; i++)
     memcpy(a + i * j MB, b + i * j MB, j MB);

   gettimeofday(&t2, NULL);

   timersub(&t2, &t1, &t3);

   printf("%d %f\n", j, mem / ((t3.tv_usec / 1000000.0) + t3.tv_sec * 1.0));
 }
}

Credit : Data visualization using gnuplot.

First, the regular expression should be ‘compiled’ – not in the usual sense, but it’s converted to a format which increases the speed of pattern matching.

Here’s a simple example, which is used to validate rectangular dimension inputs.

#include <regex .h>

int main()
{
 char *regex  = "[1-9][0-9]\\{1,\\}x[1-9][0-9]\\{1,\\}";
 regex_t regc;

 regcomp(&regc, regex, 0);

 /* Does it match ? */
 return regexec(&regc, "800x500", 0, 0, 0) == 0;
}

Is the comment justified ? – I thought.. It felt to me if Cricket was totally batsmen centric ? Is cricket all about batting ? Batsmen don’t need to give a damn about bowling. When the regular bowlers get hit badly, do we ever shout ‘Why doesn’t xxx (a great batsmen) take the ball ?”. But see, the bowlers – They have to be able to decently bat and when all of the top order batsmen are gone, they are expected to bat well and save the game! No matter how exceptional bowler he may be he’s still a target of our criticism if he doesn’t..

Thinking more – It may not be all that bad! Bowlers kinda get an opportunity to correct their mistakes which means they don’t get OUT! But see the batsmen – they’re screwed if they lose concentration for a split second! Well, now it makes sense about why things are the way they are now..

Now, here’s something interesting. How about have a game in which the roles are reversed. Batsmen get to play for an over each, and Bowlers keep bowling till they get out (??) and .. ? .. ? Looking forward to your ideas..

Along the lines of the previous post One command called ‘clear’, I wrote a small C program that does clrscr().

/* clrscr in linux! */

#include <stdio .h>

void clrscr(void)
{
	char str[] = " [H [2J";
	str[3] = str[0] = 27;
	write(1, str, 7);
}

int main(int argc, char **argv)
{
	clrscr();
	printf("Hello World\n");

	return 0;
}

Disclaimer : May not work!

Well, I wanted to find out how it works. So the first thing I did was an strace to see what syscalls it did..

[root@cff ~]# strace clear

and I was left with..

)              = 7
exit_group(0)                           = ?
[root@cff ~]#

Well, it did clear the screen! I tried to redirect the output to a file. No use. An interesting idea struck me. What will happen if I redirect output of clear, I thought..

[root@cff ~]# clear > out
[root@cff ~]#

Wow! It didn’t clear the screen.. Let me see what’s in the file.. vim didn’t work out well. I used od.

[root@cff ~]# od -bc out
0000000 033 133 110 033 133 062 112
        033   [   H 033   [   2   J
0000007
[root@cff ~]#

Some weird ascii characters. Now, I did ‘cat out’ and I was surprised to see the terminal cleared! Wow!

Now even better, I wanted to use echo and clear the screen. Going through the echo manual, I found out how to do it. Here it is. Please replace <zero> with a 0. This damn wordpress for an unknown reason does not allow me to put a 0 in there.

 echo -e "\<zero>33[H\<zero>33[2J"

Execute this, and your terminal will be cleared.

If it didn’t, you have to do all the steps I performed and find out the byte sequence that will clear your terminal. Good luck!

Every big country spends so many billions of $$ on researching and acquiring armaments. What are we trying to solve by this ? I was wondering why no one even talks about hunger, illiteracy, health and various other problems plaguing the world ?

What we know of, is this one earth, this one family of Humans. Imagine if, in a family, the members started thinking about his/her security/welfare when others in the same family are dying out of malnutrition. What other than fools are we ?

One the same note, India’s most talked about space mission – Chandrayan came to my mind. India had produced thinkers, who were great astronomers, scientists, mathematicians etc. There are even accounts of having discovered the ‘Pythogoras theorem’ centuries before Pythogoras did etc. If the same trend continued, we should have landed on the moon, centruries before anyone else did. But it did not happen. Why ?

One can appreciate the answer if he can understand the very existence of ‘science’. Science is nothing but a study, an attempt to find about ourselves, our environment, and answer questions such as ‘Who are we ?’, ‘What are we doing here ?’. ‘What is death ? Where happens after that ?’ etc. In the east, men started searching for answers outside, but they quickly discovred its futility and started looking for, inside and this gave birth to Religion. It evolved continuously for centuries, till they found the truth, and thereafter they had no need of going out anymore as they had realised that what is within is without. But the west started and still continue to search outside. And, India where the eternal trurth was known for centuries, is trying to ‘keep up’ pace with the west, dumping all our treasure, as it were. Hey, do we need to keep up at all ? The bazillions spent on such programs could well be spent in solving India’s poverty, education and so many other problems.

Let’s make our brothers and sisters happy first. Jai Hind.

Back home I see that my BSNL broadband utilization stands at 3.1 GB out of 2.5 GB and its still 13th of the month. I was extremely shocked to see this.I quickly went to see the running processes and I see yum-updatesd taking up 200% of my CPU. I killed it right away! I believe that it is responsible for this notoriously large download. If you have to object, please comment!

I’ve disabled yum-updatesd for good. Damn it! I even once considered changing over to yum. But it was only in a fit of rage. Now I’m sticking to Fedora

This has may implications and my brain started thinking what they could be.. One thing is certain. Since its rival Novell is investing in XEN, Red Hat probably wanted to do something different. One more thing is that Red Hat did not have great influence on XEN, more less after XenSource was acquired by Citrix. So, KVM was one thing which Red Hat could potentially influence. An important thing to note is that, Red Hat already had expressed interest in Qumranet’s VDI solution a few months earlier.

The main consequence of this, I believe is that Red Hat products will have solid support for KVM. With the popularity of KVM growing by leaps and bounds, this could mean more customers and hence more revenue to Red Hat. This might also bring newer customers to KVM who might be now willing to invest in it seeing that it will be backed by Red Hat. All in all I see it as a very good move.

Another thing that surprises me is the money involved. Red Hat paid $107M in cash. Quite a big amount I feel. I feel that its not because of how much KVM is valuable now, but how much KVM will be valuable in a few years from now.

It’ll be indeed interesting to watch how things will go from here..

LZ77 is a very simple compresison method. It’s available almost everywhere. Do read about it.

/* compress.c - compression using LZ77 */
#include <stdio .h>
#include <stdint .h>
#include <string .h>
#include <sys /stat.h>
#include <unistd .h>
#include <sys /types.h>
#include <stdlib .h>

#define LBSIZE 8
#define SBSIZE 1024
#define WINDOWSIZE (LBSIZE + SBSIZE)

struct token {
	uint16_t pos:10;
	uint8_t len:3;
	char next;
} __attribute__((packed));

int main()
{
	struct token t = {0, 0, 0};
	int i = 0, len;
	char *str;
	char *sb;
	int sb_len = 0;
	char *lb, *p;
	struct stat buff;

	int match_len = 0, match_pos = 0;

	(void) fstat(0, &buff);

	str = malloc(buff.st_size);
	fread(str, 1, buff.st_size, stdin);
	len = buff.st_size;

	lb = str;
	i = 0;

	do {
		lb += (match_len + 1);
		i += (match_len + 1);

		t.pos = match_pos; t.len = match_len; t.next = *(lb - 1);
		fwrite(&t, 1, sizeof(t), stdout);		

		if ( i >= SBSIZE) {
			sb += (match_len + 1);
		}

		else
			sb = str;

		match_len = match_pos = 0;

		/* Matching code */
		for(p = lb - 1; p >= sb; p--) {
			int j = 0;
			while (*(p + j) == *(lb + j) && j < LBSIZE - 1)
				j++;

			if (j > match_len) {
				match_len = j;
				match_pos = lb  -p;
			}
		}
	} while ( i < len);
}

And, here’s the uncompressing part. Its smaller and simpler than the compressor..

/* uncompress.c - */
#include <stdio.h>
#include <stdint .h>
#include <string .h>

#define LBSIZE 10
#define SBSIZE 30
#define WINDOWSIZE (LBSIZE + SBSIZE)

struct token {
	uint16_t pos:10;
	uint8_t len:3;
	char next;
} __attribute__((packed));

	char buf[1024 * 1024];

int main()
{
	struct token *t;
	char *sb;
	unsigned int k = 0, len, i = 0, j, k1;
	char str[81];

	while((len = fread(str, sizeof(struct token), 2, stdin))) {

	t = (struct token *)str;

	for (i = 0; i < len ; i ++) {
		if ((t + i)->pos == 0) {
			buf[k++] = (t+i)->next;
		}
		else {
			strncpy(buf + k, buf + k - (t+i)->pos, (t+i)->len);

			k+= (t+i)->len;

			buf[k++] = (t + i)->next;
		}
	}
	}
	puts(buf);
}

Now on to better ones like LZ78, LZSS, LZRW1, gzip and so on. I was really amazed at what compression offers. What do you think ?

Simple Network Management Protocol (SNMP) allows a management application to retrieve statistics and status from the SNMP agent in this device.

ADSL Router - SNMP Agent

Immediately the phrase “allows a management application to retrieve statistics” caught my attention. “How about a nice GNOME panel applet that reports the unaccounted statistics dirrectly from the browser ?” – I thought.. Sounds exciting for sure. I decided to do it.

The first step is to read about SNMP. I began searching for a book, and finally got one linux administration book in my hand which seemed to have a couple of pages long description about SNMP, but I skimmed through them quickly, as I wanted a quick and dirty solution!.

I figured out that I need to install net-smp packages and did a yum install. I god a utility called tkmib which  is GUI for net-snmp.

I figured out from the list below that the interface I am interested in is No.5, as shown from .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifDescr.5 = br0

I now decieded to hack up a shell script that prints the unaccounted usage accounted in the router.

#!/bin/bash
out=`snmpget -v1 -c public  modem interfaces.ifTable.ifEntry.ifOutOctets.5 -Oq | cut -d " " -f 2`
in=`snmpget -v1 -c public  modem interfaces.ifTable.ifEntry.ifInOctets.5 -Oq | cut -d " " -f 2`
tot1=`expr \( $in + $out \)`
out=`snmpget -v1 -c public  modem interfaces.ifTable.ifEntry.ifOutOctets.3 -Oq | cut -d " " -f 2`
in=`snmpget -v1 -c public  modem interfaces.ifTable.ifEntry.ifInOctets.3 -Oq | cut -d " " -f 2`
tot2=`expr \( $in + $out \)`
echo -e "Ashwin : \t" $tot2 "MB"
tot=`expr $tot1 - $tot2`
echo -e "Balaji : \t" $tot "MB"
echo -e "Total : \t" $tot1 "MB"

It worked very well. Now, how about writing a gnome applet to display the utilization in real time ?  Here’s how I did it.

#!/usr/bin/env python

import pygtk
pygtk.require('2.0' )

import gtk
import gnomeapplet
import time
from pysnmp import role, v1, asn1
from fpformat import fix
from string import Template
import gobject

tr = role.manager(('modem', 161))

req = v1.GETREQUEST()
rsp = v1.GETRESPONSE()

msg = Template(' Ashwin = $ash Balaji = $bal' )

class ModemApplet(gnomeapplet.Applet):

lbl = None
def __init__ (self, applet, iid):
self.lbl = gtk.Label("Please Wait.." )
applet.add(self.lbl)
xx = gobject.timeout_add(10000, self.update)
applet.show_all()

def update (self):
req['encoded_oids'] = [ asn1.OBJECTID().encode('1.3.6.1.2.1.2.2.1.10.5' ) ]
(rawrsp, src) = tr.send_and_receive(req.encode())
rsp.decode(rawrsp)
vals = map(lambda x: x[0](), map(asn1.decode, rsp['encoded_vals']))
ino = vals[0]

req['encoded_oids'] = [ asn1.OBJECTID().encode('1.3.6.1.2.1.2.2.1.16.5 ' ) ]
(rawrsp, src) = tr.send_and_receive(req.encode())
rsp.decode(rawrsp)
vals = map(lambda x: x[0](), map(asn1.decode, rsp['encoded_vals ']))
outo = vals[0]
wan_tot = (ino + outo) / (1024.0 * 1024.0)

req['encoded_oids'] = [ asn1.OBJECTID().encode('1.3.6.1.2.1.2.2.1.10.3' ) ]
(rawrsp, src) = tr.send_and_receive(req.encode())
rsp.decode(rawrsp)
vals = map(lambda x: x[0](), map(asn1.decode, rsp['encoded_vals']))
ino = vals[0]
req['encoded_oids'] = [ asn1.OBJECTID().encode('1.3.6.1.2.1.2.2.1.16.3' ) ]
(rawrsp, src) = tr.send_and_receive(req.encode())
rsp.decode(rawrsp)
vals = map(lambda x: x[0](), map(asn1.decode, rsp['encoded_vals' ]))

outo = vals[0]
wlan_tot = (ino + outo) / (1024.0 * 1024.0)
self.lbl.set_label(msg.substitute(ash = str(fix(wlan_tot, 3)), bal = str(fix(wan_tot - wlan_tot, 3))))
return True

def modem_factory(applet, iid):
ModemApplet(applet, iid);
return gtk.TRUE

gobject.type_register(ModemApplet)

if __name__ == '__main__':
gnomeapplet.bonobo_factory("OAFIID:GNOME_ModemSNMP_Factory", ModemApplet.__gtype__, "modem", "1", modem_factory)

Its now a applet at the top. Doesn’t it look cool ?