Learnings

Friday, September 28, 2007

Pictorial introduction to RAID

Nice animations depicting how RAID levels work here.
Need to understand this thing well once and for all...

Struggling with Java...

Trying to understand how threads synchronize and communicate with each other with monitors, guarded blocks and what not. God, this stuff is complex to understand. Unlike many other things with software where understanding 90% of something means I can do that thing perhaps 80% effectively, programming is binary - understand the concept completely and you can write the program; understand "more or less" and you are stuck forever!
Why all this philosophy? Because I tried writing the code for the Producer and Consumer example given in the java tutorial here and it was a struggle to get it working because I didn't understand that the threads have to synchronize on a common object to get their work done. Unlike in the example where the synchronization happens on the Drop object (that is a better way of coding because it allows for multiple producers and consumers), I tried to synchronize Producer and Consumer with each other. At last got it to work at least:-)

All this stuff sounds so very similar to the locking concepts in databases and why not? Both deal with concurrent applications. Hopefully, understanding one will help understanding the other also.

Here is the inefficient code anyway...


// The Drop object contains the packet that must be passed from Producer to Consumer
import java.util.Random;
class Drop {
private boolean empty=true;
private String msg=null;

public synchronized boolean isEmpty() {
return empty;
}

public synchronized void put(String info) throws Exception {
if (!isEmpty()) {
System.out.println("ERROR! TRYING TO OVERWRITE!!!");
throw new Exception("OVERWRITE");
}
msg = info;
empty = false;
}

public synchronized String get() throws Exception {
if (isEmpty()) {
System.out.println("ERROR! EMPTY BOX!!!");
throw new Exception("EMPTY BOX");
}
empty = true;
return msg;
}
}

class Producer extends Thread {
private Drop box;
private int numMsg;
Consumer myConsumer;

Producer(Drop pkg, int n) {
box = pkg;
numMsg = n;
}

public void run() {
try {
sendMessages();
} catch(Exception e) {
System.out.println("ERROR in producer");
e.printStackTrace();
}
System.out.println("Producer exiting");
}

void start(Consumer c) {
myConsumer = c;
start();
}


private void sendMessages() throws Exception {
String msg;
for(int i=0; i<=numMsg; i++) {

// Sleep randomly
try {
Random rand = new Random();
Thread.sleep(rand.nextInt(1000));
}
catch (InterruptedException e) {
// Sleep interrupted, never mind, continue
}
if (i==numMsg) {
msg = new String("DONE");
} else {
msg = new String("Message "+(i+1));
}
writeMessage(msg);
System.out.println("PROD: Wrote:"+msg);
}
}

synchronized void writeMessage(String msg) throws Exception {
if(box.isEmpty()) {
System.out.println("PROD: Box is empty, try putting a message.");
box.put(msg);
System.out.println("PROD: Put message, notifying all.");
synchronized (myConsumer) {
myConsumer.notifyAll();
System.out.println("PROD: Notified consumer.");
}
return;
}
while(!box.isEmpty()) {
try {
System.out.println("PROD: Box is full, going to wait.");
wait();
} catch(InterruptedException e) {
}
if(box.isEmpty()) {
box.put(msg);
synchronized (myConsumer) {
myConsumer.notifyAll();
System.out.println("PROD: Put message, notifying all.");
}
return;
}
}

}
}

class Consumer extends Thread {
private Drop box;
Producer myProducer;

Consumer(Drop pkg) {
box = pkg;
}

public void run() {
try {
getMessages();
} catch(Exception e) {
System.out.println("ERROR in consumer");
e.printStackTrace();
}
System.out.println("Consumer exiting");
}

void start(Producer p) {
myProducer = p;
start();
}

private void getMessages() throws Exception {
String msg="EMPTY";
while(!msg.equals("DONE")) {

// Sleep randomly
try {
Random rand = new Random();
Thread.sleep(rand.nextInt(1000));
}
catch (InterruptedException e) {
// Sleep interrupted, never mind, continue
}
System.out.println("CONS: Trying to read msg...");
msg = new String(readMessage());
System.out.println("CONS: Read:"+msg);
}
}

synchronized String readMessage() throws Exception {
String msg=null;
if(!box.isEmpty()) {
msg = new String(box.get());
synchronized (myProducer) {
myProducer.notifyAll();
System.out.println("CONS: Notified producer.");
}
}
else {
while(box.isEmpty()) {
try {
System.out.println("CONS: Empty box, will wait till woken up...");
wait();
} catch(InterruptedException e) {
}

if(!box.isEmpty()) {
msg = new String(box.get());
synchronized (myProducer) {
myProducer.notifyAll();
System.out.println("CONS: Notified producer.");
}
break;
}
}
}

return msg;
}
}


public class ProducerConsumer {
public static void main(String[] args) {
Drop box = new Drop();
System.out.println("MAIN: Starting a loop of "+args[0]);
Producer p = new Producer(box, Integer.parseInt(args[0]));
Consumer c = new Consumer(box);
c.start(p);
p.start(c);
System.out.println("MAIN: Ended.");
}
}

Tuesday, September 25, 2007

Date formats with DB2

Have started working with DB2 again after a long time with Oracle and am getting stuck with all the basic stuff like setting date formats, command terminators, etc.

For setting date formats, here is a nice link:

Looks like you need to connect as SYSADM though and the behaviour is permanent for your CLP:-(

Multi-threading in java

Have been trying to learn how to write multi-threaded programs in java and got stuck on a nice problem yesterday. The program below tries to start multiple threads and each one is supposed to print its name within brackets. To prevent the output getting garbled, the display method has been declared as synchronized. However, the output is still garbled such as:
{{{{T3T4T2T1}}}} instead of coming out as:
{T1}
{T2}
{T3}
{T4}

Here's the code, guess where the bug is:-)

import java.lang.*;
import java.util.*;

class myThread implements Runnable
{
String tname;
int priority;
Thread t;
myThread(String tn, int p)
{
tname = tn;
priority = p;
t = new Thread(this, tname);
t.setPriority(p);
}

public void run()
{
try {
display();
}
catch(Exception e) {;}
}

public void start()
{
System.out.println("Thread "+tname+" started with priority "+priority);
t.start();
}
synchronized void display() throws Exception
{
System.out.print("{");
Thread.sleep(1000);
System.out.print(tname);
Thread.sleep(1000);
System.out.println("}");
}

}

class TestThread
{
public static void main(String[] args) throws Exception
{
int n= new Integer(args[0]);
int x = n;
myThread[] thr = new myThread[5];

Thread.currentThread().setPriority(10);
while(x >0) {
thr[x] = new myThread("T"+x, 2*(n-x+1));
x--;
}

System.out.println("Main thread start");

for (x=n;x>0;x--) {
thr[x].start();
}

for (x=n;x>0;x--) {
thr[x].t.join();
}

System.out.println("Main thread end");
}
}

Tuesday, September 18, 2007

Types of Memory - SRAM and DRAM

An excellent article on memory hierarchy is here. An excerpt from it about DRAM and SRAM...

RAM comes in two varieties - SRAM (static RAM) which is very expensive and very fast and which is mainly used for the local cache both on and off the CPU chip and DRAM (dynamic RAM) which is cheaper and slower and which is used for the main memory and graphics frame buffer (a frame buffer is just a dedicated memory chip which holds the colour information for each pixel to be displayed on the screen).

SRAM or static RAM is physically built with up to 6 transistors to store each bit or voltage signal and is a stable memory module whole signal is retained as long as it is supplied with power. SRAM is not affected by voltage fluctuations or noise. However, it is also more expensive and consumes more power than DRAM and is only used for the caches (both on and off the CPU chip).

DRAM or dynamic RAM is a simpler, slower and less stable memory module. Each cell in DRAM is stored as a charge in a small capacitor coupled with a single transistor. The charge in the capacitor decays rapidly within 10 to 100 ms and hence every bit of DRAM has to be refreshed (read and written back) within 10 ms! However, as the CPU operates in nanoseconds, this is not as bad as it sounds. Sometimes, DRAM has additional bits to encode error checking bits and error-checking circuitry (does this free the CPU of the responsibility of refreshing the bits then?). DRAM is also very sensitive to electrical noise, voltage fluctuations and even light. In fact, this last property is used to make digital cameras where the film is nothing but a bunch of super-sensitive DRAM cells.

Thursday, September 13, 2007

Data Mining in the Telecom industry

A concise paper on how data mining is used in the telecom industry to detect fraud, customer churn, marketing opportunities and also predict network failures. Telcos gather enormous volumes of data in the form of Call Detail Records (CDR's), customer information and also network messages (from all the switches and other equipment). The CDR and network status data are too detailed to be of direct use in data mining and hence the first step is to summarize this data along some useful features and to then feed this to a data mining algorithm or tool like SAS. The challenge here is that the events that one is trying to predict are all very rare events (with less than 0.5% probability) and this has to be extracted from billions of rows of data - truly finding a needle in a haystack! Such stuff is just not possible to do manually or programmatically or by specifying some analytical rules - the data has to be mined to generate both the rules and their probability (or confidence). Interesting area...

Labels:

Saturday, September 08, 2007

Computer Evolution

Summary of Chapter 2 from William Stalling's Computer Arch book:
The first generation of computers used vaccuum tubes, the second generation transistors and the third generation, integrated circuits.

Two architecture styles evolved, the one followed in IBM 7094 and mainframes and the second which was used in PDP-8 and subsequent minicompters and PC's.

The IBM machines had Data Channels (a precursor to modern direct memory access) which have a dedicated processor for handling I/O requests and which manage data transfers to one or more peripheral devices. The CPU sent a control signal to the Data Channel along with the memory location which contained the I/O request instructions and the Data Channel then executed the requests (meaning transfer of data into and out of main memory) and sent a signal back to the CPU. This greatly reduced the burden on the main CPU. As there are now multiple processors contending for main memory, a Multiplexor is required to schedule access to memory.

The PDP-8 from DEC followed an omnibus architecture where there is a common omnibus to which the CPU, the main memory, console controller and I/O modules were connected. The CPU had to manage all the co-ordination and use of the shared bus (does this make the CPU slower?). The omnibus had 96 separate signal paths to carry data, control and address signals. This architecture is highly flexible, allowing modules to be plugged into the bus to create various configurations.

Semiconductor Memory
Initially, computer memory was constructed from tiny magnetic rings called cores strung up on grids of fine wires suspended from a screen. The act of reading from memory was destructive though (the data would be lost when read and had to be re-written!). This changed in the 70's when semiconductor memory was introduced.

Microprocessors
Intel achieved a major breakthrough when it developed its 4004 which was the first chip to contain all the components of a CPU - the microprocessor was born. This chip could read or write 4 bits at a time from the CPU. Also, the registers were 4-bits meaning that two 4-bit numbers could be added at a time. This evolved into 8008, 8080, 286, 386, 486, Pentium and Itanium processors which went up from 8-bit to 16-bit, 32-bit and then 64-bit.

Microprocessor Speed
Moore predicted that the number of transistors that could be put on a chip would double every 18 months and this has held true. Chip designers are able to pack more and more into a chip but processor designers are having to innovate to make use of all that awesome power. Some techniques used are:
  1. Branch prediction - Processor looks ahead in instruction code and predicts which branches are likely to be executed next and pre-fetches them. If the guess is right mostly, the processor is kept busy.
  2. Data flow analysis - The instructions are analyzed for dependencies and an optimized schedule created (which need not be the same as the original program order) for execution.
  3. Speculative execution - Using branch prediction and data flow analysis, some processors speculatively execute instructions ahead of their actual appearance in program execution and hold the results in temporary locations.
Performance Balance
While processor speed is racing ahead, other critical components of the computer have not kept up. System designers hence have to adjust the architecture and organization to balance the mismatch among different components.
The biggest mismatch is in the interface between CPU and main memory. Dynamic RAM density has gone up a 1000 times in 15 years, the CPU speed has gone up 200 times but the dynamic RAM speed has only gone up 2 times! The interface between main memory and CPU is the most crucial pathway in the entire computer because it is this which carries instructions and data for processing. If the memory or the inteface is slow, the processor will stall in a wait state.
Also, the speed at which dynamic RAM density is increasing is greater than the rate at which main memory requirements are going up. As a result, systems typically need fewer memory chips over time. This has a negative impact as it reduces the chances for parallelism.
The CPU-memory interface problem is solved by using one or more of these techniques:
  • Make the DRAM's wider rather than deeper and use a wider data bus
  • Increase the interconnect bandwidth by using higher-speed buses and by using a hierarchy of buses to buffer and structure data flow.
  • Change the DRAM interface to include some sort of cache on the DRAM chip
  • Introduce fast and efficient cache structures on the CPU chip and in between CPU and memory.
Another problem area is the handling of I/O devices. As computers become faster, the demand for I/O also goes up rapidly. E.g., graphics application needs 30MB/s, video about 70MB/s, disk controller about 10MB/s, network about 30 MB/s and so on. While the processors are capable of handling these data volumes, the problem is with moving the data from the processor and peripheral. Again, the strategies used are buffering, dedicated processors for I/O and higher-speed interconnection buses.

Designers have to build systems which adapt to the different rates at which system components are evolving and also to the different application demands (is this why specialised systems are built, such as Pentium 3 for 3d graphics, Pentium 2 for multi-media, etc?).

IBM along with Motorola, developed the PowerPC chips used in Apple Macintoshes, RS/6000's and numerous embedded chip applications (like network management chips). The PowerPC chips had both an L1 and an L2 cache on chip and this seems to have evolved much, much more now.


Labels: