_ _ _ ____ / \ ___ ___(_) __ _ _ __ _ __ ___ ___ _ __ | |_ | ___| / _ \ / __/ __| |/ _` | '_ \| '_ ` _ \ / _ \ '_ \| __| |___ \ / ___ \\__ \__ \ | (_| | | | | | | | | | __/ | | | |_ ___) | /_/ \_\___/___/_|\__, |_| |_|_| |_| |_|\___|_| |_|\__| |____/ |___/ CS 2110 CODING COMPETITION 2009 ENTRY by Mengxiang and Chuck

-====================================================================-

Table of Contents-----------------1.) Philosophy2.) Caching3.) GraphViz4.) Root-Finding Algorithm5.) Multithreading (!)6.) Fibonacci Heap7.) Prim's Algorithm8.) Testing9.) Conclusion and Future Work

-====================================================================- - PHILOSOPHY --====================================================================-

Our philosophy behind this project was to emphasize performance, whilestill guaranteeing accurate results. We took several approaches towardachieving this goal. We owe a few clever tricks to our vast speed-up,which we will discuss thoroughly in this "Read Me" file. Thus, theneed for speed and our own unrelenting competitive spirit were ourmotivations for developing this project.

We hope you enjoy reviewing our entry as much as we did creating it!

-====================================================================- - CACHING --====================================================================-

The first and perhaps most obvious approach we took toward making theprogram run faster was to add caching of the gene and animal distances.When a distance is computed, we first check if had already been computed.If it had been, then we used a hash table to look it up in O(1) time.If it hadn't been already computed, we compute it by hand, and then westore it in the hash table for future re-use. Moreover, we use the geneand animal pairs as indexes to the hash table. Java's built-in hash tablefunctionality sufficed for this task. We realize that this speed comesat the cost of memory, but the performance gains made this trade-off wellworth it. Before caching was added, it took the program about 1.5 hoursto generate 40 graphs. Afterward, it took the program about ten seconds.Reducing the complexity down to O(1) really does pay off.

-====================================================================- - GRAPH VIZ --====================================================================-

We leveraged the Graph Viz software to provide a springboard that wouldhopefully launch us toward success in implementing our root-findingalgorithm. We wrote a GraphViz class that would generate a GraphVizoutput file, similar to the Dendroscope and TreePrinter classes. We thengenerated phylogenomic graphs for all 40 animals as roots, and began toanalyze the characteristics of these graphs in order to find some sortof metric to determine the best root animal. Example JPEG graphs andtheir corresponding Graph Viz source code are provided in this .ZIP file.

-====================================================================- - Root-Finding Algorithm --====================================================================-

We immediately noticed the aesthetics of the Parmesianian graph, whichaccording to the online assignment, was the best root. The tree was muchwider than tall, so initially we figured we could simply use the widthof the tree as a way to determine the root. Unfortunately, several treeshad the same width as the Parmesianian, resulting in ties for the bestroot. Moreover, the algorithm was not deterministic because the tieswere not resolved in any defined manner, so any tree with the same widthcould potentially be resolved as the best root tree.

Our next attempt was to consider the width-to-height ratio of the graphs.The tree with the greatest width-to-height ratio appeared to clearly bethe one with the Parmesianian root. However, this technique suffered thesame fate as the previous one; several ties with the same ratio clashedfor becoming the ideal root animal, and the algorithm was apparently notdeterministic either.

We had one last characteristic to consider, though. The tree was muchmore "balanced" for the Parmesianian than any other animal. We soughtout for a mathematical definition of balance. We needed a definitive wayto quantitatively measure the balance of a given tree. We scoured theInternet relentlessly for a way to measure the balance of a tree to noavail so we began to brainstorm on our own.

The first method up for consideration was to utilize the fact thatbinary trees increase by powers of two for each level. Thus, n-ary treesmust increase by some power of n for each level. The closer the tree isto being balanced, the more evident this relationship holds. The mainproblem of this approach though is, what is n? Should n be the same forall trees? What if one n is better for one graph and another n is betterfor another graph? This method left us with even more unanswered questionsso we looked for an alternative.

Finally, the method we chose to implement was that of a recursive algorithm.We realized that balanced trees have equal amounts of children on each side.In order to determine just how balanced a tree is, we compute a so-called"mirror index" recursively that takes this into consideration. The mirrorindex algorithm traverses each sub-tree of a given node, counts its children,and adds the differences between this sub-tree and the other sub-trees atthat level to the mirror index. Then the algorithm recurses to the next leveland counts the "sub-sub-trees", adding the differences in children to theindex accordingly. The algorithm worked! As you can see below, the mirrorindex is by far the lowest for the Parmesanian animal:

Frilly_Sea_Sprat: 70Asian_Boxing_Lobster: 292Policle: 330Jelly_Belly: 198Ballards_Hooting_Crane: 262Pompous_Snark: 262Fuzzy_Trible: 174Sextopus: 356Gilligans_Squimp: 292Ballards_ProtoDuck: 222Shy_Frecklepuss: 216Bards_Star: 292Larval_TreeNymph: 192Globe_Floater: 216Snuffling_Blat: 152Big-Billed_Peacock: 262

Spotted_Ghila: 216Gray_Floop: 216Leaping_Lizard: 152Sprats_Butterfly: 192Munkles_Mouse: 330Strats_Squirrel: 262Biscuit: 262Nocturnal_Mourningbird: 152Green_Herring: 286Nocturnal_Plexum: 262Green_SnapDragon: 222Common_Mudfly: 262Striped_Salamander: 286Paradise_Rockfish: 216Hairy_Rock_Snot: 58Darwins_Tortle: 292Hallucigenia: 292Parmesanian: 24Swamp_Slime: 140Pink_Ziffer: 286Toothy_Ballonfish: 216Elephant_Snark: 152Translucent_Tridle: 88

-====================================================================- - Multithreading (!) --====================================================================-

We decided to go off on a limb here and do multithreading. After all, now thatMoore's Law is quieting down and we are approaching the physical limits of whatgood-old silicon transistor CPUs can actually do as far as clock speed goes, thechip manufacturers still want to release innovative products so their idea is,"Throw more cores on it!" Unfortunately, computer scientists haven't figured outhow to completely take advantage of having additional cores yet and parallelizationis still an active research topic. PC games such as Crysis and the Source Enginehave only recently added support for multithreading to their 3D engines. Now, GPUsare being utilized computationally for the same purpose: parallelization.

We were admittedly tired of seeing the CPU usage in the Windows task manager onlygoing up to 50% on my dual-core Thinkpad laptop especially back when generatinggraphs took a really long time before optimizations were put in place. We wantedto desperately double the speed of the program by using near 100% CPU usage theentire time, and we were inspired by Professor Birman's lecture on multithreading.

We decided to jump on the bandwagon here and implement the gene algorithms inparallelizable form. In order to do this, we divided our program up into threestages that must run in serial: the distance computations, the animal speciesgraph generation, and the root finding algorithm. We then wrote multithreadedimplementations of these algorithms. Luckily, the computations were extremelywell-suited for parallelization; the algorithms could work on different animalsat the same time since the data is completely independent of itself.

We wrote a ThreadManager class (not sure if this fits some kind of design pattern)that dispatches out worker threads with allocated workloads that work in tandem toaccomplish the three serial tasks in parallel. We ran into the problem of decidingwhen each thread is done. Our solution was to put the main thread to sleep andperiodically "wake up" to poll the other threads to see if they were completed

every 100 ms. We realize that Java has built-in notification/wait functionalityfor threads, but alas we ran out of time now with only three hours to go beforedeadline. We needed a way to make sure that all of the threads were completelyfinished before moving onto the next serial task so we implemented our ownSemaphore class with atomic operations for increasing and decreasing the semaphorecount (P and V).

Multithreading code can be hard to debug. We discovered this ourselves the hardway with this assignment. The programmer's mantra of "code for an hour, debugfor a week" rang quite true for us. We first ran into problems with Java'sbuilt-in HashMap class not being thread-safe. Luckily for us, Java comes equippedwith a ConcurrentHashMap class that alleviates these issues. Switching to theconcurrent step-child of HashMap was not difficult at all. We also ran into dead-locks and even more thread safety issues. HashSet just wasn't cooperating with usand caused the threads to deadlock halfway through. We took advantage of Java's"synchronized" keyword to make the Phylogeny tree generation code run atomicallyin each thread, and this resolved our deadlocking issues. At times, thefrustration became such that we almost gave up on the idea of using threads, butwe finally managed to work out all the bugs and come up with a parallelimplementation of the project.

-====================================================================- - Fibonacci Heap --====================================================================-

The online assignment web page suggested that if we were truly crazy, we coulduse our own Fibonacci Heap implementation to generate the MST tree. Since weare, in fact, self-professedly crazy, we thought, "Sure! Why not?" This task didnot prove to be nearly as easy as we thought it would be. The Wikipedia page articlewas vague in explaining the Fibonacci Tree operations, so we had to sort of reverseengineer the diagrams on there. Moreover, we ran into problems with having markedroot nodes and our alpha version of the implementation frequently violated theheap invariant. JUnit testing came to the rescue here, and we were able towork out all of the bugs and reap the benefits of using a Fibonacci Heap. We wroteour own Priority Queue implementation that took advantage of this Fibonacci Heapclass and used it for our next task: using Prim's algorithm.

-====================================================================- - Prim's Algorithm --====================================================================-

The project web page mentioned using Prim's algorithm instead of Assignment 4'sNaive MST algorithm. We figured, what better way to put our Fibonacci heapimplementation to use?

Prim's algorithm turned out to be a bit more of a challenge than we had thought.We ended up having to rebuild the PriorityQueue after each iteration to reflectthe new distances of the animals that were going to be added. Moreover, we hadto scrap our code three times and rewrite it because things just were notworking properly.

Implementing the lexicographic tie breaker turned out to be the hardest part.Initially, my Fibonacci Priority Queue class was designed to be more like atraditional Priority Queue by using numerical priorities instead of comparators,which we thought of as a clumsy solution. Nevertheless, we had to resort tousing generics and supporting the comparator interface in our code albeit at

the cost of more bloated, more complex code. Once we switched to using thecomparator interface, we merely had to write our own comparison routine tocheck for and break ties.

We thought of two ways of getting around Professor Birman's siblings infinitedistance "hack". The first was to use some sort of look-up table where wecould tell instantly if animals were siblings, and then do some sort of cleverwork-around if they were in the PhylogenyTree Prim's algorithm implementation.The second idea was the one we ended up using. When we build the Phylogenytree, we first ignore all siblings when looking for a closest distance for theminimum spanning tree. If we cannot find a node because they are all siblings,then we check for the first sibling and use that instead for the animal withthe closest distance. The end result is that we no longer need to set thesiblings' distances to the ad hoc infinity value that was needed before, andwe still get exactly the same MST.

The resulting Prim's algorithm implementation seemed to be a lot more stream-lined than we had expected. It was clearly cleaner than the Naive MST-buildingalgorithm and about half the length in code.

-====================================================================- - TESTING --====================================================================-

Our attitude toward testing was "test early and test often". Thus we devisedas many tests as we could to try to break our program. We were successful inmany instances, which helped to improve the stability of our code. Throughoutthe project, we used the Subversion version control system that Chuck hadinstalled on his OpenBSD box at home to help aid in collaboration. Eclipseeven had a plugin that allowed it to use SVN as a development tool. Thisallowed us to simultaneously write tests and run them in hopes of discoveringbugs. We came up with some pretty cool ideas for tests!

Multithreading necessitated a unique kind of testing we called "stresstesting". The idea was throw 20 threads in the ring and have them duke it outand try to deadlock each other or reveal any race conditions. For the latter,we repeated the test for multiple trials and checked to make sure the rootanimal was the same each time. This spot checking turned out to be very usefulfor detecting small variations in the tree. Such variations would manifestthemselves later on in the root finding algorithm, yielding completelydifferent results.

Furthermore, the Fibonacci heap needed thorough testing if we were going toboldly replace Java's venerable PriorityQueue class with our own hack. Ourbest idea was to try constructing multiple random heaps and perform ourown random set of operations on them, checking the heap invariant after everyone. This test proved to be quite effective. Many hidden bugs lurking withinthe heap implementation were swiftly and surely brought to light by thistest. As a result, we gained some confidence that our own heap solutionwas worthy enough to contend with Sun's (wishful thinking! :).

Menxiang wrote many of the rote validity tests in the code. They test themethods for correctness and fault tolerance.

-====================================================================- - CONCLUSION AND FUTURE WORK --====================================================================-

Time is a scarce resource at Cornell. Some of our most ambitious ideas didnot make it into the final product, but that can be said about many projectlife cycles in the real world. We had thought of writing a 3D OpenGL treevisualization tool for the GUI, but we were one day short of actuallyincluding it in our project. JOGL would have facilitated this, along withprior experience with OpenGL in other projects.

Also, we thought Birman's cloud/distributing computing stuff was pretty neatand were wondering if we could somehow dispatch our threads on other machinesusing Java's web services functionality. Unfortunately, multithreading aloneproved to be ambitious enough, and we were not able to implement this, buthey, it's still a pretty cool idea nonetheless to crunch out large DNA datasets "in the cloud" much like how protein folding is being carried outnowadays.

Overall, we thought the project was pretty successful. Our greatest triumphwas hands down the multithreading, but all in all, the rest of the projectwent just as smoothly and we seemed to work quite nicely towardaccomplishing our goals here even if it meant being overly ambitious at times!

("`-''-/").___..--''"`-._ `6_ 6 ) `-. ( ).`-.__.`) (_Y_.)' ._ ) `._ `. ``-..-' _..`--'_..-_/ /--'_.' ,' (il).-'' (li).' ((!.-'