I’ve been meaning to fix the RSS aggregator at http://www.tchpc.tcd.ie/dri-planet for a while now, it’s fixed! Note that it’s just something that aggregates news that I think are useful for work. The aggregated links may or may not be affiliated with my work.
It seems that ZFS On Linux reached a significant milestone, that is the software is stable to use on Linux. This is pretty useful and significant for the preservation and archivists community as it provides a more reliable platform to build on. The LLNL guys must really want to mitigate against silent data failures in their systems (they’re running Lustre on top of ZFS). If ZFS is trustable or not we will know over time.
At work we like to do lots of testing and behaviour driven development since we have a number of stakeholders and institutions all working on the same application. To make sure everyone is getting what they want we’re using cucumber to write our specifications; that is we’re primarily doing outside in developement of our system. As such we like to test things in a near production like environment… Having chosen to use capybara-webkit to test the interface (at a very functional and simplistic level) on our workstations.
I’ve been recently reading up on Amazon Redshift, at first I thought it was a parallel/distributed datastore like HDFS. At a second look in reality its more of a distributed relational database. This in itself is pretty cool for scaling applications with large data sets that happen to need to be in a database; which is quite a few things. From the initial reading of the architecture and docs, it looks like Amazon built a job queuing system around postgres to schedule queries out to its nodes in the cluster.
I was cleaning up some files and I found this time lapse that I did when we were building Kelvin, it’s a few years old now. Even by current standards it’s still pretty respectable. The timelapse We had to unpack and install all the infiniband cards ourselves, cabled, racked, installed, configured and burnt it in for production usage. The cluster has 100 nodes, each node has 2 sockets, each socket has 6 cores and 24gb of ram.
After a few months of budgeting, negotiations and finding venues we’ve at Trinity College Dublin have committed to hosting a HydraCamp for europe! So I’m shamelessly plugging it here. Trinity College Dublin as a part of Digital Repository of Ireland will be organising Hydra Camp in the week of April 8th till 12th, 2013. This will be a week long training course, which will be aimed at developers who are interested in the Hydra framework for developing repositories.
We’ve been ramping up our development work on the project that I have been on in the last month or so. One of the issues that we’ve come across is the not so good XML validation and parsing libraries that are available in the ruby world compared to the java world. So as an exercise I decided to see if I could make our prototype work with jruby with the view of doing a test deployment on tomcat or some other application server.
Having a few days of time off from work I’ve committed to migrating my archlinux based laptops to using btrfs. I’ve to date been just using ext4 and nilfs2 (on an SD card) on my eeepc and plain old ext4 on the bigger laptop. The main motivation was that the two devices were pretty outdated and I felt lucky with doing a major upgrade (replaced sysvinit with systemd as recommended by the archlinux people)
It was clear at this year’s supercomputing conference that there wasn’t as much excitement as previous years. There wasn’t much surprise as nothing too revolutionary and radical was announced. In the past when Bluegene/L and P arrived after the earth simulator there was an arms race to being number 1 in the top500 list. Even things like GPGPU’s aren’t as cool anymore, everyone is selling effectively the same systems when it comes to clusters.
Having spent the best part of my Sunday afternoon playing with ansible just to learn and see what all the fuss is about, I was pleasantly surprised with it. I had installed ansible on my OSX laptop and vagrant-ansible for my vagrant test environment. The plan was to try and re-create my current ruby on rails development and test virtual machine with vagrant. A secondary goal was to get it to work with both Ubuntu Precise (LTS) and Scientific Linux 6.