I recently picked up a cheap Acer Chromebook 11inch - CB3-131. It runs Chome OS. While it’s nice as a machine that just works for small things and travel. I found the environment to be a bit restrictive (less fun) to play with. I did try crouton, but it wasn’t what I had hoped for. I proceeded to investigate installing a full linux distro onto my Chromebook and discovered that on some (the one that I got) has a physical write protect screw on the mainboard.
I’ve recently went and updated a bunch of things that I’m working with, and one of the things I decided to refresh was an automated installer. In the past I’ve relied on either hand rolled configurations or more recently Ubuntu MAAS. This time I decided to give Cobbler a try on my Ubuntu Precise installations. Given Cobbler is pretty (or was) RHEL specific, I was not too disappointed with the Debian packaging.
Having recently changed jobs (after a very long stint at TCD - just short of 10yrs!) I’ve moved to a small startup. I’ve been working on a few small bits and pieces of code, infrastructure, etc… I had to make some stuff work and make it work well. So I decided to use autotools to configure and build the application that I’m working on. As great as it was on my nice fast intel based machine, it was dog slow on my target platform, especially when I had to regenerate the autoconf scripts.
I’ve been happily hacking at some packages for hashdist, it’s pretty nice, there other build systems out there for dealing with building applications and libraries with different combinations of compilers and numerical libraries. Out of the lot I think hashdist has been the most satisfying to use so far. It’s still missing some bits and pieces to allow users to use different compilers for key components (or everything). Without explaining too much, it’s basically taking inputs which define a package and then generating an output hash to store the output of the build.
After going to SC13 and being at a few BoF’s and hearing some people talk about their operations and potentially using Salt to replace the likes of puppet and chef, I decided to learn a little more about Salt. In particular since I have an old laptop lying around at home, I decided to setup a little private cloud. I followed this blog post. It mostly worked apart from some buggy behaviour in the seeding process.
It’s probably no secret that we use Ansible at our work place for the project that I am working on. So far we’ve used it to deploy and maintain state for our Open Nebula deployment, Jenkins CI system, Ceph’s radosgw and our digital repository. In fact I currently have a Jenkins job which deploys our software stack using Ansible to a test system in our Open Nebula cluster. This has been hugely beneficial to myself so far to be able to teardown and bring up systems quickly to make sure our application is well tested and debugged.
It seems that ZFS On Linux reached a significant milestone, that is the software is stable to use on Linux. This is pretty useful and significant for the preservation and archivists community as it provides a more reliable platform to build on. The LLNL guys must really want to mitigate against silent data failures in their systems (they’re running Lustre on top of ZFS). If ZFS is trustable or not we will know over time.
I was cleaning up some files and I found this time lapse that I did when we were building Kelvin, it’s a few years old now. Even by current standards it’s still pretty respectable. The timelapse We had to unpack and install all the infiniband cards ourselves, cabled, racked, installed, configured and burnt it in for production usage. The cluster has 100 nodes, each node has 2 sockets, each socket has 6 cores and 24gb of ram.
It was clear at this year’s supercomputing conference that there wasn’t as much excitement as previous years. There wasn’t much surprise as nothing too revolutionary and radical was announced. In the past when Bluegene/L and P arrived after the earth simulator there was an arms race to being number 1 in the top500 list. Even things like GPGPU’s aren’t as cool anymore, everyone is selling effectively the same systems when it comes to clusters.
I’ve been eyeing crowbar recently, it looks pretty useful and interesting for deploying servers and applications. I haven’t seen much if at all any documentation out there which suggests that people in the digital preservation and archiving fields are implementing systems at scale, I’m under the impression that most systems/sites are building systems up one piece at a time without much automation. It seems to use chef in the backend for all the automation.
I shall be going to SC2012 next month, I plan on hitting a few of the storage vendors for possible collaborations and flagging to them that we’re on the look out for storage systems. One of the first observation that the reader will note is “where is that link between HPC and Digital Preservation and Archiving”. It’s probably not obvious to most people, one of the big problems in the area of preservation and archiving is the the amount of data involved and the varied types of data.
A co-worker of mine (Paddy Doyle) had originally hacked at a perl script for reporting balances from SLURM’s accounting system a year or two ago and he had figured out that it might be possible to do some minimalistic ‘configuration’ and scripting to get a system that’s very basic but functional. It was just one of those things that funding agencies wanted to justify how the system was being used, GOLD was clunky and obtrusive and complicated for what we wanted.
Given that I have a number of old 64bit capable desktop machines and a collection of hard drives at home, I could have run Tahoe-LAFS like I do in work for backup purposes. In fact Tahoe works quite well for the technically capable user. Recently I’ve decided that I need a more central location at home to store my photo collection (I love to take photos with my Canon DSLR and Panasonic LX5).
After my last failed attempt at Installing Ceph on SL6 or rather my attempt at configuring Ceph for a test failed miserably. It hasn’t deterred me to test more. As a result I setup a number of Vagrant Virtual machines and got together a few puppet scripts to provision machines. Here’s a sample manifest for puppet to automate the deployment of a machine to build Ceph. It requires that you SL6 environment to have at least the epel repository enabled.
As an exercise of a Friday afternoon of not starting something big before heading off to a conference. I’ve decided to spend an hour or two at seeing how ceph is installed and configured on an SL6 based machine (RHEL6 clone). The install target is just an old desktop running a 64bit install of SL6x, so it’s nothing too fancy. Following the instructions at http://ceph.com/wiki/Installing_on_RedHat_or_CentOS, I ended up doing this
I’ve been happily using tahoe-lafs for my backup needs for a while now in work. It’s not a huge cluster, but just a small hive cache from workstations dotted around the lab. The project recently did a release which fixes a number of bugs and issues which has made this software much more pleasant to use. I’ve also been happily using git-annex to manage my files which reside in tahoe-lafs.
I’ve been following the development of the new git-annex assistant work that Joey Hess has been working on over the past few weeks. Even in this early state of development, it is slowly becoming more usable and accessible for less technical users. As soon as as the issues with the limitations of kqueue and OSX’s silly limits are resolved (without the need for a user to do a sysctl -w WHATEVER) it will be pretty cool.
I was waiting for my backups to be done hence this post, as I was using git-annex to manage my files and I decided I needed to have git-annex on a SL5 based machine. SL5 is just an opensource clone/recompile of RHEL5. I haven’t tried to install the newer versions of Haskell Platform and GHC in a while on SL5 to instal git-annex. But the last time I checked when GHC7 was out, it was a pain to install GHC on SL5.
It seems Joey Hess is in the process of getting git-annex assistant crowd funded. I had not been keeping up to date too much in the last year with git-annex partly because it met my needs and I never needed to upgrade to the latest tip. There are a number of really cool features now which is making git-annex more accessible. I particularly like how the merge and sync workflows have now been pretty much scripted so there is less to go wrong!
Should you dogfood your own project that you are developing? The answer is probably yes, especially if you have no clear cut requirements from the stakeholder in a project with a greenfield for development. There is a lot to be said about having a working implementation that can be presented and refined. Sometimes the project that you are working on won’t have clear requirements for implementation, so you should probably take basic assumed cases and run with it.
For those that care about having related posts on their Octopress blog. It’s actually quite easy to turn it on, it’s nice to have and useful. But it’s not enabled by default in Octopress. This feature already exists in jekyll, enabling this feature in Octopress is a trival task. Firstly add this to your _config.yml file lsi: true Then create a file such as source/_includes/custom/asides/related.html with the following content, add it to one of your asides in _config.
Why would one want to target other platforms when building applications on the server side? This came out of a conversation with the ex-CTO of Creme software (he is also a friend of mine), the conversation started out with why I like to use Macs and OSX as my laptop or workstation. I’ve been a long time Linux user of pretty much most of the major distributions ranging from RHEL, Debian/Ubuntu, Gentoo, ArchLinux as well as a number of other derivatives, not to mention other systems like the BSD’s which I have a soft spot for.
I’ve talked about cports in the past, it’s basically a collection of makefiles which mostly automates the process of downloading, configuring, building and installing applications and libraries for High Performance Computing systems that use environment-modules. One of the key-features that cports offers is the automated modulefile generation, and the fact that the makefiles acts as documentation to how software is configured, built and installed. It’s currently being used on the clusters at my work place, it has been a boost to the productivity of the systems admin team.
It’s all over the internet that RHEL6 and newer releases will have a life cycle of 10yrs. This is pretty good news for projects that are deploying systems that have a lifetime of 5yrs or more. Namely Digital Preservation projects, not having to worry about migration between point releases of an operating system platform reduces time and costs. I’m now more likely to target the systems we develop in work towards a RHEL system (or a RHEL like clone).