Archive for the ‘Uncategorized’ Category

Strata 2011 – Big Data

07 Mar

The internet exerts an unprecedented equalizing force in bringing access to information to everyone on the planet. More information is available (and mainly for free) now than ever before, and yet it is becoming clear that access to information is not enough. The infrastructure to store and share data within sectors is a vital part of the ecosystem, and yet it is often treated as an afterthought. We need a radical change in the way we develop infrastructure in the higher education sector, to ensure that services consumed and funded by the public can do their job as efficiently as possible and at the best possible price.

This is key, we’ve found that having data isn’t enough (although its a great start!). Making that data available in some meaningful way is more of a challenge. With MusicNet we’re striving to make readily available Musicology information available in a single place under a single search. We’re leveraging Linked Data technologies to better allow others to integrate their own data with our project outputs.


MusicNet & LinkedBrainz Meetup

25 Oct


Last Friday the MusicNet team headed to QMUL to meet with Kurt Jacobson & Simon Dixon from the LinkedBrainz project. LinkedBrainz is also funded by the JISC Expose (#jiscexpo) programme and is working, in conjunction with MusicBrainz, to produce an official Linked Data mapping for the MusicBrainz database. You can follow their progress on the project blog at

What does a collaboration look like?

As well as learning a bit more about each others projects we were able to look at a few ways in which we might be able to collaborate over the coming months. We also came away with a data export from the most recent MusicBrainz database for all Classical musicians. This will essentially allow us to link our exposed composer URIs directly to the MusicBrainz (or LinkedBrainz) equivalents. This will greatly increase the utility of our URIs, especially as organisations such as the BBC are already using the MusicBrainz IDs.

Adding “same-as” links to LinkedBrainz is only one side of the solution, ideally it would be great if we could convince the MusicBrainz community to provide the reverse linking. This is likely to be a longer term outcome and one we should approach once the sustainability of URIs issue has been resolved (

How will we align our URIs to LinkedBrainz?

We’ll use out custom built Alignment Tool! Over the last few months we’ve spent quite a while engineering the tool and making sure its as re-usable as possible, we plan to add the LinkedBrainz data as though it were just another partner’s catalog. This means that once our Musicology expert has performed the alignment we’ll not only know the overlaps between our partners catalogs but we’ll also know how they map to MusicBrainz and by proxy to Wikipedia & the BBC etc.


Alignment Tool Implementation

19 Oct

In this post we’ll discuss a little about the implementation of the relatively simple server component of the Alignment Tool. You can read more about the tool in previous posts (Beta Release, Assisted Manual Data Alignment), or download the source yourself and have a play.

Server Application Component

Our servers run a typical LAMP (Linux, Apache, MySQL, PHP) stack & although it can also run python, perl & ruby we decided that due to the experience of the project team we would develop the server component in PHP. Usually when we need to write a PHP driven application we would reach for the Kohana Framework.

Kohana is an elegantĀ HMVC PHP5 framework that provides a rich set of components for building web applications

HMVC (or Hierarchical-MVC) is an extension to the more commonly used MVC (Model View Controller). HMVC is essentially useful to help build more modular “widgets” that make up a webpage, we won’t be discussing this as it doesn’t serve our purposes for MusicNet.

In MVC, each object in a system is separated into one of the following groups:

  1. Model: Objects which make up the datastructures used in the system
  2. View: Typically the UI
  3. Controller: Where application specific code is implemented

MVC allows for proper code separation and makes for easier design and maintenance.

Using Kohana, each HTTP request to the server is interpretted as a method call on a constructor object. For example:

This URL equates to calling the public function get_tags() on the controller object Api.

Lightweight PHP Framework

We felt that requiring the Kohana Framework for the server component of the Alignment Tool was a bit heavyweight but still wanted the flexibility of the a lightweight MVC architecture in which to quickly code the AJAX API used by the the Javascript Client. So taking inspiration from Kohana’s URL interpreting we wrote a lightweight framework of our own.

To achieve this we needed 3 distinct parts:

  1. URL Interpreting
  2. Controller Object
  3. Abstract UI Rendering

URL Interpreting

To enable Kohana style requests we first needed to route all URL requests through a single PHP gateway script. As we’re using the LAMP stack this is easily done using ModRewrite. Our .htaccess file in our /ajax folder looks like this:

# Turn on URL rewriting
RewriteEngine On
# Installation directory
RewriteBase /
# Allow any files or directories that exist to be displayed directly
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
# Rewrite all other URLs to index.php/URL
RewriteRule .* /ajax/index.php/$0 [PT,L]

The key part to this script is the last line, here we tell Apache to send all requests that are made below the ajax folder to the index file.

For our purposes we only need a single controller object so our bootstrapping code (index.php) only needs to work out the method/action and the arguments. Our bootstrap script looks like this:

$path = $_SERVER['PATH_INFO'];
$parts = explode('/', trim($path, '/'));
   $method = array_shift($parts);
   $args = $parts;
$ajax = new Ajax();
call_user_func_array(array($ajax, $method), $args);

Controller Object

Now that all the routing is taken care of we just need a simple Controller class with a public function for each method/action in our API:

class Ajax
   private $status = 200;
   private $message = "Success";
   private $data = array();
   // Fetch all Ungrouped items
   public function ungrouped()
   { }
   // Fetch all Grouped items
   public function grouped()
   { }

We also need an output() function as this is what the bootstrap script calls to send output to the client.

public function output()
   if($this->status == 404)
      header('HTTP/1.0 404 Not Found');
   $output = array(
      "status"	=> $this->status,
      "message"	=> $this->message,
      "data"		=> $this->data,
   header("Content-Type: application/json");
   echo json_encode($output);

And its a good idea to implement the __call method incase the client makes an unrecognised request:

public function __call($method, $args)
   $this->status = 404;
   $this->message = "Unknown method: '$method'";

Abstract UI Rendering

The final piece of the system is to enable abstract UI rendering. In one of the calls in the Alignment Tool, the server is required to return HTML rather than JSON. To remove this rendering from the Controller class and to enable 3rd parties (we hope the Alignment Tool will be useful to others too!) to write their own Views for their own data we use PHP’s Output buffering:

$html = ob_get_contents();

By including the file in this way the View script has all the same variable scope as the method in the Controller object. Here’s an extract from our View file to give an idea of how it can be used.

<?php foreach($this->data as $item): ?>
	<div class="item" id="info-<?=$item->id?>">
		<ul class="metadata">
			<?php if(isset($item->metadata->Birth_Date)): ?>
				<li><span class="title">Birth Date</span><?=$item->metadata->Birth_Date?></span></li>
			<?php endif; ?>
			<?php if(isset($item->metadata->Death_Date)): ?>
				<li><span class="title">Death Date</span><?=$item->metadata->Death_Date?></span></li>
			<?php endif; ?>
<?php endforeach; ?>

Minutes of Face-to-Face Project Kick-off Meeting

09 Jul

MusicNet project kick-off meeting held on 24 June 2010 at Southampton.

Completed actions to report:

mc has call with DFF confirming project plan and offer.
Project Blog Established with initial 7 posts by Joe; update posted to DFF.
DFF email confirming request to JISC for Project Offer Letter requested.


Data Preparation

Action: David and Joe to meet on 9th July to synchronise on the data triage task (WP1).
Deliverables of this action will be a document plan of the core data resources to be used within the project, detailed what they offer, and how they will be used.

Workshop Plan

Goal: have a workshop to demonstrate project, including how to use it and what it offers. Identified participants, including stakeholders as listed in bid document. Planned to be hosted in London.

David: Decide on a date for the workshop.

David: Approach IMR about hosting the workshop, negotiate suitable date in May with stakeholders. Due: August.

Update bid partners

David: Write to authors of letters of support to thank and update them. Due: 9 July

Dan: Write to other letters of support thanking them. Due: 9 July


David: To write a short publicity announcement for the music news web page, and to send around to us to check, and we can decide on releasing it through Joyce/ECS also. Due: 16 July

Joe: Set up Blog to Twitter posting. Due: 9 July


Joe: To reconnect Gabriel at OSSWatch and confirm a date. Due: 9 July

Project Management

Joe: Upload Gantt chart in a format we can all look at that shows which tasks are outstanding and next.