Great Endeavours Beginning

PHP RSS Parsers

2008-04-04T11:09:47Z

So, the next thing to build in my site is an RSS parser. I'm uploading video files to a hosting provider, who generates their own unique ID for each video. However, when users browse my site they want to play these videos, which means that I need to know the hosting provider's ID for the video the user wants to watch.

My video host provides an RSS feed of my uploaded content, which includes their IDs. So I need a scheduled task which will poll this feed and parse it so the IDs can be written back to my database.

The first try was using Magpie RSS, which took about 10 minutes to download, install, and get working.

require_once 'rss_fetch.inc';
$url = 'http://myprovider.com/feed.rss';
$rss = fetch_rss($url);

$rss is now a structured array containing all the data from the provider's RSS feed, and can be accessed like any other array:

print "Channel: ".$rss->channel['title'];

The big drawback is that the array only contains the data from the XML elements in the feed - it doesn't include the attributes. My provider's feed conforms to Yahoo's Media RSS specification, which includes a element where all the thumbnail information is held in attributes:

So the next candidate was Simple Pie. This proved to be much more flexible and capable, but a bit more complex to get up and running, but got there in the end:

require_once('simplepie.inc');
$rss = new SimplePie();
$rss->set_feed_url('http://myprovider.com/feed.rss');
$rss->enable_cache(false);
$rss->init();

foreach ($rss->get_items() as $item) {
  $user_data = $item->get_item_tags('http://myprovider.com/WebService/userdata/0.1/', 'id');
  $video_id = $user_data[0]['data'];

  $thumbnail = $item->get_item_tags('http://search.yahoo.com/mrss/', 'thumbnail');
  $thumbnail_url = $thumbnail[0]['attribs']['']['url'];
}

What tokens are available in Drupal?

2008-03-31T08:41:14Z

Drupal has a very powerful module called Tokens, which makes various pieces of data available in other modules. As an example, the Pathauto module will create friendly URLs for any piece of content based on some feature of that content, using token substitution. By default, a new piece of content will be available at the URL http://www.mysite.com/node/5 (assuming you have enabled clean URLs). However, you can create a pathauto rule which creates an alias to make that content available at http://www.mysite.com/reviews/run_fatboy_run - much more SEO-friendly.

This pathauto rule looks like this:
reviews/[title-raw]

So far, so good. However, what happens when you want to use some feature of the content other than the title? What other data is available?

It turns out this is pretty easy, once you know where to look. In the modules/token directory is an include file for each module which exposes tokens. I wanted to use the user name as a token, so checked in token_user.inc, and found

$values['user-raw'] = $account->uid ? $account->name : variable_get('anonymous', 'Anonymous');

so I've now created a pathauto rule on user account page paths of
users/[user-raw]
so every user gets a nice URL to their account page, like http://www.mysite.com/users/bob

Incidentally, the token_user.inc file also includes a bunch of help text in the overridden hook_token_list function, so it looks as though there is support for displaying this information somewhere in the Drupal admin interface. I can't find it anywhere though, and can't find any documentation around it (other than in the API.txt file to say "This function is used to provide help and inline documentation for all of the possible replacement tokens.", but no clue as to where this help and inline documentation may be). All answers on a postcard please...

Importing Data Into Drupal CCK

2008-03-20T14:46:42Z

I've now got a bunch of structured data in a relational database, and need to get it into Drupal. I've created my content types using CCK. Fortunately, there's a great overview here which highlights some of the core Drupal tables I need to populate which aren't immediately apparent.

Everything is pretty straightforward, apart from populating deltas for repeating fields in CCK.

An example will help here.

Suppose I have a content type of person, who can have multiple phone numbers. CCK lets me add a field of type text, which is defined as a repeating field. To the end user, this looks as though you can simply enter as many phone numbers as you like for this person.

In the database, CCK creates a table called content_type_person, which holds simple fields (i.e. fields where there can only be one value - things like first name, surname). It also creates a separate table for repeating fields, so phone numbers will be stored in content_field_contact_telephone. This new table is keyed on vid and nid (i.e. the current version of the node this person belongs to), and delta. Delta is a zero-based autoincrementing field, which increments for each new value for this nid.

Again, an example will help.

I create a new person, which is internally given nid 1 and vid 1, and added to content-type-person. If I add three phone numbers, these are added to content_field_telephone with nid and vid of 1, and deltas of 0, 1 and 2. A second person with nid 2 would get deltas 0, 1 and 2 again:

nid	vid	delta	phone
1	1	0	01234 567890
1	1	1	01234 567891
1	1	2	01234 567892
2	2	0	01234 567893
2	2	1	01234 567894
2	2	2	01234 567895

I've got hundreds of people, and several phone numbers for each. I can create the content node OK and get the right nid and vid, but how do I then insert all these into the database, and get nice zero-based incrementing delta values?

]]> triggers, and this is the key.

Some experimentation and testing led to this trigger on the content_field_contact_telephone table:
delimiter |
create trigger generate_deltas before insert on content_field_contact_telephone
for each row begin
set new.delta = (select max(delta)+1 from content_field_contact_telephone where vid = new.vid);
end;
|

Bingo! Specify your insert values for nid, vid and phone, and you get nice neat auto-incremented delta values for each nid/vid. Just remember to take the trigger off again once you've finished bulk loading - I'm not sure what impact this would have on the main CCK content handling, but I suspect it wouldn't be pleasant! Even better, load this data into a staging table to generate the deltas, check them over, and then transfer them to your main Drupal tables.

Drupal Tutorials

2008-03-19T09:38:01Z

I'm starting to get the hang of Drupal now, largely thanks to Robert Safuto's video tutorials (especially the one about CCK and Views) - nice one Robert!

Getting started with Drupal

2008-03-12T20:20:45Z

A few weeks ago, I decided that the best foundation for the site I'm working on would be Drupal. After some discussions around the scale of the site and the likely complexity of our customisations, this seemed to be the natural choice. One of the key factors in this decision was the perceived strength of Drupal's architecture. As a fairly seasoned developer, I want an architecture which enables me to customise existing functionality and develop new functionality in a consistent and predictable way, with consistent and predictable results.

My last project used OSCommerce as a foundation, and while I've been able to build a good (if fairly simple) e-commerce site with it, the architecture is somewhat inconsistent. Different parts of the system have varying coding styles and variable naming conventions (which isn't surprising, given it's open source roots), meaning that I have to adopt different coding styles and mindsets when debugging or enhancing different parts. Not insurmountable, but not a good foundation for great endeavours.

The other key reasons for choosing Drupal were

product maturity
rich third party module library
very active developer community, including conferences
large-scale implementations (like MTV UK)
open source (i.e. debuggable, extensible, customisable and free!)

So today I've been getting into the nuts and bolts of it. In our sprint planning, we've included some basic user profile functionality, as I thought this would be a good place to introduce myself to the Drupal architecture. Wrong, wrong, and wrong.

One of our requirements is to support multiple user types, each of which will have different attributes. A good analogy (although not an actual example) would be to have

car drivers (who might want to add their car make and model to their profile)
mechanics (who might want to add specialisms such as engine tuning to their profile)

The core Drupal profile module doesn't support this, but a quick run through the modules list reveals Advanced Profile, which does. However

Advanced Profile requires Panels, CCK and Node Profile
Node Profile requires Node Family and Subforms

Although it's all pretty easy to install and configure, I'm now trying to get my head around what each of these modules brings to the party, and I suspect this is making my supposed "gentle introduction" more difficult than it really needs to be. Still, I'm sure it will start to make sense tomorrow.

Connecting C# to MySQL

2008-03-01T20:37:12Z

My previous lives have left me pretty handy with C#. I only really used .net 1.1 for serious heavy lifting, so I'm finding all kinds of nice suprises (like generics - strongly typed ArrayLists at last!) in .net 3.5 now that I've started using Visual Studio 2008 (and finding that the free Express Edition is fine for commercial development came as a rather nice suprise too!).

One of the big down sides with the whole Microsoft ecosystem is the cost of getting off the ground, so as a startup we are going down a pretty standard open source tool route (i.e. LAMP servers). However, we have some fairly sophisticated spidering to be done to get structured data from a fairly large number of web sites of widely varying quality, so in the interests of getting up and running quickly, I've developed the spider in C# (with a fairly massive leg-up from Jeff Heaton - thanks Jeff!). I need to get the spidered data back into MySQL so that our other applications can use it, which brings me to today's problem.

]]> connector, and away you go:

using MySql.Data.MySqlClient;
...
...
MySqlConnection conn = new MySqlConnection(builder.ConnectionString);
conn.Open();
MySqlCommand cmd = new MySqlCommand("select description from products", conn);
MySqlDataReader reader = cmd.ExecuteReader();
while (reader.Read())
{
MessageBox.Show("Product description: " + reader.GetString("description"));
}

reader.Close();
conn.Close();

The only other complication was enabling remote connections to the MySQL server, which (thanks to a little help) turned out to be as simple as changing /etc/mysql/my.cnf from

bind-address = 127.0.0.1
to the actual IP address of the server:
bind-address = 192.168.187.128

And away we go!

Sprint 1

2008-02-29T21:10:19Z

Let the sprints begin!

We've had our first sprint planning session, and it's gone pretty well. Over the past few weeks we have come up with an initial product backlog, and I've estimated all the stories using story points. The only snag is there are over 200 stories! Still, we managed to pick out five to develop and test in the first sprint. We're starting off with two week sprints, and we'll see in the retrospectives how well that works out. I'll be trying various ways of presenting stuff on the project whiteboard, and I will post here how well each one works for us.

Let the endeavours begin...

2008-02-27T11:08:03Z

So, day one dawns. I've left the heady world of corporate software development, and begun the startup adventure!

In this blog I'll be writing about the whole process of getting a new site developed and launched. I'll be looking in some depth at all kinds of different tools to do what we need, and writing a fair bit about managing the software development process too. I'll be using Scrum to keep my business partner's creativity in check, and it will be interesting to see how well that works - my last corporate project was something of a Scrum disaster, but I think it can work much better in the startup environment. Time will tell!