I have wanted to write a blog post about how we work on our client projects for a while now. I’m proud of our team and how we manage our work but I don’t want to stop there. Writing out our steps might help me to see any areas that could be improved. I’d also love to learn how other similar businesses manage their work too, so if you have any comments or suggestions, please get in touch. This blog post could also be a useful introduction to how we work to potential future clients.

Our team

We are currently a team of 3 talented software developers which include me (Gordon), Alex and Paco. Each of us works from home in different countries, luckily with very little time difference to contend with.

We build web applications using PHP and MySQL, often with a mix of other languages such Javascript and Python. Codeigniter is our PHP framework of choice.

The applications we create are often built from scratch for other businesses to resell to their clients. We’re very lucky to have happy customers in Ireland, England and the US. In the near future we’re going to add some blog posts highlighting some of the work we have done for customers.

The tools we use, Basecamp, Springloops and Dediserve.

http://basecamp.com,  http://springloops.com and http://dediserve.com/

Basecamp is a project management tool and Springloops is an SVN code repository. I prefer Springloops version 1, not the newer version 2, there are too many things in v2 that I don’t need.

Basecamp and Springloops work very well together. Springloops can connect to the Basecamp API allowing to-do items assigned to you to show up in a ‘To Do’ tab in Springloops where client code is stored, specific to each user.

In Springloops each item is given a number and when some code is committed, one or more item numbers can be added to the commit  comments which will cross off the appropriate items in Basecamp. (It would be cool if the commit comments would add comments to Basecamp but it doesn’t unfortunately.)

On top of this, we use the Basecamp comments and we email each other a heck of a lot. I try to keep the emails to during work hours so that we’re not all working all hours of the day but I don’t always succeed.

We are using Dediserve more and more over the past 6 months to host sites.  Its a great service allowing us to create or modify the servers we need to suit the project.

All projects we develop have a Test site and a Live site. All code commits are automatically deployed to the Test site for testing online. Our clients can use the Test site any time they like, knowing its a sandbox and could be broken at any time or showing debug information.

We don’t phone, skype or IM each other much but those options are there if needed. Writing items down in Basecamp or emails allows us to keep track of everything and refer back to them.

I’d highly recommend the combination of Basecamp, Springloops and Dediserve to any developers creating software for their clients.

Customer communication

When meeting a customer face to face or else over the phone, I take as many notes as I can in my notebook. Right after the meeting I write them all in to Basecamp in a new list usually with a title along the lines of ‘meeting notes dd/mm/yy’. I type them up as quickly as I can after the meeting so that I hopefully don’t forget anything between my notes and what I remember myself.  Sometimes I have meetings back to back which can be a bummer for remembering everything.

Once everything is added to Basecamp I use the ‘Print’ option in Basecamp to give a clean list of notes and I email this to the client and ask them if I have noted everything. I usually get it all right (or else it’s not read at all! )

Clients can access Basecamp too and add/edit to-do lists, most prefer phone or face to face meetings though.

From here we work away on the to-do items added to the list. Usually we would get in contact with the client when the items are all finished and we have deployed the changes Live.

Sometimes the list can be quite long and we should contact the owner more often to let them know how we’re getting on. There is room for improvement here on my part; I sometimes don’t get in touch with a client as often as I should.

In my mind I’d prefer to get in touch when I have everything completed and deployed or ready to deploy. This could leave the client wondering where we are though if it has been a while since last contact. The clients can always get in touch by email or phone but it can put things on a bad footing.

I should be contacting them first with a progress update. This happened at most when it was just me. With 3 of us on the team now there is room for me to communicate more, daily if possible but that’s not always practical, especially across several client projects.  I’d spend all my day talking if I contacted people daily and I wouldn’t get any programming done. We’re hired to programme, first and foremost.

Deploying completed work

One process we have that I’m very happy with is what I call a ‘deploy list’. I’m not sure when this started but when a list of to-do items are coming to an end we create a deploy list for a certain date. In this list there are usually the same steps listed below. The intention is to put all recent changes Live to a client’s application in a responsible way. A handy feature of Basecamp is that it allows to you create a To Do list template, rather than typing out the same list every time.

A typical deployment list:

- Back up Live database (When this step is completed, a comment is added showing the backup file name or database location if backed up to another database. We have automatic backups too but this is a very up to date snapshot.

- Any specific SQL queries. (This could hold specific sql queries to change data in some tables, such as deleting a bunch or records)

- Sync Test database structure to Live database. (This allows me to synchronise items such as new tables, new fields and new field types from the test database to the live site. The data in the database isn’t copied.)

-   Note current Live revision (This means to take note of the current revision in which is deployed to the Live site at the moment. If things go bad, I can revert to this revision very quickly thanks to Springloops.

- Deploy the Latest revision (This means to use Springloops to deploy any recent work to the live site. The new current revision number is noted in a comment)

- Any file or folder permissions (This usually means setting Write permissions to new images folder)

- Test live site (This would be a test of any of the items that have been added or changed. It usually goes hand in hand with the next item, I write the email to the owner as I test each item)

- Update the owner. (I usually email the owner with the subject ‘Application updated’ showing a list of all items recently fixed or added)

If after testing there are any bugs found, these are given a list of their own which needs to be worked on immediately because these bugs could be visible to the client or their customers. It’s possible at this stage to revert back to an earlier release and get back to the drawing board.

Thankfully any few bugs that do pop up are usually small and are related to Live customer data so reverting back wouldn’t allow us to see them easily in the Test environment again. If reverting back, we’d might copy all data from the Live site to the Test site to try to simulate the problem in the Text environment.

Once it’s all deployed and tested, I usually talk to the client about what they need next if needed.

Overall, I’m very happy with our current team. Our 3^rd member is new to the team and we’re all working together very well so far.

I don’t know if these steps are comparable to Scrum or other work methods. Hopefully as I read more about them I might learn some new methods to add to our process. Right now though, I’m very happy with our team and our process. From some recent conversations, our clients are too!

Yahoo has a new (to me) Content Analysis API which can perform text analysis on some text or a URL. I read about it this evening on ProgrammableWeb and had to try it out.

Its limited to 5,000 API calls per 24 hour period per IP Address, thats about enough leg room to try it out with some PHP code.

Below is some PHP code if anyone would like to use it to get started. Its very basic but helps show what the API can do.

Given a string of text, it can pick out words or statements (such as ‘Computer programming’ in the example) and provide a category for the content it understands with a score, with links to Wikipedia.

It shows the words it has found and the start and end character numbers. It even provides related links for that terms. It can spot peoples names too and link to Wiki articles about that person.

I’m only scratching the surface but I can think of some cool usages for this in other applications. Sign in on the http://developer.yahoo.com/contentanalysis/ page with your Yahoo ID and definitely try out the Console for testing Queries.

/**
* Function to use Yahoo to analyse some simple text
* @param String $text
* @param String $format
* @return String $content
*/
function yahoo_content_analysis($text, $format=’json’)
{
$url = “http://query.yahooapis.com/v1/public/yql”;

$query = ‘SELECT * FROM contentanalysis.analyze WHERE text = “‘ . $text . ‘”‘;

$characters = array(’ ‘, ‘=’, ‘”‘);
$replacements = array(’%20′, ‘%3D’, ‘%22′);

$query = str_replace($characters, $replacements, $query);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, “q=$query&format=$format”);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
$response = curl_exec($ch);
$headers = curl_getinfo($ch);
curl_close($ch);

return $response;
}

// Text taken from wikipedia
$text = ‘Computer programming (often shortened to programming or coding) is the process of designing, writing, testing, debugging, and maintaining the source code of computer programs.’;

$response = yahoo_content_analysis($text);

echo $response; // json

Codeigniter is an excellent lightweight PHP framework with great documentation. This post assumes you already know a little about Codeigniter. To learn more about it, visit codeigniter.com.

This tutorial will walk you through building your own URL shortening service using Codeigniter. The end result will be a basic application allowing you to enter a long URL in to a form and produce a short URL such at http://domain.com/abcd. It uses jQuery to place the shortened URL in a DIV under the form rather than reloading the page.

The source code for this application can be found on Github here : https://github.com/murrion/Codeigniter-URL-Shortener

I started by downloading a fresh copy of the current version of CI, version 2.0.2.

Update your database.php in your config folder to connect to your own database.

We’ll need a table to store the URLs. Create a table called ‘links’ in your database with the following structure:

CREATE TABLE `links` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`alias` varchar(6) CHARACTER SET utf8 DEFAULT NULL,
`url` text CHARACTER SET utf8,
`created` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `alias` (`alias`)
) ENGINE=InnoDB  DEFAULT CHARSET=latin1 AUTO_INCREMENT=1;

We’ll need 2 controllers, shorten.php and redirect.php, place these files in the application/controllers folder.

We’ll need 3 views: header.php, footer.php and form.php, place these in the application/views folder.

We’ll need 2 JavaScript files too, one is jQuery itself and the other is javascript.js which is used to contain the jquery event handler.

The Shorten.php controller does the hard work; it creates the short URL if none already exists for the URL. Redirect.php redirects a short URL to the original full length URL.

The header.php view file contains basic stuff, just a link to the jquery file located in the javascript folder and the javascript.js file containing .click event which is called when the ‘shorten’ button is pressed in the form.

The form.php file contains the form which a user enters their longer URL.

The footer.php file just contains the closing body and html tags.

Redirect.php

<?php

if (!defined(’BASEPATH’)) exit(’No direct script access allowed’);

class Redirect extends CI_Controller

{

/**

* Method to redirect from an alias to a full URL

*/

public function index()

{

$alias = $this->uri->segment(1);

$this->db->select(’url’);

$query = $this->db->get_where(’links’, array(’alias’ => $alias), 1, 0);

if ($query->num_rows() > 0)

{

foreach ($query->result() as $row)

{

$this->load->helper(’url’);

redirect($row->url, ‘refresh’);

}

}

else

{

echo “Alias ‘$alias’not found”;

}

}

}

/* End of file redirect.php */

/* Location: ./application/controllers/redirect.php */

Shorten.php controller

<?php

if (!defined(’BASEPATH’))

exit(’No direct script access allowed’);

class Shorten extends CI_Controller

{

/**

* Show a form to shorten a URL

*/

public function index()

{

$this->load->helper(’form’);

$this->load->view(’form’);

}

/**

* Take in a URL from the form and shorten it

*/

public function create()

{

$short_url = “”;

$url = prep_url($this->input->post(’url’));

$link_length = $this->config->item(’link_length’);

// Check to see if this URL has an Alias

$existing_alias = $this->alias_from_url($url);

// Generate a new alias if needed

if ($existing_alias == “”)

{

$this->load->helper(’string’);

$alias = random_string(’alnum’, $link_length);

while ($this->does_alias_exist($alias))

{

$alias = random_string(’alnum’, $link_length);

}

$this->save_new_alias($url, $alias);

$short_url = $alias;

}

else

{

$short_url = $existing_alias;

}

// display the short url

echo base_url() . $short_url;

}

/**

* Method to see if a generated Alias already exists in the table

* @param type $alias String to check to see if it exists

* @return Bool True or False

*/

function does_alias_exist($alias)

{

$this->db->select(’id’);

$query = $this->db->get_where(’links’, array(’alias’ => $alias), 1, 0);

if ($query->num_rows() > 0)

{

return true;

}

else

{

return false;

}

}

/**

* Save a new Alias to the table

* @param type $url URL to shorten

* @param type $alias  The new Alias for this URL

*/

function save_new_alias($url, $alias)

{

$data = array(

‘alias’ => $alias,

‘url’ => $url,

‘created’ => date(”Y-m-d H:i:s”)

);

$this->db->insert(’links’, $data);

}

/**

* Return an existing Alias, if any

* @param type $url String, the URL to check

* @return type $lias String, the alias, if any

*/

function alias_from_url($url)

{

$alias = “”;

$this->db->select(’alias’);

$query = $this->db->get_where(’links’, array(’url’ => $url), 1, 0);

if ($query->num_rows() > 0)

{

foreach ($query->result() as $row)

{

$alias = $row->alias;

}

}

return $alias;

}

}

/* End of file Shorten.php */

/* Location: ./application/controllers/Shorten.php */

Header.php view file

<html>

<head>

<title>Codeigniter URL Shortener</title>

<script>var base_url = ‘<?php echo base_url(); ?>’; </script>

<script language=”javascript” src=”<?php echo base_url(); ?>javascript/jquery-1.6.2.min.js”></script>

<script language=”javascript” src=”<?php echo base_url(); ?>javascript/javascript.js”></script>

</head>

<body>

Form.php view file

<?php $this->load->view(’header’); ?>

<form name=”ajax_form” id =”ajax_form” method=”post”>

<?php

$data = array(

‘name’ => ‘url’,

‘id’ => ‘url’,

‘value’ => ”,

’size’ => ‘50′,

’style’ => ”,

);

echo form_input($data);

$data = array(

‘name’ => ’shorten_url’,

‘id’ => ’shorten_url’,

‘value’ => ‘Shorten’

);

echo form_submit($data);

echo form_close();

?>

<div id=”alias”></div>

<?php $this->load->view(’footer’); ?>

Footer.php view

</body>

</html>

javascript.js

$(document).ready(function(){

$(”#shorten_url”).click(

function(){

var url=$(”#url”).val();

$.ajax({

type: “POST”,

url: base_url + “index.php/shorten/create”,

data: “url=”+url,

cache:false,

success:

function(data){

$(”#alias”).html(data);

}

});

return false;

});

});

You probably won’t want the index.php file to appear in your shortened URLs. To remove it, update your config.php file and set the index_page item to contain no value, eg : $config['index_page'] = ”;

Also, you’ll need to add a .htaccess file to the root of your application, I use the one from the Codeigniter wiki. You can copy and paste it from there if you like rather than re-writing it here.

Lastly, but most importantly, we’ll need to update the routes.php file.  If we don’t update this file, we’ll have a short link like this, which isn’t very short : http://domain.com/ shorten/index/abcd. We’ll want to remove the ‘shorten/index’ bit.

Update the routes.php file within the config folder, add the following  lines:

$route['(:any)'] = “redirect/index/$1″;

This rule will forward any item to the redirect.php controller and index method, which performs the forward.  For example : domain.com/abcd will be routed to domain.com/redirect/index/abcd where it will find the full URL and perform the redirect.

$route['shorten/create'] = “shorten/create”; // overwrite the previous route

This second route is important. It is there to overwrite the first rule to that the creating of a short URL by the shorten.php controller will work. If we left it out the previous route would think that ‘shorten’ is the short text it must look up to see if there is a full URL to go with it.

$route['default_controller'] = “shorten”;

This route will point the user to the main form to create a new short URL if they simply go to the main domain such as domain.com.

Thats it. Deploy the app to your own domain name and its now a URL shortening service.

It doesn’t record any usage logs, keep an eye on the github repository for that.

In a recent blog post I wrote briefly about some ideas relating to using popular APIs to help identify and prevent spam that can happen with forms on a website.

I wanted to follow up on this with some further thoughts and some code examples for anyone that might like to use them.

Here is an example of using the Rapleaf utility API with PHP to determine the gender of a name.  This could be handy if you are unsure of whether to write ‘Dear Sir’ or ‘Madam’ in a reply if you are inclined to reply so formally!

The following example sends a name to the name_to_gender service and receives the JSON response in to a PHP array called $response_array.

$name = urlencode($name);
$response = file_get_contents(”http://api.rapleaf.com/v4/util/name_to_gender/$name”);
$response_array = json_decode($response, TRUE);

Output:

Array ( [status] => OK [answer] => Array ( [input] => Bill [gender] => Male [likelihood] => 0.992195 ) )

Here is an example of using the Rapleaf API to gather more information about an email address:

$response = file_get_contents(”https://personalize.rlcdn.com/v4/dr?email=XXXXX&api_key=YYYYY&show_available”);
$response_array = json_decode($response, TRUE);
return $response_array;

In this sample, you will need to replace XXXXX with your email address and YYYYY with your own API Key which you can sign up to for free.

Output:

Array ( [location] => Cork, County Cork, Ireland [gender] => Male [influencer_score] => 71-80 )

Here is a PHP example of connecting to the Lymbix toneAPI using CURL to determine the sentiment of a piece of content:

$url = “http://gyrus.lymbix.com/tonalize”;
$header_information = array(’AUTHENTICATION: XXXXXXXXXXX’,
‘ACCEPT: application/json’,
‘VERSION: 2.1′);

$data_information = array(
‘article’ => $message,
‘return_fields’ => ‘[]‘);

$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADER, $header_information);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_information);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
$result = json_decode(curl_exec($ch), TRUE);
curl_close($ch);
return $result;

In this example, I am sending the content $message to the Lymbix API located at http://gyrus.lymbix.com/tonalize. XXXXXXX would need to be replaced with your own API Key.

A sample output from the ToneAPI for the quote:

“Determine never to be idle. No person will have occasion to complain of the want of time who never loses any. It is wonderful how much may be done if we are always doing.” - Thomas Jefferson

Output:

Array ([article] => Determine never to be idle. No person will have occasion to complain of the want of time who never loses any. It is wonderful how much may be done if we are always doing. - Thomas Jefferson
[ignored_terms] => Array ( [0] => No [1] => Thomas )
[affection_friendliness] => 0.45
[enjoyment_elation] => 1.27
[amusement_excitement] => 0.82
[contentment_gratitude] => 0.43
[sadness_grief] => -2.85
[anger_loathing] => -5.82
[fear_uneasiness] => -0.74
[humiliation_shame] => -0.71
[dominant_emotion] => anger_loathing
[average_intensity] => 0.92
[article_sentiment] => Array ( [sentiment] => Negative [score] => -0.57 )
[coverage] => 32
[intense_sentence] => Array ( [sentence] => No person will have occasion to complain of the want of time who never loses any.
[dominant_emotion] => anger_loathing
[intensity] => 0.2 )
[clarity] => 61.65 )

Its quite a detailed response showing the different sentiments it can pick up in the text and how much of the text it understood.

Adding additional information to incoming emails

While working on this I thought it could be great to add information to my regular emails, not just those that come in via forms on a website.

There are already some great tools for a users email inbox, such as Xobni which provides a profile with additional information about the person you are communicating with.  I use Mozilla Thunderbird for my emails so unfortunately this add-on isn’t available to me.

I connect to my email server using IMAP so I wondered if I could connect to my mail server and using some of the APIs, append some additional information to incoming emails directly in to the email content rather than using an add-on to the email client.

Using PHP, I was able to connect to my IMAP inbox and determine the email address of the sender. Using Rapleaf, I looked up the email address and added the results to the body of the incoming email. Its handy if I receive an email from someone I have never communicated with before.

If an API could be developed to add additional information to a phone number and display it on the screen of a phone during an incoming call, now that that would be cool!

Can social APIs help prevent fraud?

Recently, I was working on a client’s website, updating some older code. Their site made great use of a set of APIs combined to provide a service to their customers. It included an online payment gateway to charge the customers and their system works quite well.

My work involved replacing a deprecated API.  This changed the overall process of the system quite significantly and I spent some time updating the logging and notifications the owners used to monitor the system.

Their site receives an amount of traffic which tries to game or take advantage of their system to try to avail of their service without paying, so it’s important to have plenty of logs and techniques to monitor and protect the system.

In a situation like this it could make a lot of sense to also incorporate an API which might add additional information about the user making a transaction. Perhaps payment gateway providers such as Paypal or Worldpay already do this behind the scenes to help prevent fraud?

In a case where the identity of a buyer is in question, a system adding additional information such as age, gender and location could help a lot in deciding whether to block the transaction or allow it to proceed.

Combining 3 different APIs (Rapleaf, ToneAPI and my own Murrion APIs) I wanted to see if they could be of any use in helping identify spam that comes through forms on websites such as Contact Forms.

I built a simple form that could be on any website, asking a user for a name, email address, subject and a message to send.

You can try out a demo here. The form doesn’t submit the information anywhere other than back to the user on the screen to see the results.

• The Name field is passed to the Rapleaf Utilities API to determine if the name is Male, Female or Unknown
• The Subject and Content message are sent to the ToneAPI to try and determine the positive or negative tone of the content.
• The Email address is passed to the Rapleaf API to see if it can return any information about the user.
• The IP address (which the user isn’t asked for but is determined automatically) is sent to an API of my own which uses the Maxmind location database to convert an IP address to a geographical location.
• The Browser agent (Ie the web browser programme the user is using to view the page, such as Internet Explorer) is sent to another API of my own which can tell if a browser agent is a normal user, a web bot, a mobile device or an Unknown browser agent.

Combining these results could allow a website to filter or block the information coming through their form in any number of ways.

For example if the Name wasn’t Male or Female, if the tone of the Subject and/or Content message couldn’t be determined, if the Location couldn’t be determined and the browser agent type was unknown, then that message could be labelled as spam and not delivered to the users inbox.

If the tone of the Subject or Message as overly negative it could be sent directly to the complaints department in an organisation.

Its probably not going to solve all of the worlds spam but it was an interesting experiment nonetheless. At the very least, these APIs can provide additional information to incoming information from a website.