Archive for the ‘software’ Category



Trying out Yahoo’s Content Analysis API with PHP

Posted on December 23rd, 2011 in development, software | 2 Comments »

Yahoo has a new (to me) Content Analysis API which can perform text analysis on some text or a URL. I read about it this evening on ProgrammableWeb and had to try it out.

Its limited to 5,000 API calls per 24 hour period per IP Address, thats about enough leg room to try it out with some PHP code.

Below is some PHP code if anyone would like to use it to get started. Its very basic but helps show what the API can do.

Given a string of text, it can pick out words or statements (such as ‘Computer programming’ in the example) and provide a category for the content it understands with a score, with links to Wikipedia.

It shows the words it has found and the start and end character numbers. It even provides related links for that terms. It can spot peoples names too and link to Wiki articles about that person.

I’m only scratching the surface but I can think of some cool usages for this in other applications. Sign in on the http://developer.yahoo.com/contentanalysis/ page with your Yahoo ID and definitely try out the Console for testing Queries.

/**
* Function to use Yahoo to analyse some simple text
* @param String $text
* @param String $format
* @return String $content
*/
function yahoo_content_analysis($text, $format=’json’)
{
$url = “http://query.yahooapis.com/v1/public/yql”;

$query = ‘SELECT * FROM contentanalysis.analyze WHERE text = “‘ . $text . ‘”‘;

$characters = array(’ ‘, ‘=’, ‘”‘);
$replacements = array(’%20′, ‘%3D’, ‘%22′);

$query = str_replace($characters, $replacements, $query);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, “q=$query&format=$format”);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
$response = curl_exec($ch);
$headers = curl_getinfo($ch);
curl_close($ch);

return $response;
}

// Text taken from wikipedia
$text = ‘Computer programming (often shortened to programming or coding) is the process of designing, writing, testing, debugging, and maintaining the source code of computer programs.’;

$response = yahoo_content_analysis($text);

echo $response; // json



How to build your own URL shortener with Codeigniter

Posted on August 14th, 2011 in development, software | No Comments »

Codeigniter is an excellent lightweight PHP framework with great documentation. This post assumes you already know a little about Codeigniter. To learn more about it, visit codeigniter.com.

This tutorial will walk you through building your own URL shortening service using Codeigniter. The end result will be a basic application allowing you to enter a long URL in to a form and produce a short URL such at http://domain.com/abcd. It uses jQuery to place the shortened URL in a DIV under the form rather than reloading the page.

The source code for this application can be found on Github here : https://github.com/murrion/Codeigniter-URL-Shortener

I started by downloading a fresh copy of the current version of CI, version 2.0.2.

Update your database.php in your config folder to connect to your own database.

We’ll need a table to store the URLs. Create a table called ‘links’ in your database with the following structure:

CREATE TABLE `links` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`alias` varchar(6) CHARACTER SET utf8 DEFAULT NULL,
`url` text CHARACTER SET utf8,
`created` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `alias` (`alias`)
) ENGINE=InnoDB  DEFAULT CHARSET=latin1 AUTO_INCREMENT=1;

We’ll need 2 controllers, shorten.php and redirect.php, place these files in the application/controllers folder.

We’ll need 3 views: header.php, footer.php and form.php, place these in the application/views folder.

We’ll need 2 JavaScript files too, one is jQuery itself and the other is javascript.js which is used to contain the jquery event handler.

The Shorten.php controller does the hard work; it creates the short URL if none already exists for the URL. Redirect.php redirects a short URL to the original full length URL.

The header.php view file contains basic stuff, just a link to the jquery file located in the javascript folder and the javascript.js file containing .click event which is called when the ‘shorten’ button is pressed in the form.

The form.php file contains the form which a user enters their longer URL.

The footer.php file just contains the closing body and html tags.

Redirect.php

<?php

if (!defined(’BASEPATH’)) exit(’No direct script access allowed’);

class Redirect extends CI_Controller

{

/**

* Method to redirect from an alias to a full URL

*/

public function index()

{

$alias = $this->uri->segment(1);

$this->db->select(’url’);

$query = $this->db->get_where(’links’, array(’alias’ => $alias), 1, 0);

if ($query->num_rows() > 0)

{

foreach ($query->result() as $row)

{

$this->load->helper(’url’);

redirect($row->url, ‘refresh’);

}

}

else

{

echo “Alias ‘$alias’not found”;

}

}

}

/* End of file redirect.php */

/* Location: ./application/controllers/redirect.php */

Shorten.php controller

<?php

if (!defined(’BASEPATH’))

exit(’No direct script access allowed’);

class Shorten extends CI_Controller

{

/**

* Show a form to shorten a URL

*/

public function index()

{

$this->load->helper(’form’);

$this->load->view(’form’);

}

/**

* Take in a URL from the form and shorten it

*/

public function create()

{

$short_url = “”;

$url = prep_url($this->input->post(’url’));

$link_length = $this->config->item(’link_length’);

// Check to see if this URL has an Alias

$existing_alias = $this->alias_from_url($url);

// Generate a new alias if needed

if ($existing_alias == “”)

{

$this->load->helper(’string’);

$alias = random_string(’alnum’, $link_length);

while ($this->does_alias_exist($alias))

{

$alias = random_string(’alnum’, $link_length);

}

$this->save_new_alias($url, $alias);

$short_url = $alias;

}

else

{

$short_url = $existing_alias;

}

// display the short url

echo base_url() . $short_url;

}

/**

* Method to see if a generated Alias already exists in the table

* @param type $alias String to check to see if it exists

* @return Bool True or False

*/

function does_alias_exist($alias)

{

$this->db->select(’id’);

$query = $this->db->get_where(’links’, array(’alias’ => $alias), 1, 0);

if ($query->num_rows() > 0)

{

return true;

}

else

{

return false;

}

}

/**

* Save a new Alias to the table

* @param type $url URL to shorten

* @param type $alias  The new Alias for this URL

*/

function save_new_alias($url, $alias)

{

$data = array(

‘alias’ => $alias,

‘url’ => $url,

‘created’ => date(”Y-m-d H:i:s”)

);

$this->db->insert(’links’, $data);

}

/**

* Return an existing Alias, if any

* @param type $url String, the URL to check

* @return type $lias String, the alias, if any

*/

function alias_from_url($url)

{

$alias = “”;

$this->db->select(’alias’);

$query = $this->db->get_where(’links’, array(’url’ => $url), 1, 0);

if ($query->num_rows() > 0)

{

foreach ($query->result() as $row)

{

$alias = $row->alias;

}

}

return $alias;

}

}

/* End of file Shorten.php */

/* Location: ./application/controllers/Shorten.php */

Header.php view file

<html>

<head>

<title>Codeigniter URL Shortener</title>

<script>var base_url = ‘<?php echo base_url(); ?>’; </script>

<script language=”javascript” src=”<?php echo base_url(); ?>javascript/jquery-1.6.2.min.js”></script>

<script language=”javascript” src=”<?php echo base_url(); ?>javascript/javascript.js”></script>

</head>

<body>

Form.php view file

<?php $this->load->view(’header’); ?>

<form name=”ajax_form” id =”ajax_form” method=”post”>

<?php

$data = array(

‘name’ => ‘url’,

‘id’ => ‘url’,

‘value’ => ”,

’size’ => ‘50′,

’style’ => ”,

);

echo form_input($data);

$data = array(

‘name’ => ’shorten_url’,

‘id’ => ’shorten_url’,

‘value’ => ‘Shorten’

);

echo form_submit($data);

echo form_close();

?>

<div id=”alias”></div>

<?php $this->load->view(’footer’); ?>

Footer.php view

</body>

</html>

javascript.js

$(document).ready(function(){

$(”#shorten_url”).click(

function(){

var url=$(”#url”).val();

$.ajax({

type: “POST”,

url: base_url + “index.php/shorten/create”,

data: “url=”+url,

cache:false,

success:

function(data){

$(”#alias”).html(data);

}

});

return false;

});

});

You probably won’t want the index.php file to appear in your shortened URLs. To remove it, update your config.php file and set the index_page item to contain no value, eg : $config['index_page'] = ”;

Also, you’ll need to add a .htaccess file to the root of your application, I use the one from the Codeigniter wiki. You can copy and paste it from there if you like rather than re-writing it here.

Lastly, but most importantly, we’ll need to update the routes.php file.  If we don’t update this file, we’ll have a short link like this, which isn’t very short : http://domain.com/ shorten/index/abcd. We’ll want to remove the ‘shorten/index’ bit.

Update the routes.php file within the config folder, add the following  lines:

$route['(:any)'] = “redirect/index/$1″;

This rule will forward any item to the redirect.php controller and index method, which performs the forward.  For example : domain.com/abcd will be routed to domain.com/redirect/index/abcd where it will find the full URL and perform the redirect.

$route['shorten/create'] = “shorten/create”; // overwrite the previous route

This second route is important. It is there to overwrite the first rule to that the creating of a short URL by the shorten.php controller will work. If we left it out the previous route would think that ‘shorten’ is the short text it must look up to see if there is a full URL to go with it.

$route['default_controller'] = “shorten”;

This route will point the user to the main form to create a new short URL if they simply go to the main domain such as domain.com.

Thats it. Deploy the app to your own domain name and its now a URL shortening service.

It doesn’t record any usage logs, keep an eye on the github repository for that.



Using social networking APIs to prevent spam + code samples

Posted on March 15th, 2011 in development, software | 1 Comment »

In a recent blog post I wrote briefly about some ideas relating to using popular APIs to help identify and prevent spam that can happen with forms on a website.

I wanted to follow up on this with some further thoughts and some code examples for anyone that might like to use them.

Here is an example of using the Rapleaf utility API with PHP to determine the gender of a name.  This could be handy if you are unsure of whether to write ‘Dear Sir’ or ‘Madam’ in a reply if you are inclined to reply so formally!

The following example sends a name to the name_to_gender service and receives the JSON response in to a PHP array called $response_array.

$name = urlencode($name);
$response = file_get_contents(”http://api.rapleaf.com/v4/util/name_to_gender/$name”);
$response_array = json_decode($response, TRUE);

Output:

Array ( [status] => OK [answer] => Array ( [input] => Bill [gender] => Male [likelihood] => 0.992195 ) )

Here is an example of using the Rapleaf API to gather more information about an email address:

$response = file_get_contents(”https://personalize.rlcdn.com/v4/dr?email=XXXXX&api_key=YYYYY&show_available”);
$response_array = json_decode($response, TRUE);
return $response_array;

In this sample, you will need to replace XXXXX with your email address and YYYYY with your own API Key which you can sign up to for free.

Output:

Array ( [location] => Cork, County Cork, Ireland [gender] => Male [influencer_score] => 71-80 )

Here is a PHP example of connecting to the Lymbix toneAPI using CURL to determine the sentiment of a piece of content:

$url = “http://gyrus.lymbix.com/tonalize”;
$header_information = array(’AUTHENTICATION: XXXXXXXXXXX’,
‘ACCEPT: application/json’,
‘VERSION: 2.1′);

$data_information = array(
‘article’ => $message,
‘return_fields’ => ‘[]‘);

$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADER, $header_information);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_information);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
$result = json_decode(curl_exec($ch), TRUE);
curl_close($ch);
return $result;

In this example, I am sending the content $message to the Lymbix API located at http://gyrus.lymbix.com/tonalize. XXXXXXX would need to be replaced with your own API Key.

A sample output from the ToneAPI for the quote:

“Determine never to be idle. No person will have occasion to complain of the want of time who never loses any. It is wonderful how much may be done if we are always doing.” - Thomas Jefferson

Output:

Array ([article] => Determine never to be idle. No person will have occasion to complain of the want of time who never loses any. It is wonderful how much may be done if we are always doing. - Thomas Jefferson
[ignored_terms] => Array ( [0] => No [1] => Thomas )
[affection_friendliness] => 0.45
[enjoyment_elation] => 1.27
[amusement_excitement] => 0.82
[contentment_gratitude] => 0.43
[sadness_grief] => -2.85
[anger_loathing] => -5.82
[fear_uneasiness] => -0.74
[humiliation_shame] => -0.71
[dominant_emotion] => anger_loathing
[average_intensity] => 0.92
[article_sentiment] => Array ( [sentiment] => Negative [score] => -0.57 )
[coverage] => 32
[intense_sentence] => Array ( [sentence] => No person will have occasion to complain of the want of time who never loses any.
[dominant_emotion] => anger_loathing
[intensity] => 0.2 )
[clarity] => 61.65 )

Its quite a detailed response showing the different sentiments it can pick up in the text and how much of the text it understood.

Adding additional information to incoming emails

While working on this I thought it could be great to add information to my regular emails, not just those that come in via forms on a website.

There are already some great tools for a users email inbox, such as Xobni which provides a profile with additional information about the person you are communicating with.  I use Mozilla Thunderbird for my emails so unfortunately this add-on isn’t available to me.

I connect to my email server using IMAP so I wondered if I could connect to my mail server and using some of the APIs, append some additional information to incoming emails directly in to the email content rather than using an add-on to the email client.

Using PHP, I was able to connect to my IMAP inbox and determine the email address of the sender. Using Rapleaf, I looked up the email address and added the results to the body of the incoming email. Its handy if I receive an email from someone I have never communicated with before.

If an API could be developed to add additional information to a phone number and display it on the screen of a phone during an incoming call, now that that would be cool!

Can social APIs help prevent fraud?

Recently, I was working on a client’s website, updating some older code. Their site made great use of a set of APIs combined to provide a service to their customers. It included an online payment gateway to charge the customers and their system works quite well.

My work involved replacing a deprecated API.  This changed the overall process of the system quite significantly and I spent some time updating the logging and notifications the owners used to monitor the system.

Their site receives an amount of traffic which tries to game or take advantage of their system to try to avail of their service without paying, so it’s important to have plenty of logs and techniques to monitor and protect the system.

In a situation like this it could make a lot of sense to also incorporate an API which might add additional information about the user making a transaction. Perhaps payment gateway providers such as Paypal or Worldpay already do this behind the scenes to help prevent fraud?

In a case where the identity of a buyer is in question, a system adding additional information such as age, gender and location could help a lot in deciding whether to block the transaction or allow it to proceed.