Part 7: Connecting local expertises

“Local expertise” should not be understood as “geographically local” but as “expertise in a particular niche”. On top of being a series of tubes, the web is a tapestry of niches (e.g. “make money online”, “web 2.0 startups”, “reviews of gagdets”, “flash games”, “travel guides”, “maps” etc.). The goal for every webmaster is to make his/her website the leader in his/her particular niche.

Being a leader means having some influence on your followers or on other leaders. Now what’s the best way of having that kind of influence than making your followers use your data? Let’s take an example: Techcrunch, the famous blog commenting the web 2.0 startups ecosystem, has quickly become one of the leader blogs in the niche (quite a tremendous amount of RSS subscriptions). They developed a whole blog network totalizing 15 websites so far, including an interesting project: Crunchbase, a free and organized database containing updated information about startups and startup founders. In a nutshell, Crunchbase is to Techcrunch what Freebase is to Wikipedia: an organized abstraction of pure and cross-linked data. But Crunchbase goes even further: it is possible to embed Crunchbase information in any post, just like you can embed a Youtube video in your blog or a Scribd document in your website (more information here).

Here is an example of Crunchbase widget for Flickr:

This is quite a smart move from Techcrunch: now blogs or websites in the same niche (or elsewhere) can directly embed Crunchbase data into their posts or pages. For Crunchbase data, it’s a first step outside through parameterizable widgets: it’s outside of Crunchbase’s official website, but the data is still enclosed in the widget’s HTML.

Recently, Crunchbase released an API. For Crunchbase data, that’s the second step outside, this time in complete freedom since developers can access pure JSON. Now third-party developers can build layers on top of Crunchbase data, which makes the latter a new reference. The example of Techcrunch is interesting: for it’s usually not easy to get accurate information about private equity fundings and acquisitions, opening access to a comprehensive and cleverly built database simplifies the developer’s life and, in a way, prevents him from reinventing the wheel (although it’s not exactly like an algorithm but it’s more of an information provider).

By extending its reach through widgets and an API, the Techcrunch network is aiming at becoming a comprehensive reference i.e., in the hot web 2.0 startups niche, a local expert. They’re not alone though: seeing the threat coming, ReadWriteWeb recently launched its own companies database too.

Still, the strategy can’t only consist in opening wide access to core data: there is an equilibrium to find between locking all the data (bad move: competitors opening their data could reach your audience) and granting full access to it (bad move too: competitors could artificially bloat by absorbing your content or bury your web application under several layers of mashups).

From APIs to mashups

For an application, a blog or any website, releasing an API may help becoming a local expert. For third-party developers, using APIs makes it possible to connect local expertises.

Say you want to build a cool webapp (who doesn’t?) and provide users with geolocalized pictures of places they’ve been to. The easiest and best way to do so would certainly be to use existing services through their APIs:

• Facebook for profile data and places where the user has been (requires the user’s permission)
• Flickr to find the pictures
• GoogleMaps to display the pictures in a neat map of the world.

The service could work as follows:

1. the user signs up and grants access to his Facebook profile data
2. your web application parses his/her profile and finds relevant places,
3. then gets photos from Flickr tagged with these places (and gets geolocalization too),
4. then create a custom GoogleMap with pictures on it,
5. and displays it back to the user for him/her to interact

Here is a schema to better understand this example of a connection between local expertises (namely expertises in profile data, photos and maps):

Conclusion

Through this series of posts, we’ve been reviewing APIs both on technical and business sides. In these intense times when every hour could bring you a new competitor from the other end of the world willing to lead your niche, it is important to build the basements of your user base and, if your business allows you to release an API, to do so and extend your reach.

A lot of nice people are here to help you building your API:

• experts such as Google’s Joshua Bloch (and his famous keynote on APIs)
• companies such as Mashery, specialized in helping you rolling out your own API
• more advanced tutorials on how to design your own API (now that REST, XML-RPC and SOAP are understood)
• tons of classified APIs at ProgrammableWeb for you to get inspiration
  

I sincerely hope you enjoyed this series of posts. Please feel free to leave a comment!

Related posts:

1. The pursuit of APIness (part 6) The first five posts of series "The Pursuit of APIness"...
2. The pursuit of APIness (part 1) Say you need to upload a set of 100 pictures...
3. The pursuit of APIness (part 5) XML-RPC is damn easy to use and many projects still...

Part 6: Web APIs strategy

APIs are a fantastic way to make computers talk together and build great business layers on top of other business layers on top of etc. On the human side of the world, where things tend to be less binary, APIs represent a two-way strong strategic tool (”two-way” because releasing an API impacts both the original developers of a webapp and third-party developers building a third-party webapp).

Let’s start with the third-party developer’s view. In the first post of this series, we considered a fictitious dude named Joe, willing to programmatically get data out of a website called community.com (fictitious website). For him, as for any third-party webdeveloper, using an API is great because it allows to:

• backup existing data: say Joe is not confident in the future of community.com, he can programmatically backup his data stored in there through the API
• leverage existing data and, for example, display it in a more innovative way: Joe could build a great third-party application relying on community.com’s API and showing an interactive US map with the members’ names updated in real time
• catch new users: say Joe builds this app, then (at least!) a percentage of original community.com users will be interested in it and will start using it
• unload some server resources: some web services do things well and fast, so why bother reinventing the wheel and consume server resources? For example, you could save storage for your pictures by using Flickr’s API (or Amazon S3’s), you could create charts using Google’s Chart API etc.

If we look at things from community.com’s point of view, releasing an API is a strong signal and should be part of a well-thought strategy, aimed at:

• extending reach by bringing in new users from third-party apps using his/her API (there’s a mutually beneficial exchange of users between the original application and the third-party application, each one becoming a new entry point for the other)
• becoming a local expert: this is a really important point which will be analysed in the next (and last) post of this series; in a nutshell, releasing a clever API can erase competition from your niche
• fighting against web scraping: third-party webdevelopers would now have an official and documented way to get data out of an application (of course they can still keep on scraping your web pages but releasing an API greatly simplifies their lives and encourages them to keep to the straight and narrow)

API design

When releasing an API, developers should follow a comprehensive checklist to ensure third-party developers will start using it smoothly. In particular, it is essential to:

• offer the widest range of protocols: some developers love REST, so you have to release a RESTful version or your API; some other developers can’t live without XML-RPC so open a XML-RPC access to them; some other have existing libraries built for SOAP so release a SOAP API; some other love JSON so build a custom “JSON-RPC” API for them etc.
• document all the API’s functions for every protocol: yes, this is a huge work and this is were you will spend most of your time; documentation is key and the clearer it is, the more developers will use you API. Check out Flickr’s or Facebook’s APIs documentation: they’re real jowels for webdevelopers, filled with substantial code snippets helping them to get quickly started with the API.
• version your API: remember that once released, an API is forever. This means you can’t go backwards and change existing functions in your API because this would be a disaster for third-party applications built upon it and, therefore, for your users and your reputation. With an API, you can only go onwards. So it is extremely important to explicitly define versions for your API, for example by inserting version numbers right in the URL
• make it secure: web APIs have to be secure, both technically and strategically. Technically because accessing the data should happen the way it was meant to be without disrupting existing processes on the server or causing mess in the database. Strategically because releasing a web API is opening a documented door to the very heart of your content. An easy way to enhance security for your API is to release API keys: third-party developers have to request a key (usually a hash-like sequence of digits and letters) and then explicitly show this key in every single call made through the API. Doing so allows you to better monitor traffic and usage of your API.
• anticipate traffic surge: releasing an API means facing traffic spikes and, if everything goes well, facing a global increase in the load. It is crucial to anticipate this and avoid outages.

Protecting web property

Here is a simple schema to better understand what’s going on when releasing an API:

In this example, website 1 is the usual interface developed to display the data (for example: website 1 is http://community.com displaying the community.com data). Website 2 is a parallel website accessing community.com’s data but not developed by community.com. For example, it’s an aggregator designed to consolidate profiles from different online communities. For community.com, releasing a web API is interesting to extend its reach and bring in new users (for example, users of the aggregator service at website 2 who didn’t know about website 1 until they discover some of its members) but it also means organizing a documented data leak towards third-party apps, hence the need to focus on technical and strategic security.

The pursuit of APIness continues next week (last episode will be part 7). Stay tuned!

(go to Part 7: Connecting local expertises)

Related posts:

1. The pursuit of APIness (part 7) Already the seventh post of series "The pursuit of APIness",...
2. The pursuit of APIness (part 2) Last week we looked at web forms and we observed...
3. The pursuit of APIness (part 5) XML-RPC is damn easy to use and many projects still...

Part 5: SOAP: Hot or not?

Like XML-RPC, SOAP is a protocol for XML messaging over HTTP. Let’s start with the basic structure of a SOAP message (without HTTP headers for the moment):

<?xml version="1.0"?>
<soap:Envelope
  xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
  soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
  <soap:Header>
    ...
  </soap:Header>
  <soap:Body>
    ...
  </soap:Body>
</soap:Envelope>

Note: as you can see, a SOAP message is made of an Envelope containing an optional Header and a mandatory Body. No need to get crazy about namespaces: you can basically chose your own namespace for any SOAP request. Look at the elements of the Envelope tag: in this example we chose “soap” and we had to tell that this “soap” namespace follows some standards (the two lines literally mean “the ’soap’ namespace we use has to be considered to be in the normal SOAP namespace” and “the ’soap’ namespace follows some standards for data types”). Instead of soap, we could have used “SOAP-ENV”, “env”, “s” as well as “mynameisjoe”. The important thing here is only to declare the namespace.

Now let’s look at a more realistic SOAP request with HTTP headers and with a different namespace for the envelope, let’s say a mere “s” as used for the Flickr API. The Header element being optional, we won’t use it here:

POST /soap HTTP/1.1
Host: api.community.com
User-agent: script
Content-type: text/xml
Content-length: 332

<?xml version="1.0"?>
<s:Envelope
  xmlns:s="http://www.w3.org/2003/05/soap-envelope"
  s:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
  <s:Body>
    <a:GetMembersList xmlns:a="http://api.community.com/soap">
      <age>23</age>
      <city>Indianapolis</city>
    </a:GetMembersList>
  </s:Body>
</s:Envelope>

Note: as you can see, the HTTP headers must have a Host, a User-agent and a Content-type set to text/xml. Now look inside the Body element: what lies there is application-specific data which has to be identified using a different namespace (here we used the SOAP API’s entry point of website community.com). Developers have complete freedom to define what goes inside. Here we chose namespace “a” for all this application-specific data and we described which method had to be called (i.e. GetMembersList) along with parameters age and city.

So far we have seen the basic structure of a SOAP request. Now here is what the SOAP response could look like:

HTTP/1.1 200 OK
Content-type: text/xml
Content-length: 391

<?xml version="1.0"?>
<s:Envelope
  xmlns:s="http://www.w3.org/2003/05/soap-envelope"
  s:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
  <s:Body>
    <a:GetMembersListResponse xmlns:a="http://api.community.com/soap">
      <members>
        <member>Anna</member>
        <member>Lisa</member>
      </members>
    </a:GetMembersListResponse>
  </s:Body>
</s:Envelope>

Note: few things have changed in the response. The method name has changed (following the usual convention of appending “Response” to the request’s method name) and the appication-specific data has changed too (it now returns the names of the two members aged 23 and living in Indianapolis). Apart from that, all the namespaces and definition thing remains the same.

Keep in mind that SOAP is all about abstraction and that its complexity is the price for its flexibility. For example, we chose to design the application-specific data inside the first childnode (by calling it a:GetMembersList) but we could have decided as well to specify it along with the data in the following way (the Flickr way):

<a:Request xmlns:a="http://api.community.com/soap">
  <method>GetMembersList</method>
  <members>
    <member>Anna</member>
    <member>Lisa</member>
  </members>
</a:Request>

In the request example above, we did not specify data types for the parameters we sent along with the request. Without going too much into details, here is the way to specify that <age> is an integer and <city> a string:

POST /soap HTTP/1.1
Host: api.community.com
User-agent: script
Content-type: text/xml
Content-length: 482

<?xml version="1.0"?>
<s:Envelope
  xmlns:s="http://www.w3.org/2003/05/soap-envelope"
  s:encodingStyle="http://www.w3.org/2003/05/soap-encoding"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema"
  >
  <s:Body>
    <a:GetMembersList xmlns:a="http://api.community.com/soap">
      <age xsi:type="xsd:int">23</age>
      <city xsi:type="xsd:string">Indianapolis</city>
    </a:GetMembersList>
  </s:Body>
</s:Envelope>

Note: as you can see we need two more specifications defined as attributes of the Envelope tag. First we need to defined the namespace for XML Schema Instance and then the namespace for XML Schema Datatypes, and then we have to add the corresponding attributes to the <age> and <city> tags (the latter step being quite self-explanatory).

That’s it for this introduction to SOAP. So.. hot or not?

In many cases, SOAP is just bloated XML-RPC. If you can stick to XML-RPC then do it; no need to start using SOAP. On the other hand, if you want to expand the reach of your API or if you need very specific data types for your requests and responses, then go for SOAP. The final choice usually depends on the available libraries: as SOAP is more complex to manipulate, using a good library becomes more important than for XML-RPC where you can go down to making your PHP script directly spit the XML.

(go to Part 6: Web APIs strategy)

Related posts:

1. The pursuit of APIness (part 4) Last week we reviewed the principles behind a RESTful architecture....
2. The pursuit of APIness (part 1) Say you need to upload a set of 100 pictures...
3. The pursuit of APIness (part 2) Last week we looked at web forms and we observed...

Part 4: Review of XML-RPC

XML-RPC stands for XML Remote Procedure Call: a XML-RPC request is a HTTP POST request which body is formatted in XML and a XML-RPC response is a XML file sent back by the server. Contrary to REST, the method called by a XML-RPC request does not appear in the URL but appears in the XML POST body of the request, in-between tags <methodName>. Therefore, XML-RPC works based on a single entry point which is the unique URL to be called when using XML-RPC.

Before we dive into the XML-RPC specifications, here is how our favorite call (Joe’s call) to community.com’s API could be designed:

POST /xmlrpc HTTP/1.1
Host: api.community.com
User-agent: script
Content-type: text/xml
Content-length: 243

<?xml version="1.0"?>
<methodCall>
  <methodName>community.members.getList</methodName>
  <params>
    <param><value><int>23</int></value></param>
    <param><value><string>Indianapolis</string></value></param>
  </params>
</methodCall>

Note: several details matter in this sample request. First, we decided to use http://api.community.com/xmlrpc/ as the single entry point. Then, XML-RPC requires a User-agent (can be the name of the library you’re using, the name of your script or the name of your dog ) and a Content-type set to text/xml. Last, method names must be unique: had we simply called this method getList, we could have faced difficulties when designing the call to retrieve, for example, cities instead of members. This is why we prepended the method name with community.members so as to be aware that we are using community.com’s API and that we are looking for members.

And here would be the successful response:

HTTP/1.1 200 OK
Date: Tue, 01 Jul 2008 15:22:00 GMT
Content-Length: 313
Content-Type: text/xml
Connection: close

<?xml version="1.0"?>
<methodResponse>
  <params>
    <param>
      <value>
        <array>
          <data>
            <value><string>Anna</string></value>
            <value><string>Lisa</string></value>
          </data>
        </array>
      </value>
    </param>
  </params>
</methodResponse>

Now if we assume there was some issues while transmitting or processing the request, here is how an error response could be formatted:

HTTP/1.1 200 OK
Date: Tue, 01 Jul 2008 15:22:00 GMT
Content-Length: 384 
Content-Type: text/xml
Connection: close

<?xml version="1.0"?>
<methodResponse>
  <fault>
    <value>
      <struct>
        <member>
          <name>faultCode</name>
          <value><int>1</int></value>
        </member>
        <member>
          <name>faultString</name>
          <value><string>Invalid request.</string></value>
        </member>
      </struct>
    </value>
  </fault>
</methodResponse>

Note: there is no official specification for error codes and error messages. Each API designer should conscientiously create error codes and error messages for each possible scenario. Still, I could find one attempt to make error codes more official on this webpage.

This is typical of a XML-RPC protocol: instead of relying on an existing standard, XML-RPC adds a new abstraction layer. A RESTful approach would have used the HTTP status tool (namely: a HTTP/1.1 400 Bad Request) to notify the client that the request didn’t work; instead, the XML-RPC approach clearly overwrites HTTP and puts the error message in the very XML response delivered. As a matter of fact, the HTTP status of the response is still HTTP/1.1 200 OK even though the request resulted in a failure.

The three examples above illustrate how XML-RPC looks like both in terms of requests and responses. The underlying structure is easy to guess: a request has to be contained between <methodCall> tags and specify both the <methodName> and optional <params> (parameters). Symmetrically, a response must be contained between <methodResponse> tags and speficy optional <params>. Parameters are contained in-between <params> tags and each parameter must be surrounded by <param> and <value> tags, and then tags corresponding to their data type. For example, an integer parameter will be sent like this: <param><value><int>23</int></value></param>.

XML-RPC defines several types of vanilla data types:

• int or i4: 32-bit integers (e.g. <int>23</int> or <i4>23</i4>)
• double: 64-bit floating-point numbers (e.g. <double>2.7812</double>)
• boolean: boolean (true (1) or false (0): e.g. <boolean>1</boolean>)
• string: ASCII or Unicode text (e.g. <string>Hello World!</string>)
• dateTime.iso8601: a datetime (e.g. <dateTime.iso8601>20080701T15:22:00</dateTime.iso8601>)
• base64: some base64-encoded information (e.g. <base64>SGVsbG8gV29ybGQh</base64>)

And two types of elaborated data types:

• array: as shown in the “sample successful response” above, arrays further enclose data in <data> tags with as many <value> sub-items as there are values in the array (together with their data types)
• struct: as shown in the “sample failure response” above, structured data is made of <members>, each of them having a <name> and a <value> with a specific data type (a bit like an associative array in PHP)

Note: both struct and array elements are recursive (I.e. a struct element can contain an array containing struct elements etc.): this opens endless possibilities for data formatting.

Programmatically speaking, creating XML-RPC clients and servers using PHP and relying on cURL and SimpleXML is absolutely not more complicated than the numerous code snippets explained in parts 1, 2 and 3. Only the XML syntax changes. Also note that there is a specific PHP extension for using XML-RPC (more information on php.net).

The pursuit of APIness continues next week with a quick review of XML-RPC’s successor: SOAP.

(go to Part 5: SOAP: Hot or not?)

Related posts:

1. The pursuit of APIness (part 2) Last week we looked at web forms and we observed...
2. The pursuit of APIness (part 3) In part 2, we saw how could be designed an...
3. The pursuit of APIness (part 5) XML-RPC is damn easy to use and many projects still...

Part 3: Let’s take a REST

REST (REpresentational State Transfer) is actually more than a plain API structure. When coining this concept, Roy Fielding described an entire style of software architecture based on:

• protocol constraints: the protocol has to be client-server, stateless, cacheable, layered and allowing code-on-demand
• resources: resources are fundamental informational elements on which operations can be realized (representing, linking, modifying, including, searching, caching etc.)
• universal syntax: resources are uniquely addressable using a universal syntax
• representation: resources are not data and therefore need to be represented (through pictures, HTML or content of any kind)
• uniform interface: the transfer of state is executed through well-defined operations and content-types

An good example of RESTful architecture is the World Wide Web: HTTP is client-server, cacheable, layered, stateless when used RESTfully (i.e. without cookies or sessions), able to carry code-on-demand (e.g. JavaScript or Java applets) and accesses resources through a uniform interface (HTTP methods such as GET/POST/PUT/DELETE, MIME content-types, headers and URIs). Still, HTTP can be used more or less RESTfully and we can imagine other RESTful architectures than HTTP.

Let’s now apply the REST principles to the code snippets we have written in part 2. A RESTful API doesn’t have to comply with a given XML format but with an architectural style. So here is how we could rewrite our API call in a RESTful manner, i.e. using the full potential or URIs and HTTP methods:

GET /members?age=23&city=Indianapolis HTTP/1.1
Host: api.community.com

As you can see, the URL has changed: we are no longer POSTing XML to an opaque “submit” resource but rather calling a meaningful “members” resource using the GET HTTP method and specifying meaningful parameters directly in the URL.

HTTP/1.1 200 OK
Date: Tue, 17 Jun 2008 20:24:00 GMT
Content-Length: 85
Content-Type: application/text-xml

<?xml version="1.0" encoding="UTF-8"?>
  <data>
    <name>Anna</name>
    <name>Lisa</name>
  </data>

The response format does not need to change: as we’ve seen, REST doesn’t define a XML format but gives general principles of architectural design.

Programmatically speaking, there is little change in the script consuming the web service (we’re still using PHP, cURL and SimpleXML). This is what our code would look like:

<?php

  // STEP 1: SEND THE REQUEST
  $url = 'http://api.community.com/members?age=23&city=Indianapolis';
  $handle = curl_init();
  curl_setopt($handle, CURLOPT_URL, $url);
  curl_setopt($handle, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($handle);
  curl_close($handle);

  // STEP 2: PARSE THE RESULT AND DISPLAY THE NAMES
  if ($response) {
    $result = new SimpleXMLElement($response);
    $val = array();
    foreach ($result->name as $name) {
      $val[] = (string) $name;
    }
    print_r($values);
  } else {
      echo "An error occurred.";
  }

?>

Note: in case the parameters you provide to the URL have special caracters you need to urlencode() them. Also, note that we have added a condition to check the response before looping through it.

Let’s now have a look at the script providing the web service; here is an example of how the REST call could be handled (assuming the script name is members.php, located in /scripts and accessible through Apache URL rewriting):

RewriteEngine on
RewriteRule ^members(.*)$ scripts/members.php$1

And now the script:

<?php

  $connection = mysql_connect('host', 'user', 'password');
  mysql_select_db('database', $connection);

  $age = $_GET['age'];
  $city = $_GET['city'];
  if (!is_int($age) || !is_string($city)) {
    header('HTTP/1.1 400 Bad Request');
    exit();
  } else {
    $query = 'SELECT name FROM members WHERE age=' . $age .
             ' AND city=' . $city;
    $data = mysql_query($query);
    $members = array();
    header('Content-Type: application/text-xml');
    echo "<data>";
    while ($item = mysql_fetch_array($data)) {
      echo "<name>" . $item['name'] . </name>;
    }
    echo "</data>";
  }

?>

In a nutshell, here is why this rewritten script is more RESTful:

• we consume the web service through a GET parameter calling a meaningful members resources and specifying meaningful parameters right in the request URI
• in case there is an error (e.g. incorrect data for the age parameter), we use a dedicated HTTP status to carry the error message instead of adding a layer of XML in the response

The most important thing to remember about REST is that it’s only a set of architectural recommandations. Next week we’ll have a look at another API standard focusing on the XML message format more than the general architecture: XML-RPC.

(go to Part 4: Review of XML-RPC)

Related posts:

1. The pursuit of APIness (part 2) Last week we looked at web forms and we observed...
2. The pursuit of APIness (part 1) Say you need to upload a set of 100 pictures...
3. The pursuit of APIness (part 4) Last week we reviewed the principles behind a RESTful architecture....

Part 2: XML rocks

An Application Programming Interface is a predefined set of procedures for a program to use another one. On the Internet, an API will make it possible for a computer to use another one. Contrary to the hackers’ method previously explained, web APIs are meant to be officially supported by the webapp owners through extensive documentation and support. Your script will no longer try to imitate the human’s way with variables names and action URL but will rather take his own path to get data from the server.

Web APIs mostly rely on XML, which makes it possible to transfer structured data over the web. Basically, instead of sending raw POST variables to the server and receiving HTML, your request will send and receive clean and unchanging XML data.

At first, there is no limit to imagining and designing an API. Based on the example in part 1, we could imagine the following kind of request and response:

POST /submit HTTP/1.1
Host: api.community.com
Content-Length: 105
Content-Type: application/text-xml

<?xml version="1.0" encoding="UTF-8"?>
  <data>
    <name>Joe</name>
    <city>Indianapolis</city>
    <age>23</age>
  </data>

Note: it is quite common to design the API so that requests are sent to a subdomain like api.community.com instead of www.community.com. This way, requests at subdomain www are supposed to be sent by human beings and trigger HTML responses whereas requests at subdomain api are supposed to be sent by computers and trigger XML responses. Also note that I removed the .php extension from the submit page: through the API, we’re not really accessing a page but rather a resource.

HTTP/1.1 200 OK
Date: Tue, 17 Jun 2008 20:24:00 GMT
Content-Length: 85
Content-Type: application/text-xml

<?xml version="1.0" encoding="UTF-8"?>
  <data>
    <name>Anna</name>
    <name>Lisa</name>
  </data>

Programmatically speaking, coding the request in PHP is possible using cURL and parsing the response would certainly be easier using SimpleXML (included in PHP5). Therefore, implementing the above API call in PHP would look like this:

<?php

   // STEP 1: SEND THE REQUEST
   $url = 'http://api.community.com/submit';
   $header = array('Content-type: application/text-xml');
   $xml = '<?xml version="1.0" encoding="UTF-8"?><data><name>Joe</name>
           <city>Indianapolis</city><age>23</age</data>';
   $handle = curl_init();
   curl_setopt($handle, CURLOPT_URL, $url);
   curl_setopt($handle, CURLOPT_HTTPHEADER, $header);
   curl_setopt($handle, CURLOPT_POSTFIELDS, $xml);
   curl_setopt($handle, CURLOPT_RETURNTRANSFER, 1);
   $response = curl_exec($handle);
   curl_close($handle);

   // STEP 2: PARSE THE RESULT AND DISPLAY THE NAMES
   $result = new SimpleXMLElement($response);
   $val = array();
   foreach ($result->name as $name) {
      $val[] = (string) $name;
   }
   print_r($values);

?>

Note: in this cURL script I had to specify the Content-type header so that our request could be correctly interpreted by the server. I could have used a few more options such as CURLOPT_TIMEOUT for example.

Using an API is better than forcing a web form for 3 reasons:

• the request goes through a special path designed for computers which is clearly defined at subdomain api.community.com
• the response data is better structured and easier to parse than a raw HTML page
• XML responses being provided for machines, the developers at community.com must commit on keeping a constant XML format: this means, for example, that they can’t suddenly stop accepting <city> tags in the requests or change <name> tags for <person> tags in the responses (otherwise it would spread a terrible mess in 3rd party applications leveraging their API)

The last reason is well summed up in Google’s Joshua Bloch’s famous keynote on API design:

Public APIs are forever - one chance to get it right.

In the example above, we have assumed that community.com developers had created their API from scratch, inventing a custom XML structure for requests and responses. When it comes to designing the API, the reality is that developers tend to respect some existing API structures specifically designed for APIs in order to make their API easier to learn and use for 3rd party developers.

Today, 3 API standards are widely used on the web:

• REST (REpresentational State Transfer)
• XML-RPC (XML Remote Procedure Call)
• SOAP (Simple Object Access Protocol)

Next week we’ll have a look at these three XML standards with detailed examples of how to send and receive data.

(go to Part 3: Let’s take a REST)

Related posts:

1. The pursuit of APIness (part 3) In part 2, we saw how could be designed an...
2. The pursuit of APIness (part 1) Say you need to upload a set of 100 pictures...
3. The pursuit of APIness (part 5) XML-RPC is damn easy to use and many projects still...

Part 1: The dirty way

First thing that pops up in your mind is: let’s make my computer fill the forms alone! The idea is pretty straightforward: you write a script able to login to your Flickr account using your credentials and, once there, to fill the form fields. To fill the form fields? Does it mean I will see my computer in action and the cursor moving from one field to another one, opening the file selection dialog box, double clicking on one file, moving on to the next field etc.? Well, not really, for all this is pure client-side stuff.

In a form, only three things matter: the method (usually GET or POST), the names of the fields and the action URL (i.e. the URL your browser sends a request to when you press the “submit” button). At the end of the day, “automatically filling out a form” simply means sending a GET or POST request to the action URL with correct field names and values.

Let’s move on to a simple case study. When filling out a GET form on cool website http://www.community.com (fictitious example), here is what 23 years old Joe from Indianapolis would send back to the server:

GET /submit.php?name=joe&city=indianapolis&age=23 HTTP/1.1
Host: www.community.com

In case the form was a POST, here is what Joe would have sent back:

POST /submit.php HTTP/1.1
Host: www.community.com
Content-Length: 33
Content-Type: application/x-www-form-urlencoded

name=joe&city=indianapolis&age=23

Note: the requests above are simplified ones, containing only the required headers for GET and POST methods. In fact, many more headers would usually be sent along with these ones (see the list of HTTP headers on Wikipedia).

That was great. Now the funny thing is: upon receiving the request, the servers looks up the database to find other people aged 23 living in Indianapolis and sends a webpage back to Joe with the list of names on it. After Joe submitted his personal information, he will see a webpage featuring members Anna and Lisa.

HTTP/1.1 200 OK
Date: Tue, 10 Jun 2008 19:38:07 GMT
Content-Length: 268
Content-Type: text/html

<html>
   <head>
      <title>Community.com rocks!</title>
   </head>
   <body>
      <h1>There are 2 members aged 23 and living in your town:</h1>
      <ul>
         <li>Member #1: Anna</li>
         <li>Member #2: Lisa</li>
      </ul>
   </body>
</html>

Note: the webpage above is a simplified one. Come on there’s not even a DOCTYPE declaration in it !

That was so cool. By filling out a plain form, Joe was able to know some of his neighbours also belonging to http://www.community.com. Rephrased version: by sending out a request, Joe received a reply with data in it. A human (supposedly non geek) would read the data as displayed on his browser; but a script could parse the HTML and extract relevant bits of information.

Now Joe is a clever guy knowing about computers so he decides to write a script exploiting this form and aimed at retrieving the members from http://www.community.com aged 23 and living in Indianapolis. The script would essentially consist in emulating the web form by sending a request to http://www.community.com and then parsing the HTML result. Such a script could easily be used to know when new users of the same age and living in the same town become members of http://www.community.com: to do so, Joe would have to run it daily or make it a daily CRON task.

Joe knows PHP and will use a wrapper called cURL to write the requests and then a regex to parse the results. Here is what the script could look like:

<?php

   // STEP 1: SEND THE POST REQUEST
   $url = 'http://www.community.com/submit.php';
   $handle = curl_init();
   curl_setopt($handle, CURLOPT_URL, $url);
   curl_setopt($handle, CURLOPT_POST, 1);
   curl_setopt($handle, CURLOPT_POSTFIELDS,
               'name=joe&city=indianapolis&age=23');
   curl_setopt($handle, CURLOPT_RETURNTRANSFER, 1);
   $result = curl_exec($handle);
   curl_close($handle);

   // STEP 2: PARSE THE RESULT AND DISPLAY THE NAMES
   preg_match_all("|<li>Member #[0-9]+: (.*)</li>|", $result, $match);
   print_r($match);

?>

Note: it is safer to use more options when writing a cURL request (especially options like CURLOPT_TIMEOUT).

In this script, Joe decided to simply print_r() the array of names, but he could have decided as well to insert them in a database or send them by email. The important thing is: Joe was able to get structured data out of a request using variables.

The next step would be to extend this script to all ages and cities in the US. Using two nested loops featuring all ages between 21 and 100 and all main cities of the US for example, it would be possible to rebuild a fair part of http://www.community.com’s members database.

As a matter of fact, the script Joe is writing goes well beyond the normal use of the initial form and is certainly not supported by the website owners. Providing that they no longer want such scripts to automatically exploit the form and spit members’ names, the developers at http://www.community.com have several weapons at their disposal:

1. first thing they can do is to regularly change the names of the form fields, which means that Joe would have to update his own script each time there’s a such a change otherwise the script would blindly keep on sending outdated variables to the server
2. second thing they can do is to regularly update the HTML code of the delivered webpage: for example, adding a simple class to the lines would break the regex
3. third move: limiting the number of requests in a given timeframe, i.e. making it impossible for a single IP to send more than one request every 10 minutes for example
4. lethal weapon: adding a captcha, which is a much trickier obstacle to skip

The script Joe has written is a hacker’s self-made API of http://www.community.com’s members database: it’s a bridge towards the content stored in there. Now wouldn’t it be nicer if, instead of sniffing variable names and updating a regex, it could be possible to send a structured query to http://www.community.com and get a structured, officially supported and constant response?

(go to part 2 : XML rocks)

Related posts:

1. The pursuit of APIness (part 2) Last week we looked at web forms and we observed...
2. The pursuit of APIness (part 3) In part 2, we saw how could be designed an...
3. The pursuit of APIness (part 5) XML-RPC is damn easy to use and many projects still...