Archived: Change to Absolute URLs in API Responses

On Monday 13th February, we will make a minor change to our Unified API responses so that they will no longer return absolute URLs as references to other API entities. They will now return a relative URL instead.

We don’t anticipate many developers are relying on these URLs being absolute, but if we’re wrong, please let us know as soon as possible in the comments section below or through the API portal.

url-code-screenshot-2
Next Monday we’re making a minor change to our Unified API responses, so that they will no longer return absolute URLs as references to other API entities

Sing for absolution

Our API responses for the Places, StopPoint, BikePoint and Roads API all return a “Url” field which is a reference to the returned entity in the REST API. This is intended to be a convenient way to navigate the API e.g. search for a BikePoint by name, then get that BikePoint’s full details by following the URL to that entity’s GetById endpoint.

For example:

$ curl "https://api.tfl.gov.uk/BikePoint/Search?query=waterloo" | jq .
 {
 "id": "BikePoints_154",
 "url": "https://api-neon.tfl.gov.uk/Place/BikePoints_154",
 "commonName": "Waterloo Station 3, Waterloo",
 "placeType": "BikePoint",
 "lat": 51.503791,
 "lon": -0.112824
 },
 {
 "id": "BikePoints_160",
 "url": "https://api-neon.tfl.gov.uk/Place/BikePoints_160",
 "commonName": "Waterloo Place, St. James's",
 "placeType": "BikePoint",
 "lat": 51.506633,
 "lon": -0.131773
 },

If I want the details of the first result, I just follow that URL, and get the whole resource with all of its additional properties:

$ curl https://api-neon.tfl.gov.uk/Place/BikePoints_154 | jq .
{
 "id": "BikePoints_154",
 "url": "https://api-neon.tfl.gov.uk/Place/BikePoints_154",
 "commonName": "Waterloo Station 3, Waterloo",
 "placeType": "BikePoint",
 "additionalProperties": [
 {
 "category": "Description",
 "key": "NbBikes",
 "value": "10",
 "modified": "2017-02-06T13:26:42.063Z"
 // etc
 }]
 "lat": 51.503791,
 "lon": -0.112824
}

As you can see, this leaks an implementation detail that the backend environment was “api-neon.tfl.gov.uk”, not api.tfl.gov.uk, which is what we started with. Users of our API may have noticed argon, radon, or even xenon in the responses.

URI fever

These environment names are no great secret, but its better for developers if they don’t know (or even have to care) about these details. Ideally everything external points to api.tfl.gov.uk, and we route to the correct backend at the time we receive the request.

Developers that save these absolute URLs could also run into problems if the environment becomes non-live – as they regularly do for our blue-green, continuous delivery rotation.

We debated a few ways of fixing the absolute URL to reflect the caller’s original domain, however in the end we settled on just returning a relative URL and letting the caller resolve it as required. The previous example now becomes:

$ curl "https://api.tfl.gov.uk/BikePoint/Search?query=waterloo" | jq .
 {
 "id": "BikePoints_154",
 "url": "/Place/BikePoints_154",
 "commonName": "Waterloo Station 3, Waterloo",
 "placeType": "BikePoint",
 "lat": 51.503791,
 "lon": -0.112824
 },

We hope that this change doesn’t impact you significantly. Please let us know in the comments section below if you have any questions or concerns.

4 Comments

  1. I must admit that I use this kind of PHP code…


    private static function fixPathForURL($strSrc, $strFixHost, $strFixScheme)
    {
    $arrNode = urlGetContents::ParseURLfilled($strSrc);
    $arrReconfig = ["scheme" => $strFixScheme, "host" => $strFixHost, "path" => "", "query" => ""];
    foreach ($arrReconfig as $strWhat => $strWhere) {
    if ($arrNode[$strWhat] == "") {
    $arrNode["scheme"] = $strWhere;
    }
    }
    $arrNode["path"] = strtr("/" . $arrNode["path"], ["//" => "/"]);
    $strSrc2 = $arrNode["scheme"] . "://" . $arrNode["host"] . $arrNode["path"];
    if ($arrNode["query"] != "") {
    $strSrc2 .= "?" . urlencode($arrNode["query"]);
    }

    return $strSrc2;
    }

    public static function ParseURLfilled($strInput)
    {
    $arrTester = parse_url($strInput);
    foreach (["scheme", "host", "extension", "query", "path"] as $strWhat) {
    if (!isset($arrTester[$strWhat])) {
    $arrTester[$strWhat] = "";
    }
    }

    return $arrTester;
    }

    to fix-up the URLs from relative to absolute to ensure that things don’t go wrong if the remote end changes.

  2. Hi, this doesn’t affect me, just a question.

    What was the argument for not changing the URL to be absolute and point to api.tfl.gov.uk, and then you determine where to route the request to as mentioned in the post?

    Cheers

    1. Hi Dennis, glad to hear you’re not affected.

      In most cases, setting to api.tfl.gov.uk would be correct, however when accessing the API from our internal domains it might not be. On the server-side, behind many layers of cache, load balancer and CDN, it’s difficult to accurately get the referring domain of the original request. We thought just making it relative was by far the easiest approach, since the caller knows the context. Similar to using relative URLs in most places on a normal website where you want to caller to stay within the domain.

Comments are closed.