Glitch proxy for API requests in Observable

Give me a reason: CORS headers

CORS (Cross-Origin Resource Sharing) is a fundamental web security feature that acts like a gatekeeper, controlling how web pages can request resources from different domains. For example, when working in an Observable notebook and trying to fetch data directly from an external API (as I did), you are making a cross-origin request because your notebook is running on observablehq.com while trying to access data from another domain. Your browser will reject this “cross-origin” action unless the server explicitly allows it!

We may think of CORS like an international border crossing. Your Observable notebook is in one country (observablehq.com), and the API you want to access is in another country. Just as you need proper documentation to cross borders, your web requests need proper CORS headers to access resources across different domains. When these headers aren’t present, the browser blocks the request for security reasons.

A wingman for the API

A proxy server solves this by acting as a diplomatic embassy. Instead of trying to cross the border directly, you first go to your embassy (the proxy), which then handles the international communication on your behalf. In technical terms, thr proxy server makes the API request server-side, where CORS restrictions don’t apply, and then returns the data to the notebook.

To illustrate how this can be set up I use Glitch in the below example. Glitch is a web-based platform that allows you to create, host and modify (remix) existing web applications. It provides a collaborative development environment and free hosting for small projects.

Here’s how to set up a simply proxy at Glitch:

1. Remix this project: https://glitch.com/~hello-express.

   Hello-express is a minimalist node app based on express.js.

2. Implement the proxy functionality by inserting the following to the server logic in server.js:

   ::: {.cell}

   //| eval: false
   // define CORS proxy endpoint
   app.get("/proxy", async (req, res) => {
       const targetUrl = req.query.url;
       try {
           const response = await fetch(targetUrl);
           const data = await response.json();
           res.header("Access-Control-Allow-Origin", "*");
           res.json(data);
       } catch (error) {
           res.status(500).json({ error: "Failed to fetch data" });
       }
   });
   

   :::

   This achieves the following: When we make a request (req) to the app’s /proxy endpoint, it reads the target URL from the query parameters. The server then fetches data from that URL on behalf of the client, adds the CORS headers that allow browser access (as needed for embedding in an observable notebook), and returns the data in JSON format in the response res. If anything goes wrong, we return a 500 error.

   I’ve implemented steps 1 and 2 in my personal account, you can view (and use) the app ‘proxyme’ here.

3. In the observable notebook, use fetch to send requests through the proxy server, which acts as a middleman to fetch and relay the data safely.

Example: Fetching and plotting Fraunhofer ISE data

The next chunk shows how to fetch data from the Energy-Charts API of the Fraunhofer ISE.¹

I access data on average daily solar energy production for Germany (solar_share_daily_avg). The target url is https://api.energy-charts.info/solar_share_daily_avg?country=de. The API response schema is:

// response schema 
{
  "days": ["dd.mm.yyyy"],
  "data": [float],
  "deprecated": bool
}

Since the fetched data is in JSON format, it’s helpful to transform it into a structure that’s readily usable with D3 for plotting. Using .map, we can achieve this transformation in a chained operation.² The next chunk implements the following logic:

• Combine the proxy URL and target URL to form the complete request URL and fetch the data from the API.
• Handle any network errors that occur during the fetch operation, parse the response as JSON.
• Transform the fetched data into a format suitable for D3 plotting:
  • Convert the date strings to Date objects.
  • Pair each date with its corresponding value.
• Catch and log any errors that occur during the fetch or transformation process, returning an empty array in case of failure.

solarData = {
  const proxyUrl = "https://proxyme.glitch.me/proxy?url=";
  const targetUrl = "https://api.energy-charts.info/solar_share_daily_avg?country=de";
  const url = proxyUrl + encodeURIComponent(targetUrl);
  
  // fetch the data from the Fraunhofer API using a Glitch proxy
  return fetch(url, { headers: { accept: "application/json" } })
  
  .then(response => {
    // error handling
    if (!response.ok) throw new Error(`network error: ${response.statusText}`);
    // return as JSON
    return response.json();
  })
  .then(data => {
    // transform to d3 format
    return data.days.map((d, i) => ({
      date: new Date(d.split('.').reverse().join('-')),
      value: data.data[i]
    }));
  })
  .catch(error => {
    // logging
    console.error("Error fetching data:", error);
    // empty array in case of error
    return [];
  });
}

I’ve implemented all steps in this observable notebook, which also shows how to visualize the fetched data in an interactive D3 plot³ that

• allows user to compute and plot moving averages of the data.

• display detailed data point information on mouseover.

Here’s the result:

Credit: German solar energy production Share by Martin C. Arnold

… and thats it for now :-).

References

Quitzow, R. 2015. “Dynamics of a Policy-Driven Market: The Co-Evolution of Technological Innovation Systems for Solar Photovoltaics in China and Germany.” Environmental Innovation and Societal Transitions 17: 126–48. https://doi.org/10.1016/J.EIST.2014.12.002.