Solving problems on the fly

The local government decided to switch technological providers for this and upcoming elections and as usual there is some adjustment time to get things working smoothly. 

At La Nación, we wanted to make a news app that would allow readers to perform real time analysis of the results of the elections showing as many details as possible, therefore we wanted to show the results by polling station.

Frontend isolation

Due to the mentioned technical change the local government was running late in providing newsrooms with details on how the results were going to be made available during the realtime vote counting process.

So we decided that we needed to blackbox the frontend needs and specify a data format that was required by the frontend leaving the transformation tasks to the backend and assuming that it was going to be doable after all.

That way we could move forward in defining the final visualizations and the different reader interaction capabilities and drilldown analysis.

After being really persuasive with government officials, we ended up having some documentation on the 21st of April… we were going to have an API to query but not many details yet.

We started defining the backend structure out of thin air, trying to be modular so that we could be efficient when changes were necessary.

Trying to divide the functionality in modules helps when you find an error, because you can easily go directly to the module that is causing the error and correct it. But on the other hand, it results in more difficulties in keeping the state of the overall process simple enough. We relied on User-Defined Exceptions to keep track of the overall process while executing code in a module deep down the execution stack.

We did not have data yet to test the process so we developed an ad hoc simulation script that would randomly produce voting results distributions. This process was lead by our Opennews Fellow from Spain Juan Elosua, and we invited his wife to the rescue… Marta helped us develop the results simulation and that was a key decision in the overall success of the project since it allowed us to start testing the backend well ahead of the API availability.

Finally we were given an endpoint to test the government API by friday 24th at 18h… fun weekend ahead.

As a real time application (running on EC2 machines) we knew that it was almost as important to have an easy way to keep track of what was going on in the backend on real time to be able to act quickly in case something unexpected happened.

We developed a logging strategy that was published each time the process was run (each minute), making it accessible through HTTP.

For monitoring purposes we used Postman, a chrome app that lets you define and save collections of HTTP requests:

• One collection for the government API endpoints

• One collection for the backend monitoring

Concurrency >100.000 simultaneous visitors

Usually on an election date, LA NACION receives a huge amount of traffic.So we had to consider together with the infrastructure and architecture team all that we needed to take into account in order to be successful: HTTP caches, Content Delivery Networks configuration and static files versioning…

To be able to handle that, we decided that we needed an “almost” automatic way of deployment that would decrease the probabilities of human errors while going through a somewhat stressful situation. We used gulp to automate the deployment process, letting it handle the minification, uglyfying and versioning for us. Extract of the gulp deployment process:

Realtime news app

At the end, on the election day, we were “anxious” to check if the kind of creative way of connecting the dots was going to work out… and it did!!

After sometime while the API was not responding we were able to get a full response to our backend and transform it to the frontend expected format… that was at 22:30 Sunday night… we even published the initial results before the official government page that was suffering from availability issues.

We have cleaned up after ourselves and have released the code of the project which Juan, our fellow, published on github here

Calm after the storm… Nope!!

After the adrenaline came down a bit we realized that had not completed our task since we wanted to let the readers go deep into their local details… we needed the results by polling station.

Luckily Manuel Aristarán (former Opennews Fellow in LA NACION) had worked on a similar project for the 2013 elections and we had worked in advance to obtain the geolocation of each polling station.

Thanks to that we could delivered a map that showed the results for each polling station the same day we got the official results from the City of Buenos Aires Government.

 

We used cartodb.js API require.js in order to keep the javascript development modular.

To avoid having too many HTTP requests we finally integrated the requirejs optimizer into the gulp deployment pipeline so that we have the benefit of modularity in development but with a nice performance on production.

Here is the open sourced code for that map.