LA NACION´s Shopping Cart. Monitoring 2600 supermarkets and 170 product prices

LA NACION´s SHOPPING CART: monitoring 2600 supermarkets in 170 product prices

CONTEXT

In March 2018, Argentina was one of the five countries with highest inflation according to the International Monetary Fund. The index climbed up to 47.6% anually according to official numbers. The jump of the exchange rate, higher than 100% also had an impact on the rise of prices on the supermarkets every week.

Given the country’s inflationary context, there are several sources of information about price variation, but these end-user reports are abstract numbers that do not reflect their daily reality and do not represent the real impact on their economy.

To understand how supermarket product prices evolve independently of the official measurements monthly informed, and to bring this measurement closer to the daily economy of the audience, we created “Changuito LA NACION“  (LA NACION Shopping Cart), a price monitor that allows a weekly and monthly monitoring of goods product by product such as noodles, soft drinks and liquors and toothpaste, among others. Its name refers to the colloquial way of naming the shopping cart.

The premise is to make price variation -an abstract subject that impacts on every household in an inflationary context- more contained in terms of everyday aspects and give control to end users . According to the study made in March of this year by D’Alessio IROL/Berensztein, inflation is the first concern of Argentine citizens, followed by uncertainty about economic situation and insecurity.

The audience of Changuito LA NACION is potentially massive: as price variation is an issue for every Argentine citizen, any person with devices may access the app for which links are often provided from our home page.

El Changuito presents a group of 170 food, beverage, cleaning and toiletry products from leading brands that are marketed nationally in 2,561 points of sale and are representative of household consumption in Argentina. Each one of the goods has a “premium” option and a “standard” option, differentiated by price.

To make interaction easier, we created four pre-set baskets, based on our official statistics methodology. They are: for people living alone, for couples, for a four-member family, and for retired persons. Thus, if the user selects this option, he/she can access a pre-set basket of food, toiletries and cleaning items that are similar to the consumption of the type of household chosen. In addition, you may modify: add or remove products and decrease and increase quantities. Each of them has its nominal price (total price) and its variation (on a weekly and on monthly basis).

The user also has the option of assembling a “changuito a medida” (taylor-made shopping cart), that is to say, to make a virtual tour among the monitored products to personalize his/her own basket. To enable an individual tracking, each reader registered on the Web or through the mobile app of LA NACION may save his/her own selection.

In addition, by clicking on each product, the user may see its current price and the weekly and monthly price variation with a linear graph.

 

INNOVATION

This is the first time in Argentina that a platform has been developed by a data journalism and the business editors team,  and it allows the user to compare price evolution every week in consumer centers in the most important areas of the country. This is particularly useful because the country is exposed to constant inflationary movements throughout its history, something that has worsened in the last six years. This is rare and valuable in a country where monthly price variations reached 6.5% in 2018.

It is different from any type of application that analyzes price variations because it has a personalized approach, because it invites dialogue with the user and because it “brings back” to reality and individualizes a product (initially, an abstract concept), but that affects the pocket of all Argentine citizens.

It is also innovative from its genesis. To begin with, we select key products to be included in an average consumer basket of supermarkets. We made a different selection and we were helped by experts, but also by potential users of the app: we consulted both consumer specialists and consumers.

In addition, we use an official database, “Precios Claros” (clear prices) that publishes daily thousands of prices in thousands of supermarkets in Argentina, but doesn´t open this historical prices in an API or replied with this data to our two FOI requests in order to obtain the complete dataset. So we decided to add value. Every day this database presents a picture of the current status of prices in different points of sale, but you can´t go back to the price of the day before, it´s lost. Instead of this still picture, we make a film: we collect data once a week and build a dataset, so we may analyze its evolution, we add frequency and systematization with data analysis over time as well.

In the process, for each of the functionalities we created (such as the generation of individual baskets or the possibility of ordering products according to higher or lower price) we held meetings to determine the user experience. To do this, we consulted different profiles, from consumer specialists, designers, economic journalists and programmers, among others.

IMPACT

The average time of the app is 4.51 minutes. The Changuito LA NACION was presented several times on TV, on news sections of LN+ and Lanacion.com. Since it is an atemporal platform, always useful , it may apply to any event that may alter prices or that may focus on them, such as strong devaluation or a monthly report on inflation made by the Instituto Nacional de Estadísticas y Censos (National Statistics and Census Institute – Indec).

Besides, its use is also highlighted on special occasions, such as Christmas Season. In December last year, we set a special Christmas basket of products that could be weekly checked to see when it was convenient to purchase each product.

SOURCE AND METHODOLOGY

The Changuito LA NACIÓN consists of a following-up of a group of 170 products (food, soft drinks and liquors, cleaning and toiletries items of leader brands locally commercialized in 2.561 points of sale.

The project began in the middle of 2018. After several requests for access to information to obtain a complete historical database we decided to scrap the products by ourselves from the official Precios Claros website. To do so, we left activated a software to be executed twice a week and record the information in a database.

To create the basket, consumer specialists were consulted and the final selection of the products was made. In all cases, prices are collected by LN Data from the information available in Precios Claros, the official system that is assembled from the data provided daily by stores and supermarkets to the Dirección Nacional de Defensa del Consumidor (National Board of Consumer Protection).

From the moment we began to save the data in the database, we checked the official website to verify and control the information. In addition, as we found inconsistencies in the prices published on the site, we decided to discard the outliers.

The survey excluded items on offer, specific promotions and products that make up the official program of Precios Cuidados (through which fixed prices are agreed with the Government for more than 500 products) to avoid distortions. Meat, fruit and vegetables were not taken into account either, as they tend to lose track and distort the sample.

The price of each product arises from averaging its price in the 2.561 points of sale and its sum gives the total price of the basket. LA NACION has been following this basket since August 3, on a weekly basis, and will continue to publish information with the same periodicity.

Another feature is that you can select from pre-determined baskets according to each user’s profile, or else, create individualized baskets. The basket may be saved for tracking over time. It is also possible to choose between premium or standard products, add or remove products, and, in each case, it graphs the variation of each product and that of the entire basket. 

Also, the user may order the basket by different attributes: higher or lower price, higher or lower price increase, premium or standard products. Another useful function is to be able to see the products with their photos or in a list.

For each of the functionalities we held different meetings to determine the user experience and to do so we consulted different profiles, from consumer specialists, designers, economic journalists and developers among others.

The opdating process is fully automatic, since data is automatically collected twice a week, saved in a database, and then an update impacts on the front end .

On the other hand, if it is necessary to add or remove products for users to see, we have an admin and back-end where the same journalists may modify the arrangement of a basket or create new ones to follow some other project, such as a school basket when school time begins, a Christmas basket or a set of products related to holidays.

Beside these special baskets we arranged a specific basket that we called “Canasta LN” (LN Basket), that we use for reporting in which we included a list of 43 products. Thus, every week another script runs automatically, and creates an article based on a template together with our visualization and it determines the products that increased the most or were reduced the most. In addition, it embeds a visualization on Tableau that is automatically updated for each publication.

 

TECHNOLOGIES

On its website “www.preciosclaros.gob.ar” the National Government publishes a price list of more than 70,000 products throughout the country. This list may be consulted to find the best price of a product in supermarkets and stores nearby, as it uses geolocation. It may be accessed through the web portal or through an API.

The government application offers for each product several descriptive data such as name, packaging presentation, minimum and maximum price that may be found for that product and branches of supermarkets where the user may get them with their corresponding list price. All retailers are required to report the price list on a daily basis so the site is updated every day.

Taking into account the inflationary problem of Argentina, LA NACION used these data to calculate the price variation for each family type. The initial question for this project -and its answer- is: “What was the price variation for me (or my family) this month?”

For doing so, first of all, we developed a scraper for some products on the website Precios Claros since 2018 to build our dataset. This database contains prices in supermarkets and stores in each city of Argentina. It was also necessary to set up common products between cities, since not all brands and packaging presentations cover all the territory. For this reason, there was a selection of 170 products that were representative and that may be found in almost all the stores.

This scraper updates data every week and the user may thus obtain the price variation of each product. First we developed one for the “LN Canasta” in SQL so we could prove the concept, Then for “LN Chanquito” we needed to automatize it and connect it with boths outputs,  so we developed another with Python and hosted in a Lambda with CloudWatch and PostgreSql , using in RDS for the database.

An ADMIN developed with Django/Python is used as a back-end to be able to control that all processes work correctly. Besides, journalists involved in the project may make decisions such as which products are included in each basket and which items need to be added, for example, according to the seasons of the year.

From the back-end, after updating the data weekly, a json file is created and hosted in S3 with the prices of the last four weeks for each product in each city. At the moment, the data of the City of Buenos Aires and the Greater Buenos Aires are being used, but in the future more cities will be added.

For the front-end an application developed with VUE.js, D3.js, CSS3 HTML5 was programmed. This one uses the json files published by the back-end in S3. Each user has different alternatives to know what is the price variation of the products he/she consumes through sets of goods that we call baskets.

A “basket” is the group of products that make up the monthly consumption of a person or family, as appropriate. There are five basket options: “family” (couple plus two children), “couple” (two adults), “retired persons” (elderly couple), “single” (adult person) and “taylor-made” (possibility of choosing which products make up the user’s monthly basket).

When selecting a basket or choosing the products that the user consumes, the user may see on the front-end the variation of their prices based on the selection made.

The application is connected to the login service of the LA NACION website and the user has the possibility of saving his selected basket for future visits. If the user is not registered, he is invited to register in order to save his basket.

Sin comentarios

FIFA World Cup Russia 2018

On the occasion of the 2018 FIFA World Cup in Russia, a complete coverage was planned. The production had to demonstrate the highest standards of journalism in an interactive and digital design. The objective was to inform and engage the audience in  two different stages:

The first stage consisted of a series of playful and social visualizations prior to the beginning of the World Cup. The articles were strategically conceived according to the schedule prior to the event: selection of the 23 players that were going to travel to Russia, the 11 players that were going to play on the first match,  a draw simulator of the different possible groups in which each country soccer team could be selected and their possible opposing team, and a result forecast for the people to play with and bet.

Instead, the second stage was focused on the real time results and matches: a fixture with the times, teams and matches, a calculator for the audience to estimate results and evaluate the possibility of passing to the next round, a game that allows the audience to score the players after each match, a live statistics visualisations of the amount of shoots and passes between players and their positions on the field throughout the game, the possibility to vote for the best goal of the world cup and a monitor of the speed and the distance runned by each player.

Russia 2018

Visualizations published before the Soccer World Cup

 - DRAW SIMULATOR

Days before the event and with the purpose of informing the methodology of the draw, an interactive was made that simulated the possible combinations that could result from the draw and thus allow the user to speculate with which teams could face the Argentine football team.

 

- CHOOSE YOUR 23 SOCCER PLAYERS

The day the coach of the Argentine football team informed the previous list of 35 players of which only 23 would be included in the final list, an interactive development allowed the reader to choose his/her 23 favorite players for the football team. With the data provided by the users, a ranking was formed in real time with the most and least chosen players. During the day the app was active, it obtained 113,340 unique users and an estimated time of 4.22 minutes.

 

- CHOOSE YOUR 11 FOOTBALL PLAYERS

After informing the list of 23 football players confirmed to play in the World Cup, users were asked to put together a team of 11 players to play the matches. With the data provided by users, a ranking was made in real time with the most and least chosen players. It was an interactive that also was monetized through a sponsor.

 

- ALL THE WORLD CUP FOOTBALL PLAYERS

Once the players of all the football teams that would play the World Cup were confirmed, a database was created with the nationality and the football club where they played. Through an interactive, user could explore data and get different views on the players on the list.

 

- FOOTBALL WORLD CUP DRAW

Three weeks before the beginning of the event, this app was released and the users could predict the results of all the football matches. With the data provided by the users and by means of a score system based on the hits obtained, a list of positions was created that, at the end of the World Cup, informed a winner.

In addition, the app offered the possibility of creating groups for users to set up tournaments with their friends. More than 30,000 forecasts were created, and the app had a high level of participation during the 45 days that the app was active. Total users 1,028,134. The average time was 08:33. It was an interactive that also was monetized through a sponsor.

 

Visualizations published during the Football World Cup

 FIXTURE

During the World Cup this app was released with the schedule, groups and real-time results of all World Cup matches. In addition, each group had a calculator to speculate the results of the matches and evaluate the chances of classification of the teams according to the speculation made. It was the most successful piece of all, with a user record of 3,376,065 and an average time of almost 10 minutes. It was a development that was also monetized through a sponsor. For the real time it was used the data service of the OPTA company which was processed and enriched to adjust to the needs of the proposed development.

 

GIVE A SCORE TO PLAYERS

An interactive that allowed users to give a score to players after Argentina’s matches during the World Cup in Russia. The data provided by the users allowed to give each player an average score. This resulted in a platform with a reusable collective score for each match of the Argentine football team.

 

INTERACTIVE KEY

To get excited about the elimination phase of the World Cup, this app was released so the user could speculate on the results of the groups and create the tournament key. In this way, the user was given the possibility of determining which selection the user thought would be the winner. For real time the data service of the OPTA company was used, which was processed and enriched to adjust to the needs of the proposed development.

 

LIVE STATISTICS

For the coverage of all the soccer matches of the World Cup, a series of visualizations were developed in real time with the statistics and incidences of each match. For real time, the data service of the OPTA company was used, from which the raw data were taken to obtain original and exclusive visualizations of LA NACIÓN: shots to goal, passes between players and movements on the field .

 

FIRST ROUND BALANCE

Based on the data provided by OPTA, a database was created to develop this visualization, which was released just after the first round of the World Cup, in which the performance of the Argentine soccer team players was evaluated: shots to goal, distance covered and speed for each match of the first phase.

 

PENALTIES IN KNOCKOUT ROUND

FIFA’s official data was analyzed for all the matches that were defined by penalties in the knockout round in the history of the World Cups. The result of the analysis was the conclusion that the teams that had most defined their matches by penalties were Argentina and France, both rivals in the following match of the World Cup. With the data, a visualization was developed that analyzed in detail these two teams in their definitions by penalties.

Sin comentarios

Argentina’s Senate rejected the bill to legalize abortion

ONE BY ONE, HOW DID EACH SENATOR VOTE FOR THE BILL ON VOLUNTARY INTERRUPTION OF PREGNANCY

ABORTION: THE VOTES BETWEEN BOTH HOUSE OF REPRESENTATIVES AND THE SENATE, ACCORDING TO PROFESSION, AGE AND GENDER

CONTEXT

On August 8, the Senate voted the Bill on voluntary interruption of pregnancy after a preliminary approval at the House of Representatives a few weeks before. The bill established the possibility of having an abortion during the first 14 weeks of pregnancy at public hospitals. The bill was not approved that day and Argentine legislation only allows abortion in case of rape or in a life risk situation of the mother.

The day the abortion bill was defeated in the Senate

That bill divided Argentine society and the debate moves to the media, politics and society and it involves deeper feelings of each individual, since it questions the origins of human life, the position of a woman and her capacity to decide on her own body, among other subjects. There appeared “green scarfs” as a symbol of the fight of those women who were in favor of the decriminalization of abortion and, as opposed, there also appeared “light blue scarfs” showing another position against this legislative proposal and in favor of saving both lives (mother and child).

The day the abortion bill was defeated in the Senate

Women of all ages stayed long hours at the surroundings of the Congress waiting for a resolution

As days went by, the city of Buenos Aires was painted in both colors and, on the day the bill was voted in detail at the Senate, women of all ages stayed long hours at the surroundings of the Congress waiting for the legislative session. It was a cold day of August and the session lasted ten long hours. It rained, but nothing stopped those persons. The mere fact of being debating a taboo subject like this one was a historical milestone in our country. One could be in favor or against it, but it was impossible to be on the sidelines.

The whole country observed and listened to each Senator as he/she presented the reasons for his/her vote. The country was divided: half was green, and the other half was light blue. That day we saw how audiences grew in their interest on this topic and we decided to react and present on real time two projects to the audience: the online following up of the debate and a comparison between the vote at the House of Representatives and the Senate.

The day the abortion bill was defeated in the Senate

Argentina’s Senate rejected the bill to legalize abortion 

INNOVATION

Citizens followed the presentations of Senators as closely as we were watching a suspense movie, to be on the spot was very important. That is why, La Nación reused code to build a visualization that could follow that rhythm; it was important to know the final result of the vote and the vote of each one of the Senators on real time as well. The first interactive shows the faces of those persons representing us at the Senate, some of them are unknown for the audience in general. Moreover, the interactive adds the possibility of crossing variables to understand whether his/her age, religious belief, politic orientation or university background were related to their vote.

ONE BY ONE, HOW DID EACH SENATOR VOTE FOR DE BILL ON VOLUNTARY INTERRUPTION OF PREGNANCY

The second interactive, published early in the morning the day after, followed the same logic as the first one, but this time it analyzed and compared the votes between both House of Representatives and the Senate (257 Representatives/72 Senators), trying to find a pattern that answered the questions that many of us asked ourselves: did senior legislators vote against the bill and the younger ones, in favor?; do the Catholic ones support the 2 lives? Are lawyers against it? These and many more questions find their answers in both visualizations.

ABORTION: THE VOTES BETWEEN BOTH HOUSE OF REPRESENTATIVES AND THE SENATE, ACCORDING TO PROFESSION, AGE AND GENDER

 

IMPACT

The project concerning how each Senator voted based on each Senator’s different demographic indicators became the most read article of the day. In social media, gender-filtered visualization was shared because it was clear that 60% of all male Senators voted against abortion, while the percentage among women is 50%. It was remarkable for the female audience since it was a fundamental public policy for women and they claimed that it had not been approved because it had been left in the hands of the men in the House (who are in the majority).

At the same time, the filter of age and political orientation was also shared repetitively. The only women under 40 years old voted in favor, while among the rest of the senior Senators, there was much disparity in the vote. And as regards the orientation and political parties, it was very clear that the pro-government party has the majority of its Senators against the bill.

The day the abortion bill was defeated in the Senate

Moreover, the visualization regarding the vote of the Senators compared to that of the members of the House of Representatives allowed to give a wider visibility to the subject, since it included the whole Congress as well as the possibility that the user understands in a visual way how each legislator voted, who were absent and whether the socio-demographic factors had any impact on each chamber.

SOURCE AND METHODOLOGY

Google Spreadsheet and Google Spreadsheet API was used to upload each one of the votes in real time. The session lasted until 3 o’clock in the morning and data were completed as each speaker gave his/her opinion and we verified it with the minutes published by each House.

As regards socio-demographic data: age, gender, political orientation, province, university degree and termination of office, a database made for the opening day of sessions at the Congress on March 1st was reused.

It was a comprehensive survey, a data journalist telephoned the 257 members of the House of Representatives and 72 Senators, a total of 329 individuals. This was manually performed for two months, since many had been elected a few months ago.

TECHNOLOGIES

The first interactive was made using HTML5 and CSS3. It was also used a JavaScript library named VUE which enables a faster uploading and its fragmentation. Thus, during the vote at the Senate, part of the interactive was highlighted on the home of La Nación website (bars and counting), and the complete interactive (filters and faces) could be seen inside the article. The second interactive was made on Tableau and both visualizations used the Google datasheet. One person uploaded the votes on real time and both visualizations were automatically updated (Google Spreadsheet and API).

The day the abortion bill was defeated in the Senate

 

Sin comentarios

At what price?, behind the scenes of the investigation

 AT WHAT PRICE? : A PUBLIC PROCUREMENT OBSERVATORY 

In this post we propose to gather the step by step of At what price?, a public procurement observatory created by LA NACION Data in alliance with CHEQUEADO and ACIJ, which compiles the purchases of HIV drugs, contraceptives and vaccines made by the Health Office of the argentine Government and the city of Buenos Aires.

In this first approach we focused the work on the purchases of 2016 and 2017.

Seguir leyendo

Sin comentarios