Building Surveillance in Bangalore leaderboard using Node-Red and CouchDB

I wrote the Surveillance in Bangalore leaderboard using Budibase a while ago. However, I was not impressed with Budibase's latest changes, so I replaced it with the good old Node-Red flow. Node-Red is already running at my home, so installing it was not required. Since the data was always stored on my instance of CouchDB, I didn't have to do any migration there. I moved the automation to Node-Red And viewed it using HTML and CouchDB Views.

Design

The design remains the same; there is no change there.

Automation

Step 1: Pull data from OSM

I pull data from OSM/Overpass using the HTTP Request Node's GET (Overpass API) method and then transform the response using the Function node into a format I want. The transformation code also remained the same. I just had to read it from msg.payload and return msg.payoad that's all. 

nodes = JSON.parse(msg.payload).elements
const totals = {}
for (let node of nodes){
    user = node.user
    if (totals[user] == null){
        totals[user] = 1;
    }else{
        totals[user] = totals[user] + 1
    }
}
return { "payload" : { "_id":new Date().toISOString().split('T')[0],
 "type":"top_contributors", 
"content":Object.entries(totals).map(([user, count]) => ({user,count})).sort((a, b) => b.count - a.count)
 } };

Step 2: Add a record into CouchDB

CouchDB databases natively support REST. So, inserting is just an HTTP POST. I added some NTFY-based alerts based on the status of the response.

View

Get Latest

I wrote a simple CouchDB view that returns documents with the type "top_contributors" because we don't need to deal with meta or design documents. The database is set up for public reading. Writing or making changes needs Auth. So my HTML can directly make a call to this View order by key in descending way to get the latest and limit the records returned by it to 1.

https://data.thejeshgn.com/sib/_design/listing/_view/top_contributors?descending=true&limit=1

Display

I used DataTable to display. I have done experiments with DataTable and CouchDB before. I reused that piece of code. I created a simple HTML page so I could iframe it anywhere I wanted. Piece of code that creates the DataTable is below.

$(document).ready(function() {
$('#sib_contributors').dataTable( {
  "ajax": {
    "url": "https://data.thejeshgn.com/sib/_design/listing/_view/top_contributors?descending=true&limit=1",
    "dataSrc": function ( json ) {
      data = json.rows[0].value.content;
      console.log(data);
      return data;
    }
  },
   "columns":[
            {"data":"user"},
            {"data":"count"}
          ],
  order: [[1, 'desc']]
} );

Final Results and Conclusion

You can see the final results here and on the Surveillance in Bangalore project page. It's simple, but I like it.

I have realized more than once that the existing tools are better than the shiny new ones. Existing tools have a track record of working and working the way you want them to. New tools are great, but they need to be used in non-critical functions before production.