HOPCOMS API

Introduction

HOPCOMS (Horticultural Producers’ Co-operative Marketing and Processing Society Ltd) was established to help farmers to sell their products. They connect farming community and the consumers. They are based out of Banagalore. They have hundreds of stores in five districts of Karnataka - Bangalore Urban, Bangalore Rural, Kolar, Ramanagar and Chikkaballapura Districts.

Like many Bangaloreans I buy my fruits and vegetables from HOPCOMS. I love them. For sometime I have been thinking about how to make them more accessible and visible. One of the idea was to build some alerts. HOPCOMS as such doesn't have any APIs¹ on which I could build things. But they had a web page where they publish the daily rates. Well I took it upon myself to build an API so others can use it too. It also reduces burden on HOPCOMS servers. All in all serves HOPCOMS, Consumers and Farmers.

There are two ways to use the API. Use the code provided and run your own. Or email me and get access to my small API server.

API

Data is hosted on CouchDB. APIs are simple CouchDB restful APIs. For the external world it's mostly GET access only. All responses are in JSON format.

Do remember the returned Content-Type is text/plain even though the information returned by the request is in JSON format. To get the content-type application/json set need to explicitly set your request Accept header as Accept: application/json .

Assuming CouchDB runs at https://mycouchdb.ext there are two databases available.

hopcoms_daily

This stores the daily rate data. The primary key _id is date in the YYYYMMDD format. For example Jan 16, 2018 will be 20180116. It's accessible at https://mycouchdb.ext/hopcoms_daily/20180116 with a simple GET. The response data is JSON and is simple key:value pair of item_code and rate.

{"_id":"20180116","_rev":"1-6177e5919a7f6a22dae47b44e4ff5a18","1":110.0,"4":198.0,"7":189.0,"9":240.0,"11":29.0,"12":62.0,"13":80.0,"14":54.0,"19":58.0,
"20":75.0,"21":86.0,"22":40.0,"27":100.0,"28":118.0,"29":142.0,"31":75.0,"32":425.0,"36":148.0,"37":32.0,"39":77.0,"40":76.0,"42":34.0,"43":48.0,
"44":100.0,"46":80.0,"48":60.0,"49":148.0,"51":53.0,"52":57.0,"54":96.0,"55":68.0,"56":65.0,"57":95.0,"58":70.0,"59":160.0,"63":120.0,"66":21.0,
"67":27.0,"69":39.0,"70":42.0,"78":20.0,"79":20.0,"81":80.0,"84":52.0,"87":88.0,"88":250.0,"101":60.0,"102":19.0,"108":42.0,"109":26.0,"110":34.0,
"112":24.0,"113":42.0,"114":40.0,"115":22.0,"116":16.0,"117":17.0,"121":12.0,"122":42.0,"123":37.0,"125":82.0,"126":50.0,"127":30.0,"131":60.0,
"132":45.0,"135":37.0,"136":36.0,"137":28.0,"139":14.0,"140":80.0,"142":26.0,"145":34.0,"147":148.0,"149":70.0,"150":80.0,"153":24.0,"155":54.0,
"156":29.0,"157":30.0,"158":60.0,"160":200.0,"162":188.0,"163":56.0,"167":140.0,"168":80.0,"171":13.0,"173":16.0,"174":23.0,"179":18.0,"180":80.0,
"181":26.0,"182":28.0,"183":18.0,"184":74.0,"185":56.0,"186":59.0,"187":46.0,"190":10.0,"191":40.0,"193":65.0,"196":50.0,"201":25.0,"202":66.0,
"203":38.0,"204":58.0,"205":40.0,"206":40.0,"208":34.0,"213":38.0,"215":56.0,"218":16.0,"220":4.6,"221":33.0,"226":32.0,"227":145.0,"250":45.0,"251":25.0}

You could do other CouchDB key based queries to get data worth of a month or year or between specific dates etc. See queries section for some examples.

hopcoms_meta

Item Details

hopcoms_meta table stores other meta data. To get the item details. You can issue you a GET request like https://mycouchdb.ext/hopcoms_meta/item_details. It returns a simples JSON with item_code as the key and names in different languages. A partial response looks like this

For the rate list published before 2019 Sept 23, use item_details.

{
  "_id": "item_details",
  "_rev": "3-b3808e206cd2263b5ae67e7fd543f245",
  "1": {
    "name_en": "Apple Delicious",
    "name_kn": "ಸೇಬು ಡಲೀಷಿಯಸ್"
  },
  "2": {
    "name_en": "Apple Simla",
    "name_kn": "ಸೇಬು ಶಿಮ್ಲ"
  },
...
...
...

}

For the rate list published after 2019 Sept 23, use item_details2. Read this blog post for details of this change.

{
  "_id": "item_details2",
  "_rev": "1-7a682471c33b9e0a449dd09babb827b4",
  "1": {
    "name_en": "Apple Delicious",
    "name_kn": "ಸೇಬು ಡಲೀಷಿಯಸ್"
  },
  "2": {
    "name_en": "Apple Simla",
    "name_kn": "ಸೇಬು ಶಿಮ್ಲ"
  },
...
...
...

}

So using both you can figure that on 20180116 rate of Apple Delicious was INR 110. It's that easy. Its better to cache the item_details in your application environment.

last_modified_timestamp

hopcoms_meta also has other details like when was the DB last updated available at https://mycouchdb.ext/hopcoms_meta/last_modified_timestamp. Response looks like

{"_id":"last_modified_timestamp","_rev":"1-b9e51ee6a16a0bc711b2d7d6cf4f52bc","timestamp":"2017-11-30T11:42:00.73+05:30"}

It's the last time daily rate was updated. So gives you an idea how update the data is.

api_details

API details are available at https://mycouchdb.ext/hopcoms_meta/api. Response looks like

{"_id":"api_details","_rev":"1-b9e51ee6a16a0bc711b2d7d6cf4f52bc","version":"1", "date":"20180313", "author":"Thejesh GN", "author_url:"https://thejeshgn.com", "documentation":"https://thejeshgn.com/projects/hopcoms-api/"}

Hosted version of API

I do host the version of API where I run the server and maintain it. As of now its running on a small server and hence not public. Email me, I will send the access details.

Self host the API

You can host it yourself. All you need is CouchDB, ability to run python programs and CRON for scheduling. I am going to explain the broad steps.

1. Install CouchDB, make it web accessible or may be use hosted CouchDB from IBM Cloud. Whatever works for you. This is not the best place for it.
2. Create hopcoms_daily and hopcoms_meta databases. Setup auth etc.
3. Install python, virtualenvwrapper, git
4. Clone https://github.com/thejeshgn/hopcoms
5. Go to hopcoms folder and create a virtual env
6. pip install all packages in requirements.txt
7. Edit load.py and update the db_full_url with your CouchDB url
8. Run python script load.py to load historical data and meta data
9. Then setup a cron to run daily.sh to run everyday post 11AM IST. This is the one which keeps the data updated on daily basis
10. Setup your firewall, CouchDB database security etc. This is not the best place to say everything about it
11. You should have hopcoms_daily and hopcoms_meta with data available now

Queries

Get all for year 2016

https://mycouchdb.ext/hopcoms_daily/_all_docs?include_docs=false&startkey=%2220160101%22&endkey=%2220161231%22

By date range

https://mycouchdb.ext/hopcoms_daily/_all_docs?include_docs=false&startkey=%2220160101%22&endkey=%2220161231%22

By months

https://mycouchdb.ext/hopcoms_daily/_all_docs?include_docs=false&startkey=%22201601%22&endkey=%22201602%22

By year

https://mycouchdb.ext/hopcoms_daily/_all_docs?include_docs=false&startkey=%222016%22&endkey=%222017%22

Specific day

https://mycouchdb.ext/hopcoms_daily/20160101

Item details

https://mycouchdb.ext/hopcoms_meta/item_details

Updates

You can keep track of what I am working on here. I have TODO list.

TODO

• Add meta data table
• Add store list
• Add store lat/long locations in the format of GEOJSON

This API documentation was last updated on Oct 4, 2019 @ 12:07 PM.

Contact

Contact me using the email or contact form for any feedback or comments.

Footnotes