Quest for Open Data

Making Open Data Discoverable

by

Thejesh GN

Who am I?

I am Thejesh GN, Technologist from Bangalore.

Co-founder of DataMeet and Architect at Mavrix.

At some point I have worked for NextDrop, A Political Campaign, IWP, Infosys etc

  

The Problem

Type 1 - Demand

Shafeeq Rahman Thu, 11 Dec 2014 03:00:31 -0800

Dear All

Is there any link where the boundaries of AC and PC can be found for 1955 & 
1964 delimitation commission?

Regards,

Shafeeq
            
Pradeep Bhatt Mon, 24 Nov 2014 10:08:18 -0800

Hi All,

I am looking for address of district collector's office for all districts
in our country.

Which is the best place to look for this?

Regards,
Pradeep

            
Renaud Misslin Tue, 02 Dec 2014 05:50:38 -0800

Dear all,

I am currently trying to find a map of wards in Delhi that would match the 
census data of 2011. I tried to join ward wise population data from the 
census website with a shape file I found in this group (see this page : 
https://groups.google.com/forum/#!searchin/datameet/ward/datameet/1b_VhZ8leJE/dRhs5M3FprAJ).

            

The Problem

Type 2 - Supply

Anand Chitipothu Sat, 06 Dec 2014 10:16:28 -0800

Hi,

I've published a dataset containing all states, districts, assembly
constituencies and polling stations in India.

https://github.com/anandology/election-directory

Not sure if this is done before, but it has been a big pain to find this
info. Hope this will benefit some of you too.

Anand
http://anandology.com/

            
On Tuesday, September 23, 2014 1:23:32 PM UTC+5:30, Raphael Susewind wrote:

Hi Saurabh, 

you might have a look at my dataset - it is of varying quality (because 
raw data from the ECI was), but perhaps it does what you need: 

http://dx.doi.org/10.4119/unibi/2674065 

Best, 
Raphael 

            

Solution

Communication

Solution

  1. Allow creators to communicate to middleman
  2. Middleman will publish the metadata and provide search engine
  3. Users will use search engine

Simple

Not really

  • Format for communicating metadata
  • Way to receive the that programatically
  • Many things to Research
  • Many things to Build

Solution

Quest

Research

  • Put the idea in-front of the world - opendata.json – Format for making Open Data Discoverable
  • Similar projects by US Gov, OK etc
  • Adopt or start from the scratch - Clone or Fork FOSS way - Reuse - Discussions on Hackpad
  • Standard Vocabulary by GoI
  • Way to receive the that - Standard Ping (used by Google, Wordpress etc)
  • Build when required else reuse

Format

  • Forked - Project Open Data Metadata Schema v1.1
  • JSON, JSONLD
  • Well Researched, Public Domain, Already in use
  • http://quest.datameet.org/schema/v1.1/catalog.jsonld
  • http://quest.datameet.org/schema/v1.1/dataset.json
  • http://quest.datameet.org/schema/v1.1/distribution.json

Quest System

Creator

  • Purpose: Make it easy for the data publishers to create JSON files
  • Online form based system to create the JSON files
  • Offline - Excel based form to create the JSON files
  • Python and/or PHP libraries to include as part of your services

Validator

  • Purpose: Make it easy for the data publishers to validate JSON files
  • Online form based system to validate the JSON files
  • API End point to receive and validate the JSON
  • Python and/or PHP libraries to include as part of publisher's services

Ping Receiver

  • Purpose: Make it easy for the data publishers to send us updates
  • Online form based system to receive the ping
  • API End point to receive the ping
  • Python and/or PHP libraries to include as part of your services

Analyzer

  • Purpose: Internal, once you receive the file, analyze and insert into DB
  • Analyze the metadata
  • Check for spam and other rules
  • Insert into DB for later use and display
  • Build index for search

Publish/View

  • Purpose: Make it easy for the users to view catalog, dataset
  • Online, UI based
  • API

Search

  • Purpose: Make it easy for the users to find data set
  • Search based on Source org, License, Geography etc
  • Online, UI based
  • API

Community

  • Purpose: Enhance the quality, include community
  • Curation, Rate, Mark as spam etc
  • Online, UI based

Contribute

  • Writing code, GitHub
  • Creating example JSON - Email me
  • By being early publisher - Email me
  • Sharing - quest.datameet.org
  • Like any Data{Meet} project, its FOSS

Thank You

BY Thejesh GN / thejeshgn.com

Watch this movie when you find time. Thank you.