SQLite – Open Format for Open Data
I scraped Gujarat rainfall data recently. The end result was 6 lakh rows of structured data. I released the data as an SQLite file instead of say a bunch of CSV or excel files. It kind of surprised some of my friends. But I think it’s a great file format for releasing open data.
- Database contained in a single file
- Easy to send, store and share. It’s one file after all
- Like any file, you can compress and encrypt it using standard ways
- Control read/write using file read/write permissions
- Size of the database depends on the size of the file which is virtually unlimited by hardware
- Simple UI-based software available to explore. Just the way you open .xls files with Excel, you can open .sqlite files with softwares like DB Browser for SQLite. It works on Windows, Mac and Linux.
- Supports standard SQL
- Supports standard SQL data formats
- Supports export and import from other formats (like CSV, TSV)
- Standard libraries available for all programming languages
- Transparent encryption support through plugins
- GIS support though plugins
- Above all it’s a public domain software; you can do whatever you want to do with it. It’s going to stay here forever
- SQLite database file format is open
- Above all SQLite just works
So from here on my default file format for Open Data is SQLite.