A container for all my views with excerpts from technology, travel, films, books, kannada, friends and other interests. I am Thejesh GN, friends call me Thej.
One of the primary jobs as a Data Archivist at DataMeet is to download and archive the data from the internet. Mostly from government websites. I usually use python scripts to download, scrape and clean the data. But sometimes, I just need to download many files and store them. I could still use python, but its an overkill. So here are some of the methods that I use.
I wanted to write 100 posts in 2021, and I am nowhere close to that. I tried to look at the posts by year and see how I have performed over the years. Of course, I could have done that manually by looking at the year archive count or running a query on the database. But recently, I have started using Xidel, so why not use it? :)
So It’s in my nature to convert everything to a web-service. The biggest reason is that you can call a web service from anywhere, and it’s easy to share the processing power and logic with anyone who has access to a web browser or curl and nothing else.
As you would know, I scrape a lot of web pages as a Data Archivist at DataMeet. I usually use BS4 for this, and it’s beautiful, simple, and works. But often don’t want to write a python script to do that, and I need a simple tool to get data out of HTML.
There are many ways to subscribe to this blog. Email is probably the best.
Thejesh GN (ತೇಜೇಶ್ ಜಿ.ಎನ್) "Thej" is an Independent Technologist, Hacker, Maker, Traveler, Blogger, InfoActivist, Open data and Open internet enthusiast from Bangalore, India. He loves experimenting with all things life and hence some times he is called hacker and other times duct tape. You can read more about him here. Email [ i @ thejeshgn dot com]. PGP/GPG Keys are here.
Fediverse or Mastodon: @[email protected]
(C) Thejesh GN.The views expressed on this site are mine (Thejesh GN) alone and do not necessarily reflect the views of my employer. Please read the terms and conditions before reading or commenting on this blog.
