Python – dcScrape v1.3 project back in progress

I took a few months break from coding lots of stuff and brushed up on my Python again off and on the past week, a tiny bit of rust but Python retention is very high. Started working on this again and went back to v1.2 to fix a few bugs before redoing it entirely. v1.3 will use OOP classes/objects design and be more flexible with inputs allowing multiple searches to occur. I have had a few people email me in the past few months wanting to know if any updates on this ‘universal’ version, it’s happening.

Besides the OOP new design an outputted database will be created as an option allowing you to scrape and see whats changed with some date/time and other info. The database file will be created and named by whatever url you are scraping (it’s an idea at least) with the arguments you used for your previous searches. This database part might come after the OOP refactor maybe in v1.4 if not in 1.3.

I was thinking of adding the ability to ‘search again’ with history of previous searches ability too once the database feature is enabled. This would scan the database for all the search arguments and allow you to simply redo the search, quicker than a new search. I have done this many times with other scripts where a numbered list will be outputted and you simply input a number to redo/update that search.

I am wanting a GUI down the road and looking into some Python modules for that.

Leave a Comment