Today I Learned
Date : 20/April/2020
Archival, quality, deletion of unnecessary data are very important when you are processing lot of data. When you are writing and testing, it might perform well for the input provided, even for full data and also completes in few seconds or minutes, it might be ok at that time, eventually when quantity increases, performance degrades and those seconds & minutes become hours of unnecessary processing. Here, archival is about moving data that need not be processed anymore but required, so its better to move it to history tables, quality is about every field or column, being in up-to-date state as sometimes, the row is processed because of that column and after processing, its still not updated. Reasons could be, that row should have been archived, missed to update that field, updating wrong data to the field. Deletion of unnecessary data, really helps in tablespace scans in the worst case scenario, makes the table active rather than one table having history and active rows. Housekeeping of data is really a important thing and hate to say this, its always ongoing task.
Sometimes progress is not about refactoring the same code, so, its best to think about performance at the beginning, it might not be possible all the times, if you are working in a new technology or new domain. Below are some lessons learnt this week,
- Refreshing of whole table is a waste of time, if you are doing it everyday because and only few rows changed. So think about processing/updating specific rows. MQTs(pre calculated tables) are not the answers always
- Cleansing data is a everyday process, don't make a habit to do it every day, weekly or monthly is fine
- There are always best practices that are proven to work, read about them, test them.
Date : 17/April/2020
Learnt the below technique in python to show the line which caused the exception. By default, when you handle exception, it doesn't show the line number. Now, adding this to all my programs.
Date : 08/April/2020
Problem being a computer programmer is, your life is so perfect(you think), so you create a problem for yourself and work on solving that problem. It gives a good feeling when you solve it. But its entirely waste of time at some point you might wonder. Problem never existed.
Date : 07/April/2020
Jupyter Notebook - Previously when i try to write a program, i also create lots of small programs to test features, functionality, understanding concepts and finally implement them in the big program with Notebook everthing can be done in one big place and tests go really cluttered and its everywhere, eventually i spend lot of time in deleting cells which should have been in a separate program, as i wrote that for testing and understanding purposes.
Have to learn a approach to avoid this.