One incident which I can recall is when I was working as a Senior Technical Lead for a commercial bank CBS project. The bank had gone live about one month ago, and I was providing the post production support. There had been a problem reported with the migrated Term Deposits on first day after going live. Once this was found out, the bank was provided scripts to correct the issue. My team gave the bank the correction scripts to be executed on particular dates. But on 24th January, the bank somehow missed the execution of the script in the live environment.
When I found this out, realizing the implications of the mistake, I immediately called up my team and explained the problem, because the script that was provided was not designed to work the next day, due to date change. I proposed two solutions and asked them to start working on it immediately. The possible solutions were:
1) Identify the affects and provide a list to the bank, which will then take according manual action in live environment. However, identifying the effects across the system was a time consuming process. But this was a safer way in terms of not affecting the transactions in progress during the day.
2) Create a new script to take care of the issue. Even though creating such a script was complex, it was a faster way of fixing the issue, but had the risk of affecting the transaction processing during the day.
I informed the bank about their mistake, and its consequences. Knowing the urgency, the bank asked me to come into a meeting where they would decide which method to go with. By the time the meeting began, we were ready with the script for the second approach. In the meeting I proposed both the solutions. The bank decided to go with the first approach, even though it was time consuming, as they did not want to take the risk of running a new script (even though I assured that the script would correct the issue and was a faster method).
By the time the first approach was cleared on the test environment, it was 7 PM. Due to the delay, the EOD support representative from the bank suggested postponing the remaining activities on live environment to the next day, but knowing the issue will cause further problems on next day, I insisted that the remaining activities be completed on same day. I provided them support and by the time everything was completed, it was 11.30 PM. I made a report and sent the same for verification, and left at 12 AM, tired, but relieved. The next day the bank’s business leader for deposits came and congratulated me for taking the correct decision of not postponing the activity by another day and avoiding a lot of trouble as well as further effort for the bank. Since I did this in parallel to my regular duties of handling the issues reported during the UAT for the next project, I was appreciated by my peers as well.
English
日本語
Русский