Don’t panic – this is not another blog about GenAI. This one’s about client data disasters!
If you think your data is bad, trust me – you’re not alone.
Over the course of a 20-year career working with financial data, you get to see a lot. From pristine, best-practice, cleaned, tagged, and structured data whizzing via APIs across the globe… …to a hard drive storing data worth £250,000 being used as a coffee cup holder!
As part of understanding a client’s data engineering and AI needs, I ask a lot of questions. They’re pretty straightforward – designed to narrow in on where tech can help.
They’re also designed to rule out where tech is* not needed. More than half the time, common sense beats software.
Typical questions include:
- “Have you backed up your data recently?”
- “Do you have failover if your servers go down?”
- “Is there more than one person in your firm with access to the AWS keys?”
Our job is to shine a light on bottlenecks and weak links – then suggest the* right *tech (or sometimes, just the right habit change).
Below are some real anonymised client exchanges I’ve had over the years…
☔️ Client A: Flooding
Insig: “What would you say the biggest risk to your data is?”
Client A: “Flooding.”
Insig: “Flooding?”
Client A: “Yeah. We store all our data in hard drives in that server room… which is unfortunately also where we keep the boiler. It’s leaking.”
Insig: “How bad would it be if it actually flooded?”
Client A: “We‘d have to close the company.”
Solution: Migrated all server data to a cloud-based infrastructure, backed up in two separate locations. The server room is now used to store staff bicycles.
🧮 Client B: Fat Finger
Client B: “We’ve lost £150,000,000.”
Insig: “Did the markets take a turn? Were you not hedged?”
Client B: “No – the PnL Excel file is missing £150m. I don’t know where it’s gone. This could trigger redemptions.”
Insig: “Has anything changed in the file recently?”
Client B: “Yes, I cleaned up some old formulas that were slowing it down.”
Insig: “Have you tried CTRL + Z?”
Client B: “You are a genius! The £150m is back.”
Solution: *Shifted all source data into a database, replicated the PnL logic in Python. Also helped the client book a long-overdue holiday.
💾 Client C: “Our data is gone… all of it.”
Client C: “Do you by any chance have a backup of our data?”
Insig: “Which data?”
Client C: “All of it.”
Insig: “What happened?”
Client C: “We switched providers. The job failed. Everything’s gone.”
Insig: “What does that mean for your fund?”
Client C: “We have to shut down—immediately.”
Insig: “We took a snapshot of your data during our last integrity check. It’s 48 hours old. Will that do?”
Client C: “I could marry you right now.”
Solution: *Migrated everything to a secure, cloud-based setup with redundant backups. Crisis averted.
📊 Client D: Running Out of Rows*
Insig: “What’s your biggest concern in your role?”
Client D: “That my spreadsheet is running out of rows to store our track record.”
Insig: “Excel supports up to a million rows. How many have you used?”
Client D: “997,132.”
Solution: Migrated all Excel history into a database. Centralised the data feeds. Automated the calculations using Python.
☁️ Client E: “We already use the cloud.”
Insig: “Have you considered moving to the cloud?”
Client E: “We already use the cloud extensively.”
Insig: “Great – AWS, Azure, AliCloud?”
Client E: “No, Google.”
Insig: “Ah – Google Cloud Platform?”
Client E: *”Um… no. Google Drive. We store all our data there.”
Solution: Migrated all files into a secure cloud based infrastructure.
📋 Client F: CTRL C + CTRL V
Insig: “Where does your team spend the most time in the trade process? Research? Backtesting?”
Client F: “Copying and pasting trade details.”
Insig: “Copy/paste? Doesn’t your PMS handle that?”
Client F: “It handles 80% of securities. But not real estate, aircraft, or private CLOs. We export that data, then copy/paste into the aggregate Excel file.”
Insig: “How long does that take?”
Client F: “About one week a month.”
Insig: “For all four of your analysts?”
Client F: “Yes.”
Insig: “So you lose one person-year to copy/paste.”
Client F: “I never thought of it like that—but yes.”
Solution: *Automated aggregation via central database + Python logic. Team now spends time thinking, not copying.
🧹 Final Thoughts
These are just a few of the data disasters I’ve helped untangle. Sometimes it takes tech, often, it just takes a bit of common sense and a CTRL + Z.