Big Data and Schlep Blindness

When I was in university, I thought a typical day at a software engineering job would be full of solving complex integrals & developing our own version of algorithms. I was wrong.

I got into my first software development role, and my days were a mix of:

  1. shelling scripts to run cron jobs & monitoring log files
  2. running SQL queries to extract data & combine tables
  3. monitoring disks, message queues, adapters and log files for errors/space

So I figured I was just enjoying the apprentice part of the job & need to rise up the organization to get to work on the "good stuff". After years of enhancing & bug fixing other people's documented code, I got to work on my first new project. Now my days were a mix of meetings & some actual work (i.e. extracting, transforming & loading data for point to point interfaces, and building micro services to enable orchestration of business processes).

I worked on Enterprise Application Integration on an Oracle powered big data project that would one day revolutionize how our customers made decisions. We made tons of integrations, dashboards and ultimately supported a monopoly business update their systems (Digital Transformation).

I jokingly said to my colleagues over a smoke break that we could f this all up, and still our client would keep printing money like nothing happened. Since they had a strong regulatory capture situation going on. Luckily, that didn't happen, and we all moved on to other roles.

Fast forward five years, and I got to a client whose IT team has messed up everything royally, during a billing system "transformation" they had gone live and were sending the wrong bills to 70% of their customers. So we started to investigate how a blunder of such epic proportions could have occurred.

We went to the Business Intelligence & Project Management Organization leaders and we handed over gigabytes of data + documents, but no answers other than vague finger pointing about how a company of 20,000+ employees could have screwed up so badly. They had a lot of owners, sponsors, dashboards, and reporting but nothing that would say what to do once your system starts printing the wrong bills. Finally, we reviewed the Statement of Work and did some basic statistical sampling to show that testing was documented well, but not executed well.

That couple of days of statistics made me a small hero within our client, who offered me a new job right away if I wanted to leave the consulting firm (all for doing 1st university statistics).

Finally, I became a startup founder, and now we had tons of data streams giving us information about customers trying out our product, us building up new features, invoices being paid (sometimes) and expenses being incurred in addition to all the time stamped data.

What I learned was that: the larger the company, the more process driven they become and it's less about making the best decision, and more about not f-ing up since that is where it hurts.

Every large company has a data transformation program that has taken 4 times as long and costs 4 times as much, but none of them have been able to improve decision making to make the company go 20% faster each year.

The problem is the promise of big data, being a silver bullet that will resolve a company's problems. Instead, what's needed are small streams of data that leaders can accurately use to identify opportunities for growth, and understand risks that the business is taking.

Computers are not bad, but they cannot just decide things on their own.

Famous companies like Palantir or PayPal were highly successful because they produced systems that combined in a complementary way both human capabilities (decision making) & computer capabilities (processing large quantities of data quickly).

Meanwhile a large business, sometimes works when people focus on one metric. One famous example here is Chamath Palihapitiya at Facebook saying ignore all the k-factors and just get a person to add 9 friends as quickly as possible. Small data wins over big data, despite all what your technology consulting friends tell you.


Thanks to Faizan Siddiq, this started with mindless typing. He reviewed several drafts & gave feedback to bring out the good parts. Thanks for your help.

1) Schlep Blindness by Paul Graham

Show Comments