#395 - Data Journal Project History

This Column is more squarely aimed at Aaron Gillespie as the target audience than most. Proceed with caution. We’re going to cover the past, the present, and the future of “The Data Journal” - or if it’s more fun to think about it this way, the past, present, and future of Aaron’s mental breakdown.

Why

A book I read 3 chapters of before giving up on told me to “start with why”. So, why track dozens (eventually, over 100) data points in your every day life? What’s the point? Why go through the effort of writing things down, analyzing data, learning to code, and keeping it up with it all?

Fear, mostly.

Fear and boredom.

I started “The Life Tracker” as it was then known a little over a year after I left college. I realized I was in a rut of going to work, coming home, wasting the night away, eating whatever, going to sleep whenever, then starting the cycle all over again. The way I was living I would have trudged through life, slowly forgetting things I once knew, as I got fatter and less engaged, and failed to acquire any amount of equity… and it was all on me to stop it. I was afraid of slowly becoming somebody I’d regret without realizing it.

Also, I wasn’t really doing much else. Playing around in Excel seemed slightly more productive than playing around on a PS3.

Enter the Data Journal

The solution - write it all down. I didn’t really know exactly what I needed, but as my high school science teacher once said:

“If you aren’t measuring it, you aren’t going to learn about it.” John Torrence said something like this

So. I wrote it down. Rather than bore you to death with paragraphs of details that nobody in their right mind should read - here’s some hand-drawn history for that VISUAL FLAIR!

The Past a Visual History of the Data Journal

V1

V2

Reason for new version: no need to carry around (and sometimes lose) a piece of paper and a pen.

V3

Reason for new version: I got tired of typing in my daily food summaries, sleep details, and other things.

V4

Reason for new version: I got tired of tracking the same 25 Daily Goals after 5 years. Was ready for what’s next. Also I figured out how to use a free weather API.

V5

Reason for new version: A couple more data points I thought might be nice.

V6

Reason for new version: A redefinition/refinement of the new data points.

V7

Reason for new version: Accommodating the switch from Android to iOS & getting into tracking at a finer granularity than “day”.

V8

Reason for new version: I realized I wasn’t fat & that writing down what I ate wasn’t really the reason why. I honed in more on self-actualization, rather than just self-sustainment.

The Present - V9

Reason for new version: A house divided against itself cannot stand. Tracking at both the daily and “realtime” level was causing data conflicts that I believed I could resolve by way of refactor.

This, though, doesn’t tell the whole story. Version 9 has not worked yet. Today, I got to the point where I expected to fully “transition” from V8 to the new normal… and it just fully went sideways.

Here’s where I’d like to help the maybe one person if I’m lucky that will ever read this and benefit from it.

How It Work(ed)

When the Data Journal worked well, it was set up such that one row = one day. This is the easiest, most natural way to do a quantified self/habit tracking/whatever you want to call it project. Each row corresponds to a given time period. The “Daily” sheet rolled up to “Weekly” and “Monthly” sheets. All of those rolled up to “Yearly” sheets. The columns defined the measures I took. Data types were implicitly covered by cell number formatting. The formulas defined how those measures rolled up.

Each morning a script I wrote would run and create space for new data. Each night, I input a Google Form which filled out that space with answers to questions I asked myself & data from the data services I learned to use.

In a graphic:

How It Doesn’t Work

As soon as you remove the foundational principle upon which everything else was built, it all becomes much more difficult to manage. In other words: one row ≠ one day is an order of magnitude more cumbersome. Treating each row as an individual data point is not how spreadsheets are meant to operate. This contributes two compounding problems:

1. The data set that formulas have to look through for relevant data increases by dozen(s) of rows a day.
2. There are suddenly thousands more formulas to calculate, and they ALL are forced to recalculate each time data is changed.

Or, in a graphic:

The Future

There are 4 paths forward I can see:

Choosing the path is where I sit today. I may do one. I may do them all, eventually. The question is back to “why”. Why do I want to continue tracking things? Is it merely to continue having a journal of references? Is it because I care about correlations and metrics? Is it to force me to think about my life? Or is it for fun? or because I want to build a product?

Who knows. If you know what I want, please write in. Email me here..

Top 10: Lessons I’ve Accidentally Learned in Service of Learning about Myself

1. Not planning for failure is the same thing as planning to fail. The first couple years of the tracking my system was completely intolerant of anything other than perfect data completeness for an entire week. When I inevitably lost data for a single day, the whole week had to go. Now I’ve got data gaps that are much wider and more noticeable than just missing food for a single day. Don’t write yourself into a corner with what data you choose to track. Don’t expect to know today all the data points that you’ll care about in 5 years. Leave your system room to grow, shrink, or refocus as time goes by.
2. Tracking something is sometimes all the reason you need to make it successful. I achieved many things simply because I didn’t want to record a failure for a run of successive days.
3. Tracking either needs to be easy - or important enough to be difficult. Every data point I didn’t fully automate that wasn’t absolutely essential eventually got nixed.
4. Subjective scales from 1 to 5 are less useful than ones from 1 to 10. Neither of them are as useful as objective measures that tell the same story. “How do you feel, physically” is less useful than “what medicines did you have to take”. The gold standard to aim for is a subjective qualification on top of an objective fact. “I ate Taco Bell today, and that’s poor food” is better than “I ate poorly”.
5. Tracking things is a great way to spot trends and correlations, for sure - but its real value comes from forcing you to think about your life. Are the things you do day-to-day aligned with your values and long-term ambitions? Track according to what’s important to you. How can you measure your goals? How can you measure how well you’re tending to the areas of responsibility in your life?
6. Another, more obvious, but less appreciated benefit of tracking things in your life is that it serves as an incredible set of reference materials for questions like “when did we buy the house?”, “when was my last doctor’s appointment?”, “how long have I had this cough?”, “when was the last time I saw my parents?”
7. Anything piece of data that isn’t closely bound to time, shouldn’t go into a time-indexed tracking system. I tried, for a bit, to use the Data Journal to hold everything. Turns out that some notes are better organized by subject than by “when” they happened to get written down.
8. Being good at spreadsheets can take you places. My 4 year, multi-thousand dollar degree has helped me less in my day-to-day life than have the couple of weeks I’ve collectively spent learning how to use static references, why VLOOKUP is usually not the best option, and learning how to string together functions to do basically whatever I want.
9. Spreadsheet manipulation is an awesome place to learn basic scripting. It’s easy. It’s self-contained. It’s obvious how some simple coding can do incredibly powerful things.
10. I learned how to speak, write, and think in JavaScript + HTML + CSS. Also how to learn how to make APIs and new environments work. If I stick with something long enough, I usually figure out how to make it work. Or I find something else that does the job.

Quotes

That’s mildly important but let’s get back to the gum dilemma.
Joe