The Column
I’m relieved - but not thrilled - to be writing this.
Prelude
I start grad school in a few weeks. Right as a new year begins, I’ll add a large new rock to my bag of responsibilities and demand on my time. I have been worried about managing it. Worse, my system for managing things has not been what I hoped it would be.
Lately as I’ve struggled to use the Personal Data Warehouse & Data Journal system I just spent literal months re-writing I’ve found myself questioning “why am I even using this at all?“. I was putting data in, but never really pulling value back out. I thought about stopping, but the thought of not keeping track of things scared me for the same reasons I started in the first place.
So - literally one Column after having written about my problem with building systems, I find myself writing about a system I built. Irony? Hypocrisy? Idiocy? All of the above?
The Data Journal is Dead
The thing I spent 100s of hours over the past 4 years building, writing about, drawing diagrams for, documenting, and rethinking is dead. I’m not using it. I don’t plan to use it any more. This is the last time I plan to write about it directly[^1].
All of the variants & releases, the blog posts & pages, the diagrams & documentation, the models & artifacts, and the ad hoc data files, all of that has, as of this moment, become irrelevant - a relic of the past from a long and extraordinarily complicated experiment/period of my life.
The system I built was too complicated. So I killed it and started again. The “simpler” thing I built was still too complicated, so I killed it and started again, again.
Long Live the Data Journal
I built my new system in a day. My new system is essentially just my spreadsheet from 4 1/2 years ago, with minor tweaks. It’s so much less impressive than the things I’ve built over the preceding 4 years, but it’s better.
I won’t go into details of how it works here - because I just wrote this page to describe it in detail.
Epilogue: Mastery
Do I regret building all that stuff I am not going to use? Not at all. I learned an immense amount - about little techniques, gigantic concepts, and about myself[^2]. Mastery is knowing what you actually need. I feel like I well and truly reached the final stage of maturity on this. The end of the bell curve:
Top 5: Things I Designed the Data Journal (v11) to Do
My design principle going into this build was to identify the very few things the system actually has to do, then write the least amount of code possible to make those things happen.
5. Store data for multiple levels of granularity
Some data are day-long, like total sleep, for example. Other data are year long, like days spent travelling. Yet other data may happen more often than once per day, like movies watched. My approach is to have one sheet per level of granularity from years to days, then have independent sheets for the few other things I’d like to track multiple times per day.
4. Allow posting new data for today via a REST call
Creating an HTTP POST request to my webapp’s URL and supplying two parameters (what data to put in, and which sheet to put it in) will add those data in the correct spot.
3. Allow retrieving today’s data via a REST call
Simply navigating to my webapp’s URL will fetch the data for today from all the sheets, then respond with them in a sensible JSON format.
2. Create new row(s) every night
Google Apps Script lets you trigger code execution on a schedule. Each day a new row is added to my “Days” sheet. On Mondays I get a new row in the “Weeks” sheet. Etc.
1. Function like a normal spreadsheet
I am not writing my own highly-specialized magical functions & calling them from within the workbook. The spreadsheet is a spreadsheet. It’s doing what regular spreadsheets do, using regular relatively simple spreadsheet functions.
Quote:
I rebuilt my Data Journal thing from scratch. The last one was too complicated, so I made it way simpler. Now I don’t have to touch it ever again! - me, to Melissa
I feel like that’s exactly what you said last time. - Melissa, in response
[^1] And I’m sure this comment will age well.
[^2] Booooooo