Data Journal

I have maintained a life-tracking project since 2013. If you’re interested in that, this page is for you.

Questions

Who

I did this, which means you could do it too.

What

Data Journal¹ is a system I built to help me keep track of stuff that happens in my life. Its form has changed greatly over the past decade², but its function has remained the same: to house data about things I do. It’s a default place to make note of things that might not otherwise have a place. I’ve tracked lots of different things over the years.

My Data Journal allows me to answer questions like:

• “When did this pain in my back start?”
• “How many workouts have I done this year?”
• “When was the last time I hung out with this person?”
• “How many drinks have I had the past few weeks?”
• “Have I eaten out more or exercised more this month?”
• “Have I met this goal I set for myself?”

When

I started with a pen & paper back in 2013 and haven’t stopped since. You could start today.

As far as when to track things - tracking inputs happen as frequently as you want them to. I track stuff from once/week or fewer all the way down to multiple times a day. Depends on the thing, how easy it is to track, and how much value I get by tracking it.

Where

In concept, the Data Journal could be built using anything that can hold data. Some Bullet Journalers use their physical notebooks to do something similar. It’s easier to scale over time if you use technology. Easiest of all if you put it in the cloud^[3]. For those options, Google Sheets or Notion both make great choices.

Why

There are a bunch of reasons “why”…

• fantastic memory recall of the past
• better attention paid to the present
• having a framework around which future goals can be built and maintained
• specifically in my case - having a great testbed application and reason to learn to code, system design, and how to make things work

…but honestly, my best response to the question of “why” is ”why not?”

How

This is why I made this page at all. This is what separates this page from my 5 year, 7 year, 10 year, and most recent retrospectives on the project.

If even one person benefits from reading this I will be very pleased.

How To

How I Recommend Starting

First, ask yourself:

• What you want to capture?
• What you want to do with what you capture?
• What technical skills do you have now, and what skills are you interested in developing?

There are plenty of options to get you started. I’m not going to go into any of them in detail, but try any of these to begin with before trying to make something fancy.

• Just carry around a notebook and a pen. I tracked with a pen and paper for a year, manually transcribing to Excel each weekend. That worked fine.
• Use a Google Sheet, or a few Google Sheets, maybe along with Google Forms and/or IFTTT.
• Spin up a Notion database based off some habit tracking or journaling template that you like.
• If you’re mostly interested in habit tracking, I’ll plug one of my favorite apps Streaks on iOS.

Track things to the best of your ability, either throughout the day or just at night before bed. Put it in your calendar or to-do list. Review after a month or so. Make adjustments.

This is the best place to start. Don’t jump in and try to perfect things out the gate. You won’t know what’s working until you try it, so just get started with something basic. You can evolve your system from there. Don’t make it complicated, because Highly Specialized Solutions are Highly Brittle.

How I Do It

In my experience, the best bang for your buck toolset is Google Sheets + Google Apps Script (the macro-like scripting engine behind Google’s productivity suite) + Siri Shortcuts (Apple’s wonderful scripting application).

Google Sheet

The whole thing is built around a single “master” Google Sheets workbook. I made this blank copy for you to look around in.

There is one sheet per each level of granularity you care to track. At a minimum I’d recommend sheets for Days, Weeks, Months, and Years. Each row in these sheets corresponds to one period of time, and you don’t skip rows³. If you decide some things you want to track more frequently than “once/day”, then the best thing to do is add a sheet dedicated to tracking that thing. Each row in that sheet then represents one instance of that type of event.

Create columns for each piece of data you want to track in whatever sheet it belongs in. If it’s something that happens every day, put it in the Days sheet. Create additional columns with formulas in them to pull data from other sheets as desired. You could, for example, use an averageif() formula to pull a daily “Health” rating from the Days sheet into the Weeks sheet to see how your health is trending over the year. I include several “index” columns to make formulas easier for pulling data between spreadsheets. Each of the “one row per occurrence” sheets share a common schema for the first 6 columns:

This allows you to use formulas like countif() and averageif() to select rows that belong to a given period of time.

Google Apps Script Web App

Create a Google Apps Script using code like this⁴ to do 1, 2, or 3 things - in order I’d recommend:

1. oneAM() - Setup a nightly Trigger that auto-generate rows for you each day/week/month/year. This is a relatively simple process that makes the whole thing much less tedious.
2. doPost() - Publish a webapp to allow for writing data to the workbook from anything that can generate an HTTP POST request (e.g. Siri Shortcuts, IFTTT)
3. doGet() - Include a route on the webapp to allow for retrieving data. I kept mine simple and it only supports getting today’s data.

The secret sauce is really the automations to always have new rows created for each period of time AND to setup the doPost API route to allow automated data writing from Siri Shortcuts. These two things alone ensure you’ve got space to put stuff and that you’re putting at least something in that space. Once you have some data in a spot it becomes motivating to try to fill the blanks.

Siri Shortcuts

The first part of the secret sauce was automatically creating one row per day, the second part was making it crazy easy to get data into your system using your phone, watch, or voice via Siri Shortcuts. It’s as easy as:

Siri Shortcuts lets you input data from any Apple device by talking, typing, or even doing nothing at all (when you set up automations). Getting data in has never been so easy.

After setting up the WebApp, with literally one action in a Siri Shortcut you can create an easy-to-trigger, easy-to-automate, easy-to-reuse Siri Shortcut for adding new data to your Data Journal.

I have about a dozen Siri Shortcuts that look like that, each for inputting a specific kind of event data. Most of them I trigger using my voice or keyboard, but some are fully automated & I never trigger manually. For example - every night at 3:30am my phone sends to the Data Journal the city I am in. This Sleep Location metric I use to keep track of how often I’m traveling, and it happens without me doing anything.

Conclusion

Making the Data Journal was (and still is) a continual process of learning, refinement, and optimization. Using it did all those same things when it came to my lifestyle. It has been my 4th best constant companion for the past many years. If I hadn’t had such a good reason to learn all these things, I reckon it would have never happened.

Epilogue: Short List of Technical Things I’ve Learned Thanks to this Project

Here’s an incomplete list of the technical things I’ve learned thanks to this project.

• Basic Web Development
  • JavaScript
  • HTML
  • CSS
• Slightly less basic web dev
  • TypeScript
  • I wrote my own JavaScript Framework NPM Package (that you shouldn’t use)
  • NPM publishing
  • Tailwind
  • Svelte
• Local development
  • NodeJS
• Siri Shortcuts - a surprisingly powerful tool.
  • Siri Automations
• Google Apps Script - another surprisingly powerful tool
• Scriptable App
  • Custom widget creation
• REST API consumption
  • Google Fit
  • OpenWeatherMap
  • Oura
  • Google Sheets
• REST API creation
  • Node + Express
  • SvelteKit
  • Firebase Functions
  • Google Apps Script Webapps
• Authorization implementation
  • Firebase + Firestore
  • Custom tokens
• NoSQL Databases
  • MongoDB
  • Firestore
• Serverless compute
  • Vercel serverless functions
  • Firebase Functions
• Unit Testing
  • And why/when it’s worthwhile
• High-level software concepts
  • Architecture
  • Separation of concerns
  • Standardization & interface-orientation
  • Documentation
  • Designing & packaging plugins
• Hardest lessons of all:
  • YAGNI
  • Avoiding complexity is the #1 most important thing
  • Sometimes living with small problems is the correct solution
  • Options & features don’t deserve to exist because you thought of them

Footnotes

1. Formerly known as Data Journal. Formerly known as Life Tracker. Formerly known as Demetri List. ↩

2. From pen & paper, to a spreadsheet, to an expansive set of web apps, databases, utility functions, custom user interfaces, and translators, back to a humble spreadsheet. ↩

3. It’s better to have blanks on days where you failed to track something that it is to just delete the row for that day. ↩

4. Which should be included in the sample “blank” Data Journal workbook, by the way. Under “Extensions” > “Apps Script”. ↩