Python script failing via cron

The issue

I had setup a cron task to run my Squirrel Archiver script once per day, but for some reason the task was not executing. Nothing, including any errors, was showing up in any logs. At first I thought there was something wrong with my crontab, but other scripts set up to run in the same way were running without issue.

Copying the commands in crontab to the terminal and running it manually worked. It was only when cron was triggering the run that it failed.

Just to be sure, I changed my script to just print('hey'), and this worked! So there must have been something wrong inside the script, and I started the tried and true debugging process of deleting chunks of code until something works.

Side note: Since the script had to run via cron, while I was debugging it was set to run once per minute on the dot. I had to rush to make changes in time, but I also had to wait to see results, which made for an interesting experience.

The culprit

I had been declaring openai_client = OpenAI() as a global variable, outside of any function. This worked fine when running the script manually, but when cron triggered it, it failed silently.

“Why?”
I have no idea.

“Is initializing objects outside of functions commonly known to be a bad practice?”
Probably.

I wish I knew more about the reasons, but moving the declaration inside a function fixed the issue.

So, maybe to many it is obvious that “yes of course you shouldn’t do that”, but I’m recording it here in case anybody else out there is like me.

What I’m doing /now

Personal Life

Nothing much has changed for our family these past few years, and that is by design. Work is stable, life is stable, everybody is happy. I am very thankful for that.

Work

Since 2022 I have been leading a department that is tasked with making the company better at creating new games. This involves encouraging all staff to submit game ideas at any time, working with people to strengthen game ideas and create attractive pitches, coaching teams through early development phases, maintaining a corporate strategy for the release of new games, and more. It allows me to work directly with a lot of people throughout the company, and has been tremendously fun.

Hobbies

In 2022 I joined a Haiku study group. We decide on a few prompts every month and submit our poems in advance of a meet-up, where we comment on each other’s pieces and have our teacher critique them. Haiku is fascinating because its extreme minimalism (only 17 characters allowed) makes it feel like putting together a puzzle. It is also a great way to learn about nature and Japanese culture.

Last year I set up GoToSocial as a solo instance to join the Fediverse, and that has been a lot of fun. It’s nice to be social on the internet again. I think this has helped motivate me to update this site more, and I’m having a lot of fun making little updates to the site as well.

AI tools have been super useful for programming as a hobby. It has made it so much easier to finish and ship my little projects, and I have already developed a slew of little utilities that I use often. I’m still writing words by hand though, because I think being able to organize my thoughts into communicative sentences is such an important skill.

I also started keeping a physical journal at the beginning of this year, and one month is the longest I have ever kept up the habit so I am hopeful that it will continue. I had always admired people who keep physical journals; I thought it was a wonderful habit. I never could get into it though, and had resigned myself to thinking that I just wasn’t that kind of person. But my wife convinced me that if that is the kind of person I want to be, then it’s worth putting in the effort to make the change.
So far, I am really enjoying it. I’m using the most simple interpretation of Bullet Journal. It’s turning to be a super helpful framework to guide the kinds of information I should keep, and the format for recording it.

Katamari Damacy postmortem

By Game Developer Magazine:

We recently received a request to publish the Katamari Damacy postmortem from the December 2004 issue of Game Developer. It has been posted here in full for the first time in 2024 to celebrate the game’s 20 year anniversary. Please enjoy.

This article was written by the game’s director, Keita Takahashi, and has been translated into English from the original Japanese. I wish I could find the original text, but this is a great read regardless.

It’s easy to take a successful game and say “Here are its unique features and the reasons why we think they led to success.” But this portmortem also includes regrets from the game’s development, which gives a really rare, close look into what the team’s ambitions were and how they thought about the game’s design.

CrossPooter - Poot your toots around the web

CrossPooter is a simple script for personal use, that reads the RSS feed of this website and posts the latest article to GoToSocial and Bluesky. It was inspired by EchoFeed, but I wanted to try my hand at making something similar myself.

You can see the source code here.

Features

CrossPooter reads the RSS feed of this website and checks if the latest article has been posted yet. If not, it will grab the article’s OpenGraph tags to construct a post.

The post will include a link to the article, the article’s title, and the article’s description. It will also attach an image if one is available.

The script is inteded to be run via cron every hour or so. I’m assuming that I won’t be posting new articles faster than that, so the script only checks the latest article.

Things I learned building this

Bluesky was very straightforward since I just used the official python sdk. GoToSocial probably would have worked with the Maston library, but the API was simple enough that I wanted to try just sending HTTP requests.

I’m still very inexperienced when it comes to creating complex HTTP requests, so I stumbled a lot with setting the correct headers and formatting the data in the correct way. But it was a good exercise in reading source code and documentation, and now I have a stronger handle on how to do it.

The geometric nests of the white-spotted pufferfish

The white-spotted pufferfish, or Amami hoshizora fugu, is a species of pufferfish newly discovered in 2014 near Japan’s Amami-Oshima Island. The males spend roughly a week constructing an elaborate circular nest about 2 meters in diameter to attract females.

Divers had known about these structures that appear in early summer every year, but apparently it took a while to catch the fish in action.

Hobbies that I am no good at

A distinguished actor appeared on TV and talked about their love for juggling. They had taken lessons at various times and always sets aside time to practice because they enjoy it so much. They then recounted how they recently met their juggling teacher of ~10 years past, displayed their current juggling skills, and was met with astonishment at how they had not improved one bit in all that time!

It was a funny story, but what struck me was just how nice it was that juggling was a constant source of enjoyment for this person despite apparently being no good at it. That’s a hard thing to do.

I think that the internet is partly to blame, since the entire world’s top talent is now at our fingertips. It is too easy to see the best of the best, compare ourselves to them, and end up thinking, “Well what is the point of me doing this at all, then?”

But we must allow ourselves to enjoy things, even in the face of our clear mediocrity. Is there anything I enjoy doing – and let myself enjoy – that I am not good at? Am I denying myself of any hobbies just because of a lack of skill? Let’s dig in.

Hobby: Web development

This one is tricky, because I used to be decent at it but am now well behind the curve. I can slap together some HTML + CSS + JavaScript and make an OK web page. I was adept at Ruby on Rails 10 years ago and can still fuddle through making a web app.

But I touched python for the first time last year. I used Docker for basically the first time last year. I have no idea how to fix a Linux server when things don’t go exactly according to Getting Started guides. I have never touched a JavaScript framework. I have a small comfort-zone and am totally lost outside of it.

These things don’t bother me though, because I enjoy the time I spend tinkering. Occasionally I decide I want to try something new, and I stumble and fumble my through it and maybe learn a little along the way. This is possibly the only creative activity that I can enjoy in a pure, for the love of the craft kind of way. I like that.

Not-yet-a-Hobby: Drawing

Sometimes I get a hankering to draw a picture, and I always enjoy the time I spend doing it. But I am not a good artist. I’m especially terrible at drawing faces. Awful.

I can’t call it a hobby, since I rarely make time to draw. I have a suspicion that I would enjoy it if I let myself, and that the main thing holding me back is that I think I’m no good at it.

Hobby: Haiku

A few years ago I developed an interest in Haiku. The hyper-minimal structure makes it so that changing just one preposition, or modifying the word order a little, can have a huge impact on the poem’s overall meaning and impact. I think that’s fascinating.

I was lucky to find a group of people that meet regularly for Haiku. We decide on a few prompts, spend a month writing 3 Haiku each, and come together to comment on each other’s pieces and have our teacher critique us.

I think that I’m not bad at Haiku, but to be honest, I thought that I would’ve gotten a little better at it by now. This isn’t discouraging though, as I find myself not really caring how skillful I am. The exercise of distilling a scene to its barest core and painting its picture with my words is fun. I feel like there are some low-hanging fruits for improvement if I put more effort into study, but I’m pretty happy just puttering along at my own pace.

Dragonsweeper - A lovely minesweeper game

This is a really excellent take on minesweeper. Spaces can contain monsters of various strength, and you need to spend HP to defeat them. By collecting EXP and hearts to level up and heal yourself, the goal is to become strong enough to slay the dragon.

It took me a few tries to get a handle on the strategy, but this is super fun!

The Sociable Weaver

The sociable weaver is an African bird that creates giant, community nests in trees. The nests look like a giant woolly mammoth hanging in a tree, and have many entry holes on the bottom. These nests can house over 100 pairs of birds, spanning multiple generations at a time.

Becoming a parent introduced me to stress

I have always been a calm and happy person. Aggressively optimistic, even. I made it through the first 1/3 of my life without feeling much stress, only getting angry on rare occasions. Anger management, emotional therapy, mindfulness exercises, these were all things that I had no need for.

I thought that this was a personality trait, but now I realize that I was simply blessed with an easy life. It’s not that I was particularly good at dealing with stress, I just didn’t have much stress coming my way.

Well, parenting is not easy. It is demanding. uncontrollable, scary, and stressful. I love my kids dearly and would not want them to change for the world, and that seems to make it even harder. I want so badly to be the parent they deserve, to guide them with a gentle hand and encourage their every exploration.

But it is hard.

And I am feeling the stress that had nothing to do with me until now. I feel frustrated when things don’t go the way I expected, I feel ashamed when I neglect opportunities to engage my kids, and I feel angry when situations get out of my control.

I have to tell myself to take a deep breath and count to ten. I have to spend time with myself to sort out my feelings and learn what my needs are. All of these techniques that I had always overlooked as having nothing to do with me, are now suddenly instructive and eye-opening.

In my head, I know that I am probably overestimating the impact that I have on my children’s growth. They are their own people, and they will live their lives to their fullest. They will learn from and be influenced by so many more sources than just their father.

But I want to give them everything I can. I want to be a rock-solid, positive influence like my parents were for me. I want to teach them the names of flowers, and insects, and constellations, and construction trucks, and super-heroes, and inventors, and mountains, and countries, and exotic foods. I want to make things with them so that they can understand how things work, how they can take things apart and put them back together again. I want to show them that they are special, and that they have so many unique things to offer to the world.

But I am not that father yet. I’ve learned how selfish and impatient I can be. I’m new to this, too, so of course it will take a lot of time, effort, and patience. I need to take things one step at a time, and also let myself recognize that I am already doing a fine job.

But now, I am stressed out.

But it will be worth it.

Squirrel - Archive webpages so I can find them again

I created a thing that lets me save webpage contents in a smart way. I know, there are a million bookmark managers and web archivers out there, but this one is for me.

The Motivation

From Pinboard:

Do you half-remember interesting things you saw online six months ago, but struggle to find them in a search engine?

Yes, yes, a hundred times yes. This happens to me all the time. Searching my history doesn’t work because usually the keywords I remember are not in the title. Google et al. aren’t much help either.

I want a way to have the contents of the pages I visit be searchable, in a smart way.

My Requirements

1. Minimal friction to saving content

If I need to go through an entire workflow or remember some specific key combinations, I’m probably not going to use it. I want this to become a subconscious action whenever I finish reading something interesting.

Pinboard’s javascript bookmarklet is very good, but unfortunately I then need to manually add tags and a description. I mean, I don’t have to, but the bookmark is significantly less useful if I don’t.

I used the Obsidian Web Clipper for a while, and it was actually very good. The article text is saved in a new note, and tags and summaries are extracted from the page’s metadata if they exist. But, and I know this is a nitpick, it takes a few steps each time:

1. Click the button
2. Click “Add to Obsidian”
3. Say “Yes I want to allow this site to to open Obsidian”
4. Obsidian opens automatically with the new note
5. Navigate back to the browser, to continue what I was doing

Every time. It’s friction.

2. Easy to run a text search across the full text

Titles and URLs are not enough. I want to expose the full article text to search, too.

Pinboard again, only allows me to search titles + manual summaries + bookmarks. I can pay $39 a year to enable full text search, but that’s kind of expensive and leads to the next point:

3. Not expensive

I mean, a couple of dollars a month isn’t that much. But they add up. This seems like the kind of problem that I, a non-power user, should be able to solve pretty cheaply.

Tangentially, I also really liked the idea behind Browser Parrot, which seems to have been replaced by Browser Gopher. It automatically saves every page from your browser history, and can extract the full text.
The problem with this was that it ended up using lots of CPU, and the archive took up a lot of space, too. I think that in practice, 90% of the pages I visit are not worth saving.

Other things that aren’t needed but might be nice

1. Some kind of tagging system (minimum friction of course) so that I can link related information and re-discover old items.

2. An accessible format to save the data. I prefer plain text to a database, because I can browse and open text files from Finder, or use them later for whatever I need.
   Would a database be more byte-efficient and make searching faster? Maybe. But since I’m not going to be saving millions of files, it probably will not make much of a difference.

Based on the above, my solution

1. Put any webpage I want to save into a text file

I set up a Shortcut to grab Safari’s URL and add it to a text file urls.txt, which lives on my iCloud drive.

2. Run a script that periodically looks at urls.txt and processes new items

Setting a cron task to run this script once per day is fine. The text file is wiped after each run, and URLs that failed for some reason are saved to a separate file.

3. Access the URL to save the title and article text

Python libraries can do this easily. There is a lot of optimization that could be done to check for errors and edge cases, but as a personal project I don’t need it to be bullet-proof. Just a simple grab and dump.

4. Use ChatGPT to generate tags and a short summary automatically

This sort of thing is what AI was born to do. Having an automatic tagging system is such a luxury, and doing it directly through the API is dirt-cheap.

5. Package into a text file, and output to a local folder

Now I have a single folder on my hard drive that holds all of the articles that I thought were worth reading.

6. Open the folder as an Obsidian Vault

Searching the article contents is very fast, and I can use graph view to explore tags. I can probably do a lot to make it more useful, but for now it’s pretty.

Bonus

It also was super easy to set up an edge case for YouTube videos. Instead of grabbing the page body’s text, it instead uses the YouTube API to grab transcripts from the video. This is turning out to be a great way in general to extract information from YouTube and make it accessible to me later.

The Script

Is on Github

My intention is not to promote this for wide use, but you can feel free to fork it if you think it might work for you.

My archive now

I went through the bookmarks I had saved in various places and ran them all through this script. After pruning dead links, I had about 500 articles saved over the course of 17 years. My most used tags are:

1. Technology
2. Innovation
3. Design
4. Sustainability
5. Environment
6. Development
7. AI
8. Community
9. Creativity
10. Javascript

Sustainability and Environment were a surprise. I guess that when I read articles about those topics, I feel compelled to save them?