Downloading Callahan clips from YouTube

The first time

Last year, I was roped into the Callahan video¹ for the frisbee team I play on at Carnegie Mellon. The hurdle was that many people had spent hours pouring over footage finding great clips of awesome moments, but it was incredibly time-consuming to download all the videos and clip them to the timestamps.

Long story short, I put on my programmer hat, pulled out Swift because I just love it, and hacked together a script that used yt-dlp to do the job. The video turned out great, and I moved on with my life.

Then I got a message in the middle of the summer from a teammate who was playing for another team asking about the script. He needed to do something similar, so I put my code in a zip file, wrote a long explanation about how to make sure you’ve installed Swift, and sent it off. Reflecting on the wall of text I’d just sent, I added “AI could probably rewrite it if Swift doesn’t work or make sense.”

I never did hear how things went. But I did realize that 1. this is a somewhat useful tool and 2. Swift might not be the best language if I was going to share this. I tucked “Write something about Callahan scraping” into the Someday section of my todo list, and got busy with the school year.

Of course, as one does in the face of the mild stress of finals week, I decided to rewrite this. In Python.

Cathartic interlude

Python was the second language I learned, because I choose to count LEGO Mindstorm block coding as my first. I’m pretty sure the moment I learned a third language (which was C++, which definitely has its own irritations), Python has been annoying. But I’ve also had to write far too much Numpy code, and I’m pretty sure more of my college robotics courses have been spent tracking down bad array shapes and type issues than writing actual logic. At some point, I definitely need to come to terms with the fact I’ve chosen to major in a field dominated by a language I frequently rediscover my distaste for…

The one redeeming quality of Python, though, is just how many libraries exist for it.

The standard library

In high school, I wrote a bunch of scripts to send emails and scan QR codes as a ticketing system for our prom.² Everything that made that possible was a bunch of libraries: Google API libraries, QR code generation libraries, computer vision libraries, UI application libraries. I remember wanting to add a little chime sound for when a check-in was successful. First thing I googled was “audio libraries python.” That project would not have happened in any reasonable amount of time if not for the enormous amount of human effort that has been put into making Python libraries for everything under the sun, and the language’s ability to act as a simple glue for all of it.

When I rewrote this project, I knew Python would be the best choice, just because it would be able to be run by anyone with very little setup. In the spirit of this goal, I figured I should also try to limit myself to using standard library things, because trying to explain good Python environment management if people need to install packages to use it would get me back to a similar wall of text to the one from Swift. Honestly, I’d forgotten the Python standard library’s level of game.

I looked at my cobbled-together Swift CSV parsing code, said “I do not want to write that much Python,” and stuck import csv at the top of my file. Someone else (or rather, many someone elses) did the work so I could pull out the things I need with code I think my grandmother could probably half-understand.

clips = []
with open(csv_path, 'r') as csv_file:
    reader = csv.DictReader(csv_file)

    for row in reader:
        index = int(row['Index'])
        vid_id = extract_video_id(row['Link'])
        timestamp = extract_timedelta(row['Timestamp'])
        clips.append((index, vid_id, timestamp))

Originally, I was a little worried I’d have to reimplement my mess of Swift code (admittedly, I was not on my game in writing this Swift code, so it could definitely be improved upon).

var fiveSecondsBefore: (Int, Int, Int) = (timeStampComponents[0], timeStampComponents[1], timeStampComponents[2])
fiveSecondsBefore.2 -= 5
if fiveSecondsBefore.2 < 0 {
    fiveSecondsBefore.1 -= 1
    fiveSecondsBefore.2 += 60
}
if fiveSecondsBefore.1 < 0 {
    fiveSecondsBefore.0 -= 1
    fiveSecondsBefore.1 += 60
}

Googling the Python solution to this took a little longer than using the CSV reader, but a sprinkle of from datetime import timedelta and I was off to the races, and my five seconds before was now

zero = timedelta()
five_seconds_before = max(zero, timestamp - timedelta(seconds=5))

I had to do some regex parsing to get the timestamp into this (and I do think Swift’s amazing type-checked regex literals have Python beat a little here), but this wasn’t hard, and when it came time to spit them back out I didn’t even think to format them: the obvious way was the way.

def extract_timedelta(timestamp: str) -> timedelta:
    pattern = r'(?:(?:(?P<hours>\d+):)?(?P<minutes>\d+):)?(?P<seconds>\d+)'
    match = re.fullmatch(pattern, timestamp)

    if not match:
        raise ValueError(f"Invalid timestamp format: {timestamp}")

    hours = int(match.group('hours') or 0)
    minutes = int(match.group('minutes') or 0)
    seconds = int(match.group('seconds') or 0)
    return timedelta(hours=hours, minutes=minutes, seconds=seconds)

Running shell commands? It’s just subprocess.run.

So I very quickly arrived at a new working version. Then I wanted to have a little fun.

I’d had great success turning a “set aside time, it’ll take 20 minutes to run” Computer Vision assignment earlier in the semester into a 90 second task using joblib. As I watched the print statements slowly come in from yt-dlp chugging away, this seemed like exactly the tool for the job. However, my “no external libraries” rule meant this was out of the question. It turns out this was not a big deal, since Python has a pretty simple API for naïvely adding parallelism, and I wasn’t looking for much more than naïve.

We add new tasks as keys in a dictionary—the most Python anything can be anything thing ever—then look up which ID was completed using a blocking wait on the main thread. Voilà, concurrency.

with ThreadPoolExecutor(max_workers=4) as pool:
    futures = {
        pool.submit(download_clip, idx, vid, ts, out_dir): idx
        for idx, vid, ts in clips
    }

    for future in as_completed(futures):
        idx = futures[future]
        try:
            future.result()
            print(f"Downloaded clip {idx}")
        except Exception as e:
            print(f"Failed to download clip {idx}: {e}", file=sys.stderr)

This is by no means a perfect concurrency experience. But compared to the pthread incantations I’ve been doing in C recently, it’s a very simple abstraction that makes concurrency very easy and straightforward. And compared to the rollout of Swift’s push for concurrent safety, this shallow learning curve (look at a single example and be pretty good to go) is admirable, though certainly not as powerful in the long run.

What is there to learn from Python

Nowadays, I think a language is the sum of how it interacts with the rest of the world.

Python is not my favorite language, but it’s familiar enough that I can be effective in it (this is why I rarely use Bash). And with its enormous package ecosystem, it deserves its spot near the top.

There are plenty of other languages that beat the ergonomics of Python for me. And in a dozen other categories—performance, type checking, editing experience,³ to name a few—there are far better languages out there. But Python’s dominant collection of libraries and how widely usable it tends to be mean it remains dominant in every area where the code I write is not the real workhorse.

I do hope more languages can build out a strong standard library or the easily accessible package ecosystem that has made Python so strong. I do tend to agree with the doubters that the Python standard library may be too large (they have an entire mini-language for string formatting, for example). I’ve heard interesting things about Go’s approach to making core things like HTTP and SMTP carrier types standard library features while leaving the actual implementation to library authors.

And of course, there’s the chicken and egg issue of people not wanting to write libraries for languages without a thriving ecosystem. Perhaps the incredibly zealous Rust community may someday succeed in rewriting everything in Rust to build a comparably sized package ecosystem. We just need to create a cultish stereotype around our languages until its avid fans force it to success. Or we take the Zig and Kotlin approaches and just be effortlessly compatible with someone else’s ecosystem.

Where this rambling is going, I’m not sure. I will leave this here as a snapshot of my thoughts, though not an organized snapshot. If you have more coherent thoughts and want to discuss, share them with me through Twitter, Bluesky, or email. I’d love to know how you think the language of the future will grow to become so, or if we’re stuck forever with the executable pseudocode called Python.