How we judge Shipaton

Shipaton is, at this point, the largest mobile hackathon in the world. The apps released over the last two years during Shipaton have made a combined n dollars, we've had x people participating, and y projects submitted. That puts a lot of pressure on the judging process to find the winning apps.

Every year we explain how we judge apps, and every year we get feedback from participants who didn't know about some criteria, or who feel our judges didn't evaluate their app thoroughly enough. This article will hopefully shed some light on how Shipaton judging actually works and what we need from your submission.

The timeline for judging

Before we get into how the judging process works, it's worth looking at the timeline of the hackathon. Shipaton runs every year from August 1st to September 30th. You can register even before that, and you can submit your project anytime during the two-month period. We even encourage it, since you can keep editing your project right up until the last minute. Despite this, most projects still get sent in during the final week.

[chart showing when people sign up vs. when they submit their projects]

We've considered judging the apps that arrive before the deadline, but because you can edit your submission until the very last minute -- and because the majority of apps only show up in the final days -- we haven't done that, and we won't. It would take some of the pressure off reviewing every app in time, but it's just not something we can do reliably. As a result, the window for reviewing everything ends up being really tight.

Reviewing up to 1,000 apps a week

After submissions close, we have just two weeks to do all of the judging. That's because Shipaton winners get invited to RevenueCat's App Growth Annual in October, just weeks after the hackathon wraps. Since winners can come from anywhere in the world, we want to give them enough time to sort out their travel plans.

Last year, that meant reviewing 1,000 apps in a week for everything to go according to plan.

The judging process

Shipaton judging works as a funnel. Thousands of submissions come in, and each stage narrows the field, so that by the time a judge sits down with an app, they're choosing between a handful of genuinely strong contenders instead of wading through everything. Here's how a submission travels from "submitted" to "winner."

Stage 1: Intake and rolling screening (August-September)

You submit your project through DevPost, and from the moment it lands, it goes onto a shared tracking sheet. Throughout the two months, Charlie -- one of our developer advocates -- along with a few teammates and some automation, screens submissions as they arrive: flagging spam and invalid entries, and tagging each project for the categories it's eligible for. Doing this continuously, all the way through, is what makes the tight October timeline possible at all.

Stage 2: Prescreening (October 2-5)

Once submissions close on October 1st, every entry gets assigned roughly evenly across a group of RevenueCat screeners. Over the next four days -- two weekdays and a weekend -- they narrow the field down to a size the judges can realistically handle. Two things decide whether a submission moves on:

• All requirements are met: the app is live in the store, every asset is included, and the category questions are answered
• It clears our pass/fail rubric: a pre-defined bar designed to send only high-quality submissions through to the judges

This is where the biggest cut happens, so making sure your submission is complete matters more than almost anything else you can do.

Stage 3: Judge scoring (October 6-7)

Now the apps reach the judges. For every submission assigned to them, each judge must at minimum:

• Read the entire description
• Watch at least two minutes of the required video
• Review all of the screenshots
• Score the app from 1 to 5 (5 being best) in each category

Judges are also welcome to download the app from the store and try it in person (it's encouraged), but not required. For this reason it is important to state in the first two minutes of the video what your app does, and what challenges you are targeting. This also applies to the Devpost submission description. Once everything is scored, each judge nominates their top two apps per category.

Even after all this filtering, the categories add up: across everything we run, close to 100 apps still make it to this final round of judging.

Stage 4: Final winner selection (October 8-9)

For each category, RevenueCat (or the category's sponsor) meets to deliberate and pick a first, second, and third place from the judges' nominees. Before anyone wins, we go back to the source material one more time, reading the full description, watching the video, and reviewing the screenshots, and at least one RevenueCat employee downloads the app to confirm it matches what the video shows.

On October 9th, winners are finalized, and we reach out to first-place winners to start coordinating their flights to App Growth Annual in NYC.

A note before you build

This process is subject to change, and it gets a little sharper every year. If anything here is unclear, or if you have feedback or questions, let us know, the whole reason we wrote this is so that no one walks away surprised by how their app was judged.