How not to run a Hackathon. [Updated with response from Salesforce.] By Alicia Liu (Senior Software Engineer, Lift Worldwide)
A month ago, Salesforce announced that this year’s Dreamforce, their big splashy annual conference, was going to feature a hackathon with the biggest single prize ever—a cool $1 million dollars in cash. I didn’t think twice about the news. To me, it was just another marketing gimmick, like so many corporate-sponsored hackathons before it, to trick otherwise sane developers into working for free—with copious amounts of junk food and caffeine and the lure of dubious prizes—to develop apps on top of APIs and platforms they otherwise wouldn’t bother with.Corporate-sponsored hackathons are all-win for the sponsoring company. They get a big influx of fresh ideas, and developers working themselves to illness (not quite death, but a diet of soda and pizza, plus sleep deprivation and being sedentary for long periods is the definition of unhealthy), just to meet a crazy deadline. In return, the company puts up a couple prizes which is far far less than what it would cost them to get the same result in-house, paying developers fair wages, assuming they even have developers of the caliber that their hackathon might attract. On top of it all, it’s also a recruiting event for the company.
But a couple weeks after the initial announcement, Matt Thompson—a respected iOS developer and mobile lead at Heroku, a Salesforce acquisition—wrote a blog post about why people should sign up for this big hackathon. He laid out his arguments about developing for the enterprise, reasons I was more than familiar with since my first startup developed enterprise software and I have seen first hand just how bad the software is that most people have to use to do their jobs.
Matt also mentioned entering the hackathon was now free. This should actually have set off alarm bells, because anyone who was anywhere around the Moscone Center during Dreamforce, can see just how much money is poured into this conference. Why would they charge developers $99 for a chance to add value to Salesforce! To their credit, I believe developers who did pay the $99 to enter were reimbursed.
I started to mull it over and pinged my partner Ben about doing this hackathon together. Ben, like me, was quite skeptical about entering a corporate hackathon. The lure of the prizes at these events is never worth the effort one has to put into these projects. At traditional developer-run hackathons, you join because it’s a chance to work with new technologies in a setting where you can collaborate with others, and it’s damned exciting to build something and to do it under severe constraints. Developers love a good challenge. Winning is much more about the pride of having the best “hack” that could be developed in such a short amount of time—the prizes were just icing.
At corporate-sponsored hackathons, though, the goal is to have developers build on top of the sponsor’s platform and use their APIs. After all, having a platform is useless if there are no developers building for it. Enterprise software has a knack for having terrible usability and their APIs are rarely better. It’s difficult to attract good developers to build on top of shoddy platforms. But then companies figured out that they could get these developers under the guise of a hackathon. They hijacked the goodwill of developers who just want an excuse to build cool stuff.
But I was swayed by Matt’s blog post:
Honestly, if you’re reading this, your chances are probably pretty darn good. If I weren’t an employee (and thus legally prevented from entering), I’d be all over this.
Ben and I figured we can make a nice side project out it, and I had been looking to dive back into iOS anyway—having a deadline will give us some good motivation.
After nearly two weeks of evenings spent coding and designing, and one all-nighter, we submitted our app according to Salesforce’s extremely long and meticulous Rules. Rules which include (emphasis mine):
The application you or your team submits must: • be a mobile application • use the Salesforce Platform • have been developed solely as part of this Hackathon • be a completed working application at time of submission • be functional and comply with these Official Rules
We went by the Developer Zone at Dreamforce a half hour after the 6pm deadline to see if we could see the judging, but event staff were shutting things down and starting to kick people out! We thought that was weird, there wasn’t even anywhere to see all the submissions. At typical hackathons, this is usually the most exciting period. You get a chance to see what everyone else came up with, and hear what the judges think of the submissions, and get feedback about your app from judges and other participants.
At the Salesforce Hackathon, the picking of the five finalists was done completely in secret, with no clue about who did the judging, or what was evaluated.
Finally, later that evening, all the submissions were up on ChallengePost, the company that handled the submissions for the hackathon, and Ben and I started checking out the competition according to Salesforce’s judging criteria:
Innovation: Is it an amazing new scenario?
Business Value: Does the application create value? Does it save money, create opportunity, connect customers, partners, and employees?
User Experience: Does the app have a user interface that is gorgeous, easy to use, and fast?
Use of Salesforce Platform: Does the application take advantage of the features and capabilities of the Salesforce Platform?
Some apps were real contenders, but many apps did not appear to meet all the judging criteria. Then there were those that clearly did not even adhere to the rules, because a couple entries looked like companies who just repackaged their existing product for the hackathon! We hoped the judges, whoever they were, would see through those. The rules clearly state:
Participants may begin forming their teams and working on their entry (each, an “Entry” and, together, the “Entries”) as early as October 25, 2013. Each Entry shall consist of a mobile application with a link to the working source code, the application login credential, a written description and a brief demo video.
(They actually required the working source code to be zipped up and uploaded. A requirement included being able to compile and run your app.)
The next morning I saw a bland standard rejection email, which was sent just past 3am that night, specifically stating no judging feedback will be given. I quickly checked Testflight and our app data to see if it had been run. While Salesforce does say not all apps will be run, I thought we would at least pass the video stage and have someone actually check out our app, but the Testflight email with our build wasn’t even opened. (Not having your app run later turned out to be a wide-spread phenomenon, and some reported that their video was never even watched. We can’t tell if our video was viewed by judges or not.)
Having spent collectively over 150 hours working on something, giving up nights and weekends, and not even have someone actually look at our app left a distinctively sour taste, but hey, what the hell. Judges at hackathons can be of questionable quality and it’s not a rigorous competition. I was satisfied knowing we made a good effort and I was proud of what we achieved in two weeks.
I tried to see who the finalists were, but all the submissions on ChallengePost had completely disappeared overnight. Strange. I followed along on Twitter as the winners were announced, and was dismayed to see a tweet that said the winner was a former Salesforce employee. Could this be true?
While Salesforce rules say employees as of Sep 1, 2013 could not enter the hackathon, it appears people who were employed not just as recently as this year, but who is “Formerly an engineer at Salesforce for 9 years. Lead engineer on Salesforce Analytics. Tech lead for Custom Report Types and many other reporting features” sure are allowed to participate and win. And guess what their app was? A “tool that lets you easily create and edit [Salesforce] reports on a mobile device.”
I think it’s a bit of stretch that a reporting tool was the most innovative submission, but most "gorgeous"? C’mon. Within enterprise there is a ton of favoritism of the “you scratch my back, I’ll scratch yours” kind. (It’s no coincidence the iPad giveaways at industry conferences always land in the hands of a key contact at a customer or partner. As a company spending marketing dollars, you want to get your money’s worth!) It’s naive to think that Salesforce wouldn’t give a leg up to favored developer partners.
Ben and I were disappointed that this hackathon was no different, but what makes it even more egregious and outright wrong, is that the winners had demoed their app at Meetup on Oct 8, well before the hackathon even started, which is in violation of the rules.
If Salesforce doesn’t adhere to their own rules, there’s not much that can be done. It’s their event. But the way this hackathon was run leaves all the projects completely hidden away. Compare this to every other hackathon where the submitted projects are up on the website, people can check out other people’s demos, learn from others, and see for themselves what was good. It’s a place for developers to showcase what they built.
There were 149 submissions, which is not such a huge number that Salesforce could not handle more hands-on judging at such a hyped-up hackathon, especially considering how many employees worked this event. Other hackathons manage to do this and have way more teams up on stage actually present.
Finally, what I find most regrettable is all the developers who submitted their source code, in conjunction with this provision in the rules:
You also understand that we will not restrict work assignments of representatives who have had access to your Entry. By entering this Hackathon, you agree that use of information in our representatives unaided memories in the development or deployment of our products or services does not create liability for us;
And there you have it folks. Companies like Salesforce have hijacked hackathons to get free projects, apps, and ideas (and in this instance, code) out of developers in exchange for some cold pizza, without any of the spirit of what hackathons were originally supposed to be about.
If you’re a developer, don’t fall for one of these traps like I did.
Salesforce has finally responded to the allegations of cheating, which they have termed "questions about why the winning team was allowed to submit an app that may have used pre-existing code." They reference this answer they provided on their discussion board in response to a participant who asked for clarification about the rule "The application you or your team submits must have been developed solely as part of this Hackathon":
The intent of that provision is that submissions would not be apps that have been previously released or in development. Reusing code you may have written before is fine, provided that you were the author of that code, itdoesn’t comprise the majority of your app and its use does not violate any third party’s rights. You could modify an existing product to integrate with Salesforce and submit that, however you’d be judged on just that component, not the pre-existing product (emphasis mine).
From what I can tell, Upshot, the winning team, already had their product built before the hackathon was announced, so at most they made their app "mobile" for the purpose of this hackathon.
Upshot did not build a native app. For them, making it a mobile app meant that their app, by which I mean website, can now be (if it couldn’t be before) opened on a mobile device and look okay. Kind of. You can tell it’s not even done very well, because in the video the graph is still wider than the width of the phone screen, and they had to pinch the screen to see all of it. In four weeks, they couldn’t get the graph on the primary screen to resize properly? From their submission: “Our mobile web app is built using HTML5.” Maybe they should also have used some CSS3! (Sorry, had to get that jab in there.)
They also sped up the footage of the actual demo. I’ll give them the benefit of the doubt that it was done just to save on time, but it sure has the nice effect of making a mobile website appear faster than it actually is.
So according to the rules then, they should have been judged on their shoddy implementation of a responsive website. Pretty sure implementing a responsive CSS grid system doesn’t rank anywhere on any of the judging criteria.
This is all moot, though. The judges can all claim plausible deniability that they had no idea what was implemented before or after the hackathon started, and no one’s going to go through Upshot’s commit logs, so Salesforce is just going to hang on to their straw man and wait for this to blow over.
But I’ll leave you with this: There are obviously a whole bunch of Salesforce users who think this Upshot reporting tool is a godsend, because creating and editing reports is such a pain in Salesforce. Yet the CTO of Upshot was the "Tech lead for Custom Report Types and many other reporting features" at Salesforce for, presumably, multiple years. You’d think his job description would include making creating and editing reports not such a pain. If I was Salesforce, I’d be pissed an employee was so bad at his job he left a gaping hole in the product, then he leaves and starts a company that conveniently fills that hole? Yet, Salesforce gives him a million dollars. Either Salesforce is an investor in Upshot, or they’re dumb.
This post originally appeared on Medium. Photo credit: heckNY.org via Flickr. About the blogger: I build web and mobile apps, currently @liftapp. I love food, books, and adventure. @aliciatweet, http://alicialiu.net.