Collaboration Software Development:

Hire, Rent, or Buy?

hi@talky.io • talky.io

If you’re looking to build a communication app or add collaboration features to your application, you have some important decisions in front of you.

We’d like to share with you some of what we’ve learned in order to help you make the best decision.

If you read no further in this document, the most important thing to understand is that especially when it comes to collaboration applications, production quality software is infinitely harder than building a nice looking demo. We’ll explain why that’s the case later in this whitepaper. (See “Why is this stuff so complicated?”) But for now, let’s talk about your options.

You’ve got an overwhelming number of options—but they mostly fit into these buckets: hire, rent, or buy. This whitepaper will walk through why each of those options could be a great decision for you or could be a poor one.

The advice in this document is the result of years of consulting with numerous clients ranging from small startups to some of the biggest companies in the world, and we have helped clients who have decided to go each of these routes.

Hire Build your own team of experts and get crackin’ on the full stack!

Hiring is the first option most companies consider when looking to build these kinds of products and features. And it’s a good option for many companies for good reason.

Hands down, hiring is the least expensive route by measure of hourly cost. But it’s of course much more complex than that.

Employees are not like water faucets that can be nonchalantly turned on and off. It’s impossible to know how long it will take to acquire the right talent, let alone onboard them and form a team—and then you’ve got to retain them and manage them. And once they ship the product, you’ll need to keep them in order to maintain and improve the product. All of that is expensive, especially when you looking out over the course of years.

But especially when you’re talking about specialized capabilities, it’s very important to decide whether the most effective choice is hiring someone as an employee.

In some circumstances, it is paramount for you to hire in-house expertise, but in many cases, you would do well to invest elsewhere.

The approach:

• Open job postings for a combination of design, frontend, backend, architecture, and ops roles.

• Recruit developers you need.

• Retain them against constantly being recruited by other new startups and enterprises.

Estimated cost:

• Development: $350,000+ annual salaries and benefits, plus management staff expenses

• Infrastructure: $1000/month per 300 concurrent users (depending on technology choices and media types used)

• 3 year total cost of ownership: $1,000,000+ (not counting infrastructure costs)

Pros:

• Cheapest option on an hourly basis.

• You’ll have absolute control over every detail.

• Hiring great people and building a team is an extremely rewarding process.

• You know exactly what you’re getting every single day and have the ability to change everything about it if you don’t like it.

Cons:

• Slowest time to market of available options.

• Most expensive option over the long run.

• Highest degree of potential volatility among available options.

• Recruiting is hard and may be costly. A lot of demand for this kind of expertise means retaining developers is hard, too.

• Onboarding an employee can take a significant amount of time—often a minimum of 1-3 months and in some cases 6 months or even a year. And helping a set of people to become an effectively collaborating team is difficult.

• This is a new and rapidly growing field—it’s difficult to assess candidates unless you already have an expert on board, and this may also make managing them difficult.

• Realtime applications are hard to design and build well.

• Deceptively easy for many developers to create demo-quality work, infinitely harder to ship dependable production quality.

• High likelihood of engineers attempting to “reinvent the wheel” based on demo-quality knowledge instead of using tried-and-true approaches, which rely on knowledge of established protocols.

• If your application requires HIPAA compliance, this may require additional knowledge that isn’t available in-house.

• Your team will need to stay up on top of browser bugs and inconsistencies that have

been commonplace in these technologies.

Recommended approach to mitigate risks / cons:

• We recommend a standards-based approach to development as opposed to reinventing the wheel. We have helped companies who have “rolled their own” collaboration backends to move to a more stable, consistent, and standards based approach using established signaling protocols.

• You can choose a “rent” approach for the backend and make your options simpler, though this still will require hiring or contracting to build the frontend.

• Contract with experts for consulting, architecture, and development assistance with an experienced team can help maximize the value of your hired team’s efforts. Talky’s team has provided this kind of consulting to enterprises like AT&T, Microsoft, and Google, and to startups like Precision Nutrition and Flatiron School, and even to businesses like Interactive Intelligence who have built their business around collaboration.

Who it’s best for:

• Companies looking to create new technology in the communication/collaboration space in order to be acquired by a larger company.

• Startups and non-profits comfortable with shipping limited, demo-quality software.

• Well-funded startups who are focused on collaboration as their central feature who can afford to take their time to deliver a quality product.

• Enterprises where this is the core

business value.

Who it’s the wrong choice for:

• Companies looking to add collaboration functionality to an existing application.

• Companies who want the fastest time to market.

• Companies who want to limit their longterm operational expense.

How Talky can help:

• We’d love to help you build your team with qualified engineers, and can assist in your interviewing process.

• Talky offers CTO as a Service and support plans— we can advise your team on proven approaches to solve the hard problems and serve as a guide along the way.

• You might choose to use some of our well-established open source tools, listed on GitHub at github.com/andyet and github.com/otalk. We also offer training on using these tools.

• We publish blog posts and whitepapers. You can keep in touch with what we publish by subscribing to our newsletter, &you.

We’d be happy to have a quick free 15 minute consultation where you can find out if we’d be useful to your project. Drop us a line!

Rent Use a popular service for the collaboration backend and hire a team to build just the apps.

“Platform as a Service” (PaaS) products are a core part of modern application infrastructure. They can save you a lot of money and your dev team can be up and running in no time. You’re benefiting from someone else being responsible for the backend of your application, so you can focus on the experience.

However, PaaS options are not the best choice in all cases—and not all PaaS offerings are created equal. It can be especially nerve-wracking to lock your product into one vendor in an industry with so much volatility. A high percentage of Collaboration PaaS services tend to get “acqhired” and the services get shut down, requiring a transition to another service which can mean considerably more time and money. That said, there are some vendors (like TokBox) that we can expect to be around for quite some time.

One important thing to note: with collaboration software, both the backend and the frontend are very complex undertakings—a PaaS service just gets you halfway there. Building a production-quality realtime application is not something the average developer is comfortable and experienced with.

The approach:

• Choose a collaboration backend service

• Open job postings for a combination of design and frontend roles with people skilled at building advanced realtime applications.

• Recruit developers you need.

• Retain them against constantly being recruited by other new startups and enterprises.

Estimated cost:

• Development: $100-250K/year

• Infrastructure costs vary considerably based on your number of users and their usage:

› TokBox’s base price is $50 for the first 10,000 minutes and $0.00475 after that, with a slight discount as you increase in volume.

› 100 users participating in five 20-minute calls a month will use your first 10,000 minutes.

› 1000 users who participate in calls for 20 minutes a day, your monthly cost is roughly $2000.

› 1000 users who spend a third of each workday using collaboration features, your monthly cost is roughly $15,000.

• 3 year total cost of ownership: $500,000+ (not counting infrastructure costs).

Pros:

• Super cheap to get started.

• No need to worry about hiring backend developers, realtime system architects, or system operations support.

• Just need to deliver the frontend web and mobile applications.

• Complete control over user interface design.

• At no additional expense, you’ll get access to any improvements to the PaaS backend

or new features.

Cons:

• Can be prohibitively expensive depending on your business model and level of usage. If you have a lot of users or are expecting daily usage by them, the numbers add up quickly.

• Locked in to the vendor—realtime Platform as a Service offerings have been bought and shut down left and right over the last few years. And anytime you’re locked into a vendor, you’re subject to potential rate increases.

• You’re still only partway there. If you’re trying to create a great user experience, developing the clients will be just as hard as the backend—you’ve not simplified everything, just eliminated half of the work.

• Realtime application development is hard, and while most developers are capable of delivering demo-quality realtime software, performant and production reliability realtime apps require more expertise.

• Bound to the cloud—minimal flexibility to move to your own server if the security needs of you or your customers require it.

Recommended approach to mitigate risks / cons:

• We recommend TokBox for a few reasons:

› One of the longest established services built on quality engineering.

› They’ve already been acquired—by Telefonica several years ago. That seems to have been a positive acquisition for the team and stable partnership.

• Know what your projected usage is going to be, and understand how the pricing works. (For example: one “minute” means one *user* minute—so if you have 3 people in a 10 minute conversation, you’ve used 30 minutes.)

• If your team isn’t experienced with building realtime applications, you can hire a veteran consultancy (like t) to build the hard parts of the frontend.

How Talky can help:

We have worked with several of the popular PaaS offerings, and in most cases we can develop your application more quickly and less expensively than hiring full-time team members to build these features.

Who it’s best for:

• Companies who are never going to need more than a limited amount of usage.

• Companies who have a clear revenue model and aren’t intimidated by potentially high operating expenses when they hit scale.

• Companies just dipping their toes in.

• Companies eager to just start building (literally this minute!) without wanting to think about the backend.

Who it’s the wrong choice for:

• Companies planning on significant numbers of users, particularly if you expect those users to spend a lot of time using collaboration features.

• Companies who don’t want to build the client portion of their application or hire out that work.

Buy Choose an already built option covering backend, frontend, mobile, scaling, operations, and maintenance.

One more approach worth considering is buying a complete solution which covers all your needs and allows for rapid integration into your product (or which provides the foundation of your product).

One of the challenges with the “Hire” approach is you have full responsibility for some extremely complex pieces of code, which can make it extremely costly and perhaps take a long time to replace expertise capable of delivering and maintaining features built on these technologies.

The “buy” approach allows you to not only offload the responsibility for maintaining that expertise, but also deliver these features much faster than if you were to hire for these positions.

Ideally the solution chosen should allow for:

• The potential for deep integration that allows for customized look and feel as well as seamless integration into the user experience.

• A path for control of intellectual property rights.

• The option to run the service on your own infrastructure.

• Options which allow for a service level agreement and support plans.

• Source code escrow in order to protect your investment.

Talky Core is an example of an end-to-end realtime collaboration solution which meets each of these ideals.

The approach:

• Choose an end-to-end solution.

• Collaborate with the solution providers to integrate into your application.

Cost:

• Development: between a few days and a few weeks.

• Licensing: One-time $20-90K license (based on features), plus an optional SLA/support license renewable yearly ($5-30K).

• Infrastructure: Low marginal server costs based on your estimated usage, or pay an additional license fee and run everything on your own servers.

• 3 year total cost of ownership: $40-150K (not counting infrastructure costs).

Pros:

• Depending on approach, can avoid vendor lock-in. (Talky Core, for example, is built on open standards.)

• Fastest time to market for collaboration features.

• Proven backend—Talky Core, for example, is based on Talky, one of longest running and most popular collaboration tools of its kind.

• Most cost-effective option available based on total cost of ownership.

• Frontend user experience entirely complete and ready to be added.

• Customizable interface.

• Ability to transition to running on your own servers (for a one-time installation fee).

• At no additional expense in some circumstances, you can get access to any improvements to both the backend and the frontend.

• A team of proven veterans are responsible for these feature and working to keep updated and compatible with changes in the spec and nimbly responding to browser bugs.

Cons:

• Requires an up-front capital investment. (In some circumstances, we may be able to arrange for payment options to reduce the up-front expense.)

• Will not gain organizational knowledge of the underlying technology.

• Control of user experience is limited to the available options of the solution.

Where to go to get started:

Just reach out to us at hi@talky.io. We’d love to help you evaluate whether Talky Core is a good fit for your needs.

Who it’s best for:

• Companies who are comfortable making an upfront investment in order to get to market faster and reduce their total cost of ownership.

• Companies eager for quick, reliable delivery of collaboration features.

Who it’s the wrong choice for:

• Companies who cannot afford to make an up-front investment.

• Companies building their value around their expertise in the underlying technology itself, rather than the features.

HIRE RENT BUY

TIME TO MARKET Year Months Days or weeks

DEVELOPMENT COSTS

3 year total cost of ownership

$1,000,000+ $500,000+ $40,000 – $150,000

$350,000+ in annual developer salaries and benefits, not counting management

$100,000–$250,000+ in annual developer salaries and benefits, not counting management

Up-front investment plus support contract

INFRASTRUCTURE COSTS

Monthly fees $1,000+ $50 to $20,000+ per month $300+ per month

Monthly operational staff cost

$10,000+ 0 0

How infrastructure is billed

Pay for the servers you use, plus operations team to manage them

Pay per minute of usage with very profitable markup

Pay for the servers you use, plus a margin to cover operational support

Can run on own servers Yes No Yes

INTERFACE

Control of UX Total control Control of client Customizable UI

Implementing / maintaining UI

Fully responsible Fully responsible Included

Dealing with long list of edge cases

Fully responsible Partially responsible Included

MAINTENANCE Fully responsible for backend and client(s)

Fully responsible for client(s) Included

PROS • Cheapest option hourly • Absolute control of all aspects of

the user experience • Own the technology stack

• Super cheap to get started • No need to worry about hiring

realtime systems architects or operations staff

• Offload responsibility for the hardest parts

• Fastest time to market • Cheapest option based on total

cost of ownership • No need to hire expensive/

difficult to retain experts in underlying technology

• No vendor lock-in

CONS • Slowest time to market • Highest total cost of ownership • Difficulty recruiting/retaining/

onboarding • Responsible for developing,

maintaining, and supporting all the hard parts

• Married to the vendor • Infrastructure costs can become

expensive, depending on business model

• Still need to invest time and money into building features

• Requires up-front investment • Will not gain organizational

knowledge of underlying technologies

BEST FOR • Well-funded companies looking to create underlying technology

• Startups comfortable with prototype-quality software

• Companies determined to invest a lot of time to deliver a unique and deeply considered custom user experience

• Companies experimenting and prototyping

• Companies with a clear revenue model that can scale with usage fees

• Companies eager to start building without worrying about the backend

• Companies comfortable in making an upfront investment in order to get to market quickly and reduce total cost of ownership

• Companies wanting good, reliable delivery of collaboration features

NOT FOR • Companies looking to add collaboration features to an existing application

• Companies focused on fast time to market

• Companies wanting to limit long-term expense and exposure

• Companies who don’t want to building/maintain the client portion of their app

• Companies who want the potential of running on own infrastructure

• Companies who can’t afford to make an up-front investment

• Companies building their value around the underlying technology (like PaaS companies)

Why is this stuff so complicated?

In the past, realtime collaboration applications required downloadable plugins or dedicated thick clients, along with in-depth knowledge of technologies such as SIP that were unfamiliar to mainstream developers.

With the introduction of the WebRTC standard, it has become easier than ever to build realtime media applications. However, many challenges remain. We want to share Talky’s experience with building such applications, including experience gained building and operating our free video conferencing service and numerous customer projects.

This should be easy...

That’s what we thought, too. In 2013, excited about the possibilities of WebRTC, we wrote a JavaScript library called SimpleWebRTC and a socket.io signaling server called SignalMaster, then proceeded to launch a video conferencing demo called conversat.io (which later became Talky).

It worked great! Most of the time. Except when it didn’t - because of NATs and firewalls, browser incompatibilities, WebRTC API changes, and a dozen other things.

In the end, the problems boiled down to this: realtime applications are annoyingly complex and really, really hard to get right.

You’re probably reading this whitepaper because you, too, are excited about the possibilities. And that’s great! Unfortunately, it’s difficult to go from a mostly-working demo to a production-quality service that people can depend on every day.

You can spend a lot of time and effort getting it mostly right—but that’s not good enough. These days, people’s impatience with software that doesn’t work is high. With realtime communication tools, that impatience is even higher.

Building realtime features requires an understanding of distributed computing, messaging protocols, signaling semantics, and media streaming. Creating a solid user experience requires

knowledge of advanced JavaScript applications that goes well beyond the requirements of most web apps. And operationalizing an always-on service requires hand-on experience with unfamiliar infrastructure like signaling servers, media relays, and selective forwarding units.

Enthusiasm is not enough. Your users and your business might not have the patience for you to deliver these features. Learning all the hard lessons on your own can be expensive and risky. Let’s see what some of those lessons are.

We’ve learned firsthand just how hard it really is

With a realtime application, we’ve learned from experience how difficult and expensive it is to do it yourself—even if you’re well-versed in building these kinds of applications.

For many years, our team has built standards-based realtime applications. But for quite some time, the developer experience when creating these applications was poor, and working with the existing tools required knowledge and understanding of those standards—which meant junior developers couldn’t contribute without an intricate knowledge of complex protocols.

As a result, there came a point several years ago when we decided to build a realtime chat application entirely from scratch—without relying on these standards. It had a very simple target feature set. Also understand that members of our team had been building these kinds of applications for many years and several have helped author and improve Internet standards for chat.

What our veteran team discovered is that even though we had an enormous amount of experience and knowledge to draw on, accurately implementing around all the necessary edge cases took our veteran team over a year. Even two years in, we still periodically found nasty edge case bugs that took weeks to even be able to identify and repeat—let alone fix!

If you’re building a realtime application, we can tell you from painful experience: you really don’t want to reinvent the wheel. And the axle. And the drive shaft. And the transmission. And the clutch. And a million other necessary parts.

But you don’t have to take our word for it. In this talk from &yet’s 2011 conference on realtime technologies, veteran software developer Leah Culver, founder of realtime chat app Grove.io, talks about just how hard chat is.

99% of chat is magical kittens and rainbows and something you can probably build in a week. So if you’re thinking about building chat, go for it and build it adhoc in a weekend and 99% of it is awesome! Then there’s that 1% of chat which is basically just horrible. 1% of chat is actually really really horrible.

The worst part of chat is presence. Passing messages is easy, its knowing when people come online, and worse than that, knowing when they go offline.

And that’s just something as simple as text chat! When you begin talking about multiparty audio/video, you’re adding a whole new level of difficulty. Realtime audio/video signaling is entirely based around that 1% of presence!

As for the rest of the story for our team? Eventually, we went back to well-trodden and long-proven paths (such as XMPP) and we’ve been quite pleased with the results—but only because we’ve invested massive amounts of time and money in building tools that enable us to rapidly build great and stable features. And the developer experience with these tools is a world ahead of where it was years ago.

So why is it so complex?

There are many forms of both hard and messy complexity involved in building a realtime communication system.

The following are just a few examples and are by no means exhaustive.

• Audio, video, screen sharing, and other form of realtime media are extremely difficult to get right.

› Signaling to set up and manage sessions involves codec selection, IP address sharing, complex state machines, etc. Very few developers are able to define all the right semantics on their own.

› NAT traversal requires running STUN and TURN servers on all the correct ports, and if you don’t do that then media simply doesn’t flow. Even a 1% call failure rate will annoy your users.

› UX for audio and video is hard (even just things like active speaker detection and notifications).

› Multiparty audio and video requires that you run a selective forwarding unit or a multipoint control unit, and integrate that with a conference focus so that there is coordination between the signaling path and the media path.

› Ops is a significant challenge. Web ops folks don’t know about TURN servers and video bridges and the like. Major optimizations (e.g., TCP/UDP/OS tunings) are needed to offer any kind of reasonable scale.

• Text chat sounds easy, but it’s not just basic messages.

› There are UX and routing differences between 1:1 and multiparty messages.

› Storing and paging through history for “infinite scrollback” is not straightforward.

› Users want keyword matching (e.g., notification of @-mentions across all channels), inline images, last message correction, formatted messages, autocompletion of names and emojis, and a host of other features.

› Internationalization might seem unnecessary until you realize that emoji support requires good Unicode support.

• Realtime application development is just different.

› Events can fire at any time.

› Notifications and messages need to be handled in the proper order.

› State machines are complicated.

› Delivering on both web and mobile is not easy.

› There are huge difficulties with browser differences and incompatibilities, new browser versions breaking existing apps, etc.

• Security and privacy are even harder when it comes to realtime.

› It’s necessary to secure both the signaling path and the media path.

› Selective forwarding units and multipoint control units act as termination endpoints for encryption.

› Authentication of users is critically important (e.g., you might need custom authentication methods).

› Do you even want to know what DTLS-SRTP is? We didn’t think so.

Conclusion

In evaluating your options, we really like Dan McKinley’s concept of innovation tokens as a way to think this through:

Let’s say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while. You might get a few more after you achieve a certain level of stability and maturity, but the general tendency is to overestimate the contents of your wallet.

Understand that the collaboration technology space is full of multiple things which take spending innovation tokens, as the “Why is it so complicated?” section demonstrates.

If you’re aiming to build a company which offers the underlying technology that empowers collaboration software, it makes great sense to choose the “Hire” option and invest your innovation tokens (and money) into the problems to be solved in that space. (In fact, you would be unwise not to do so.)

If you’re wanting to have deep control of the interface and put your innovation tokens into the frontend while reducing your initial spend, the “Rent” approach is a good option.

The “Buy” option empowers your team to spend the least number of innovation tokens, and very likely has the least total cost of ownership of available options. This is especially the best option for companies looking to add collaboration as a feature and not a core technical competency.

Whichever option you think might be best for you, we’d love to talk to you and offer a free consultation to help you make your decision.

Get in touch!

hi@talky.io • talky.io