How To Evaluate Your Systems Integrator

System integrators (SIs) are the consultants you use to implement and integrate software systems for your business.

The term is a bit of a misnomer because they do more than just “integrate” systems–they configure, test, and support them for you too. They can also play an overarching strategy role.

Systems integrators are common in modern IT. Working with them to accomplish your technology goals is mostly unavoidable (at least for now), so you should make an effort to evaluate how your SIs are doing and hold them to high standards.

This post will help you think through how you can evaluate a systems integrator. It applies to evaluating a current SI, and it can also be used when evaluating new ones.

Why are you using a Systems Integrator at all?

Before you start evaluating a current or potential SI, you should start by asking yourself why you even want to use an SI. 

Understanding why will help you think through if the tradeoffs that come with an SI are worth it. It’ll also help you sharpen the lens through which you evaluate them.

The following are usually good reasons to consider an SI:

• Speed is more important than cost and the SI can deliver fast
• You don’t have the in-house skillset and you can’t hire it fast enough
• You don’t have enough in-house people with the necessary skllset
• You don’t know how to recruit the needed talent or can’t afford to
• You don’t view the skillset as mission critical enough to hire for it
• You have only temporary demand for the work

These aren’t the only reasons you should (or shouldn’t) work with an SI, but it’s a reasonable starting point to check your thinking.

Why You Should Evaluate Systems Integrators

SIs will be a big part of your team and a big influence on whether you successfully achieve your business goals. You should take time to evaluate how any current or potential SIs will work for you.

But, to break that down further, here are a few reasons why evaluating SIs is important.

Systems integrators are expensive.

Systems integrators are not cheap. Fundamentally, you are paying for peoples’ time. People are expensive. 

If you want the best of the best SIs, expect to really pay for them, but even more modestly priced SIs are still going to be expensive. 

With such a big expense, it’s critical that you track whether you’re getting appropriate value in return for the cost.

Measuring an SI’s performance is hard.

Measuring a systems integrator’s performance is difficult, which makes that cost-value assessment even more complicated.

If an SI is negligent or clearly bad, you’ll know it. If they are 1% exceptional you probably will too.

Most fall in the mushy middle where it’s not entirely clear how well they are performing. This is the danger zone–where you may be on the losing side of cost-value without even realizing it.

The SI you pick is a huge decision.

When you make the decision to work with a specific systems integrator, you’re entering a long term (maybe permanent) relationship with a team that will have a big impact on your business.

Big decisions like this can have big upsides, but if you get it wrong, the decision can have a huge negative impact (blown budgets, delays, etc.). It must be managed, which means it must be measured.

How Systems Integrators Make Money

To evaluate the performance of a systems integrator, it helps to understand how they make money. This section will dive into the unit economics of that business model, including some of the costs you may not realize are there.

Costs & Profit Margins

At the simplest level, an SI seeks to sell you someone else’s time for money. Ideally for them, they’ll charge you more than they pay that person.

Let’s look at the unit economics a bit closer.

Unit Economics

For a simple example, if an SI pays an employee $50/hour and then bills that employee as a consultant to you at $80 an hour, they make $30 on every hour.

In reality, few consultants bill 100% of the working hours in a year. You have to subtract vacation time, bench time (more on this shortly), and non-billable time (internal meetings, etc.) from that $30 of profit margin. You also have to consider salaried versus hourly employees, benefits for full time employees, and support overhead.

In effect, systems integrators are simply performing labor arbitrage, trying to pay for expenses that don’t generate revenue (non-billable overhead) with the markup they bill to you, ideally with some left over as profit.

Every company is different, but you’ll usually see an SI trying to manage to metrics like 70-80% utilization (consultants’ billable hour percentage) and 30-40% gross margins (project revenue minus project costs, mostly labor).

This is important to understand, because it is probably the biggest thing driving how an SI works, negotiates, and sells to you. They may be staffing people on your team unnecessarily or having them do double duty with another project, for instance.

Managing The Bench

One of the most important thing a consulting company like a systems integrator has to do is called “managing the bench”.

When a consultant is “on the bench”, it means they are being paid by their employer (the SI), but they aren’t deployed to any billable projects. In other words, their time costs the SI money, but their time isn’t being used to generate any revenue.

If you have too many people on the bench for too long, it usually leads to layoffs. If don’t have enough people on the bench, it makes it harder to take on new client work or absorb unforeseen scope changes for existing clients. Usually the former is a bigger problem for an SI.

Bench time is the inverse of the 70-80% utilization we mentioned earlier. In other words, on average, the SI will try to keep their consultant bench time to 20-30%.

This is important to understand. Your SI is very dedicated on keeping people off the bench–deploying them for billable work. Again, this might be why they are trying to bill a consultant to you. Do you really need that extra person or do they need them off the bench?

Hourly vs. Fixed Fees

There are a lot of ways SIs will structure projects, but most of the time it’ll be either as a “fixed fee” (sometimes called “fixed bid”) or “time and materials” (sometimes called “hourly”) projects.

These are two strategies for managing scope, timeline, and hours. The SI will deploy either strategy in order to set clear expectations with you, the client, and to establish a working relationship that protects them from spending too much time (cost) relative to what you’re paying them (revenue).

With a fixed fee, the project will be structured around certain requirements milestones and deliverables and they will be assigned a known, upfront cost. 

With an hourly project, you ring the register for every hour one of the SI’s consultant works on your project. Rates will vary by seniority and role. 

Neither is better than the other, but it’s useful to understand how the SI considers uses the strategies. It will speak to their incentives and help you understand why they ask certain questions, set certain boundaries, and generally behave.

This article gives a pretty good comparison of the two.

Most important to understand, though, either arrangement can result in cost overruns for you. And, the SI will usually charge you more before they eat internal cost overruns.

Hourly cost overruns occur when:

• The SI underestimated the amount of work it would take to deliver
• The scope of what needed delivered wasn’t complete or clear and it grew
• The balance of senior (expensive) roles to junior ones was wrong

Fixed fee cost overruns occur when:

• Scope changes add new functionality
• SIs price in excessive buffer to protect their internal costs
• The timeline, therefore continued overhead cost, extends unexpectedly

Typically an SI will choose fixed-fee when they are confident they can deliver with less labor (i.e. make a large profit) on the project. They'll also usually estimate some extra fluff in there to protect their margin. On the other hand when scope is loose, they'll usually bill hourly, assuming you'll have them do way more work than originally intended.

Project vs. Ongoing Services

SIs usually engage in two general formats: project work and ongoing work.

Projects are usually temporary and relatively short lived, though in the enterprise projects can be years long. They are work oriented around building, implementing, or changing software. There’s a tangible goal and when it’s achieved, the project is complete.

Ongoing services are bounded by timeline instead of scope. Support is usually a good example of the kind of work an SI will structure as an ongoing engagement. You pay a monthly retainer and get X hours of support under Y circumstances. The project is over when parties agree there are no further months.

In both project and ongoing services work, fixed and hourly billing strategies may be used.

Partnerships w/ ISVs

SIs also commonly make money via partnerships with Independent Software Vendors (ISVs). An ISV is the company that makes one of the software products that is being implemented and integrated. If you’re implementing a new ERP (say NetSuite) and working with an SI to get it done, Oracle (who makes NetSuite) is the ISV.

SIs usually get new work when someone buys a new software system that they have expertise in. So, naturally, SIs partner with the ISV who makes that software system. In doing so they get the following:

• Privileged knowledge of product features, documentation, and roadmap
• Sometimes, the opportunity to provide feedback
• Co-marketing assistance to sell services side-by-side with the system
• Referrals for business

The last one is key. The ISV often doesn’t have the capacity or maybe even the desire at all to take on the work of implementing or integrating the software they sell. Instead they refer that work to the SIs. How and when that happens is very different by ISV, but there’s usually a heavy dose of “offline” relationship management involved.

ISVs will also usually pay SIs for new customers too. If an SI registers a new lead with the ISV that turns into a software sale, there would be a kickback. Sometimes the SI acts as a “reseller” which is a deeper version of that relationship. This post dives into that more than we will here.

It’s important to know that you are the ISV’s customer and you are the SI’s customer. However, there is likely some kind of privileged relationship between the ISV and SI that you aren’t part of.

Recruiting & Placement

We aren’t going to spend a lot of time on this, but SIs also sometimes offer recruiting or placement services. The casual term “body shop” is often used. These firms aren’t really delivering projects or strategic value for clients. They are simply staffing people for specific roles with a rate markup.

Criteria For Evaluating Systems Integrators

How to precisely evaluate an SI is very dependent on your business, the nature of the work the SI does, and a host of other factors. Instead of being overly specific, this section will give you guidance on what a good SI and a not-so-good SI looks like.

Use these general guidelines to craft more specific criteria for your business.

What does an effective systems integrator look like?

There are common qualities you’ll tend to see in a systems integrator who is effective. They aren’t all required, nor do they always show up exactly as described below. But you’ll tend to see them more in better SIs.

• Quality Delivery: This one is probably the simplest. When you ask them to do work, they deliver high quality work that does what you asked.
• Rate and Resume Transparency: They are clear and open with who they staff for your project work, specifically their experience relative to what they charge you for the work.
• Clear Communication: They understand how to communicate progress and especially challenges during the project. 
• Effective Delivery Methodology: They run an intentional process that makes sure they get work done in a timely and incremental manner.
• Demonstrated History of Success: They can point to past successful projects and past happy clients–ideally clients that look a lot like you!
• ISV Referral: It’s not usually in the ISV’s best interest to refer a bad SI to you (it does happen though) so this can be a positive indicator.
• Cultural Overlap: They work how you work, talk how you talk, and generally get a long with your team. This all helpful, especially when you get to the hard parts of projects or need to have difficult conversations.

It Still Might Not Matter

Even if you find an SI that checks all these boxes, it’s no silver bullet. There are even downsides to using a really effective SI:

• Even the best SIs have projects go off the rails and incorporate hidden costs into their projects.
• Really good SIs are rare. If they know they’re good, then they will be expensive.
• Really good SIs also tend to be very busy and with all that demand, they are pushed to take on more work, potentially stretching their consultants’ bandwidth.
• Really good SIs tend to make a lot of money and therefore tend to get acquired by larger companies. When that happens, everything may change quickly.
• It’s easy to rely on a good SI, but then you’ve built really critical knowledge about your business in someone else’s company. When they leave, it walks out the door.

What does an ineffective systems integrator look like?

In some ways, an ineffective SI is just the inverse of the list above of effective SI traits. But, there are some nuances worth calling out. An ineffective SI will tend to have:

• Lack of clarity about prices and delivery method. They may even be setting these on the fly.
• Related to pricing, they may not be forthright with you about when they are using third-party contractors or off-shore labor.
• They maintain a strictly transactional relationship with the ISV and don’t seem to have or to make an effort to have a strategic relationship.
• They estimate or charge for excessive project overhead. Every project has some amount of project management, but if that’s getting close to 50% you should ask questions.
• When technical challenges arise or difficult project conversations need to be had, the SI is not effective having those conversations with you.
• They have high employee turnover.
• They aren’t really able to communicate to you what their long term goals are nor how they want to work with you for the long term.

All SIs have some amount of these, but if these are really showing up, they are good indicators that you have a questionable SI.

Conclusion: Trust But Verify

Systems integrators are an common part of the modern tech landscape. Many of them are great, even if still pricy. Much more of them are less than great.

It’s important to build a habit of evaluating these SIs, because they will play a key role in the success of your business. Letting the wrong SI in the door can cause years of strife.

Tired of dealing with SIs? At Doohickey AI, we're building the first AI-powered alternative. Join the waitlist to get started as soon as its ready.