The Illusion of Objectivity

When you launch a directory, the premise seems straightforward: list the good ones, exclude the bad ones. Simple, right?

Nope. Because “good” and “bad” are subjective, and the moment you start making judgment calls, you’re no longer a neutral platform. You’re an arbiter. And arbiters get blamed by everyone, for everything.

Here’s the uncomfortable truth: what works for one person may not work for another. A therapist who challenges their clients might be exactly what one person needs and completely wrong for someone else. A contractor who works slowly and methodically might frustrate a client on a deadline but delight someone who values craftsmanship over speed. A restaurant that seasons aggressively will have fans and detractors in equal measure.

This isn’t a bug in the system. It’s the system.

The Restaurant Review Problem

Consider two reviews for the same restaurant, posted on the same day:

Review 1: “Don’t eat here! The soup was way too salty. I could barely finish it.”

Review 2: “Don’t eat here! The soup was so bland it needed salt. Tasteless.”

Same chef. Same day. Same soup. Opposite complaints. Both one-star reviews.

Who’s right?

The answer, of course, is both of them and neither of them. They’re not wrong about their experience. They’re just two people with different experiences because they have different palates. The soup isn’t objectively good or bad. It’s a matter of fit for that specific customer.

Now imagine you’re running a directory of restaurants, and both reviewers demand you remove the listing for the restaurant for serving awful food. What do you do? If you remove it, you’re saying the restaurant is objectively bad. If you don’t, you’re ignoring “legitimate” complaints. Either way, someone’s unhappy.

And this is just soup. Imagine the stakes when you’re dealing with something that actually matters.

The High-Stakes Version

Let’s say you’re running a directory of therapists, lawyers, financial advisors, or medical professionals. The stakes are infinitely higher. These are services that impact people’s mental health, their families, their futures.

The same dynamic applies, but it’s amplified. A therapist who “pushes too hard” on one client might be “finally someone who challenged me” for another. A lawyer who “didn’t fight hard enough” might have been practicing restraint that saved the client money. A financial advisor who “was too conservative” might have protected someone else from catastrophic losses.

When someone complains about a high-stakes service provider, they’re often processing a difficult experience. That’s valid. Their feelings are real. But their experience isn’t necessarily representative, and it definitely isn’t objective.

The Data Problem

Here’s where it gets really uncomfortable: the data you’re working with is fundamentally skewed.

Think about your own behavior. When was the last time you went out of your way to leave a positive review for a service that simply worked? That did what it was supposed to do? That met your expectations without drama?

Probably not recently. Maybe never.

Now think about the last time you were motivated to leave a negative review. A flight got canceled. A package arrived damaged. A customer service rep was rude. That energy is different, isn’t it? That’s “I need to warn others” energy. That’s “someone needs to know about this” energy.

This is human nature. People love to complain about bad experiences but rarely talk about good ones thus getting good reviews is significantly more difficult than getting bad ones. The psychology is well-documented: negative experiences activate our threat-detection systems and motivate action and a desire to protect people from potential harm. Positive experiences just fade into the background of life.

So when you look at the feedback for any listing in your directory, you’re not seeing a representative sample. You’re seeing who bothered to write in, and that population skews heavily toward the dissatisfied.

As a directory provider, ask yourself this: How many complaints does it take to blacklist someone? Or, as a consumer, how many bad reviews before you don’t consider the provider? Three? Five? Ten? Whatever number you pick, ask yourself a follow-up question: how many satisfied clients never bothered to provide feedback? Could be dozens. Could be hundreds. You’ll never know, because they had no reason to tell you anything.

This is why some businesses try to incentivize positive reviews, and why the FTC and other regulatory bodies have deemed the practice of buying positive or negative reviews as illegal so avoid that at all costs. The temptation to do so exists precisely because of this data imbalance and thus highlights the core problem: you’re making decisions based on incomplete data, and the voices you’re hearing are the loudest ones, not necessarily the most representative ones.

The Legal Dimension

Beyond the ethical complexity, there’s a practical concern: liability.

When you position yourself as the arbiter of who belongs in a directory, you’re taking on responsibility for that judgment. If you remove someone, you’re making an implicit statement about their quality or trustworthiness. If that statement is wrong, or even if it’s just perceived as wrong, you could find yourself on the receiving end of a defamation claim.

Conversely, if you keep someone listed who later causes harm, you might face questions about why you didn’t act on the complaints you received.

This isn’t hypothetical. Directory operators have been sued for both including and excluding listings. The legal landscape varies by jurisdiction, but the principle is consistent: the moment you start making editorial judgments, you lose some of the protections that pure platforms enjoy.

We’re not lawyers, and this isn’t legal advice. But we’ve seen enough to know that “we’ll just remove the bad ones” is not a strategy that accounts for the real-world complexity of running a directory.

The Social Cost

Even if you never face legal action, there’s a social cost to being the arbiter.

Communities talk and industries have networks. When you remove someone from a directory, word gets around. Sometimes that’s fine, maybe the person really was problematic and everyone knows it. But sometimes you’ve just made an enemy, and that enemy has friends, and those friends have platforms. Worse, other members start wondering: “If it happened to them, what’s to say it can’t happen to me?”

The same is true when you refuse to remove someone. The complainant expected you to take their side. You didn’t. Now they’re telling everyone that your directory protects bad actors, and other members start wondering if their complaints would be taken seriously either.

Either way, you’re in the middle of a dispute you didn’t create and you can’t resolve. And either way, you’ve given people a reason to question whether they want to be part of your community at all. Being the arbiter means being the target.

How to Avoid the Pitfalls of Running an Online Directory

We’re not saying directories are impossible or that you shouldn’t build one. (In fact, we’re saying the opposite, and we’ll get to that later.) We’re saying you should go in with clear eyes about what you’re taking on.

Here are some principles we’ve learned:

No secret directories. This goes both ways. You can’t list someone in private because they don’t know they’re being officially listed, and your listing might reach the wrong audience. And you can’t blacklist someone in private because they have no opportunity to defend themselves. If the company or provider can’t see what’s being said about them, you shouldn’t be saying it in an official capacity. To be clear: you’re always free to share your personal opinion about a provider as an individual. That’s just conversation. But when you’re consolidating reviews and providing a platform for others to do the same, doing it behind closed doors changes the nature of what you’re doing. Transparency isn’t just good ethics, it’s good protection. Hidden lists invite accusations of bias, backroom deals, and worse.

Transparency over curation. Instead of deciding who’s “good enough” to be listed, consider being transparent about your inclusion criteria and letting users make their own judgments. “We list all licensed providers in this field” is a defensible position. “We list only the good ones” is not.

Let users see the data. If you collect reviews or feedback, consider showing all of it, positive and negative, rather than making editorial decisions about what counts. Users are smart and they can read between the lines. They can recognize when one negative review is an outlier versus a pattern.

Here’s an important distinction: on a directory site where users post their own reviews, everyone is responsible for their own posting. It’s their experience, their feelings, their words. That’s defensible. But when you consolidate those individual experiences into a judgment (for instance, when you decide that three complaints means someone gets removed) that’s where the issues start. You’ve taken ownership of a conclusion that no individual reviewer was making.

Be clear about what you’re not. A directory is not a recommendation engine unless you explicitly make it one. Make sure your users understand what being listed means and what it doesn’t. And if it is a recommendation engine, you need to be absolutely clear about your criteria for inclusion. What qualifies someone? What disqualifies them? Write it down and publish it clearly in plain language. Vague standards invite vague accusations, and vague accusations are very difficult to defend.

Have a policy and stick to it. If you’re going to remove listings under certain circumstances, define those circumstances in advance. “Three complaints and you’re out” is a terrible policy, but at least it’s a policy. “We’ll use our judgment” invites challenges to every judgment you make, and “You may disappear overnight at our whim” is dangerous for your members and for you.

Consider the source. Not all feedback is created equal. Anonymous complaints carry different weight than signed ones. Broad accusations are different from specific, documented concerns. If you’re going to act on feedback, have a framework for evaluating it.

Let businesses respond. This one’s a trade-off. Allowing listed businesses to reply to reviews can turn into a shitshow of arguments, and we’ve seen it happen. But it also gives the business a chance to defend themselves, show they’ve changed, or demonstrate improvement. A one-star review from 2019 hits differently when there’s a response saying “You’re right, we struggled early on. Here’s what we’ve done since.” It’s messy, but it’s fairer. And if they write back with something attacking the customer? Well, that tells future customers all they need to know about the business.

Consider timing. Reviews need to be taken in the context of when they were written. Having worked with a number of restaurants, we know that soft openings can be rough. The kitchen is still finding its rhythm, the staff is still learning the menu, systems are still being debugged. A flood of one-star reviews from opening week doesn’t tell you much about what the restaurant is like six months later. If your directory displays reviews, consider showing the date prominently or even weighting or sorting older reviews differently. Trends matter more than blips.

Document everything. If you do end up in a dispute, social or legal, you’ll want a record of what you knew, when you knew it, and why you made the decisions you made.

The Uncomfortable Truth

Here’s the thing nobody tells you when you launch a directory: the hardest part isn’t building the database or designing the interface or marketing the platform. The hardest part is the moment someone asks you to make a judgment call, and you realize there’s no right answer.

You’re not judging quality. You’re judging fit. And fit is personal.

The therapist who was “too pushy” for one client might be “finally, someone who gets it” for another. The contractor who was “too slow” might be “thorough and careful” to someone else. The restaurant that’s “too salty” might be “perfectly seasoned” to the next customer.

None of them are wrong. They just needed different things.

And that’s the dilemma. When you become an arbiter, you’re not just managing a database. You’re navigating an impossible landscape of subjective experiences, incomplete data, legal exposure, and social dynamics. Every decision you make will be wrong to someone.

The question isn’t whether you can avoid this complexity. You can’t. The question is whether you’re prepared for it, and whether the value your directory provides is worth the cost of being in that position.

Why We Still Recommend Building Communities

If you’ve made it this far, you might be wondering why anyone would bother building a directory at all. We get it. We just spent a lot of words telling you everything that can go wrong.

But here’s the thing: we still think you should do it.

When you build on someone else’s platform, you’re at the mercy of search engine algorithm changes, AI-driven content decisions, and the ever-present risk of social media deplatforming. Your reach can evaporate overnight because someone in Silicon Valley tweaked a setting. Your email list, your membership site, your directory? Those are yours. No one can take them away.

Communities like these are sticky. They provide enormous value to your members and to your organization and keep customers engaged and coming back for more. They create loyalty that social media followers simply can’t match. We believe in them, and we help clients build them all the time.

But – and this is the point of this entire post – you need to know what you’re getting into. You need policies. You need to think through your inclusion and exclusion criteria before the first angry email lands in your inbox. And you need to spell all of this out clearly in your privacy policy, terms of service, and community guidelines.

This isn’t just about protecting yourself legally, though it does that. It’s about setting expectations, so your members understand what they’re joining and what the rules are. Clarity up front prevents conflict later.

We’ve done this before. We know where the landmines are. And we’d rather help you build it right than watch you learn these lessons the hard way. If you don’t have policies in place yet, that’s where to start, and we’re happy to help you think through what yours should say.