What the heck is going on over at Proton?

26 Jul 2024 by cthos
About 5 min

tl;dr I’m no longer recommending Proton services to anyone and have moved off to other services.

Right, so up until recently I was a happy Proton customer at the annual unlimited tier. They were missing some convenience features and the integration of a document editor into the Drive product was pretty great.

That is, until they decided to shove an LLM into their core product and upend their security model and break some trust by the way they rolled it out. Let’s take a quick look at their timeline / speedrun of adding LLM features (and Crypto wallet, but we’ll get there in a sec).

1. June 5th, 2024 - Proton releases the results of their 2024 community survey
2. June 17th, 2024 - Proton announces transitioning to a non-profit structure
3. July 8th, 2024 - Eamonn Maguire posts about building "privacy protecting AI" - Signaling that they were "thinking about" the problem (but clearly had already built the thing).
4. July 18th, 2024 - Proton releases Proton Scribe - this is the LLM product.
5. July 24th, 2024 - Proton releases a Bitcoin Wallet

Now that we’ve established the timeline, let’s break down the path from Survey to “releasing products that no one asked for”.

So I took this survey, and I’m now kicking myself for not screenshotting the questions, because I have a lot to say about shitty survey design.

I’m going to do some conjecture and speculation here, but I think that the survey was designed when they already had the Scribe product already well into development and that the questions were designed to make it seem like the product they were already developing was by popular demand. Like, you don’t develop a “privacy preserving” (it’s not) LLM in 1.5 months. They were already building this thing.

Let’s talk about those results. They provide this graph in their survey results post:

My recollection of this question is that it was a multiple-choice, but not stack-ranked question. I’m not sure if that’s how it actually was, just how I remember it.

Regardless, I want to point out 2 things:

1. The LLM answer in there doesn’t actually mention an LLM. It mentions a “writing assistant”. There are tons of things that do not use Large Language Models to do writing assistants, like checking grammar and spelling. The way they worded that answer was extremely misleadging.
2. Only 29% of respondents said they wanted it.

So let’s move on to the second point they try to make in this survey that absolutely does not say what they say it does.

And here’s their analysis of that data:

Generative AI is one of the most significant developments in recent history, and it is supposed to lead to incredible gains in productivity. As more and more AI assistants come online, we asked the Proton community what they thought of these tools. Around 42% of respondents use an AI service regularly (at least once a month), and another 18% have never tried AI but are interested in it.

I just. sigh.

1. The survey asked do you use Generative AI. It doesn’t ask why. It doesn’t ask if you find it useful. It doesn’t ask if you WANT IT IN YOUR FUCKING EMAIL CLIENT. This tells you nothing!
2. “At least once a month” is not very much Generative AI usage.
3. That’s still under half of your user base!

This question is poorly designed. I can’t tell if it’s poorly designed as an excuse to interpret the results the way Proton apparently wanted to, or if it was a genuine “surveys are hard to design” take, but asking “do you use Generative AI” without also asking “Do you use Generative AI for spicy roleplay” and “do you want us to put an LLM into your Email client” is ridiculous.

So, I don’t think this survey is a killer “Our users want this” result.

In their announcement post they make the following assertion:

In our 2024 community survey, more than 75% of Proton’s business users said they are interested in generative AI tools, but most were also concerned about a lack of data protections. Scribe was designed to be a secure alternative.

Right. Okay, I don’t think that’s what those survey results actually say (since you didn’t ask “do you want a secure alternative if we build one”).

When the backlash to this feature started over on Mastodon, they had the following response:

This excuse is flatly ridiculous. “We thought ‘hey they’re gonna do it anyway, let’s do it so they can be safe’”, is what that amounts to. That’s not a smart way to roll out features - and I don’t actually buy it.

Meanwhile, over on a Pivot to AI blog post on this subject, Amy and David point out the following:

Proton’s descriptions of Scribe are vague and waffly about their threat model. Your prompt — that is, the email you’re writing — is kept in plain text on their server, unlike emails you’ve sent or received, which are secure at rest. Proton promises they don’t log the prompts — but services like Apple, which many Proton users were trying to get away from, make only the same level of promise.

Proton then goes on to point out that the Model can run locally…. but only on Chrome, and only on systems with high enough system specifications. They went on to respond to this post with the following rebuttal on Mastodon:

I want to break down some of these things.

The feature is not “opt-in” if you’re on affected plans. It shows you this dialog:

Notice how that dialog doesn’t have a “no I don’t want this” option. To turn it off, you have to go into the settings. That’s an opt-out feature. Not opt-in.

The second thing is that there’s no such thing as an “Open source” model. The weights may be open, but Mistral has been super cagey about where they got the training data (also known as, they got it the same place all the other companies did - the internet, without permission).

Finally, if you or someone you’re chatting with decides to use the “zero logs server” the text is sent to them in the clear…. which I think very much does break their zero-knowledge model.

You’ve totally broken your privacy model for your users, willfully. For an LLM.

Ugh.

But that’s not the most ridiculous thing about this whole saga.

Literally six days later, Proton announces they’re launching a Bitcoin wallet of all things!

Pivot to AI covers this too and points this out:

If Proton was taking privacy seriously they’d have used Monero or Zcash — two cryptos that use zero-knowledge proofs to make transactions untraceable through the blockchain data trail. At least these have a use case, even if it’s buying Russian research chemicals off the darknets.

Yeah, exactly that. Bitcoin isn’t private, and no amount of you providing a non-custodial wallet is going to change that.

Here’s the security model. This is a reasonable attempt at the functionality, but it doesn’t make the basic idea any less dumb. Also, they recommend you use a bitcoin mixer so the Feds can bust you for money laundering too when they match you to your bitcoin address.

… That’s bad.

I really wish I knew. The timeline makes it clear to me that they had several potentially very unpopular features already in development and then timed the release with their developer survey. I suspect they were hoping to spin the results as a “you asked for it and guess what we delivered in record time!”.

The spaces I hang around in are very skeptical of both Generative AI and Crypto, and I think that Proton’s core individual customer is too. It feels like they’re speedrunning a “alienate our core customer base” but I honestly don’t know what they’re doing here.

It’s possible that their business users really do want this thing and have been asking for it for a long time. It’s also possible that the people running the show are adding features for a future investment or a sale. That doesn’t really jive with moving to a non-profit.

It’s also possible that they’ve always loved LLMs and Crypto and just happy to shove it into everything.

But, it doesn’t functionally matter for me. They’ve broken my trust, and that was the primary currency and reason I’d be willing to shovel money towards them. That takes a while to build. It takes an instant to destroy.

I think about that a lot. Oh well.

tech rant blog