Or, for that matter, anything else.

This post is meant to be two things:

1. a PSA about LessWrong's current security posture, from a LessWrong admin^[1]
2. an attempt to establish common knowledge of the security situation it looks like the world (and, by extension, you) will shortly be in

Claude Mythos was announced yesterday. That announcement came with a blog post from Anthropic's Frontier Red Team, detailing the large number of zero-days (and other security vulnerabilities) discovered by Mythos.

This should not be a surprise if you were paying attention - LLMs being trained on coding first was a big hint, the labs putting cybersecurity as a top-level item in their threat models and evals was another, and frankly this blog post maybe could've been written a couple months ago (either this or this might've been sufficient). But it seems quite overdetermined now.

LessWrong's security posture

In the past, I have tried to communicate that LessWrong should not be treated as a platform with a hardened security posture. LessWrong is run by a small team. Our operational philosophy is similar to that of many early-stage startups. We treat some LessWrong data as private in a social sense, but do not consider ourselves to be in the business of securely storing sensitive information. We make many choices and trade-offs in the direction that marginally favor speed over security, which many large organizations would make differently. I think this is reasonable and roughly endorse the kinds of trade-offs we're making^[2].

I think it is important for you to understand the above when making decisions about how to use LessWrong. Please do not store highly sensitive information in LessWrong drafts, or send it to other users via LessWrong messages, with the expectation that LessWrong will be robust to the maybe-upcoming-wave-of-scaled-cyberattacks.

LessWrong is not a high-value target

While LessWrong may end up in the affected blast radius simply due to its nature as an online platform, we do not store the kind of user data that cybercriminals in the business of conducting scaled cyberattacks are after. The most likely outcome of a data breach is that the database is scanned (via automated tooling) for anything that looks like account credentials, crypto wallet keys, LLM inference provider API keys, or similar. If you have ever stored anything like that in a draft post or sent it to another user via LessWrong DM, I recommend cycling it immediately.

It is possible that e.g. an individual with a grudge might try to dig up dirt on their enemies. I think this is a pretty unlikely threat model even if it becomes tractable for a random person to point an LLM at LessWrong and say "hack that". In that world, I do expect us (the LessWrong team) to clean up most of the issues obvious to publicly-available LLMs relatively quickly, and also most people with grudges don't commit cybercrime about it.

Another possibility is that we get hit by an untargeted attack and all the data is released in a "public" data dump. It's hard to get good numbers for this kind of thing, but there's a few reasons for optimism^[3] here:

• From what I could find, probably well under half of data breaches result in datasets that get publicly circulated in any meaningful sense.
• Many of those that do are "for sale", not freely available. Someone with a chip on their shoulder might download a freely available dataset, but is much less likely to spend money on it (and also risk the eye of the state, if they then try to use that purchased data for anything untoward).
• Datasets like this often don't ever really "go away", but they often do become unavailable, especially if they're large. Storage is expensive, hosting sites generally take them down on request, torrenting is risky, and there isn't much motive to keep re-uploading terabytes of data that you aren't even selling. (Monetizable datasets tend to be stripped down and much smaller, but also wouldn't include approximately any of the information that you might be concerned about here.)

FAQ

What "private" data of mine could be exposed in a breach?

• Your email address(es)
• A hashed version of your password
• Your previous display name, if you've changed it (not technically a secret)
• Analytics data about e.g. what pages you've visited on LessWrong, and in some cases what you've clicked on
• Any information that may have come from your OAuth providers (Google, Github, Facebook)
• Messages to other users
• Draft posts and comments
• Deleted comments
• Draft revisions of published posts
• Your frontpage tag filter settings
• Your voting history
• Your location data (if you provided it for e.g. being notified of nearby events)
• Posts you've read
• Your bookmarks
• Posts you've hidden
• Information you've given us to enable us to pay you money (if you provided it for e.g. Goodhart Tokens), such as a dedicated Paypal email address. (We do not store any e.g. credit card information that you would use to pay us money.)
• Your notifications
• Your account's moderation history (if any)
• Actions you've taken in previous Petrov Days
• Your user agent and referer
• Any messages you've sent to LLMs via one of the two embedded LLM chat features we've built, and responses received
• Probably other things that aren't coming to mind, though I'm pretty sure I've covered the big ones above. If you're curious, our codebase is open source; you're welcome to examine it yourself (or sic your own LLM on it).

Can I delete my data?

No*. Nearly all of the data we store is functional. It would take many engineer-months to refactor the codebase to support hard-deletion of user data (including across backups, which would be required for data deletion to be "reliable" in the case of a future data breach), and this would also make many site features difficult or impractical to maintain in their current states. Normatively, I think that requests for data deletion are often poorly motivated and impose externalities on others^[4]. Descriptively, I think that most requests for data deletion from LessWrong would be mistakes if they were generated by concerns about potential data breaches. Separately, most data deletion requests often concern publicly-available data (such as published posts and comments) which are often already captured by various mirrors and archives, and we don't have the ability to enforce their deletion. I'll go into more detail on my thinking on some of this in the next section of the post.

* If you are a long-standing site user and think that you have a compelling case for hard-deleting a specific piece of data, please feel free to message us, but we can't make any promises about being able to allocate large amounts of staff time to this. e.g. we may agree to delete your DMs, after giving other conversation participants time to take their own backups.

Is LessWrong planning on changing anything?

We have no immediate plans to change anything. There might be a threshold which the cost of auditing our own codebase can fall under that would motivate us to conduct a dedicated audit, but we are not quite there yet^[5].

The Broader Situation

Epistemic status: I am not a security professional. I am a software engineer who has spent more time thinking about security than the median software engineer, but maybe not the 99th percentile. This section necessarily requires some extrapolation into the uncertain future.

A proper treatment of "what's about to happen" really deserves its own post, ideally by a subject-matter expert (or at least someone who's spent quite a bit more time on thinking about this question than I have). I nonetheless include some very quick thoughts below, mostly relevant to US-based individuals that don't have access to highly sensitive corporate secrets^[6] or classified government information.

Many existing threat models don't seem obviously affected by the first-order impacts of a dramatic increase in scalable cyber-offensive capabilities. Four threat models which seem likely to get worse are third-party data breaches, software supply chain attacks, ransomware, and cryptocurrency theft.

I'm not sure what to do about data breaches, in general. The typical vector of exploitation is often various forms of fraud involving identity theft or impersonation, but scaled blackmail campaigns^[7] wouldn't be terribly shocking as a "new" problem. One can also imagine many other problems cropping up downstream of LLMs providing scalable cognition, enabling many avenues of value extraction that were previously uneconomical due to the sheer volume of data. If you're worried about identity theft, set up a credit freeze^[8]. Behave virtuously. If you must behave unvirtuously, don't post evidence of your unvirtuous behavior on the internet, not even under a very anonymous account that you're sure can't be linked back to you.

Software supply chain attacks seem less actionable if you're not a software engineer. This is already getting worse and will probably continue to get worse. Use a toolchain that lets you pin your dependencies, if you can. Wait a few days after release before upgrading to the newest version of any dependency. There are many other things you can do here; they might or might not pass a cost-benefit analysis for individuals.

Scaled ransomware

Everybody is already a target. They want your money and will hold the contents of your computer hostage to get it.

This probably gets somewhat worse in the short-term with increased cybersecurity capabilities floating around. The goal of the attacker is to find a way to install ransomware on your computer. Rapidly increasing cybersecurity capabilities differentially favor attackers since there are multiple defenders and any one of them lagging behind is often enough to enable marginal compromises^[9].

To date, scaled ransomware campaigns of the kind that extort large numbers of individuals out of hundreds or thousands of dollars apiece have not been trying to delete (or otherwise make inaccessible) backups stored in consumer backup services like Backblaze, etc^[10]. My current belief is that this is mostly a contingent fact about the economic returns of trying to develop the relevant feature-set, rather than due to any fundamental difficulty of the underlying task.

As far as I can tell, none of the off-the-shelf consumer services like this have a feature that would prevent an attacker with your credentials from deleting your backups immediately. Various companies (including Backblaze) offer a separate object storage service, with an object lock feature that prevents even the account owner from deleting the relevant files (for some period of time), but these are not off-the-shelf consumer services and at that point you're either rolling your own or paying a lot more (or both).

If you are concerned about the possibility of losing everything on your computer because of ransomware^[11], it is probably still worth using a service like this. The contingent fact of scaled ransomware campaigns not targeting these kinds of backups may remain true. Even if it does not remain true, there are some additional things you should do to improve your odds:

• Set your 2fa method to rely on TOTP, not a code sent by email or SMS.
• Do not install the app generating TOTPs on your computer.
• Do not check "Remember this browser" when entering your 2fa code to sign in to their website. If you've already done that, delete all the cookies in your browser for the relevant domains.

This increases the number of additional security boundaries the ransomware would need to figure out how to violate, in order to mess with your backups.

Scaled cryptocurrency theft

Everybody is already a target (since the attackers don't know who might own cryptocurrency), but this mostly doesn't matter if you don't own cryptocurrency. The threat model here is similar to the previous one, except the target is not necessarily your computer's hard drive, but anywhere you might be keeping your keys. I am not a cryptocurrency expert and have not thought about how I would safely custody large amounts^[12] of cryptocurrency. Seems like a hard problem. Have you considered not owning cryptocurrency?

My extremely tentative, low-confidence guess is that for smaller amounts you might just be better off tossing it all into Coinbase. Third-party wallets seem quite high-risk to me; their security is going to be worse and you'll have fewer options for e.g. recovery from equity holders after a breach. Self-custody trades off against other risks (like losing your keys). But this is a question where you can probably do better than listening to me with a couple hours of research, if you're already in a position where it matters to you.

All of these probably deserve fuller treatments.

---

Habryka broadly endorses the contents of the LessWrong's security posture section. Instances of the pronoun "we" in this post should generally be understood to mean "the members of the Lightcone team responsible for this, whatever this is", rather than "the entire Lightcone team". I'll try to be available to answer questions in the comments (or via Intercom); my guess is that Habryka and Jim will also be around to answer some questions.

1. ^{^}

   Me!

2. ^{^}

   I won't vouch for every single individual one, not having thought carefully enough about every single such choice to be confident that I would endorse it on reflection. Many such cases.

3. ^{^}

   Which unfortunately are contingent on details of the current environment.

4. ^{^}

   Though I won't argue for that claim in this post, and it's not load-bearing for the decision.

5. ^{^}

   If you think you are qualified to do this (and are confident that you won't end up spamming us with false-positives), please message us on Intercom or email us at team@lesswrong.com. We do not have a bug bounty program. Please do not probe our production APIs or infrastructure without our explicit consent. We are not likely to respond to unsolicited reports of security issues if we can't easily verify that you're the kind of person who's likely to have found a real problem, or if your report does not include a clear repro.

6. ^{^}

   This does unfortunately exclude many likely readers, since it includes lab employees, and also employees of orgs that receive such information from labs, such as various evals orgs.

7. ^{^}

   We technically already have these, but they're often targeting the subset of the population that is afraid of the attacker telling their friends and family that they e.g. watch pornography, which the attacker doesn't actually know to be true (though on priors...) and also won't do since they don't know who your friends and family are. These attacks can become much scarier to a much larger percentage of the population, since personalization can now be done in an substantially automated way.

8. ^{^}

   This won't help with e.g. fraud against government agencies, or anything other than attackers opening financial accounts in your name.

9. ^{^}

   This is not intended as a complete argument for this claim.

10. ^{^}

    This is not the case for things like OneDrive/Dropbox/Google Drive, where you have a "sync" folder on your machine. It is also not the case for targeted ransomware attacks on large organizations of the kind that ask for 6-7 figures; those are generally bespoke operations and go through some effort to gain access to all of backups before revealing themselves, since the backups are a threat to the entire operation.

11. ^{^}

    Or hardware failure, or theft of your computer, or many other possibilities. But the further advice is specific to the ransomware case.

12. ^{^}

    I'm not sure when the "hunt you down in person"-level attacks start. Maybe six figures? At any rate, don't talk about your cryptocurrency holdings in public.

Discuss