Musk says nothing can shut down open source. That is also the AI safety risk.

“Nothing can shut down open source,” Elon Musk wrote on X on 13 September. It is a short defence of openness that also points to one of AI safety’s hardest problems: what happens when a powerful model is released and its creator can no longer control the copies?

The serious concern is the combination of dangerous capabilities and distribution that cannot reliably be reversed. Calling every open-source project a threat misses that distinction. Dismissing the risks because openness has benefits misses it too.

What Musk was responding to

Musk was replying to Kevin Bass, who questioned his support for Dario Amodei’s call to “pace the frontier”. His reply addresses open source but does not explain how he would reconcile wider model access with safety controls. It is not a detailed policy statement.

Source: Elon Musk: Nothing can shut down open source

Our earlier coverage of Amodei, Musk and Altman explains the wider debate. This follow-up looks at the release decision: who retains control after a model leaves the lab?

Open source and open weights are not interchangeable

Model weights are the learned parameters used to produce outputs. “Open weight” generally describes models whose parameters can be obtained and run outside the original provider’s service. That does not, by itself, establish that the entire system is open source.

Under the Open Source Initiative’s definition, open-source AI requires freedoms to use, study, modify and share, supported by the relevant code, parameters and information about training data. The data requirement is not simply a demand to publish every raw training file.

For the shutdown debate, access to usable model files is the crucial distinction. An app being free to use tells you little about whether its users possess those files.

Source: Open Source Initiative: Open Source AI Definition 1.0

The risk is losing the ability to intervene

The UK AI Security Institute warns that releasing weights removes important options available to a hosted model provider: monitoring use, restricting access and withdrawing the model. Copies can operate beyond the developer’s oversight, and safeguards may be removed.

That is why “we will fix it after release” is a weaker promise when the original developer cannot ensure everyone adopts the fix. A corrected version does not make every earlier copy disappear.

That does not mean mitigation is pointless. AISI describes staged releases and full-access audits as possible risk-management tools. Removing a download can slow further distribution, even though it cannot recall existing copies. The distinction matters: reducing exposure is still useful when total recall is impossible.

Source: UK AI Security Institute: managing open-weight model risks

What the cyber evidence actually shows

In the analysis reviewed for this article, AISI found that leading open-weight models trailed the closed-model cyber frontier by four to seven months, compared with six to ten months in its earlier 2025 evaluations.

Those are evaluation results, not a forecast that any downloaded model can compromise any business. Its simulated networks lacked active defenders and some other real-world security features. AISI also cautions against applying the findings beyond cyber capabilities.

Source: UK AI Security Institute: open-weight cyber capabilities

Anthropic’s position is more specific than a blanket ban

In a statement dated 27 July 2026, Amodei said Anthropic had not advocated banning open-weight models as a category. He described models without dangerous capabilities as a public good and supported mandatory safety testing for sufficiently capable models, whether open or closed.

His concerns include cyber and biological misuse. These are risk arguments about what powerful systems could enable, not evidence that every openly released model has those abilities. He also says increased risk and the effectiveness of mitigation should be established through testing.

Anthropic is a commercial AI developer with a stake in this debate. Its position deserves accurate reporting and scrutiny; it should not substitute for independent evidence.

Source: Anthropic: Our position on open-weights models, 27 July 2026

What this means if you use AI at work

For an Australian reader choosing a tool, the useful question is more concrete than “is open source good or bad?” Ask who runs the model and who controls its access. A locally operated model and a website built around the same model can be very different products.

Consider a hypothetical small business choosing a tool to summarise documents. Before treating an “open” label as reassurance, it should establish where processing happens, what the application sends elsewhere and who is responsible for updates. This is a procurement example, not a claim that we tested a particular app.

The editorial question we would put to any release announcement is equally practical: if a serious problem is discovered next week, what can the developer still change? The answer should describe actual controls, rather than rely on the reputation of the company or the word “open”.

The release decision deserves the attention

Musk’s sentence works as a slogan. As a safety argument, it leaves the difficult question unanswered: which capabilities should be distributed in a form that cannot reliably be recalled?

Our view is that this is a serious threat to manage when models have dangerous capabilities. The evidence supports scrutiny of those capabilities before release, together with an honest account of what controls will remain afterwards. It does not justify treating every open AI model as equally dangerous—or assuming a closed service is automatically safe.