Who's Working on What — AI Safety Organizations

Open-weight Yi model family. Site now shows an enterprise decision platform rather than research; no usable research feed found on 2026-08-03.

Publishes career research and problem profiles, runs a podcast and one-on-one advising, and now directs most of its effort at moving people into AI risk-reduction work.

Global digital rights organisation working on spyware accountability, internet shutdowns, biometric ID systems and human-rights limits in the EU AI Act.

UK-focused research on AI regulation and public attitudes — foundation model accountability, biometrics law, AI in health and public services, and algorithmic audit/assurance methods.

Funds the £59M Safeguarded AI programme, building an open-source mathematical assurance toolkit for formally verified AI-generated artifacts (TA1) and applying AI-enabled formal methods to cybersecurity (TA2).

Consultancy whose self-funded alignment team publishes neglected-approaches work — self-other overlap fine-tuning to reduce deception, and attention-schema/BCI-inspired methods.

Independent LLM agent safety group focused on chain-of-thought monitorability and faithfulness as an AI control mechanism.

University of Cape Town-hosted hub building African regional research and training capacity on AI safety and its peace-and-security implications.

Acts as Spain's national market-surveillance authority for the EU AI Act, supervising high-risk AI systems placed on the Spanish market.

Designated under Italian Law 132/2025 as the AI notification authority, defining procedures for assessing, accrediting and monitoring AI conformity-assessment bodies.

Italy's AI market-surveillance authority under Law 132/2025, with inspection and sanctioning powers; publishes AI security guidelines and hosted the EU ADCO AI market-surveillance cooperation group.

Builds and promotes deliberative and collective-intelligence tooling and governance practices so democratic institutions can absorb advanced AI rather than be destabilised by it.

Trinity College Dublin lab led by Abeba Birhane doing empirical audits of training datasets and models (bias, hate speech, degenerate content) and mapping accountability gaps in AI supply chains.

Invite-to-post discussion forum (run on the LessWrong codebase) where alignment researchers publish technical write-ups on interpretability, deceptive alignment, agent foundations and evaluations.

Pan-African AI practitioner community running events and training, with some responsible-AI and safety programming.

Builds interactive explainers and live demonstrations (agent village, time-horizon and self-awareness evals) that show non-experts what current AI systems can and cannot do.

Forecasting group behind the AI 2027 scenario. Publishes as standalone scenario sites rather than a dated index; no usable feed found on 2026-08-03.

Publishes reference research on AI timelines and takeoff, including the recurring large-scale survey of published ML researchers' forecasts.

Policy research and advocacy on the concentration of economic and infrastructural power in the AI industry, plus AI's effects on labour, surveillance, and the energy grid.

Nonprofit lab studying misaligned optimisation in large-scale systems (AI, markets, bureaucracies) and building tools for collective input into AI objectives (e.g. Talk to the City).

Conducts and publishes US public opinion polling on AI regulation to inform legislators, consistently reporting majority support for slowing frontier development.

Non-profit convening policymakers and researchers across Asia, running capacity-building programmes and publishing region-specific AI safety analysis.

Long-running volunteer-run programme pairing newcomers into small remote research teams for a multi-week AI safety project; several orgs (e.g. Cadenza Labs) originated in its cohorts.

Weekly listing of AI safety fellowships, courses, conferences and deadlines, published by the AISafety.com team.

Grants $10M+ from Anthropic, Google, Microsoft and OpenAI plus outside foundations to independent researchers doing third-party frontier-model evaluations and safety/security research; a second cohort of 11 grantees received over $5M.

Grantmaking fund established by the Frontier Model Forum and philanthropic partners that funds independent third-party research and evaluations on frontier model safety and security.

Builds an Indian AI safety community through training cohorts and research programmes aimed at local developers.

Weekly-to-biweekly Substack summarising AI safety research, incidents and policy developments for a general audience.

Offers free one-to-one 'navigation calls' that point newcomers to the courses, communities and jobs that fit them.

Provides links, a Slack community and networking resources for early-career and independent AI safety researchers.

Runs weekly AI safety study sessions (benkyoukai) in Japanese and English in Shibuya and organises the Technical AI Safety Conference (TAIS).

Free online textbook (Dan Hendrycks / CAIS) used as the base text for introductory AI safety courses and reading groups.

Runs pre-deployment and post-deployment evaluations of frontier models for chemical/biological uplift, cyber-offence, autonomy and safeguard robustness, and publishes the underlying evaluation tooling (Inspect) and research findings.

Arvind Narayanan and Sayash Kapoor's newsletter critiquing AI evaluation practice, benchmark claims and predictive-AI deployments; now branded 'AI as Normal Technology'.

Maintains the AI Verify testing toolkit and Project Moonshot LLM evaluation/red-teaming tooling, and co-publishes Singapore's governance frameworks for generative AI with IMDA.

Enterprise model developer whose published work centres on grounding and attribution to reduce hallucination in retrieval-augmented deployments.

Maintains directories of AI safety projects, communities, funding and events, including the AI Safety Field Map at aisafety.world.

Berlin academic institute researching AI and democracy (including a dataset of AI projects supporting democratic processes), DSA/platform power, and AI in the workplace.

Documenting demographic bias and harm in deployed AI, especially facial recognition and biometric systems in airports, policing, and hiring; runs a public AI-harms reporting channel.

European watchdog investigating deployed algorithmic systems — content moderation labour, chatbot influence on elections and government decisions, data centre energy disclosure, and EU AI Act rollback.

Oxford-based lab working on concept extrapolation / goal generalisation out of distribution; now largely productised as 'Canvas'.

Rohin Shah's weekly summaries of new alignment-relevant papers, 2018-2022; the archive remains the standard index of that period's literature.

Prague-based group (Charles University) studying alignment as a problem of hierarchical agency and multi-agent/ecosystem dynamics rather than single-model alignment.

Theory-first alignment lab founded by Paul Christiano, working on eliciting latent knowledge, heuristic arguments / formal explanations of neural network behaviour, and low-probability estimation.

Intensive in-person bootcamp teaching the engineering skills for alignment work (transformers from scratch, mech interp, RL, evals); curriculum is public and widely reused.

Non-profit lab shipping fully open models and training data (OLMo, Dolma), which makes it one of the few places pretraining-data questions can be studied directly.

German Marshall Fund programme tracking authoritarian interference in democracies, including state-linked information manipulation, AI-enabled influence tooling and the Authoritarian Tech Monitor.

Internal AWS function producing AI Service Cards, Bedrock Guardrails content filtering and Automated Reasoning checks for hallucination detection in deployed model applications.

Litigates and campaigns against government use of face recognition, algorithmic risk assessment in criminal justice and immigration, and AI-driven benefits determinations.

Amnesty's technology programme investigates spyware targeting of activists (Security Lab), facial recognition deployment, and algorithmic discrimination in welfare systems.

Real-world agentic evaluations — running frontier models as autonomous economic agents (Vending-Bench, the Anthropic 'Project Vend' shop, embodied robot evals) to find failure modes long-horizon benchmarks miss.

Small independent R&D lab publishing reproducible training and fine-tuning methods with unusually complete write-ups.

Internal Anthropic team publishing empirical research on alignment faking, sabotage and sandbagging evaluations, model organisms of misalignment, and AI control protocols.

Internal Anthropic team that measures Claude's dangerous capabilities in cyber operations, biosecurity and autonomous replication to inform Responsible Scaling Policy thresholds.

Internal Anthropic team producing the Anthropic Economic Index on labour-market effects of Claude usage, plus research on persuasion, political even-handedness and real-world value expression in model conversations.

Independent AI safety lab that runs global research hackathons/sprints and a fellowship, converting sprint outputs into peer-reviewed papers on evaluations, interpretability, deception and agent safety.

Builds behavioural evaluations for scheming and strategic deception in frontier models, and runs evals for frontier labs and AISIs.

On-device and multimodal model research; a source of contrarian evaluation work, including the reasoning-limits papers that fed the capability-estimation debate.

Runs the ARC-AGI benchmarks and publishes verified results, including the contamination and harness caveats behind headline scores.

Global freedom-of-expression organisation working on AI content moderation, automated censorship, and AI provisions in internet governance and standards bodies (ITU, IETF).

UK NGO producing legal and policy analysis on weapons that cause unacceptable harm, with a sustained workstream on meaningful human control over autonomous weapons and targeting systems.

Runs a ten-week remote mentorship program in AI alignment research aimed specifically at women entering the field.

Work on data and AI in geopolitics — trusted data flows, AI standards diplomacy, and technology statecraft among democracies.

Monitors and analyses frontier-model capabilities including autonomous AI agents, tests new AI models and applications, and works with the Australian Signals Directorate and CSIRO, disseminating guidance through the National AI Centre.

Builds AI mathematicians and formal-verification tooling (AXLE) for machine-checkable proofs — relevant to verified/guaranteed-safe AI approaches.

Baidu's research output behind the ERNIE model family, published to Hugging Face as papers and technical blog posts rather than to its own site.

Nonprofit research org building probabilistic-programming and causal-inference machinery ('universal reasoning engine') and applying it to public-interest decision problems.

Chinese state-backed AI research academy that authored the Beijing AI Principles and convenes the annual Beijing Academic Conference on AI safety and alignment alongside its model research.

UC Berkeley center working on agentic AI foundations, LLM-agent safety and security (oversight, alignment, control, interpretability/monitoring), AI for cybersecurity and decentralization technology.

Harvard center studying platform governance, AI accountability and interpretability tooling, non-consensual intimate imagery reporting systems, child online safety and comparative AI regulation.

UK civil liberties group campaigning against live facial recognition in policing and retail, predictive policing, and expansion of state biometric databases.

FLUX open-weight image models, widely redistributed and fine-tuned — including into the uncensored forks that appear in NCII reporting.

Runs free facilitated online courses (the AI Safety Fundamentals alignment and governance curricula) that take professionals through AI risk material in structured cohorts.

Publishes on defence AI, AI and nuclear deterrence, and AI governance through its Foreign Policy and Governance Studies programmes and the AI Equity Lab.

ByteDance's foundation-model research group, publishing on video generation and agents.

Small alignment group building robust lie detectors for LLMs — eliciting latent knowledge and detecting when a model states something it internally represents as false.

Student-and-researcher group at Cambridge running AI safety reading groups, technical bootcamps and research sprints.

Funds and coordinates Canadian research on frontier AI risks through CIFAR and the National Research Council, focused on evaluation methods and misuse of advanced systems.

Operates Cybertip.ca and the Project Arachnid crawler that detects and issues removal notices for child sexual abuse imagery, including synthetic material, across hosting providers.

Its Technology and International Affairs programme maintains the AI Global Surveillance index and publishes on state adoption of AI surveillance and AI in geopolitics.

501(c)(3) research community in Cavendish, Vermont working primarily on AI safety and pandemic prevention, with a research fellowship programme.

US defence think tank whose Technology and National Security programme publishes on military AI adoption, autonomy in weapons, and AI's effect on nuclear stability and escalation.

Assesses national AI policies against democratic-values criteria (the annual AI and Democratic Values Index), files regulatory complaints and comments, and runs an AI Policy Clinic training programme.

Washington DC advocacy shop drafting model US federal legislation and briefing Congress on catastrophic risks from frontier AI systems (bio/chem uplift, cyber, autonomous agents).

Small nonprofit translating frontier-AI risk into usable risk-management practice — profiles and guidance for applying the NIST AI Risk Management Framework to general-purpose and foundation models.

Independent nonprofit doing technical safety research (WMDP unlearning benchmark, representation engineering, Humanity's Last Exam) alongside field-building and the AI Safety Newsletter.

USP centre (FAPESP/IBM-funded) working on Portuguese-language NLP, AI in health and agriculture, and contributing to Brazil's OBIA AI observatory.

US and Brussels advocacy on AI civil rights — algorithmic discrimination in hiring and benefits, AI in education and elections, and EU AI Act medical/biometric scope.

UC Berkeley research centre (Stuart Russell) developing the assistance-game / uncertain-objective framing of alignment, plus reward learning, value misspecification, and multi-agent alignment research.

Advocacy nonprofit (Tristan Harris, Aza Raskin) campaigning on attention-capture and persuasive design in social media and now on companion chatbots, youth harms, and AI product liability/litigation.

Princeton center bridging computer science and public policy on algorithmic accountability, AI evaluation and measurement, platform regulation and the societal effects of automated decision systems.

Shanghai AI Lab's safety centre publishing the Frontier AI Risk Management Framework, SafeWork safety-reasoning training systems, embodied-AI safety simulation and multi-model frontier risk evaluations.

Georgetown research center producing data-driven analysis of military AI adoption, AI test and evaluation for defence, chip supply chains, and AI-cyber capability.

Foreign-policy think tank with defence-AI, AI export control and semiconductor supply-chain analysis via its Wadhwani AI Center and Strategic Technologies Program.

Research group focused on suffering risks from advanced AI, specialising in conflict and bargaining failures between AI agents — commitment races, surrogate goals, and cooperative game theory for TAI.

University of Melbourne cross-faculty centre (law, engineering, arts, science) on regulation and ethics of AI and digital technologies, with associated postgraduate teaching.

Brussels think tank producing task-force reports on EU AI Act design, standardisation and liability, and on AI's effect on EU competitiveness and labour markets.

European think tank on emerging-technology risk for policymakers, covering frontier AI governance, cognitive integrity/manipulation, biotech, and climate interventions.

Canadian think tank on international technology governance — AI and trade rules, data governance regimes, and AI's effect on democratic institutions and information ecosystems.

Virtual interdisciplinary centre at IIT Madras doing technical and policy research on deployable responsible AI in the Indian context, including voluntary AI risk mitigation and AI governance guidance for Indian regulators.

Oxford-based research centre producing policy-facing analysis of advanced AI governance: compute governance, model evaluations as a regulatory tool, corporate and international governance structures, and frontier safety frameworks.

University of Cambridge centre on global catastrophic risk, including AI loss-of-control framing, AI-assisted research reproducibility, biological weapons governance and decolonial approaches to risk governance.

Carnegie Mellon SEI division handling vulnerability coordination and disclosure, now including work on how to assign and coordinate vulnerabilities in AI and ML systems.

Companion chatbots at consumer scale; named in litigation and regulatory attention over minors and self-harm.

International-affairs analysis of AI in security and diplomacy — AI and nuclear risk, autonomous weapons norms, and AI's effect on information environments.

Umbrella consortium of Chinese state-linked research bodies — Tsinghua University, Shanghai AI Laboratory, BAAI, Peking University, Shanghai Qi Zhi Institute, CAICT and the CAS Institute of Automation — that represents China in dialogue with foreign AI safety institutes.

Administers Canada's Pan-Canadian AI Strategy including an AI safety funding stream and the AI Insights for Policymakers programme feeding Canadian federal AI policy.

Saarbrucken security research center whose AI work covers CSAM generation in text-to-image models, image watermarking, model privacy and adversarial ML.

University of Toronto / Vector lab on trustworthy ML: machine unlearning, privacy attacks, model stealing, dataset auditing and adversarial robustness.

Publishes AI security guidance, the AI Controls Matrix and AI trustworthy-deployment frameworks through its AI Safety Initiative working groups.

Develops the open technical specification for cryptographically signed content credentials that record how a media file was captured, edited or AI-generated.

Runs the Navigating Transformative AI fund, with open RFPs for AI governance research (typically $200k+) and for capacity-building programs that train new AI safety researchers; also regranted $5M to NSF's Safe Learning-Enabled Systems program.

Cohere's research arm, producing the multilingual Aya model family and open evaluation work on cross-lingual safety gaps and benchmark reliability.

Membership organisation for entertainment concept artists that has funded lobbying on generative AI training-data consent and copyright protections for visual artists.

Beijing-based social enterprise bridging Chinese and international AI safety communities; runs a Frontier AI Risk Monitoring Platform and publishes the 'State of AI Safety in China' reports.

Runs camps and a fellowship (Condor Camp) introducing Latin American students to AI safety and other high-impact research paths.

Now a product company selling a 'Tactics' style bounded-AI architecture pitched as controllable and non-hallucinating, rather than the alignment research org it launched as.

Berkeley research centre and network hosting AI safety researchers; runs the Astra Fellowship, a visiting fellowship, and an incubator for new AI-risk organisations.

Adobe-led community driving adoption of C2PA Content Credentials across cameras, editing tools and platforms, and building open-source implementation libraries.

Campaign organisation lobbying UK and US legislators for binding restrictions on superintelligent AI development; runs legislator-signature campaigns and a policy newsletter.

Scenario and governance research, including AI emergency-preparedness work on what states would actually do in a fast-takeoff incident.

Funds and convenes research on cooperative intelligence among AI agents — multi-agent risks (collusion, conflict, destabilisation), benchmarks such as the Concordia contest, and a PhD fellowship programme.

Grants, fellowships and seminars for 'philosopher-builders' — funding technologists and philosophers building AI systems oriented around individual autonomy, decentralisation and self-direction rather than centralised control.

Research, victim services and legislative advocacy on image-based sexual abuse, including model legislation covering AI-generated non-consensual intimate imagery.

Carnegie Mellon security and privacy institute whose AI-relevant work covers adversarial machine learning, secure ML systems and usable privacy for automated decision systems.

Empirical social research on how automated and AI systems reshape labor, health care, public benefits administration, and media trust.

Participatory research project in which data workers and content moderators in Kenya, Syria, Germany, Brazil and India author first-person reports on their working conditions in the AI supply chain.

Nonprofit maintaining open interpretability infrastructure: Neuronpedia, SAELens, circuit-tracer and SAEDashboard.

Open-weight reasoning-model developer whose published output is chiefly technical reports on RL-trained reasoning (R1 series); its models have been externally evaluated by CAISI for jailbreak and agent-hijacking susceptibility.

UK cross-party think tank whose Demos Digital / CASM programme researches open-source AI regulation, online harms measurement, and public deliberation on AI policy.

Latin America-wide digital rights organisation researching state deployment of AI for policing, welfare targeting and border control, and regional AI regulation.

Atlantic Council lab conducting open-source investigations of influence operations, state information campaigns and AI-generated content in conflict and election contexts.

NTU Singapore centre designated as Singapore's AI Safety Institute node, working on AI testing and evaluation, safe model deployment, content assurance and trust technologies.

Independent research institute founded by Timnit Gebru studying AI's labour supply chain, data-worker exploitation, and the ideological and environmental costs of scale-driven AI.

Zvi Mowshowitz's high-volume newsletter giving weekly digests of AI capability releases, safety papers and policy fights.

Discussion forum run by the Centre for Effective Altruism; the main venue for AI safety org updates, funding announcements and career/field-building posts outside LessWrong.

Conference series run by the Centre for Effective Altruism whose AI-focused editions (EAG and EAGx) are a main recruiting and networking venue for AI safety orgs.

Digital civil liberties litigation and analysis, including AI-driven police surveillance (ALPR, gunshot/'distress' audio detection), age-gating mandates, and platform speech regulation.

Litigation and regulatory comment on automated decision-making, government AI systems, chatbot harms to minors, data brokers, and algorithmic pricing.

Nonprofit researching AI moral patienthood and model welfare — whether and how AI systems could have wellbeing, and what labs and policymakers should do about it.

Open-source AI research nonprofit producing open models, training datasets, the LM Evaluation Harness, and interpretability/memorisation research.

Voice cloning and synthesis; voice is the modality most used in impersonation fraud and robocall incidents.

Public benefit corporation building an AI research assistant for systematic literature review, with a stated bet on decomposed/process-supervised reasoning as a safety-relevant design.

Youth-led AI policy advocacy; drafted and lobbied for US state AI bills (including California SB 1047 and companion-chatbot child-safety laws) and litigation amicus work on chatbot harms.

Collects and publishes datasets on AI training compute, model releases, hardware and benchmark performance, plus the FrontierMath benchmark and Gradient Updates analyses.

Runs a summer research fellowship in Cambridge where fellows write policy-oriented papers on existential and emerging-technology risk, mostly AI.

ETH Zurich hub coordinating cross-departmental AI research including trustworthy ML, AI policy and the Swiss AI Initiative's open-model work.

European NGO investigating coordinated disinformation operations (e.g. Doppelganger) and tracking EU regulatory responses under the DSA and Code of Practice.

Enforces the EU AI Act's general-purpose AI rules, ran the drafting of the GPAI Code of Practice, and assesses systemic-risk models placed on the EU market.

Philanthropic re-granting fund that resources European civil-society organisations working on AI policy, algorithmic accountability, and the rights of communities affected by AI systems.

Network of ~50 European digital rights organisations coordinating campaigns against biometric mass surveillance, predictive policing and mandatory content scanning in EU law.

Non-profit that certifies generative AI companies whose training data is licensed or consented, providing a Licensed Model certification for music, image and text models.

Oxford Internet Institute project scoring platform and cloudwork companies against five fair-work principles across 40+ countries, including AI data-annotation vendors.

Berkeley AI safety nonprofit doing adversarial robustness work (e.g. attacks on superhuman Go agents), interpretability and model-tampering research, plus running alignment workshops and the FAR.Labs coworking space.

Runs structured forecasting studies (including the Existential Risk Persuasion Tournament) measuring expert and superforecaster disagreement about AI catastrophe probabilities.

Makes roughly $10k-$100k grants through its AI for Science & Safety Nodes program and Secure AI focus area, covering AI security, privacy, decentralized cooperation and epistemics, alongside fellowships and prizes.

Research on preparing institutions for transformative AI, including the transition-period questions most governance work skips.

Researches AI-enabled lock-in risk — persistent entrenchment of values or power via AI (including secretly loyal models) — and interventions against it.

Pooled fund covering AI, biosecurity and nuclear risk; verified AI grants include $50,000 to the EvalEval Coalition's 'Every Eval Ever' evaluation-aggregation project (June 2026) and $49,244 to the Institute for Security and Technology for an AI-and-nuclear-risk workshop (April 2026).

Legal non-profit supporting content moderators' litigation and unionisation in Kenya against Meta and its outsourcing contractors, and challenging UK government algorithmic decision systems.

Argentine digital rights organisation working on AI bias in Spanish-language models, state surveillance programmes, and data protection legislation.

Advocacy and grantmaking nonprofit campaigning on autonomous weapons bans, EU AI Act implementation, and frontier-model policy; publishes the AI Safety Index scoring leading labs.

Privacy and AI policy analysis across US states, EU and APAC — automated decision-making rules, chatbot suicide-detection mandates, age assurance, and cross-border data transfers.

Deepfake and synthetic-media detection company co-founded by digital forensics researcher Hany Farid, focused on enterprise defence against AI-enabled impersonation and social engineering.

Cape Town-based research centre (Rachel Adams) producing the Global Index on Responsible AI and Africa-focused work on AI and labour, press freedom, data governance, and AI safety/peace/security.

UNI Global Union-convened alliance, launched in Nairobi, organising content moderators and AI data workers across countries to bargain over psychological harm, pay and outsourcing conditions.

Public innovation lab at Universidad Adolfo Ibanez running the Algoritmos Publicos transparency platform and the Ethical, Responsible and Transparent Algorithms standard for Chilean public-sector AI procurement.

Commercial interpretability company; sells Ember/Silico for inspecting and steering model internals, and publishes SAE and feature-steering research across LLMs, genomics and vision models.

Internal Google DeepMind team publishing the Frontier Safety Framework and technical research on misuse, misalignment, mechanistic interpretability, amplified oversight and dangerous-capability evaluations.

Google's zero-day research team; its Big Sleep/naptime work applies LLM agents to memory-safety vulnerability discovery in real-world codebases.

Australian non-profit building technical tools and guidance for AI risk assessment and assurance, advising Australian government and financial regulators on AI governance implementation.

Network analysis firm mapping online influence operations and harmful online communities, including reporting on AI-generated persona networks and 'nudify' service ecosystems.

Automated red-teaming and guardrails company; runs public jailbreak arenas and agent red-teaming competitions and publishes the resulting attack datasets.

Automated red-teaming ('haizing') and evaluation of LLM applications — algorithmically searching for inputs that break safety behaviour, plus judge/eval infrastructure.

Recruits and places technical talent into US federal policy roles and think tanks working on AI and biosecurity, via fellowships and training.

Hosts the model and dataset commons most open releases pass through; blog feed ingested unfiltered, with the daily papers index tracked separately.

The community-curated daily index of newly posted ML papers — a broad view of what the field published today, distinct from the Hugging Face blog.

Coalition of music, film and creative-industry organisations advocating for consent, credit and compensation when copyrighted works are used to train generative AI.

Documents human rights impacts of AI and digital systems, including autonomous weapons (co-founder of Stop Killer Robots), surveillance of minorities, and algorithmic welfare targeting.

Australian applied research centre producing corporate AI governance guidance, facial recognition regulation model law, and advice on AI in employment and the public sector.

Develops technical standards for AI systems (the IEEE 7000 series on ethics, transparency, bias, and algorithmic accountability) used as conformity references by regulators.

Jack Clark's weekly newsletter summarising frontier AI research papers with commentary on capability trends and policy implications.

Operates a hub-and-spoke model in which partner institutions each maintain an IndiaAI Safety Institute Cell receiving work allocated by expertise, with research oriented to Indian datasets and India's linguistic diversity.

Personal-assistant models built around emotional engagement, a design target the companion-harm literature examines directly.

Brown University School of Public Health lab researching health misinformation, community-level information interventions and the effects of generative AI on the information environment.

Innovation-first tech policy think tank that argues against precautionary AI regulation and publishes on AI competitiveness, algorithmic accountability costs, and data policy.

Coordinates ANSSI, Inria, LNE and PEReN to evaluate the performance and reliability of AI models, analyse systemic AI risks from a national-security angle, and support EU AI Act implementation.

Research on compute governance, AI hardware security, frontier lab safety-policy disclosure, and AI in national-security decision-making.

University of Oxford philosophy institute producing academic work on the ethics of AI — moral status and agency of AI systems, LLMs and truth/epistemics, and AI in healthcare and democracy.

Security-focused work on AI and nuclear command and control, AI-enabled cyber operations, and frontier model weight security; runs the AI Foundation Model Transparency and NC3 workstreams.

Brazilian research institute on technology law and policy, contributing to Brazil's AI bill (PL 2338) debate, platform regulation, and AI-and-elections work.

Nathan Lambert's technical newsletter on post-training, RLHF and open-model releases, frequently covering alignment technique details.

Berlin-based European tech policy think tank on AI Act implementation, open-source AI and compute sovereignty, platform governance, and cybersecurity policy.

Publishes international humanitarian law positions on autonomous weapon systems and AI decision-support in armed conflict, including a formal recommendation to prohibit unpredictable autonomous weapons.

Ten-government body that coordinates joint testing exercises on foundation models and harmonised risk-assessment methodology; in February 2026 it published agreed consensus areas on practices for automated AI evaluations.

UK hotline that identifies and removes child sexual abuse imagery online and publishes technical assessments of AI-generated child sexual abuse material found on the open and dark web.

Independent testing and investigative reporting on video surveillance and physical security products, including AI analytics accuracy claims and ethnicity-detection features in Chinese camera systems.

Frontier AI security lab doing offensive-security evaluations of frontier models (cyber capability uplift, autonomous exploitation) for labs and governments.

Japanese government institute publishing red-teaming and evaluation guides for generative AI and coordinating with the international network of AI safety institutes.

Field-building organisation that operates SPAR and other pipeline programmes aimed at scaling the supply of AI safety researchers.

Publishes a quarterly AI Safety Forecast Report applying network-centrality analysis to global AI-safety news to derive scenarios, and runs three offices covering safety policy and external cooperation, safety evaluation, and a safety research lab.

David Krueger's academic lab doing technical alignment research and running student internships that feed people into the field.

Open dataset non-profit behind LAION-5B; central to training-data provenance and the CSAM-in-training-data findings. Blog is client-rendered with no anchors in served HTML; no usable feed found on 2026-08-03.

Runtime GenAI security — prompt-injection and jailbreak detection guardrails, plus the Gandalf prompt-injection game used as a public dataset source.

Runs a paid full-time London research programme where small teams produce a publishable technical AI safety paper over ~13 weeks.

AI governance and risk platform; built COMPL-AI, the first technical mapping of the EU AI Act into concrete model benchmarks.

Yoshua Bengio's nonprofit building 'Scientist AI' — a non-agentic, uncertainty-preserving system intended to act as a guardrail that predicts and blocks harmful actions by agentic AIs.

Interpretability company that has pivoted toward applying interpretability methods to scientific discovery on tabular data ('Disco'), extracting validated patterns from trained networks.

General rationality and AI risk forum operated by Lightcone Infrastructure; hosts a large share of informal alignment writing, org announcements and program advertisements.

Cambridge centre on the long-term implications of AI: EU general-purpose AI law commentary, capability prediction methodology, digital minds and moral status, and digital wellbeing.

Fast-turnaround grant round run by Lightcone Infrastructure that recommends funding to Jaan Tallinn and other donors for AI existential-safety projects, using the same S-Process software Lightcone built with SFF.

Non-transformer architectures for edge deployment; a check on whether scaling conclusions generalise beyond transformers.

Central London AI safety research hub providing office space and community for independent researchers and small safety orgs (Apollo, ARENA cohorts and others have been residents).

Advises major donors and runs the pooled Emerging Challenges Fund on catastrophic risk from emerging technology, including AI grants such as $100,000 to FAR AI, plus a consortium on digital sentience research.

Builds an open-source AI control proxy that sits between coding agents and the model API to log, monitor and block rule-violating actions in real deployments.

Longest-running alignment nonprofit; since 2024 its main output is communications and advocacy for halting frontier scaling rather than the agent-foundations math it was known for.

TU Berlin / BIFOLD group on offensive and defensive uses of ML in security: LLM-generated shape-shifting malware, LLM-based vulnerability discovery, personalized phishing at scale and memorization auditing.

Funds AI welfare, digital sentience and cooperative-AI work rather than classical alignment; verified commitments include $15M seed funding to the Cooperative AI Foundation plus grants to NYU's Center for Mind, Brain and Consciousness, the Center on Long-Term Risk and Eleos AI.

Makerere University centre applying AI to African health, agriculture, air quality and low-resource African-language NLP under an explicit responsible-AI framing.

Runs a regranting platform where named regrantors each control an independent budget (typically $100k+) to make $5k-$50k AI safety grants in under a week; 2025 raised $2.25M across 10 regrantors including Neel Nanda, Lisa Thiergart and Aidan O'Gara.

Interpretability research team (ex-DeepMind/Anthropic/Meta) working on 'model mapping' — turning trained networks into human-readable programs; previously known for LLM routing.

Saarbrucken/Kaiserslautern institute whose Social Computing and ML groups work on algorithmic fairness, accountability and transparency of automated decision systems.

Value-alignment methodology: democratic fine-tuning and moral-graph elicitation to derive what values a model should hold from populations rather than from policy documents.

Market-liberal research arguing for permissionless AI innovation and against state-level AI regulation; a significant voice in US preemption debates.

Internal Meta function publishing the Frontier AI Framework risk thresholds for open-weight releases and shipping Llama Guard, Prompt Guard and CyberSecEval as open safeguard tooling.

Public forecasting platform whose AI question series and tournaments produce aggregate community forecasts on capability milestones and AI risk indicators.

Third-party evaluation of frontier models for autonomous capability and dangerous autonomous replication; its time-horizon metric is now widely cited in capability-forecasting policy debate.

Internal Microsoft team that adversarially probes Microsoft and third-party AI systems, maintains the open-source PyRIT risk-identification toolkit, and published lessons from red-teaming 100+ generative AI products.

Corporate research lab publishing on agent frameworks, evaluation and interpretability; distinct from the Microsoft AI Red Team already tracked here.

Closed image-generation service at consumer scale; a named party in the artist copyright suits.

Montreal research institute (Yoshua Bengio) combining deep learning research with an AI safety and governance stream on loss-of-control risk, scientist-AI designs and model evaluation.

Miles Brundage (formerly OpenAI policy research) writes analysis of frontier lab practice, AGI readiness and safety policy.

Frontier multimodal and long-context models; part of the open-weight release wave that sets the diffusion floor for capability.

Represents Kenya in the international AI evaluation network via the Office of the Special Envoy for Technology, and published the Kenya National AI Strategy 2025-2030 which commits to local AI ethics and safety standards enforced through conformity assessments and safety audits.

European open-weight model developer that published a lifecycle environmental-footprint audit of its models and maintains system-prompt and moderation safeguards for its API.

MIT student research and reading group running alignment upskilling programs and supervised research projects for undergraduates and graduate students.

Quantitative research on computing progress and its economic consequences, including automation's effect on wages and employment and the maintenance of the AI Risk Repository taxonomy.

Cross-cutting MIT initiative producing peer-reviewed case studies and course materials on the social and ethical consequences of computing, published open-access at mit-serc.pubpub.org.

Maintains a public knowledge base of adversary tactics and techniques against AI-enabled systems, modelled on ATT&CK, with case studies of real-world attacks on ML and LLM deployments.

Runs paid cohort-based research fellowships in Berkeley that pair new researchers with established AI safety mentors for ~10-week technical and governance projects.

Publishes a free lecture-based machine learning safety course (robustness, monitoring, alignment, systemic safety) with written and video materials.

Technical newsletter summarising recent empirical ML safety papers across robustness, monitoring, alignment and systemic safety.

Open benchmark consortium; runs MLPerf and the AILuminate AI safety/risk evaluation benchmark with continuous prompt stewardship and reliability taxonomies.

Publishes the recurring AI Ethics Brief digest and commentary on AI supply-chain dependence, national AI strategy and the labour and creative-agency effects of generative systems.

Chinese frontier lab behind the Kimi model family; publishes agentic and long-context technical reports with evaluation sections, but no standalone safety framework.

San Francisco coworking and events space (1680 Mission St) used by alignment researchers, AI safety founders and adjacent communities; publishes a live event calendar.

Campaigns and grantmaking for open, accountable AI — Privacy Not Included product reviews, open-source AI definitions, and AI transparency advocacy in EU and US processes.

Chilean national AI centre hosted with PUC Chile that produces the Latin American AI Index (ILIA) and studies generative AI's effects on the Chilean labour market alongside applied AI research.

Operates the US CyberTipline for reports of online child sexual exploitation and publishes data on the growth of reports involving generative-AI-produced abuse material.

Published China's AI Safety Governance Framework (September 2024) and its bilingual 2.0 update, plus Ethics-Safety Guidelines for AI Applications 1.0 and draft sector AI-security guides for finance, health and media.

NSF-funded shared infrastructure letting external researchers run interpretability interventions on the internals of very large open models, via the nnsight library.

Commercial security consultancy whose research team publishes technical work on machine learning system vulnerabilities, prompt injection in deployed LLM applications and AI red teaming methodology.

Rates news site reliability and maintains trackers of AI-generated 'unreliable AI-generated news' sites and audits of chatbots repeating false claims.

Nonprofit that funds and incubates neglected AI safety advocacy projects by experienced professionals.

Independent open-model lab publishing on distributed training and on steering and behaviour-editing methods.

NSF-funded institute headquartered at UC Santa Barbara with 11 partner institutions, researching AI agents for cyber threat intelligence, intrusion response and security operations.

Joint NSF / Coefficient Giving / Good Ventures program (solicitation NSF 23-562) funding foundational research on end-to-end design and implementation of learning-enabled systems with high-confidence safety guarantees.

Works on AI's impact on nuclear command, control and communications and on AI-enabled biosecurity risk through its bio and nuclear programmes.

Publications from the company supplying most frontier training compute; relevant to both capability trajectory and the energy footprint of training.

Empirical work with large language models on scalable oversight, benchmark validity and debate-style evaluation aimed at longer-term risks from highly capable systems.

Astera's neuroscience-inspired AGI programme, building brain-like architectures aimed at systems whose motivations are legible by construction.

Major Indian think tank; its Centre for Security, Strategy and Technology publishes on AI governance from a Global South perspective, cyber norms, and AI in defence.

Garrison Lovely's reported newsletter on the AI industry, labor and existential-risk politics.

Intergovernmental hub tracking national AI policies, maintaining the OECD AI Principles and AI incident monitor, and hosting expert working groups on AI risk and compute.

Carries OpenAI's entire news and research feed, not only Preparedness output: model releases, safety cases, system cards and incident write-ups all arrive here.

The lab's own safety-and-alignment writing, separate from its research index: model specs, safety cases and incident write-ups.

Agent foundations research on formal-goal alignment, specifically the QACI (question-answer counterfactual interval) proposal.

Nonprofit that researched process supervision and factored cognition for delegating reasoning to ML systems; its main artefact, Elicit, is now a separate company.

Open community producing the LLM Top 10 risk list plus guidance on agentic AI threats, red teaming and securing generative AI deployments.

Oxford social science department studying AI-driven opinion manipulation on social platforms, AI companionship and emotional reliance, platform interoperability and EU AI policy.

Technical AI governance research at Oxford: frontier risk evaluation environments, chain-of-thought attacks, interpretability-driven model auditing, robotic foundation model safety and AGI economic policy.

Demonstrates concrete offensive and loss-of-control capabilities of current AI agents — shutdown resistance, autonomous hacking, spear-phishing — as public evidence for policymakers.

Pan-African digital rights advocacy — internet shutdowns, biometric/surveillance law, and the annual Londa digital rights and inclusion report across African states.

Multistakeholder body that develops voluntary practice guidance for AI developers and deployers — synthetic media/provenance framework, model deployment guidance, and work on AI and labour.

LLM evaluation and monitoring company; builds automated judge models (Percival), hallucination/agent-failure benchmarks and RL environments.

Grassroots protest and lobbying network calling for an international treaty pausing training of frontier models beyond current capability levels.

Dutch peace organisation researching autonomous weapons, the arms industry's AI investments, and 'Killer Robots' corporate accountability reporting.

Retrieval-based answer engine; its publisher disputes and citation behaviour bear on information-trust and copyright questions.

Danish national AI centre hosted at the University of Copenhagen with a research track on responsible and trustworthy AI alongside core machine learning.

Runs a full-time summer research fellowship in London placing early-career researchers with mentors on AI safety, AI governance and biosecurity projects.

Peking University lab on RLHF and post-training alignment, safety alignment and superalignment (scalable oversight, deceptive alignment), releasing open datasets and libraries such as PKU-SafeRLHF and Align-Anything.

Decentralised training and open-model infrastructure company; safety-relevant output is limited to open environments/evals and transparency about training runs rather than alignment research.

Runs interdisciplinary fellowships and an affiliate programme bringing complex-systems, physics and biology researchers into AI safety, plus its own foundations-of-intelligence research.

Litigation and investigative research on state and corporate surveillance technology, including phone extraction, biometric ID systems and surveillance exports to the global south.

Open-source LLM evaluation and red-teaming framework; ships attack plugins for prompt injection, jailbreaks and agent-tool abuse, plus a hosted enterprise version.

Alibaba's open-weight model team, publishing model cards and safety/alignment sections in Qwen technical reports covering refusal behaviour and multilingual evaluation.

Analysis of frontier AI risk for US national security including model weight security, AI-bio convergence, compute controls, and military AI decision aids.

Commercial deepfake detection company providing multi-model audio, image and video detection APIs, primarily for financial institutions and call-centre voice fraud.

Develops and stress-tests AI control protocols — monitoring, auditing and containment schemes that aim to stay safe even if a deployed model is deliberately scheming.

Multimodal frontier models and evaluation work from a small independent lab.

Builds time-bound thesis-driven pooled funds; its AI vehicle is the $9M AI for Math Fund launched December 2024 with XTX Markets, which funds AI tools to accelerate mathematics research.

African digital policy research — data governance, AI justice, linguistic sovereignty in AI systems, and African representation in global AI governance processes.

New large nonprofit alignment lab pursuing a portfolio of theory plus empirics (debate/scalable oversight, learning theory) aimed at a priori confidence in alignment before ASI.

Membership nonprofit that builds conformity-assessment tooling, maturity models and certification-style guidance to help enterprises implement AI management standards such as NIST AI RMF and ISO/IEC 42001.

One of two coordinating Dutch AI supervisors, handling technical and product supervision under the EU AI Act — conformity assessment, product safety and market surveillance — while the Autoriteit Persoonsgegevens covers fundamental rights.

Unit within RIKEN's Center for Advanced Intelligence Project researching privacy-preserving machine learning, fairness and social bias in AI, explainability, and analysis of fake news and abusive language.

Helen Toner's newsletter on AI policy, governance and how AI organisations actually make decisions.

Built an AI firewall and automated model-vulnerability scanning; work now continues inside Cisco's AI Defense product line.

Video generation models; its releases move the practical floor for synthetic video, which is the substrate of the deepfake risk.

UKRI-funded doctoral training centre (King's College London and Imperial) producing PhDs on verifiable and trustworthy AI.

Certified robustness and formal verification of deep neural networks using symbolic methods, combining machine learning, optimization and symbolic reasoning.

Develops risk-management standards and grades frontier AI companies' risk-management maturity; contributes to EU AI Act GPAI Code of Practice and ISO/CEN-CENELEC AI standards.

Evolutionary and self-modifying model methods, including automated research agents — directly relevant to unpredictable capability gain.

Volunteer superforecasting group that publishes aggregated group forecasts, including on AI timelines and AI-related catastrophic risk.

Funded an initial $10M cohort of 27 projects on characterizing and forecasting misalignment, safety measurement and intervention methods, and oversight of superhuman and multi-agent systems; PIs include Yoshua Bengio, Dan Hendrycks and Adam Gleave.

University of Toronto institute pairing technical ML security research (Papernot's group) with law and policy work on trust in human-AI interaction and responsible AI governance.

University of Illinois Urbana-Champaign group building trustworthiness benchmarks and red-teaming methods for LLMs and multimodal models, including DecodingTrust-style evaluations.

ETH Zurich lab on LLM security and privacy, sycophancy benchmarking, bias elicitation, verification-error effects in RLVR training and code-security benchmarking for coding agents.

Deepfake detection vendor that also publishes counts and analyses of synthetic media, including early public reporting on Telegram bots generating non-consensual nude imagery.

Geneva-based institute supporting UN-track AI governance — briefings for the Global Dialogue on AI Governance, human oversight of civilian AI, and capacity building for Global South delegations.

Develops theory of latent internal structure in neural networks — how models organise representations and how that structure maps to computation and behaviour (computational mechanics style).

Runs multilingual and multicultural AI evaluation and agentic-AI testing via the Moonshot toolkit, studies emotional dependency and cognitive offloading in AI use, and ran the Singapore AI Safety Red-Teaming Challenge.

Independent lab building Cogames, an open multi-agent benchmark for social intelligence, on the thesis that multi-agent social capability is needed for scalable alignment.

Internal safety function of the Grok developer, publishing model cards, a Frontier AI Framework and pre-deployment safety evaluations.

Builds consent infrastructure for training data — Have I Been Trained, the Do Not Train registry, and the ai.txt/Spawning API used by dataset builders to honour opt-outs.

Open-weight image and audio generation; its releases are central to the copyright and non-consensual-imagery disputes.

Formal verification, safe learning and control, and transparency/explainability methods for autonomous and intelligent systems, plus AI governance and human-oversight research.

Interdisciplinary research and policy analysis on AI's societal effects, including the annual AI Index measurement report and policy briefs for US and international regulators.

Stanford FSI center on social technology and AI; publishes the Journal of Online Trust and Safety and runs the annual Trust and Safety Conference, continuing the trust-and-safety research line formerly housed in the Stanford Internet Observatory.

Monitors European state and EU-level policing, borders and justice policy, including AI-driven border surveillance, Europol data systems and biometric interoperability.

Recruits and accelerates mid-career operators, researchers and policymakers into AI safety and governance roles, with programmes originally targeting India and the Global South.

Multimodal frontier lab (Step series). Publishes through model releases rather than a research blog, so the GitHub org is the live surface.

Research programme on emerging military technologies covering AI in nuclear command and control, autonomy in weapon systems, and export controls on military AI.

Coalition of ~250 NGOs campaigning for a legally binding international treaty prohibiting autonomous weapons systems that select and engage targets without meaningful human control.

Victim-support service run by SWGfL that generates on-device hashes of intimate images so participating platforms can block their non-consensual sharing, including synthetic and AI-altered images.

Music generation; defendant in the major-label training-data litigation that will shape generative copyright.

Runs biannual part-time remote mentorship cohorts pairing early-career people with AI safety and AI policy researchers on concrete research projects; now operated by Kairos.

New York-based litigation and advocacy group challenging NYPD surveillance tools, facial recognition in venues and housing, and geolocation tracking.

Allocates donor funds (chiefly Jaan Tallinn's) to existential-risk organisations via the S-Process; SFF-2025 recommended $1.607M to MIRI, $1.061M to the Center for AI Safety, $919k to FAR AI and $1.5M to SecureDNA.

Independent Indian public-policy school and think tank with a High-Tech Geopolitics programme covering AI regulation, semiconductors, and technology sovereignty.

Places fellows in European AI governance policy roles and runs an EU AI Governance Fundamentals course and policy leaders programme.

Funds and trains journalists to cover AI, placing fellows at major newsrooms and providing grants for AI accountability reporting.

Non-profit publication and convener for tech and democracy policy debate; a primary venue where AI governance researchers and advocates publish analysis.

Tencent's foundation-model family, including open-weight video generation. Releases go out via GitHub and arXiv; no dated research listing found on 2026-08-03.

UK national institute for data science and AI, with programmes on AI safety and security (including work with the UK AI Security Institute), public policy and defence applications.

UK AI Security Institute programme that funds external alignment research grants against a published research agenda; has announced funding for ~60 projects.

US writers' organisation litigating against AI developers over unlicensed use of books in training data and publishing model contract clauses on AI rights for authors.

UK policy institute advising government on extreme-risk management, with workstreams on frontier AI regulation and incident reporting, biosecurity, and cross-government risk-ownership reform.

Practical AI governance implementation work with EU institutions, OECD, GPAI and the UN, focused on the EU AI Act's general-purpose AI rules and international coordination.

Oxford philosophy-based lab (director Philipp Koralus) studying human reasoning and machine reasoning together — e.g. the PyETR reasoning framework, 'humans in, humans out' work, and AI agency/autonomy questions.

Builds detection and classification tooling for child sexual abuse material and leads the Safety by Design initiative committing generative AI developers to prevent models producing abuse imagery.

Mechanistic interpretability lab also working on new architectures and pretraining science; publishes feature/circuit-level analyses of production models.

Applies singular learning theory and developmental interpretability to measure phase transitions and structure formation during neural network training.

Open-model training and inference infrastructure; publishes on long-context and efficient training methods.

Pro-adoption policy advice to governments on AI in public services, state capacity, and national AI infrastructure; a notable counterweight voice to restriction-first AI regulation.

Security research firm auditing ML frameworks and model serialization formats, publishing on model file exploitation, ML supply-chain attacks and LLM agent sandbox escapes.

Oxford coworking building operated by the Centre for Effective Altruism that hosts longtermist and AI safety organisations and events.

Original reporting and analysis outlet (Shakeel Hashim) covering AI labs, safety policy and industry politics, including a campaign finance tracker.

Builds open tooling for understanding model behaviour at scale — Docent for automated analysis of agent transcripts, plus neuron/behaviour description and elicitation methods.

Oregon State University lab studying AI security and privacy: reward poisoning, backdoors, membership inference, indirect prompt injection against web agents and LLM detoxification.

Chinese academic institute on international AI governance; produces AI governance blue books and Track II dialogue with Western institutions on frontier AI risk and military AI.

Value-sensitive design and ethics of technology group at TU Delft working on responsible AI design, moral responsibility for autonomous systems and AI in public administration.

Worker-run organisation and review tool for Amazon Mechanical Turk crowdworkers, letting workers rate requesters and collectively contest unfair rejections and wage theft.

Timothy B. Lee's reported newsletter explaining AI systems and their real-world deployment, often fact-checking capability claims.

UN research institute whose Security and Technology Programme publishes on autonomous weapons, AI in nuclear risk, and military AI governance for state delegations.

UN Secretary-General's multistakeholder body whose 'Governing AI for Humanity' reports underpin the Global Dialogue on AI Governance and the new independent scientific panel.

Toronto institute doing machine learning research with a responsible-AI programme covering bias detection and rewriting, model evaluation and trustworthy deployment tooling.

Independent research on verification and compliance mechanisms for arms control and disarmament treaties, including how emerging technologies affect verification of weapons agreements.

Enterprise AI security platform — automated red-teaming (VirtueRed), runtime guardrails (VirtueGuard) and agentic-system security/compliance tooling.

Swedish national research programme funding humanities and social science work on AI, including global AI governance institutions, AI in policing, and AI and war.

German national research network on digitalisation and society covering AI in the public sphere, democracy and digital work.

Multi-stakeholder alliance of governments, companies and civil society coordinating national responses to online child sexual exploitation and publishing the Global Threat Assessment, which tracks generative-AI abuse material.

Human rights video organisation whose 'Prepare, Don't Panic' programme works on deepfake preparedness, detection access for frontline journalists in the global majority, and provenance standards that do not endanger at-risk documenters.

Chinese frontier lab behind the GLM model family; safety-relevant output is confined to technical reports and evaluation sections accompanying open-weight releases.