Jonathan Huang

The cost of training top models is increasing 2-3X per year, and at this rate will reach $1 billion per model by 2027. The findings are from Epoch AI and follow their recent analysis of increases in compute for foundation models running at 4-5X per year. They've also provided some interesting data on the relative breakdown of cost components like staff, hardware, and energy. A few thoughts after reading this: - I would assume staff costs won't increase as much as hardware and energy costs, so the relative contribution of human labor to future AI models will decrease, in yet another sign of capital's increasing dominance over labor in the industry (just look at NVIDIA's ever-rising stock price). - If compute is increasing 4-5X per year but cost is increasing 2-3X, is it safe to assume that training runs are getting more efficient? - Some people, like Anthropic CEO Dario Amodei, have said we'll see $1 billion training runs in the next year. Does this mean the 2-3X pace is about to increase? That Epoch is wrong in its estimate? Or that Amodei is overstating the cost, possibly to put fear into competitors who can't keep pace? - Only a few companies are going to be able to train future models. We're likely going to see further consolidation. Perhaps we get to a future with only two large general purpose models, like the Windows/MacOS and Android/iOS duopolies we've seen in the past. Article here: https://lnkd.in/g5SejXen

7

$2 H100s: How the GPU Bubble Burst Last year, H100s were $8/hr if you could get them. Today, there's 7 different resale markets selling them under $2. What happened? The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead. https://lnkd.in/gpQEidmC

2

1 Comment

Nearly a decade ago, I wrote out a complete and rigorous proof of Cantor's theorem that the set of sequences of natural numbers is bigger than the set of natural numbers...using only words of one syllable. https://lnkd.in/g4wWYVYc This is not the only time I revisited the result, much more recently I formally published a number of generalisations of this using different fundamental assumptions (for experts: taking Lawvere's fixed-point theorem and the contrapositive diagonal argument and showing how the proof works in a substructural type of logic). But this one-syllable-word proof was so much fun. I was reminded of this because someone got in touch with me, saying how it is something he periodically re-read for pure interest's sake, and asking if I could recover some material that was on the now-vanished Google+ to which I linked. I had in fact saved that discussion, which had contributions of similar pieces from a number of other mathematicians, including the Fields medal winner Tim Gowers. I will put a pdf of that thread somewhere public, soon, and link to it. But my correspondent was so taken by the exercise that he said he'd go so far as to purchase a physical book containing a collection of such "short words proofs", if one existed. The idea of publishing a slim volume of fun little proofs certainly is appealing, in some respects, because I love producing physical artifacts that reflect mathematics, but the market must surely be tiny!

7

2 Comments

Here's the technical breakdown of the AI Penalty Challenge authored by Alok Pattani (check out this video if you haven't seen the demo yet: https://lnkd.in/gsD3xjJ2) Key Takeaways: ☁️ Firestore for triggering slow-mo video capture on Android phones & uploading to Google Cloud via Firebase Storage. 🧠 Cloud AutoML for training custom object detectors (ball & goal tracking) to calculate speed & accuracy. 🗣️ Gemini Pro 1.5 on Vertex AI for personalized coaching feedback based on multiple data points. 🎨 Imagen on Vertex AI for instant, prompt-driven image editing for cool player pics. More technical insights from the project to come... #AIChallenge #MachineLearning #SportsTech #GoogleCloud #footballNOTsoccer

89

5 Comments

🔍 Quantum AI Series: Modeling Sequences with Quantum States Picking up where I left off in my #QuantumAI series, let’s talk about Modeling Sequences with Quantum States, with insights from a fascinating paper on the topic. Classical probability distributions can be modeled using quantum states—particularly entangled states that retain information from subsystems. This approach allows quantum models to efficiently represent sequences without losing essential details. Tensor networks play a key role in optimizing the process, providing a compact, low-rank factorization of data. As I've shared previously, Singular Value Decomposition (SVD) is foundational to these models, as it helps represent the complex correlations present in sequential data. Matrix Product States (MPS) capture these rich interactions by using reduced densities to efficiently compress sequence representations while preserving their structure. This is the essence of keeping representation while building efficient lightweight models. In summary, quantum probability models allow us to go beyond classical limits in sequential tasks, contributing to information retention. Sharing the paper for a deeper dive if you'd like so: https://lnkd.in/d5aqXyyS #QuantumAI #TensorNetworks #SVD #QuantumComputing #AI

5

Here's my timeline of actual and potential tipping points (2022-2032): 2022 (Actual): - ChatGPT launches, demonstrating aligned conversational AI - Stable Diffusion and DALL-E 2 democratize image generation - GitHub Copilot shows practical code assistance - RLHF becomes standard for alignment - Foundation models emerge as dominant paradigm 2023 (Actual): - GPT-4/Claude demonstrate enhanced reasoning - Multimodal understanding reaches new level (GPT-4V, Gemini) - Open source models gain parity (Llama 2, Mistral) - Efficient fine-tuning breakthroughs (QLoRA) - Local LLMs become practical - Global AI regulation begins - Real-time video generation emerges 2024 (Early developments known): - Claude 3 and successors show improved reliability - Enhanced multimodal processing - Better agent architectures and tool use - Advanced video understanding/generation 2024-2025 (Near-term potential): - Major improvement in context length (1M+ tokens) - More efficient transformer architectures - Better grounded reasoning capabilities - Enhanced deployment efficiency - Improved factual reliability 2026-2027: - New training approaches reducing compute/data needs - Better physical world understanding - More sophisticated robot learning - Enhanced casual reasoning - Reliable long-term memory systems 2028-2029: - Integration of multiple modalities at deep level - Novel architectures beyond pure transformers - More efficient hardware architectures - Better world modeling capabilities - Reliable research assistance systems 2030-2032 (More speculative): - New paradigms for AI training - Breakthrough in efficient scaling - Deep physical world interaction - Novel approaches to AI architecture - Major advances in reasoning capabilities Key Pattern Changes: 1. More gradual progression 2. Fewer dramatic breakthrough predictions 3. Focus on incremental improvements in key areas: - Efficiency - Reliability - Physical world interaction - Reasoning capabilities Removed or Reconsidered: - Specific AGI predictions - Autonomous research systems timeline - Dramatic architectural shifts - Consciousness/intelligence breakthroughs - Quantum computing specifics Core Focus Areas: 1. Practical improvements in existing capabilities 2. Efficiency and deployment advances 3. Better reliability and reasoning 4. Gradual integration with physical world 5. Incremental architectural improvements This revision aims to be more grounded in: - Current technical limitations - Resource constraints - Practical development patterns - Observable research trends - Historical development pace Would you suggest any further adjustments to make these predictions more realistic?

44

2 Comments

Humans have historically found the secret to some natural phenomena and put it on steroids. 💉 Consider the aerodynamic efficiency of a bird's wings – a concept we've amplified to create the marvels of airplanes and rockets. ✈️ 🚀 We're seeing something similar right with LLMs. The transformer approach to linking knowledge seems to produce an eerily similar result to what we do. I've seen LLMs hallucinate in a similar way to my 5 year old kid! 😵‍💫 We took the body of knowledge on the internet and created 100+ billion parameter models that hold information that any human mind can. 🧠 But there's more to the natural phenomena of intelligence. There's a few more unlocks that need to be discovered. Like the mechanisms behind novel problem solving. 🧮 It's only a matter of time – in my opinion – until we crack those, put those on steroids and then we're no longer in the pole position of the most intelligent entities. 🤖

3

6 Comments

LLama3 passed the super-proprietary, advanced test that we designed for modern LLMs! almost... You can read about that test here: https://lnkd.in/gPG8WndR When I first saw the response generating, it looked like I was going to have to come back here and apologize for slandering these machines when they were just one iteration away from solving very basic prompt injection issues. But then I looked a bit closer... can you see the mistake? #promptinjection #countingishard #LLMs #LLama3

13

4 Comments

Notion vs Glean reminds me a lot of Tesla vs Waymo... Just like I think Tesla is going to solve the 'software' problem of autonomous vehicles before Waywmo solves the 'hardware' problem... I think that Notion is going to solve 'enterprise knowledge search' before Glean solves the problem of document and knowledge creation... Why? A couple reasons: 1) Just like Tesla, Notion has a huge advantage when it comes to the amount of underlying data and documents to train on... They already have 100M+ users, and likely 10x that in terms of the number of documents, files, etc. in team's workspaces. 2) It's very hard to change people's behaviour re: where/how they create 'work'. It's why a lot of companies have come and gone in that category... As exciting as Glean's AI writing assistants might seem, most people are just fine (or even prefer) a good old fashioned editor/writer. I could be way off here, but this will be an intersting one to follow. Thoughts? Where am I getting it wrong? #ai #enterprise #knowledge #management #notion #glean

3

1 Comment

Professor Lawrence is spot on in here. Most of the ML solutions that have served us well until now have less to do with the current hype on generative AI and more with a specialized domain knowledge combined with classical modeling techniques and a great deal of software engineering. Certainly what we call Generative AI today will have an important role to play as it is introducing a new way to interact with computers to do stuff for you. However thinking that it will solve all industry/society problems is a very big stretch. https://lnkd.in/e_UX_YUd

6

Excited to share our latest work, where we demonstrate the first application of SAEs in a real-world use case that outperforms traditional methods!

11

2 Comments

BERT is back in a new optimized look (ModernBERT). I still remember the old days in 2018. Yes, 6 years in the AI realm is considered old. When BERT model was considered a "large" language model with only 340M parameters. Nowadays Phi-4 model with 14B parameters is considered a "small" language model. I think that the AI world is still in need for these encoders as the BERT architecture despite the dominance of GPT-like decoders. If they are not used as the main models for the use case, the encoders are still an important component for retrieval stage in addition to the decoders for generation stage in any RAG system design. Anyway, welcome back BERT whether you are called large or small!! 🎊

56

2 Comments

💡 ByteDance's Latest Work on Neural Network-Based Molecular Force Fields 💡 I recently read ByteDance’s work on developing a new force field, and it’s pretty good! 👍 ByteFF is a molecular mechanics force field trained on a massive dataset: 2.4 million geometries and 3.2 million torsion profiles. It leverages GNN to predict bonded and non-bonded parameters, showing strong performance in predicting molecular geometry, torsion energy, and conformational energy. This model has great potential for drug discovery and molecular simulation. I do have some concerns about using torsion scan + hessian methods to train non-bonded parameters in this framework. Non-bonded interactions often involve long-range effects, which are challenging for GNNs to capture accurately. That said, the overall results presented in this work are still quite promising. It's exciting to see tech giants like ByteDance stepping into AI for Science, and their vast resources enable them to support DFT calculations on this massive scale. (🤞 Hopefully, they will release the dataset for open use, LOL!) 👉 You can find the paper here: https://lnkd.in/gjmGV4fd 👉 And check out their open-source framework on GitHub, which is well-engineered: https://lnkd.in/gkxcTi6t #AIforScience #DrugDiscovery #MolecularModeling #ForceField #ByteDance

49

1 Comment

Ilya Sutskever confirmed at NeurIPS '24 pre-training scaling law will not be the same. Internet data, the AI fossil fuel, is running out. It is not the end of the AI hype, the rapid rise of foundational AI has outpaced our ability to fully grasp their long-term societal impact. AI agents and applications are poised to transform every facet of our lives, and we will see it happening in coming few years. There will still be improvements in model architecture, and synthetic data creation, so models will improve, just not at the lightning speed they did till now. Looking forward, the single most important milestone in AI is creating systems capable of advancing themselves—AI researchers that can innovate and improve foundational models autonomously. We need to assess how far are we from this stage.

9

The Human-or-Machine Issue In the June 2024 issue of Communications of the ACM, David Harel reflects on Alan Turing's "imitation game" and introduces the broader "human-or-machine" (H-or-M) issue. As machines increasingly mimic humans, Harel explores new social, psychological, and technical questions. Will humans care whether their interacting agent is human or machine? How will behaviors and emotional responses differ in human-machine interactions compared to human-human ones? #HumanComputerInteraction #ImitationGame #Interactions #HumanBehavior

3

2 Comments

Graph theory has a lot to contribute to optimizing graph neural network architecture for improved accuracy and training times. I haven't seen a lot of this in the literature, but a recent paper explores spectral graph properties and the modification of GNNs to incorporate Cayley graph structures to improve GNN performance. There's a lot more that spectral graph/simplicial complex theory, topological data analysis, and other graph-related tools can contribute to optimizing neural network frameworks in general. Here's the paper: https://lnkd.in/ehHVcRjQ Manny Ko, yes, very much Cayley-based tools on this one!

136

10 Comments

Here's a project I did this weekend analyzing the type of state-enforced bias and censorship on Chinese LLMs. This is probably going to be more important moving forward as more of these are not only quite strong (Qwen2 7B Instruct, which I tested is competitive with Llama 3 8B) but more importantly, being liberally licensed - Yi 1.5 and Qwen2 are both largely Apache 2.0 licensed. These are also multi-lingual models, which means it's probably going to end up being used for all kinds of things without downstream users' knowledge. Anyway, in for a penny, in for a pound, so figure I might as well share this widely, since I haven't seen a similar writeup/exploration on this yet and IMO the results are quite interesting: https://lnkd.in/g_WKscA6 Besides the writeup I also applied the recent "abliteration" (orthogonalized refusal vector removal) technique to create a model that largely (~80%) removed refusals without significantly affecting performance and a "deccp" dataset as a starting point for those looking to play around with testing Chinese state censorship alignment of their models. Model, dataset, and all related code are Apache 2.0 licensed and can be found linked from the article.

17

3 Comments

Nice. Greatly useful to improve diffusion models generalization with multi-modal data!

10

1 Comment