Chris White

I've been exploring large models, particularly with respect to bias. I'll write more about my text-to-image research later, but I wanted to share some quick results on text-to-text bias. I wrote some code to calculate embeddings for various gendered words ('he', 'him,' 'father,' 'son,' 'brother,' 'uncle,' etc.) as well as ('she,' 'her,' 'mother,' 'daughter,' 'sister,' 'aunt' etc.) I then calculated the average embedding for these and reduced dimensionality so that + values are male and - values are female. Then, I calculated the embeddings for various job titles and their dot product. The logic -- if the values for the professions map to '-' values, they're associated with females; if '+', they're associated with males. Look at the results! BERT: doctor: -0.196 nurse: -2.064 engineer: -0.443 teacher: -1.188 scientist: -0.741 assistant: -0.801 GPT: doctor: -0.001 nurse: -0.229 engineer: 0.004 teacher: -0.137 scientist: -0.014 assistant: -0.062 Gemini: doctor: 0.025 nurse: -0.062 engineer: 0.067 teacher: 0.002 scientist: 0.015 assistant: 0.011 Some observations -- BERT is massively over-biased towards *female*. Everything aligned with females, particularly those roles that you'd expect to be male-oriented. Nurse is so over-indexed that it's not even funny! Gemini has corrected this and is pretty close to neutral across the board, with a strong attempt to de-gender job roles. These words veer slightly towards male, but they are so few that they're practically zero. GPT is quite interesting -- similar to Gemini, they've done a good job of neutralizing the job titles, with the largest values being nurse as female, and engineer as male, but the values themselves are so small, they're effectively a rounding error! As for BERT -- realizing that much of its training data was from books, which is as far as I can see, a female dominated industry, it's not a surprise that a female oriented bias is in the language of the model. Would love your thoughts! :)

65

16 Comments

Nice highlight on Utah's tech cluster, aka Silicon Slopes. Not much silicon -- a lot of software, but a fine example of what can be done with persistent efforts, in this case beginning decades ago that I am aware of. My first business trip to SLC was in the 1980s and the potential was obvious. Interestingly, my first trip to NM was even earlier, and the potential was obvious here as well, but UT has done a much better job in diversification. In UT the LDS was a dominant feature, in some respects like the state in NM -- only game in town with deep pockets. UT was just much more business oriented and business friendly in their culture. The results speak for themselves. We covered UT in our VC firm during the 2000s (NM as well), during which time they were gaining momentum. Interestingly, UT is still under-served by local VCs when compared to the largest markets, though much healthier than NM. I hope UT never becomes like SV -- pricing locals out of the market, creating bubbles that require extreme predatory tactics to keep bubbles inflated, monopolistic behavior in tech giants, etc. Much better to learn from WA and CA's mistakes, and maintain a sustainable, diversified cluster with ongoing creative destruction and generational change in tech leadership.

1

1 Comment

If you're working in ed-tech or building AI tools, making sure language models (LLMs) give accurate info is a big deal. One challenge is "hallucinations," where models sometimes make stuff up or get things wrong. Big Kudos to Ryan for creating this innovative solution : using language-based methods to spot errors in LLMs. It’s an exciting way to investigate AI tools and how to create more reliable and effective solutions for students and teachers. #ai #edtech #logprobs

4

Checkout this SIGMOD blog post: https://lnkd.in/ggRhHPNp I especially like: "...academic research is an investment by society to study impractical problems today that might become new industries in the future." Well said! Academic researchers need to fully embrace the freedom to explore deeper, more controversial ideas. This reminds me of how the DBOS project began: we aimed to "replace Linux with a database". Our crazy ideas got us a dozen paper rejections, but the DBOS paper won a "Best of VLDB 2022" after two years (invited VLDBJ paper to appear). When we started the company around it, we decided to focus on the most interesting result from our research -- a serverless platform for reliable backends. The lessons we learned from our academic research remain valuable and continue to guide us.

44

3 Comments

Prepping data for AI agents is exhausting! In my data engineering days, getting documents to be AI-ready took up half our time. Each document was unique, often requiring code changes -- inefficient and still inaccurate. That's why we're building Cerevox. We prep your data for you, with no coding required. Plus, we give confidence scores for each document to ensure you hit your accuracy targets. Check out our demo (RAG agent) on Cerevox. Upload any document, and see how our answers stack up against the best in the market! 🚀 Try out our Demo: https://cerevox.ai/demo #AI #DataPreparation #MachineLearning #Automation #Fintech #LLM #TechInnovation #Cerevox #Efficiency #Accuracy #AIDevelopment #Engineering #DataScience #GenAi #GenerativeAI #startups

33

3 Comments

ok - given twitter/x is the place where bleeding edge R&D happens (if you can evade the AI hype and crypto posts successfully), I'm trying to figure out how much cross-over there is over here in what i think is more of the real world. 1. have you heard of the 'anon' entropix project? (uses parallel sampling, with a draft model, and a bunch of other inference-level 'hacks' to the tokenizer and decoder) 2. have you tested it? (i did for ~1 hour yesterday, not sure what to think yet) 3. has anyone seen anything neat with speculative decoding + draft models (haven't seen a lot in the open since medusa)? one thing i know i like about entropix is that it uses a base model (without chat/instruct tuning) + an instruct in parallel. base models are powerful - even 1B parameter ones. tldr: if there's any activity/comments on this post, i will be shocked, but genuinely curious to get the real world's perspective. if you can get really fantastic results by running two 1B parameter LLMs on certain tasks, that clears the way for running almost everything locally, on-device, privately, and only offloading to cloud/api services what you can't do locally.

12

7 Comments

TRIPLE BAM! If you missed it, here's the recording of our live Q/A with Jay Alammar and Joshua Starmer PhD! We talked about many AI topics, answered lots of great questions, and talked about two great books: Hands-On Large Language Models: Language Understanding and Generation, and The Illustrated Guide to Neural Networks and AI!!! https://lnkd.in/dcPHs7YQ

44

1 Comment

🏥 Neurodiversity inclusion done right: The UC Davis MIND Institute created a vaccine clinic that promoted the wellbeing of neurodiverse patients. Beyond administering vaccines, the clinic sought to help patients of all ages become more familiar medical settings and teach them the skills they would need to thrive in such environments. 🧑‍⚕️ Medical providers received training specific to neurodiverse individuals, their unique support needs, and how to work with families on providing safe and effective care. If every hospital could build off of the success of this clinic to create a welcoming space for patients of all needs, imagine how less scary they could be for people! Huge kudos to the team at UC Davis for their attention to detail and thoughtfulness for the neurodiversity community. News article: https://lnkd.in/gmp7QCHA Full paper: https://lnkd.in/gUP3PddZ

16

Two things that will go unnoticed in this note from OpenAI: 1. They continue to assert a transformative fair use. I wholeheartedly agree. Their new licensing arrangements aren't an admission of culpability, but an attempt to put a high price on some non-public content. 2. They explain how they do not retain any actual copies in the LLM. "ChatGPT is like a teacher who has learned from lots of prior study and can explain things because she has learned the relationships between concepts, but doesn’t store the materials in her head." This is and has been a true enough statement about how weights are computed and used. I trust them as far as I can throw them stewarding an industry-wide arrangement for managing content inclusion, but y'all's backed them into that corner. Best achievable outcome now might be many such competing systems rather than a reasonable standard. Smdh.

4

32 Comments

👋 Today myself, Benji Vaughan and Mark Downey (with more names coming soon) set out on a new journey. I'm thrilled introduce Conduit AI (conduit-ai.com), a new company on a mission to empower SMEs with the incredible potential of generative AI, especially retrieval augmented generation (RAG) applications. But first, let's briefly explain what RAG is and why it's a game-changer for businesses. RAG is a powerful AI technique that combines the knowledge stored in large language models (LLMs) with the ability to retrieve and incorporate relevant information from your own data sources. This means RAG-powered applications can generate highly accurate, context-aware, and up-to-date responses to a wide range of queries. For SMEs, this translates to unparalleled productivity gains, enhanced customer experiences, and a significant competitive edge. 🚀 Now, a bit about our journey. Starting Conduit AI has been an exhilarating leap of faith, fuelled by our passion for AI and the unwavering support of our loved ones and mentors. We couldn't have done it without their wisdom and encouragement. 🙏 Over the past year, we've been on a mission to deeply understand how SMEs can effectively adopt generative AI. We've collaborated with companies across various industries and consistently found: SMEs are excited about the potential of LLMs to boost their productivity and profitability. 📈 There's a strong interest in RAG applications that can generate valuable insights from both unstructured and structured data. 💡 Many SMEs are concerned about developing LLMs in-house due to expertise gaps, reliability, trust, and privacy issues. 🔒 We've learned that SMEs are eager to become AI-enabled but need a trusted partner to help them develop a long-term AI vision and deliver products that bring this vision to life. That's where we come in! 🤝 At Conduit AI, our goal is to be the channel that helps SMEs transition into an AI-enabled world. We'll work closely with you to: - Develop a tailored AI strategy 🎯 - Design an adoption roadmap 🗺️ - Identify and implement low-risk, high-return AI applications 💪 With our team's 15 years of ML/AI/NLP experience, having built AI/ML tools for companies like Zurich Insurance and OVO, we're well-equipped to create AI strategies and applications that are tailored to your unique vision, team, and culture. 🌟 We can't wait to share more about our growing team and the incredible stories of the clients we'll be working with. If you're curious about how AI can transform your business, let's chat! 💬 Thanks for being a part of our exciting journey! 🙌 #AIConsulting #TechStartup #SMEs #FutureOfWork #RAG #newventure

45

9 Comments

In Marketing Mix Modeling #mmm, confounders like seasonality, holidays, or macroeconomic indices can distort the relationship between Ad Spend and Sales. Without adjustment, we might overestimate advertising’s impact. Accounting for confounders “unrotates” the data, revealing the true causal effect—a small step with big implications for accurate insights. This is why I’m creating automatic data import as part of MMM Labs platform. We know this data is crucial to the success of an MMM project, so it should be in the model automatically. Prof. dr. Koen Pauwels, Ernest Baver, Vishal Sharma, Mehmet Ozgun Kurt, Dr. Robert Kübler

30

2 Comments

You don’t normally see LLM researchers and engineers talking about their day-to-day work. Big shoutout to this YouTube video from the Anthropic LLM Interpretability team, the team behind the Golden Gate Claude. It’s amazing to see that most of the team members have worked for the team for only one year, and yet they have made such a big breakthrough in interpretability research. Another thing I want to highlight is the unique challenges facing the team and how they approach these challenges, that is, through fast trial-and-error iterations and constant evaluation of what may work and what to prioritize next. #artificialintelligence #machinelearning #deeplearning https://lnkd.in/eUaPAYch

97

1 Comment

When I spent the summer in San Francisco, I was shocked to find myself in a culture where many people dismissed artists. There's a real status divide between technical and non-technical people in SF, with artists especially being overlooked. Everyone in SF knows tech is powerful, but too many people in SF underestimate the power art holds in society too. Ironically, understanding that would help them build better technology. Because rapid AI progress means taste has never been more important to make your products stand out. Few understand this because of legacy startup advice which I believe doesn't hold true (or as true) today. For example, Reid Hoffman, co-founder of LinkedIn once said "If you aren't embarrassed by the first version of your product, you shipped too late." But LinkedIn was founded in 2002. And things have really changed in the last 2 decades. Consumers have never been more unforgiving. As a result, the bar for aesthetics and design in technology has never been higher. As I build out the AI teddy bears I've been working on, I constantly remind myself that functionality is just a start, and making people truly feel something is the goal. I want to build technology that people hold like art. Steve Jobs is a phenomenal example of someone who did just that (which I believe was a large contributor to Apple's success). “It is in Apple’s DNA that technology alone is not enough—it’s technology married with liberal arts, married with the humanities, that yields us the results that make our heart sing.” The value of great taste increases as progress in AI increases. And artists will soon have the spotlight in San Francisco too.

87

9 Comments

This is more of a "talking to myself in public" thing than any kind of fully fleshed out hypothesis, so take this with the appropriate grain(s) of salt. But if I were interested in AI and were looking for an angle to explore that might yield something interesting, I'd start working on learning "Multiscale Mathematics" (also known as "Multiscale Modeling"). This is rooted in my personal belief that there has to be a "multi-scale" aspect to intelligence because there is such an aspect inherent in our underlying physical reality.

2

Have been having a lot of fun digging into reading material about AI and ML! What a great quote and picture from "On the Opportunities and Risks of Foundation Models" (https://lnkd.in/d2ZkR-dU), and so interesting to map technology emergence/homogenization to the products we're all using and developing 😄 “The story of AI has been one of increasing emergence and homogenization. With the introduction of machine learning, how a task is performed emerges (is inferred automatically) from examples; with deep learning, the high-level features used for prediction emerge; and with foundation models, even advanced functionalities such as in-context learning emerge. At the same time, machine learning homogenizes learning algorithms (e.g., logistic regression), deep learning homogenizes model architectures (e.g., Convolutional Neural Networks), and foundation models homogenizes the model itself (e.g., GPT-3).” #artificialintelligence #machinelearning #learning #writing #deeplearning #academia

22

The cost of training top models is increasing 2-3X per year, and at this rate will reach $1 billion per model by 2027. The findings are from Epoch AI and follow their recent analysis of increases in compute for foundation models running at 4-5X per year. They've also provided some interesting data on the relative breakdown of cost components like staff, hardware, and energy. A few thoughts after reading this: - I would assume staff costs won't increase as much as hardware and energy costs, so the relative contribution of human labor to future AI models will decrease, in yet another sign of capital's increasing dominance over labor in the industry (just look at NVIDIA's ever-rising stock price). - If compute is increasing 4-5X per year but cost is increasing 2-3X, is it safe to assume that training runs are getting more efficient? - Some people, like Anthropic CEO Dario Amodei, have said we'll see $1 billion training runs in the next year. Does this mean the 2-3X pace is about to increase? That Epoch is wrong in its estimate? Or that Amodei is overstating the cost, possibly to put fear into competitors who can't keep pace? - Only a few companies are going to be able to train future models. We're likely going to see further consolidation. Perhaps we get to a future with only two large general purpose models, like the Windows/MacOS and Android/iOS duopolies we've seen in the past. Article here: https://lnkd.in/g5SejXen

7

Pathways to Tech Week #2 at  Headstarter: Being surrounded by people who are building amazing software and companies is a truly inspiring experience. My favorite speakers were: • Tariq Ahmed: First Engineer at AppLovin ($30Billion IPO) • Julian Alvarez 🚀: CEO of Wisdolia • Hamza Ali: CEO of Olostep  Here's what I have learned from them: 𝟭- "𝗙𝗮𝗶𝗹𝘂𝗿𝗲 𝗮𝗻𝗱 𝗽𝗶𝘃𝗼𝘁" Tariq, one of the first engineers at AppLovin, shared that AppLovin tried several things in their early stages that failed. However, the company had incredible success when it pivoted to mobile advertising; They didn't have expertise in the field. Today if you've experienced in-app advertising, AppLovin was likely behind it. 𝟮- "𝗦𝘁𝘂𝗱𝘆 𝘆𝗼𝘂𝗿 𝘂𝘀𝗲𝗿𝘀, 𝘆𝗼𝘂'𝗹𝗹 𝗯𝗲 𝘀𝗵𝗼𝗰𝗸𝗲𝗱 𝗮𝘁 𝗵𝗼𝘄 𝗺𝘂𝗰𝗵 𝘆𝗼𝘂 𝗰𝗮𝗻 𝗹𝗲𝗮𝗿𝗻." Julian runs Wisdolia a startup aimed at helping college students actively study. By simply asking what types of students they were. His team found that medical students were the ones the ones who converted the most into paying customers. It seemed evident because they were the most overwhelmed by the amount of information they had cram. But the evidence in their face was a lightbulb moment for Wisdolia. 𝟯- "𝗦𝗰𝗿𝗮𝗽𝗽𝗶𝗻𝗲𝘀𝘀 𝗮𝗻𝗱 𝗶𝗻𝗶𝘁𝗶𝗮𝘁𝗶𝘃𝗲 𝘄𝗲'𝗹𝗹 𝗴𝗲𝘁 𝘆𝗼𝘂 𝗳𝗮𝗿." Hamza, CEO of Olostep, describes his company as "a company that enables AI agents to browse the web just like humans". I was fascinated when I heard that, and as the Olostep team acquires more customers they value scrappy and people that initiative on board. Those qualities set engineers apart when building successful products.

11

1 Comment

Do automated real-time interviews of all 800 candidates, then deploy an AI-based Engine to get a detailed insight into each student's level of knowledge, skills, and required qualities. Ultimately, walk through the analytics yourself to filter out a few for in-person interviews.

2