<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Spatial Pixel - Medium]]></title>
        <description><![CDATA[Real World Computing - Medium]]></description>
        <link>https://medium.spatialpixel.com?source=rss----9eb53d32da6b---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Spatial Pixel - Medium</title>
            <link>https://medium.spatialpixel.com?source=rss----9eb53d32da6b---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:56:45 GMT</lastBuildDate>
        <atom:link href="https://medium.spatialpixel.com/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Attached To Something Real]]></title>
            <link>https://medium.spatialpixel.com/attached-to-something-real-0068516d8c57?source=rss----9eb53d32da6b---4</link>
            <guid isPermaLink="false">https://medium.com/p/0068516d8c57</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[spatial-computing]]></category>
            <category><![CDATA[interaction-design]]></category>
            <dc:creator><![CDATA[William Martin]]></dc:creator>
            <pubDate>Mon, 23 Sep 2024 16:34:21 GMT</pubDate>
            <atom:updated>2024-09-23T16:34:21.350Z</atom:updated>
            <content:encoded><![CDATA[<h4>This Week at <a href="https://spatialpixel.com/">Spatial Pixel</a> — Sep 16, 2024</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*3OesKosy_Lks4GQE.gif" /></figure><p>1 — <strong>Real Physical</strong><br>We added Internet-connected outputs to <a href="https://spatialpixel.com/procession/">Procession</a>.</p><p>2 — <strong>Real Certain?</strong><br>Spatial Pixel spoke at CUNY / AIA Brooklyn in a talk on AI and spatial computing.</p><p>3 — <strong>Real Trustworthy</strong><br>Guest Speaker: Prem from Google, Data Commons spoke on trustworthiness in LLMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*kQf84uJxsoaG_Fc6.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/0*McdgvwhBkqBYZMx3.png" /></figure><p><em>How might we enable spatial computing beyond glasses? — In the spirit of fostering an open environment for research to enable this future, we’ll be sharing our experiments.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6XvE3mnQLr6p6iMyHnMRA.gif" /><figcaption>Some interactive sketches made with Procession</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/0*ChGflQ8f5PCWg8nq.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*qPB22xP8h4JN5N3j.gif" /></figure><h3>1 — Real Physical<br>We added Internet-connected outputs and hardware to Procession.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*qFtsKuFbO2a5CRX4.gif" /></figure><h3>NEW: Internet-Connected Outputs and Hardware</h3><p>We extended <a href="https://spatialpixel.com/procession/">Spatial Pixel’s Procession</a> with an integration with the excellent <a href="https://ifttt.com/explore">IFTTT</a> service, so we can connect to all kinds of automation routines downstream, anything supported by IFTTT. What’s exciting about this for us is Procession can now enable natural language to <strong>program physical hardware</strong>. This means behaviors like, “when I raise my hand, turn the living room light on,” or, “if you see someone enter the room, then raise the temperature to a comfortable level.”</p><blockquote>“When I raise my hand, turn the living room light on…”</blockquote><p>This is a big step for making Procession a flexible tool for nimble and accessible spatial computing and extending its capabilities beyond the immediate vicinity around the computer that’s running it. Because IFTTT is already so capable, you could control devices around your room, your home, or even around the world, with Procession interpreting the logic you want just from what you prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QXGwTQdINGAVnKZzoC0Jwg.png" /><figcaption>Procession connection to IFTTT — enables connection to sensors, actuators, and services like email and text</figcaption></figure><p>We’ve been thinking more and more about how to <a href="https://medium.spatialpixel.com/periodic-table-of-spatial-computing-3f40ed162b2e">systematize these typically complex-to-program-hardware sensors and actuators</a> in a way that’s less focused on the hardware-ness and more focused on what the hardware can do. The Google Nest makes it hotter or cooler. A Ring camera sees. Is there a future when we don’t care as much about the specifics of the hardware (or the manufacturer), and we can speak more in terms of what we want a program to do?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/106/1*ATKgTNCK4qp7W_lLmHhs3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*IJ06pg9PJMDDXGSxopscGw.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/106/1*ATKgTNCK4qp7W_lLmHhs3Q.png" /><figcaption>A room with programmed sensors (sketch made with Procession)</figcaption></figure><p>We believe there’s huge potential in creating spaces that can be interactively programmed, <em>and</em> that it’s not just up to the Amazon Echos or Google Nests, but that everyday people could write their own rules and behaviors for how their spaces should behave. It would be great to program this way:</p><blockquote>“Whenever I look at the clock, read the time. If I wave my hand, read out any upcoming meetings. When I sit in my reading chair with a book, turn on the reading light.”</blockquote><p>Students in the Physical Computing course at University of Michigan’s Urban Technologies program were testing this feature this week, and they have already started coming up with some truly innovative ideas. We’ll be sharing what they conjure as they go!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*-KBEkdlOcMl8sYO1iC5tHg.gif" /><figcaption>Procession experiment by Matt Kish @ University of Michigan, using gaze to highlight a painting.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*x65Jv3CuGevrZz6X.gif" /></figure><h3>2 — Real Certain?<br>Spatial Pixel spoke at CUNY / AIA Brooklyn in a talk on AI and spatial computing.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*x65Jv3CuGevrZz6X.gif" /></figure><p>William spoke about Spatial Pixel’s work at a joint event between City Tech (now the largest architecture school in the country, thanks to their recent accreditation!) and AIA Brooklyn in a talk series called, “Uncertainty: the Artificial Intelligence Spectrum.” The panelists were selected by the student body, so we were especially honored to be invited!</p><p>Many of the uncertainties discussed were from:</p><ul><li>What is artificial intelligence?</li><li>Potential environmental impacts of AI</li><li>Potential impact on jobs and labor</li><li>What role should AI take in our practices?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QL3dvNkFu2klEiWtTFK19Q.png" /></figure><h4>How to explore AI’s uncertainties.</h4><p>Uncertainty is a recurring theme in AI, but it might not always be a bad thing. When people are uncertain, we’re expressive about it. We might say, “I’m inclined to think…,” or “I’d bet…,” or even use qualifiers like “possibly.” We even shift our tone — expressing facts as questions, or lift our heads upward as if were pondering thought clouds above us.</p><p>If AI could be more transparent about its uncertainty, it might even make us humans more inquisitive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rPpyyaHDmHH__2j0Mr6ugw.png" /><figcaption>Clip’s from William’s talk on AI at Spatial Pixel. You can find more about his Spatial AI course and work <a href="https://awmartin.xyz/">here</a>.</figcaption></figure><h4>Where is uncertainty valuable?</h4><p>Perhaps more importantly than expressing its uncertainty is just being more deliberate about how we use AI. Where is generative AI’s non-exacting nature useful, and where is it more problematic? Is AI better as an all-knowing oracle, or a critic to help us question our work as we go?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cgv0ILnMTbh48BybxrIegQ.gif" /><figcaption>Clip’s from William’s talk on AI at Spatial Pixel. You can find more about his Spatial AI course and work <a href="https://awmartin.xyz/">here</a>.</figcaption></figure><p>At Spatial Pixel, we’re most interested in finding ways to use AI to help creatives <strong>elevate their level of expression and cognition</strong>. For a coder, looking up the quirks of a specific library on Stack Overflow might be the very kind of thing that keeps their heads in the weeds, distracting from spending more thought-time with the creative (and fun) parts of coding. Similarly for visual and spatial designers, can we use AI to help us spend less time clicking buttons to manipulate geometries, and more time exploring meaningful design concepts?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*jOljOIkn5talgkrz.gif" /></figure><h3>3 — Academic Recap — Trustworthiness in AI</h3><p><em>Academic Recap: </em><a href="https://www.linkedin.com/in/awmartin/"><em>William Martin</em></a><em> teaches “</em><a href="https://awmartin.xyz/proposing-spatial-ai/"><em>Spatial AI</em></a><em>” and “Colloquium I for Computational Design Practices” at Columbia University GSAPP. </em><a href="https://www.linkedin.com/in/violetwhitney/"><em>Violet Whitney</em></a><em> teaches “</em><a href="https://docs.google.com/document/d/1vigGsSX3gb3e0dguQjL4Alh9EfiQQHZI_rXaq4s1F6w/edit?usp=sharing"><em>Spatial UX</em></a><em>” at Columbia University,“</em><a href="https://docs.google.com/document/d/18ynHxmvIxhBs9sZL9AC9EPOLcWyJV7zMfPEuSlBuFXI/edit?usp=sharing"><em>LLMs for UX Prototyping</em></a><em>” at University of Pennsylvania,“</em><a href="https://docs.google.com/document/d/158uyTRY3nk1XDWfaTCf-7gcgoeEWcEvDQNyTVIvnJjE/edit?usp=sharing"><em>Reading and Writing the World</em></a><em>” at University of Michigan.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*dhCE6ZuRbQmwNALL.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*P4u8R-fRdZnUghd16NoOYQ.gif" /></figure><p><strong><em>Prem Ramaswami, Head of Data Commons, Google<br></em></strong><a href="http://datacommons.org/"><strong><em>DataCommons.org</em></strong></a><br><strong><em>Guest Lecture: LLMs for UX Prototyping @ University of Pennsylvania</em></strong></p><p>This week, Prem from Google, Data Commons spoke on trustworthiness in LLMs.</p><p><a href="http://datacommons.org/">DataCommons.org</a> is an open source project by Google, which aims to make all the world’s public data universally accessible and useful. It’s an open source project that uses natural language prompts to visualize and analyze open data sources with graphs and scatterplots. They’ve worked closely with the <a href="https://unstats.un.org/UNSDWebsite/undatacommons/sdgs">United Nations</a>, <a href="https://datacommons.one.org/climate-finance-files">The One Campaign</a>, <a href="https://datacommons.techsoup.org/">TechSoup</a>, and many more.</p><ul><li><strong>LLMs can be used to find more accurate data; they just aren’t being used that way</strong> — One thing that struck me about his talk was that there are most definitely ways we can use LLMs to get <em>more accurate</em> information, but unfortunately so many companies are using it irresponsibly. For example, if you’ve ever used ChatGPT to search for statistics, it will gladly point you to seemingly factual stats with links to sources. Unfortunately, if you click any of those sources, you’ll most often find the stat is absolutely bogus. As Prem simply put it, its BS-ing the expertise.</li><li><strong>New feature: RIG and RAG</strong> — Just last week the Data Commons team released a unique method for combining RIG (Retrieval Interleaved Generation) and RAG (Retrieval Augmented Generation) frameworks. If you’re curious you can read more about their method for fact checking <a href="https://docs.datacommons.org/papers/DataGemma-FullPaper.pdf#:~:text=We%20explore%20two%20primary%20methods,to%20augment%20the%20LLM&#39;s%20prompt.">here</a>.</li></ul><p><strong>Links from Prem’s talk below:</strong></p><ul><li><a href="https://datacommons.org/">Data Commons website</a> — main site to use Data Commons</li><li><a href="https://www.youtube.com/watch?v=O6iVsS-RDYI&amp;ab_channel=Google">high level explainer</a> — 2.5 minute video</li><li><a href="https://blog.google/technology/ai/google-data-commons-ai/">medium level explainer </a>— blog post</li><li><a href="https://blog.google/technology/ai/google-datagemma-ai-llm">detailed level explainer</a> — the research blog post for Data Gemma, with access to open models they use from <a href="https://huggingface.co/">Hugging Face</a> and <a href="https://www.kaggle.com/">Kaggle</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6XvE3mnQLr6p6iMyHnMRA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*uBmsIBAuSl6fXryd.gif" /></figure><p><em>Thanks for reading. Have something to share or interested in learning more?</em></p><p><em>We run </em><a href="https://spatialpixel.com/"><em>Spatial Pixel</em></a><em>: spatial computing “for your space, not for your face.”</em></p><p><em>Spatial Pixel is committed to creating and fostering an open environment for spatial design research that can enable a future for spatial computing that is inherently embodied, environmental, and collective. We hope this research can build counter-practices to the glasses-only future being created by Big Tech.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0068516d8c57" width="1" height="1" alt=""><hr><p><a href="https://medium.spatialpixel.com/attached-to-something-real-0068516d8c57">Attached To Something Real</a> was originally published in <a href="https://medium.spatialpixel.com">Spatial Pixel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spatial Pixel — Research, Research & More Research]]></title>
            <link>https://medium.spatialpixel.com/spatial-pixel-research-research-more-research-650bc398b3e0?source=rss----9eb53d32da6b---4</link>
            <guid isPermaLink="false">https://medium.com/p/650bc398b3e0</guid>
            <category><![CDATA[spatial-ux]]></category>
            <category><![CDATA[spatial-computing]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[low-code]]></category>
            <category><![CDATA[lecture]]></category>
            <dc:creator><![CDATA[Violet Whitney]]></dc:creator>
            <pubDate>Sat, 14 Sep 2024 21:29:29 GMT</pubDate>
            <atom:updated>2024-09-14T21:36:51.070Z</atom:updated>
            <content:encoded><![CDATA[<h3>Spatial Pixel — Research, Research &amp; More Research</h3><h4>This Week at Spatial Pixel — Sep 9, 2024</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>1—People are starting to test the Procession beta.<br>2—We added generative-text capabilities.<br>3—We ran a creative AI workshop with local artists.<br>4—We hosted some guest speakers in our classes who spoke about spatial interaction and AI+UX Trends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/1*JmDELk-0a9PRaN033vKADg.png" /></figure><p><em>How might we enable spatial computing beyond glasses? — In the spirit of fostering an open environment for research that can enable this future, we’re sharing experiments as we go.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/1*JmDELk-0a9PRaN033vKADg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>1—People are starting to test the Procession beta.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GCkXv-wsyNmK5FyczbYDvA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BXX8JIU90ftswA1KYkUYUw.png" /><figcaption>Left: clips from Procession, Right: Procession spatial configurations we’re working towards.</figcaption></figure><h4>Who is testing Procession?</h4><p>We’re thrilled at the inbound interest for <a href="https://spatialpixel.com/procession/">Procession</a>, a spatial computing environment “for your space, not for your face.”</p><p>It’s currently being tested as a private beta by three classes, at Columbia University, University of Michigan, and University of Pennsylvania. We’ve also provided access to a few more academic and artist labs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bJi6eggWJ2l5fhDbdZQdxA.gif" /></figure><h4>How can I get access to the Procession beta?</h4><p>We’re a small team committed full time to this dream!</p><p>Because it’s early, we’re first prioritizing developers and researchers who can expand the reach of a glasses-free spatial computing.</p><ul><li><strong>Research Beta </strong><br>If you have an idea in mind for developing a library or extension to Procession, please get in touch by messaging us through our <a href="https://www.linkedin.com/company/spatialpixel/">LinkedIn page</a>. We’re providing access to a small number of researchers who able to contribute new library capabilities. It’s our intent for contributions to first be open-source and non-commercial.</li><li><strong>Non-Research Beta (not released yet)</strong><br>Our broader focused non-research beta has not yet been released. If you’d like to stay tuned for this, you can follow along via our <a href="https://www.linkedin.com/company/spatialpixel/">LinkedIn page</a>.</li></ul><h4>A research brief.</h4><p>This week we were invited to share a research brief for Columbia University GSAPP’s Computational Design Master’s program to solicit student researchers. We’re excited to invite them to explore opportunities with us!</p><p>We’re sharing it below in case you’re also working towards a similar glasses-free future of spatial computing.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fdocs.google.com%2Fpresentation%2Fembed%3Fid%3D1hihQXyOPUZtYdcpVFmyMyeTt7LzloGoOhwshV6dig1k%26size%3Dl&amp;display_name=Google+Docs&amp;url=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fdocs.google.com%2Fpresentation%2Fd%2F1hihQXyOPUZtYdcpVFmyMyeTt7LzloGoOhwshV6dig1k%2Fedit%3Fusp%3Dsharing&amp;image=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Flh7-us.googleusercontent.com%2Fdocs%2FAHkbwyLT6BCcJh3lN2w-I1OyiihgYgXYKBkl0Q9rsgQOAsFolrP2pvLB_WMc4HP-1I6l8Xp_LXwq-aT8fpBOpFbKNfg9I8QX0vyzONWRVC9kCpn-S6TkLXHT%3Dw1200-h630-p&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=google" width="700" height="559" frameborder="0" scrolling="no"><a href="https://medium.com/media/66dcd3f297eec72d80985dbb3a98d46e/href">https://medium.com/media/66dcd3f297eec72d80985dbb3a98d46e/href</a></iframe><h4>A Special Request — Hardware Research</h4><p>We’re particularly interested in folks who are investigating camera-projector combinations and spatial awareness with sensors. If you’re working on this kind of hardware, please reach out!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>2—We added generative-text features to Procession.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h4>NEW: Generative Text — AI <em>inside</em> the AI</h4><p>This week we added generative-text capabilities to Procession. This means that in addition to using to programming with natural language to generate running code—the basic UI flow in Procession—you can also prompt Procession to incorporate text completions from OpenAI (and other APIs in the future). This means you can build sketches that generate haikus, poems, or other meaningful prose. We’ll soon be vetting function-calling to investigate the uses of generative logic.</p><h4>IN PROGRESS: RFIDs &amp; Sensor Inputs</h4><p>We’re currently working on getting more physical inputs from sensors into Procession. We’ve built a working prototype of RFID sensor inputs (commercial readers and an Arduino-based ones) into Procession and workshopped this on Friday with artists (see below). We’re also aiming to extend this to environments like IFTTT, by which we can connect to all kinds of sensors and actuators downstream. This can help spatial computing extend beyond an individual device and more into the physical hardware that can enable an intelligent space.</p><h4>FUTURE: Open Programmable Spaces</h4><p>We’re also working towards a more general beta. We’re inspired by environments like <a href="https://www.openinterpreter.com/">Open Interpreter</a>, <a href="https://processing.org/">Processing</a>, <a href="https://p5js.org/">P5.js</a>, and <a href="https://dynamicland.org/">Dynamicland</a> that are community led and open. We hope to work with other researchers to build out such an environment.</p><p>Our long term vision is to have a programming environment where <strong>artists, creatives, and spatial designers</strong> can build intelligent spaces and develop and share research in the space in an open way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JyM5hg0vVdfzApkiYAGSbA.png" /><figcaption>Spatial Pixel’s Procession: product architecture we’re building towards</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>3—We ran a creative AI workshop with local artists.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h4>AI with not for Artists</h4><p>This week we ran a creative AI workshop with a group of artists at the <a href="https://www.moabank.org/">Ministry of Awe in Philadelphia</a>. Artists came up with ideas for material objects throughout the museum that could serve as input into Procession and then eventually combine into a collective, generated, realtime mural.</p><p>The Ministry of Awe is a new immersive museum being collectively created by artists in the heart of Philadelphia’s Old City, spearheaded by the renowned <a href="https://en.wikipedia.org/wiki/Meg_Saligman">mural artist Meg Saligman</a>. She is pulling together multimedia artists from all domains, from painting to projection mapping to object-making. We’re excited to share more of our work with them as it develops.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gX0eP3sWHUSKXtNwtd9Vlw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>4—Academic Recap</h3><p><a href="https://www.linkedin.com/in/awmartin/"><em>William Martin</em></a><em> teaches “</em><a href="https://awmartin.xyz/proposing-spatial-ai/"><em>Spatial AI</em></a><em>” and “Colloquium I for Computational Design Practices” at Columbia University GSAPP. </em><a href="https://www.linkedin.com/in/violetwhitney/"><em>Violet Whitney</em></a><em> teaches “</em><a href="https://docs.google.com/document/d/1vigGsSX3gb3e0dguQjL4Alh9EfiQQHZI_rXaq4s1F6w/edit?usp=sharing"><em>Spatial UX</em></a><em>” at Columbia University,“</em><a href="https://docs.google.com/document/d/18ynHxmvIxhBs9sZL9AC9EPOLcWyJV7zMfPEuSlBuFXI/edit?usp=sharing"><em>LLMs for UX Prototyping</em></a><em>” at University of Pennsylvania,“</em><a href="https://docs.google.com/document/d/158uyTRY3nk1XDWfaTCf-7gcgoeEWcEvDQNyTVIvnJjE/edit?usp=sharing"><em>Reading and Writing the World</em></a><em>” at University of Michigan.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*fWH3Vp2xD-X0KdB8LyimIg.gif" /></figure><p><strong><em>Josh Goldblum serial entrepreneur, Founder &amp; CEO<br></em></strong><a href="https://www.bluecadet.com/"><strong><em>Bluecadet</em></strong></a><strong><em>, </em></strong><a href="https://futurespaces.com/"><strong><em>FutureSpaces</em></strong></a><strong><em>, </em></strong><a href="https://artwrld.com/"><strong>Artwrld</strong></a><br><strong><em>Guest Lecture: Spatial UX @ Columbia University</em></strong></p><ul><li><strong>Find Your People—</strong>As a serial entrepreneur, Josh talked about working on the cutting edge and “finding your people.” If you don’t have a scene make it. Josh struck me as an explorer. Someone who’d gladly venture into new scenes whether that’s the world of crypto or the world of artists.</li><li><strong>It’s okay to pivot—</strong>As someone working in emerging spaces, a big part of working in an exploratory way is being okay to change course as you learn. When asked about shifting Artwrld from NFTs to artist talks, Josh spoke about taking stock of what you have. As the market shifts, or the ethos of movement shifts, don’t be afraid to change!</li><li>Link to <a href="https://www.bluecadet.com/"><strong>Bluecadet</strong></a>—many interactive spatial experiences</li><li>Link to <a href="https://futurespaces.com/"><strong>FutureSpaces</strong></a>—talks/community on physical/digital installations</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*30TzQ8l50QBwembvfPhifw.gif" /></figure><p><strong><em>Kshitij Agrawal, Creator of <br></em></strong><a href="https://aiverse.design/"><strong><em>AI Verse</em></strong></a><strong><em> AI + UX Trends<br>Guest Lecture: LLMs for UX Prototyping @ University of Pennsylvania</em></strong></p><ul><li><strong>Research Aloud</strong>—Kshitij has been researching and categorizing interaction patterns and trends emerging around AI. His research has garnered quite the following since so much of what he does is useful those of us also trying to figure out this AI thing too.</li><li><strong>Trend: Nonlinear &amp; Agentic UX</strong>—Techniques that give a user control to configure a UX flow in a <em>nonlinear</em> fashion. Natural language programming is all about being able to structure the flow of information.</li><li><strong>Trend: Ambient</strong> — Embedded directly on top of the environment around you: the tables, the walls, the books. Most of these interaction patterns exist in the world of museum installations where Josh Goldblum (above) and Bluecadet reside. But these types of interactions are becoming much more meaningful to the world of everyday technology.</li><li><a href="https://drive.google.com/file/d/1Xf4oIQIkNDQ2HSyXcKbjyMXVSuQdbhw6/view?usp=drive_link">Link to his presentation.</a></li><li><a href="https://aiverse.design/">Link to his website</a>.</li><li><a href="https://x.com/aiversedesign">Link to his Twitter/X.</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p><em>Thanks for reading. Have something to share or interested in learning more?</em></p><p><em>We run </em><a href="https://spatialpixel.com/"><em>Spatial Pixel</em></a><em>: spatial computing “for your space, not for your face.”</em></p><p><em>Spatial Pixel is committed to creating and fostering an open environment for spatial design research that can enable a future for spatial computing that is inherently embodied, environmental, and collective. We hope this research can build counter-practices to the glasses-only future being created by Big Tech.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=650bc398b3e0" width="1" height="1" alt=""><hr><p><a href="https://medium.spatialpixel.com/spatial-pixel-research-research-more-research-650bc398b3e0">Spatial Pixel — Research, Research &amp; More Research</a> was originally published in <a href="https://medium.spatialpixel.com">Spatial Pixel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flat screen. Flat thoughts.]]></title>
            <link>https://medium.spatialpixel.com/flat-screen-flat-thoughts-2f55efa1dedb?source=rss----9eb53d32da6b---4</link>
            <guid isPermaLink="false">https://medium.com/p/2f55efa1dedb</guid>
            <category><![CDATA[lecture]]></category>
            <category><![CDATA[embodiment]]></category>
            <category><![CDATA[the-extended-mind]]></category>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[embodied-cognition]]></category>
            <dc:creator><![CDATA[Violet Whitney]]></dc:creator>
            <pubDate>Mon, 02 Sep 2024 10:49:06 GMT</pubDate>
            <atom:updated>2024-09-02T10:49:06.392Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uzyPo8_O1ehqB3cDYby2pg.png" /></figure><h4>User Interface Bureaucracy and How a Philosophy from 1900s Capitalism Could Save Us.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>Chapter 1 —Flat screen. Flat thoughts.</p><p>Chapter 2— We need space to think.</p><p>Chapter 3 —Can we study <strong>knowledge work</strong> like <strong>manual work</strong>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*eCT_2mLmfUY8c67LOs3Wuw.gif" /></figure><blockquote>I have this syncing feeling that every little task I have inside the computer is like a mini <a href="https://en.wikipedia.org/wiki/Bullshit_Jobs#cite_note-1">bullshit job</a> — a bullshit task…</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/16/1*FRdJdtoXFhMo8PzBlVlHbw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>Chapter 1 — Flat screen. Flat thoughts.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>If you’re anything like me, you spend most of your day looking at your screen. If you’re a knowledge worker it’s likely worse.</p><p>Steve Jobs said the <strong>computer is like a bicycle for the mind</strong>. <br>This is <em>my</em> “mind bicycle” right now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w3DTctS_R7mHZPjJYFx4Ng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/16/1*FRdJdtoXFhMo8PzBlVlHbw.png" /></figure><p>I think <em>my</em> mind bicycle has two flat tires going uphill both ways.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/16/1*FRdJdtoXFhMo8PzBlVlHbw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2tBQq498boKQVtB-2yjmxg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wcRrs7lp4oAooKheX0NluQ.png" /></figure><h4>Computers — like a scattered pile of papers</h4><p>What makes computer work feel <em>so</em> scattered? Well, computers today lack the spatial structure that we’re used to in the material world. So while working on a computer requires tons of context switching between apps, there’s very little spatially to help us organize the context switching between apps.</p><p>I sometimes find myself clicking around tabs having completely forgotten what I was even doing. Was it my calendar I needed? A link from an email? Was I looking up a reference on the web?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dPCkigFj7722rN4TlVIdqw.png" /></figure><p>Even more concerning is that we’re spending more and more time on our computers, often missing the world and people around us. It’s impacting how we think: <a href="https://pubmed.ncbi.nlm.nih.gov/35430923/">shortening children’s attention spans</a>, <a href="https://www.apa.org/news/podcasts/speaking-of-psychology/anti-empathy-machine">making us less empathetic</a>.</p><p>We’re burnt out and screen fatigued. <br>And yet, we’re all still here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>Chapter 2 —We need space to think.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>If you were a designer when tools like Miro and Mural emerged, it felt like a big step forward. Their killer feature: just the simple ability to <strong>zoom out</strong> on a canvas. What it meant was that you could finally organize work spatially. You could collect and group material like we do in the real world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*C6KP4AzIL654wS4jsrsPKQ.gif" /></figure><p>For knowledge workers who need to organize, research, brainstorm, or structure plans — the <strong>spatial relationships</strong> <em>are</em> the thinking. So the space helps us think.</p><h4>We think through our bodies, not just our brains.</h4><p>Humans have embodied cognition. Our thoughts, reasoning, and behaviors are affected by our physical body. How information is organized and arranged, and our sensory interaction with the environment impacts our thinking. Students in math competitions <a href="https://www.youtube.com/watch?v=y3e2DNXMq1A">use physical gestures</a> to solve the problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/1*1ul6PxjPoSqt7clg7yZReA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*6wUlU6yO1g5fGTCNgjo5kQ.gif" /></figure><p><strong>We use proximity </strong>— we’ll place tools we’re using now nearby and tools we plan to use later far away. We naturally organize our desks spatially. We put sticky notes on the sides of our monitors to remind us in a way that isn’t distracting, but close enough to help us remember. A sticky note is basically useless if it’s in the wrong place.</p><p><strong>We group spatially — </strong>We group like items together: tools and materials for sewing and laundry get stored together while tools and materials for cooking go another place.</p><p><strong>We use our gestures</strong> — I might say, “let’s brainstorm this later,” while pointing at a sketch. Or “can you make this part blue while circling an area of a drawing.” I might point at something and ask “what is it?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*8aK_ao6q3kSJMfXlUlt9rg.gif" /><figcaption>motion capture of my finger moving around</figcaption></figure><p><strong>We use imitation</strong> — an animator might demonstrate how water in a scene should behave by acting out its motions. A choreographer shows dancers how to move. Toy cars might can help us understand the the events of a car accident.</p><p><strong>We use material affordances</strong> — Building physical models or prototypes might help us understand how elements work together and behave. “<em>This</em> will slide off of <em>that</em>.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*3vxXJ9zaW2-wRI4ALM2IuA.gif" /><figcaption>thinking in the shower</figcaption></figure><p>Some of my best thinking comes when I’m <strong>away from the computer</strong>. When I’m out for a walk I can associate ideas more freely. When I make a physical prototype it quickly clarifies what works and what doesn’t. Sometimes just taking a shower and disassociating completely is where I resolve problems I’ve forgotten about.</p><p>If our thinking is tied to our bodies, our rooms, our materials, our neighborhoods, the sounds and the smells around us, then we must find ways to make our tools for thinking leverage the spatial and embodied capabilities of the natural world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7OhH-eg7yvONexmqVJo4eA.gif" /><figcaption><a href="https://www.ted.com/talks/deb_roy_the_birth_of_a_word?subtitle=en">“Wordscapes” by MIT researcher Deb Roy</a> shows how his son’s learning of various words was tied directly to the physical environment: the rooms of his home. Above shows the wordscape where his son learned the word “water.”</figcaption></figure><p>In “Externalism, Epistemic Artefacts and the Extended Mind,” Kim Sterelny shows how humans use the environment and external tools for cognitive processing. Most transformative to me was the idea that the homosapien isn’t actually much smarter than our previous distant cousins. There wasn’t some genetic breakthrough in human brain structure making homosapiens smarter. Rather a subtle change in our genome allowed us to build external tools and use them in combination with our minds to build cognitive processing capabilities into the world around us. More-so, it was the “proliferation and transmission” of these tools for thinking that led to the explosion of human’s cognitive capabilities.</p><blockquote>“Yet in all probability human cognitive powers have expanded without any significant change in human genomes. Instead, their expansion is a result of the proliferation and transmission of epistemic artifacts; of tools for thinking. As he sees it, we have got smarter, even though our naked brains have not.”<br> — “Externalism, Epistemic Artefacts and the Extended Mind,” Kim Sterelny</blockquote><p>In short, our thinking capacity is <strong>tied more to external tools</strong> than it is to our <strong>internal minds</strong>.</p><p>Thus, if our tools are what defines our intellect, <br>but our “mind bicycles” are modern day thought interrupters, <br>then its time to rethink the computer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Mac1XGgEhbYzSSnBfSBJEQ.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/1*1ul6PxjPoSqt7clg7yZReA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>Chapter 3 — Can we study knowledge work like manual work?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h4>Motion Studies of Manual Work</h4><p>In the early 1900s, Frank and Lilian Gilbreth pioneered scientific management by looking at how processes in factories — and even kitchens in the home — could be optimized for efficiency. The looked at each step of a worker’s task whether that was laying brick in construction, assembling a part of a physical product in a factory, or baking a cake in the home. Each task study was documented against a gridded backdrop with cameras and timers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HQniWgU_Zv6qXuRkC9H8dA.png" /><figcaption>Motion Study of Cooking</figcaption></figure><p>By tracking movement over time they were able to study <strong>manual labor</strong> and diagnose its parts and make recommendations to “optimize” each job. The aim was to save wasted time in the work.</p><p>Their work shaped the structure of factories and even office work today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lyq0GVkkm3ltgH-lzn-FaA.png" /><figcaption>Left: Patent of the hardware setup for Motion Studies, Right: “Therbligs” units of common tasks that made up a job</figcaption></figure><p>But while it may have saved time on a job, the drive for efficiency ignored people’s humanity, their well-being, their creativity, and the broader social aspects of work. All were simply missing.</p><h4>Can Knowledge Work Be Motion-Studied?</h4><p>Just like manual work, knowledge work takes place in sequence through time. But when the computer is introduced, the sequences are chopped into fragments: I’m going to find a file, attach it to my email, then send it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*mCFRxIMXcat2ACNdin1fNg.gif" /></figure><p>Just like manual work, knowledge work takes place through motions. But on the computer our motions get constrained to tiny finger movements on a mouse stuck on a two dimensional plane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*xYeScjIV-cUy7YOLN86t3Q.gif" /><figcaption>flat mouse movements</figcaption></figure><p>We need to view our digital interaction with the same scrutiny as the Gilbreth’s but with a fresh take. It’s not about physical labor, but a mental one.</p><p>We should investigate our own knowledge work interaction patterns. We should appropriate <strong>work measurement tools </strong>like<strong> </strong>time motion studies, employee monitoring, mouse tracking, UI heat maps, image segmentation, body tracking, gesture recognition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0bPVSnDYu8Qk9Rv3irORGQ.png" /></figure><p>But instead of focusing on “productivity,” our goal should be to <strong>reinvent</strong> <strong>knowledge work altogether-working from our own values</strong>, whether they are creativity, self-actualization, connection, or something else. We should look to the anti-work movement to understand what’s going wrong.</p><p>Like David Graeber’s postulation that much of our <a href="https://en.wikipedia.org/wiki/Bullshit_Jobs#cite_note-1">jobs today are bullshit</a>, I have this syncing feeling that every little tasks I have inside the computer is like a mini bullshit job — a bullshit task. Just UI bureaucracy playing out, but I’m just the cog in the machine, jumping through hoops to make it all work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M5Yndknthrq5Lbh0Zv-6BA.png" /><figcaption>Fun fact: “therbligs” is just “Gilbreth” spelled backwards.</figcaption></figure><p>Time Motion Studies made “therbligs” to describe and categorize the sub-tasks that made up a task like “search” or “grasp”. We can invent our own therbligs to describe and categorize the sub-tasks of knowledge work.</p><p>We can dissect the choreography of clicks, searches, drags and drops as spatial and sequenced, just as the motion studies did. Within the scatter between the apps and pop-ups, perhaps there’s a way after all to create order from the chaos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>Today’s work, knowledge work, is in crisis! It has us fixed to a chair 8 hours a day, completely screen fatigued, with a constant stream of thought patterns interrupters. Perhaps most importantly, the bounds of what we can <strong>think</strong> of is being held back by our 2 dimensional tools.</p><p>It’s time to deliver on the promise of computers as “bicycles for the mind.” We need to expand our thinking to the spatial and material dimensions, but to do so, we must untangle this 2-dimensional mess that is the UI bureaucracy of our time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p><em>Thanks for reading. Have something to share or interested in learning more?</em></p><p><a href="https://www.linkedin.com/in/violetwhitney/"><em>Violet Whitney</em></a><em> teaches “Spatial UX” at Columbia University, “LLMs for UX Prototyping” at University of Pennsylvania, and “Reading and Writing the World (Physical Computing)” at University of Michigan.</em></p><p><em>She and her partner </em><a href="https://www.linkedin.com/search/results/all/?fetchDeterministicClustersOnly=true&amp;heroEntityKey=urn%3Ali%3Afsd_profile%3AACoAAADH48YBSSkC1Q1O5-C3TZTgffpRNMNmH4k&amp;keywords=william%20martin&amp;origin=RICH_QUERY_SUGGESTION&amp;position=0&amp;searchId=0acf2f7a-9f42-4a65-a34c-4732b291ab1a&amp;sid=)Jl&amp;spellCorrectionEnabled=false"><em>William Martin</em></a><em> run </em><a href="https://spatialpixel.com/"><em>Spatial Pixel</em></a><em>: spatial computing for your space not your face.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f55efa1dedb" width="1" height="1" alt=""><hr><p><a href="https://medium.spatialpixel.com/flat-screen-flat-thoughts-2f55efa1dedb">Flat screen. Flat thoughts.</a> was originally published in <a href="https://medium.spatialpixel.com">Spatial Pixel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Programmable Space]]></title>
            <link>https://medium.spatialpixel.com/programmable-space-ef33ed5bccd1?source=rss----9eb53d32da6b---4</link>
            <guid isPermaLink="false">https://medium.com/p/ef33ed5bccd1</guid>
            <category><![CDATA[low-code]]></category>
            <category><![CDATA[lecture]]></category>
            <category><![CDATA[tangible-computing]]></category>
            <category><![CDATA[spatial-computing]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Violet Whitney]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 23:09:29 GMT</pubDate>
            <atom:updated>2024-06-23T23:10:26.355Z</atom:updated>
            <content:encoded><![CDATA[<h4>Spatial computing for <em>space</em> not your face</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8UakaDAl-V3niSyxIb_cQg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>What is “programmable space”?</h3><p>We’re imagining a “programmable space.” In a programmable space you create programs not in a computer or a screen, but rather in a physical space (a room, a street, a city).</p><p>In 2017, I was introduced to Dynamicland, a project led by Bret Victor aimed at interacting with computers in a <strong>tangible</strong> and <strong>communal</strong> way. Their pitch: the <em>environment itself</em> is the computer. This flipped the typical dynamic of using computers. Instead of typing in isolation, groups of people could work together on programs that would then change the environment around them; projections and sounds would emerge in the room.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DelYU8634HAA00HKkMQymg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*CSVCkaUpPB1XqKmGR8Ug0Q.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*ftAJtRSoJaEymOLdVvfpIw.gif" /></figure><p>Dynamicland was an experimental computing environment where physical objects and spaces were embedded with sensors, cameras, and projectors to enable real-time, tangible interaction with digital information. Users could collaboratively manipulate and create interactive programs by moving and arranging physical items like pieces of paper, thus blending the digital and physical worlds. It’s an amazing vision!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>Programmable space, a failed vision?</h3><p>Unfortunately, Dynamicland failed to gain funding and closed.</p><p>I wonder whether it was just ahead of its time. Was the technology struggling to catch up? When it first opened, Dynamicland required writing the programs in code. That meant it was painstaking for most folks (non-engineers) to really use it.</p><blockquote>To enable programmable space, we’ll need even easier ways to program.</blockquote><p>To enable programmable space, then we’ll need even easier ways to program. That means no writing code and no required software engineering degree. It should be easy for non-engineers, for designers, for the humanities, for artists, for kids to program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/80/1*qyIZUfl9YKW5n-7SRdm5Ug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AhqOudpre_kqujHcK0E-Rg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/80/1*qyIZUfl9YKW5n-7SRdm5Ug.png" /></figure><p>Dynamicland may have emerged at time when the technology for programmable space wasn’t ready, <em>however</em> it seems the technology has finally caught up.</p><p>I recently saw the <a href="https://dynamicland.org/">Dynamicland website</a> says there’s a reboot coming summer 2024! 👀</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>AI → Programmable Space</h3><p>The multi-modal AI technology emerging today seems like <em>just</em> the thing to make programmable spaces a viable reality.</p><p><strong>Multimodal AI</strong> can understand and combine different types of information, like speech, gesture, imagery, all at once. It’s more akin to how people interpret information from all kinds of context. You might point at something and say, “Hey what’s this?” Because you’re pointing at it, I know that “this” object is what you’re asking about. Multimodal AI is about that same kind of reasoning from multiple points of context.</p><p>This was an experiment we did at Spatial Pixel where you can point and ask about objects in the environment to have conversations with an AI that understands what you’re asking about.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fwww.youtube.com%2Fembed%2F_rLP7b_ScQQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fwww.youtube.com%2Fwatch%3Fv%3D_rLP7b_ScQQ&amp;image=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fi.ytimg.com%2Fvi%2F_rLP7b_ScQQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1cc24e20c54b8688fb3adaf09310bec/href">https://medium.com/media/b1cc24e20c54b8688fb3adaf09310bec/href</a></iframe><p>Instead of having to write code, we can speak instructions in natural language. Instead of having to interact with an obscure gaming environment, you could draw a shape with a pencil and assign rules to it by speaking. A programmable space will need to leverage the way people naturally interact with their environments: pointing at things, moving them around, rather than writing chunks of code.</p><p>Here are more experiments we did towards programmable spaces. It’s a way of creating interactive programs and sketches that adapt to what you say, how you move, and the environment around you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*cRUoxDBQR1y3KPVinR7LVg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*2i2dd4M6G7RV_QmtjgXLRw.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*03eUyP3bBkc2N112VxQ--w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*XtyRImjIQRMKUmHmIp3bKA.gif" /></figure><p>I really dislike being on a computer, but this was fun. You end up:</p><ul><li>interacting more with people instead of your computer.</li><li>interacting more with your environment instead of your computer.</li><li>interacting more with your body than your computer.</li></ul><p>And because you’re making the programs in a more tangible way — with your voice, your body, and objects in the environment — perhaps it makes you a bit more present. I’d like the future of computing to be more inherently collective, tangible and bodily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3><strong>Real world AI will get better</strong></h3><p>Multimodal AI has been a major investment by Google, Meta, OpenAI, and others. <a href="https://www.youtube.com/watch?v=AYSfcgVv9-U&amp;ab_channel=NVIDIA">Major companies like NVIDIA are investing in “physical AI”</a>. Physical AI means using devices like sensors or robots that leverage AI to understand and react to an environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*-97oIvwPfjyZ3seNoNvQ3Q.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*FGFxTAcrtQgTdzXxzzR3VA.gif" /></figure><p>We’re glad to see investment in technologies that interact with the real world. These will likely only get better, making programmable spaces more viable. But as these technologies develop, it will be important that artists, designers, and people outside of Big Tech shape this technology so that it’s designed for human aims, not just capitalist ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>Programmable Space Lab</h3><p>With the rise of AI, we’d like to see programmable spaces and academic labs that further explore this kind of computing in the real world, and ensure it’s done in ways that everyday people can create programs with more intuitive means like natural language and drawing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*Q31OxxrlJa0_Ve3HLQaYnw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*wCnqGqDFaeYyXX6ivl60Zg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*Q31OxxrlJa0_Ve3HLQaYnw.png" /></figure><p>We like the spirit of Dynamicland: the communal creative expression. We like the idea of co-creation and we think the technology is ripe for it. We’ve been thinking about how programs could adapt to be more environmental allowing us to more naturally reference the world around us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcEAevnJo3sC95smzwlVtQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*mS9zyxQqhsyzPKoRfrK5gw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kgw6G_ynVhROYI5zlK5jpQ.png" /></figure><blockquote>And what more would we do with programmable spaces? Well, nearly everything we do on computers today, but in a way that allows for more collective creation and more presence in the physical world. I’d be curious to see how a programmable space would change what we create and how we relate to each other. Would it make us more creative? More cooperative? Maybe most importantly, more present in this world and life we live?</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef33ed5bccd1" width="1" height="1" alt=""><hr><p><a href="https://medium.spatialpixel.com/programmable-space-ef33ed5bccd1">Programmable Space</a> was originally published in <a href="https://medium.spatialpixel.com">Spatial Pixel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spatial Primitives for AI]]></title>
            <link>https://medium.spatialpixel.com/spatial-primitives-for-ai-4171f24b0740?source=rss----9eb53d32da6b---4</link>
            <guid isPermaLink="false">https://medium.com/p/4171f24b0740</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[creative-technology]]></category>
            <category><![CDATA[spatial-ai]]></category>
            <category><![CDATA[computation]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[William Martin]]></dc:creator>
            <pubDate>Fri, 22 Mar 2024 16:03:11 GMT</pubDate>
            <atom:updated>2024-03-22T16:03:11.669Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PCz9Q-0xxymBqOio8QoRQA.gif" /></figure><h4>Catalog of computable “primitives” for spatial artificial intelligence from the Spatial AI course at Columbia University</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jDRKCrKSTB8eabfGzbkZ4w.png" /></figure><p>These are the basic building blocks we have to work with, essentially <strong>design patterns</strong> used to model spatial organizations with computable data structures.</p><p>This post reviews their semantics and respective affordances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZdcVdd24xuLJgy03-YKmvg.png" /></figure><h3>1 — Foundation Elements</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>If the primitives above are molecules, these <strong>elements</strong> are the atoms that comprise them.</p><p>Fundamentally, they are <em>data</em>, and making the distinction between data and <em>information</em> is important.</p><ul><li><strong>Pixels</strong> — “Picture elements,” the smallest units of a raster image, typically arrayed in a grid.</li><li><strong>Voxels</strong> — Volumetric pixels, conventionally cubes of “solid” in a gridded three-dimensional <strong>space</strong> of solid and empty cells.</li><li><strong>Points</strong> — A point references a location in a coordinate space, typically a singular, multidimensional collection of real numbers.</li><li><strong>Vectors</strong> — Related to points, vectors represent change, or the potential for change, along any dimensions relevant to a coordinate or other type of space.</li><li><strong>NURBS</strong> — Non-Uniform Rational B-Splines. Points in a coordinate space plus an equation that interpolates between them.</li><li><strong>Meshes</strong> — Point clouds mapped to tiled simple polygons (triangles and quadrilaterals). Meshes can be augmented with the semantics of graphs and vectors for workability.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PV7_Va5ef4fMQu7lD9D9nQ.gif" /></figure><p>Data are collections of related values of a consistent type, devoid of any meaning. For example, a value like 40ºC intuitively seems like a very hot day, but in actuality the value by itself doesn’t mean much. In the context of <em>human comfort</em>, it’s exceptionally hot, even dangerous. But in the context of computer processors, it’s quite cool, well within standard operating temperatures.</p><p>This is the essence of information, data plus meaning and context. Given information, we’re well on our way to making decisions, particularly if we can evolve information into knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sOijmmjIBYUBF_14rv0nSA.png" /></figure><h3>2 — Mappings</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>A <strong>mapping</strong> is an association from one domain of data to another, called the range (or codomain).</p><p>A “map” is the artifact that results from a mapping operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*mvzWZDIvCEimaYCfa1oQmA.gif" /></figure><p>A cartographic <em>map</em>, for example, derives from a mapping from the real world to the space of graphic representations, commonly by projecting real, measured space onto a two-dimensional coordinate space. Of course, our modern cartographic maps are more than graphic; they contain data, information, algorithms, and more.</p><p>Mappings are fundamental they enable us to associate the computable with the real world, different media with each other, and abstract concepts with more concrete methods. We really couldn’t have computational design without them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IH9QXJl2K9i6jIXxVl5u5Q.png" /></figure><h3>3 — Grids</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>A <strong>grid</strong> is a framework that maps discrete coordinates to:</p><ol><li>individual cells of a tiling, typically rectilinear, and/or</li><li>individual lines of a criss-crossed rectilinear pattern.</li></ol><p>With grids, you can index space systematically, mathematically, and consistently. That is, there’s an intuitive mapping between the index (coordinates) and the spatial position, and vice versa. They’re easily scalable, making them an effective means of control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*UqpBjmLogh7al5jxBu1gJQ.gif" /></figure><p>The grid of pixels on our rasterized screens is the obvious example, which reference each pixel by a pair of numbers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*df-uG2edynyylSXM.png" /></figure><p>Grids can have other notions added to their semantics as well, like the neighborliness used in Conway’s Game of Life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*HJ59s6kObpWFW0DA.gif" /></figure><p>The Manhattan street grid is a classic example of a gridded referential system. (Map below generated at <a href="https://maps.stamen.com/toner/#16/40.7541/-73.9860">Stamen Design</a>.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l3N9dodtiaxNCCJe.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hwlyP2usftOyUafRamlf6g.png" /></figure><h3>4 — Tilings</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p><strong>Tilings</strong> (aka tessellations) are coverings of a plane or other form with geometric shapes that don’t overlap or contain gaps, often in a symmetrical pattern.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8DU4yQUFOwT9ZxSB.png" /></figure><p>A voronoi tiling used as a heuristic to show the regions closest to the <a href="https://observablehq.com/@d3/world-airports-voronoi">world’s airports</a>.</p><p>Such tilings can also be three-dimensional or even multi-dimensional, particularly when considering the dimensions like time, resources, or experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*Z4zfAuO7Ie9PBLJqolsGXg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A40qyEBcl6zZIpiMfT08Lg.png" /></figure><h3>5 — Graphs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p><strong>Graphs</strong> are a network construct of connections, typically consisting of a set of individual nodes connected pairwise to other nodes by edges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bxGAOvFYFqc6u4Ie.png" /></figure><p>We typically represent graphs with diagrams like the above. Here, circles represent “nodes,” and the arrows between them represent edges, typically used to model connections, like social networks.</p><p>More pertinent, graphs can be also used for spatial networking, like the connectivity of street grids:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ze5OAw3NeB_7-vxE.png" /></figure><p>Graphs demonstrate such flexibility for modeling spatial relationships that they warrant much more attention. More posts on graphs coming soon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6dzPdDbKZov0xUQAPdrcEw.png" /></figure><h3>6 — Trees</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p><strong>Trees</strong> are constructs that represent hierarchies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SSMgB7QJ2q43glFX.png" /></figure><p>Decision trees represent the various options available to an agent, given a particular state of the world. Traversing increasingly complex trees is a core area of research in artificial intelligence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b0e-J3m8SYIIcfle.png" /></figure><p>The famous AI systems like DeepBlue and AlphaGo rely on abstract tree-like decision structures. Trees can also be used for static relationships like organizational charts of companies or even dynamic changes like the changing state of a complex system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*9pyVfB2-WgiccuMvI6xetw.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6WWXYjlRhKBBlN28YG5fYA.png" /></figure><h3>7 — Spaces</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>“<strong>Space</strong>” is an expanse of opportunity in which we intend to operate. For spatial AI, the categories of conceptual space and <a href="https://awmartin.xyz/computable-space/">computable space</a> are the most pertinent. Spaces set the rules for our simulations.</p><p>On the conceptual side, I define architectural space as “the physical, temporal, and experiential expanse of opportunity around our embodied selves.” Other types, like cyberspace, liminal space, and the like, are variations on this idea.</p><p>Computable categories include coordinate, metric, connective, and indexed spaces. For a more complete discussion, see <a href="https://awmartin.xyz/computable-space/">this post</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DWEdAoxwuW1AAzLVS3oDoA.png" /></figure><h3>8 — Agents</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p><strong>Agents</strong> are independent, automatic actors. A single actor can be interesting from some perspectives, but when used in collections, we can simulate the emergent behaviors of more complex systems like crowds of people or vehicular traffic, thus informing design decisions regarding factors that influence such behavior, particularly their environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bdgkaXJuGS6nMMWlYbpDLg.gif" /></figure><p>Agent-based models (ABMs) like this can simulate the schooling behavior of fish.</p><p>But such models have much more potential. Organizations and collective decision-making can be examined with ABMs. The field of AI is exploring agent models for complex tasks with systems like MetaGPT, in which multiple AI agents based on individually tuned LLMs cooperate to complete more sophisticated reasoning and tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><blockquote>This is just the start. The more primitives we can formalize, the more sophisticated patterns we can create and build upon for Spatial AI.</blockquote><p>→ I’m a design technologist and researcher. I also publish more stuff on my website at <a href="https://awmartin.xyz">https://awmartin.xyz</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4171f24b0740" width="1" height="1" alt=""><hr><p><a href="https://medium.spatialpixel.com/spatial-primitives-for-ai-4171f24b0740">Spatial Primitives for AI</a> was originally published in <a href="https://medium.spatialpixel.com">Spatial Pixel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Need a Natural Language Programming Environment!]]></title>
            <link>https://medium.spatialpixel.com/we-need-a-natural-language-programming-environment-c4dfd3efcc7f?source=rss----9eb53d32da6b---4</link>
            <guid isPermaLink="false">https://medium.com/p/c4dfd3efcc7f</guid>
            <category><![CDATA[low-code-development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[natural-language]]></category>
            <category><![CDATA[low-code-platform]]></category>
            <category><![CDATA[lecture]]></category>
            <dc:creator><![CDATA[Violet Whitney]]></dc:creator>
            <pubDate>Wed, 06 Mar 2024 18:06:57 GMT</pubDate>
            <atom:updated>2024-03-19T02:52:24.391Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5A57_oQfD0xPUFXBKiWGHA.gif" /></figure><h4>Forget code editors for Python, Javascript, and computer code. What about code editing in natural language?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>TLDR: This is the beginning of a research and design exercise for creating a natural language programming environment. My partner, William and I’ve only just started to experiment with these ideas: here’s a <a href="https://spatialpixel.com/natural-language-ide/">bare bones one for P5</a>, and <a href="https://spatialpixel.com/programmable-spaces/">one for physical space</a>. Now we’re looking for topics related to code editors, visual programming, and programming languages that can spur ideas to help in this journey. If you have references you think might help us, <a href="https://www.linkedin.com/in/violetwhitney">please send them to us</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><ol><li>Rise in <strong>Natural Language Programming</strong>.</li><li><strong>Prompting</strong> Isn’t Enough.</li><li>What to Borrow from <strong>Code Editors.</strong></li><li><strong>Emerging</strong> <strong>Experiments</strong> in Natural Language Programming</li><li><strong>Provocations</strong> for Natural Language Programming.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/28/1*54pAFbOafE2xc2XDfk49Yw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/1*f6kwC48aFZZd27HX2wsQsA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/28/1*54pAFbOafE2xc2XDfk49Yw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>1 — Rise in Natural Language Programming (NLP)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>If you write code you’ve probably spent a decent amount of time in a <strong>Code Editor</strong> or an <strong>IDE</strong> (Integrated Development Environment). Code editors like <a href="https://code.visualstudio.com/">Visual Studio Code</a> help developers write, edit, and test code. But as we see the emergence of “<strong>natural language programming</strong>” where we can write executable programs with everyday language, will we still need a programming environment for natural language? I think so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*xei_PjkW5iZG5R275fV3HQ.gif" /></figure><p>Natural Language Programming, related but not to be confused with Natural Language Processing, is a way of programming with natural-language sentences, like writing in English.</p><p>A very simple NLP: “Can you turn off the bedroom light off after 10 o’clock?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*zNRR8oDc3axPl_cikLixoA.gif" /></figure><p>What makes this a program and not just plain english? It becomes a program when it moves beyond a description and becomes something a computer can execute. Its a set of instructions meant to be computed, calculated, decided on or carried out.</p><p>Last summer my partner, <a href="https://awmartin.xyz/">William Martin</a> and I taught a course in the new Computational Design program at Columbia University GSAPP. For the first time, we used natural language (ChatGPT) to write programs rather than starting with raw code. Typically in academic programs, we have to dedicate a decent amount of time to up-skilling students in programming. In the past learning the ins and outs of Python or Javascript syntax was a prerequisite to really designing anything that leveraged computation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EOCV3Uc9GZ4s8hE3t8BfnQ.gif" /><figcaption>back: Visual Studio Code IDE, front: P5 IDE</figcaption></figure><p>However, in this go-round, <a href="https://gsapp-cdp.github.io/colloquium-1-2023/work/">students were able to develop complete websites, totally bespoke to their ideas, relying much more heavily on Natural Language Programming.</a> And for as much as ChatGPT “did their homework” developing their website code, so much of these designs were inherently theirs. Students still had to design the website, its functionality; spend days prompting to get useful responses; and struggle through actual code to develop meaningful programs. It was ultimately their ideas, their designs, and their visions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*v_kJDJ45aGxUVCnzzNpG6g.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*PIcckU9rbib-5fuhLj6Qag.gif" /><figcaption>Left: Matthew Heaton, Right: Hongqian Li</figcaption></figure><p>In all fairness NLP is not <em>that</em> good yet. Students still needed a good technical understanding of Javascript syntax to string these websites together.</p><p>But that’s changing.</p><p>Recently Jensen Huang, CEO of NVIDIA, has predicted the <strong>death of coding</strong>. He argued, “It is our job to create computing technology such that nobody has to program and that the programming language is human. Everybody in the world is now a programmer. This is the miracle of AI.” [<a href="https://www.techradar.com/pro/nvidia-ceo-predicts-the-death-of-coding-jensen-huang-says-ai-will-do-the-work-so-kids-dont-need-to-learn#:~:text=Pro-,Nvidia%20CEO%20predicts%20the%20death%20of%20coding%20%E2%80%94%20Jensen%20Huang%20says,don&#39;t%20need%20to%20learn&amp;text=Nvidia%20CEO%20Jensen%20Huang%20has,potentially%20far%20more%20influential%20audience.">source</a>][<a href="https://www.youtube.com/watch?v=2nLq6xzTx-A&amp;ab_channel=DjamgaTechEduSportHub">video</a>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*xM9FtQYR6eRUuIaEqMIrKQ.gif" /><figcaption>coding says goodbye</figcaption></figure><h4>Programming Isn’t Going Away</h4><p>So while coding may be going away, programming is not. There are useful ways of thinking that someone develops when they learn to code. They learn to think systematically about how things are connected. They learn concepts like recursion, and learn how the language itself can impact the capabilities of a program. Those ways of thinking will still be important to develop. But a lot of the nuances of coding keeps us too deep in the technical weeds and prevent us from thinking more strategically and conceptually. Natural language programming allows us to dedicate our brain power at a higher level to the problems that matter.</p><p>Today, most people aren’t writing sophisticated programs with ChatGPT, but I do believe that will come. A wave of “programming” literacy is coming via natural language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UEGjqSWu9I5OjSxB7Xx41w.png" /><figcaption>Left: Data from World Bank (2023) showing % of literacy over the last 200 years; van Zanden, J. et al. (2014) — with major processing by Our World in Data, Right: My highly speculative possible curves of future “programming literacy” worldwide.</figcaption></figure><p>Today, only 2.54% of the global workforce are software engineers. [<a href="https://qubit-labs.com/how-many-programmers-in-the-world/">source</a>] In 1820 only 12% of the world was able to read and write, but today 87% can read and write. [<a href="https://ourworldindata.org/literacy#:~:text=Historical%20change%20in%20literacy,-Global%20literacy%20has&amp;text=It%20was%20only%20in%20the,for%20the%20period%201800%2D2016.">source</a>] Like the steep incline of literacy following the Industrial Revolution, will most people be programming literate 200 years from now? Or perhaps even sooner? Today so much control of our world is only achievable by this tiny % of people who are software engineers. I welcome a future where programming is far more accessible and more people can write programs specific to their own interests and domains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>2 — Prompting Isn’t Enough</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*gq1MCjfzB5TM0XuHENZHfQ.gif" /></figure><p>Great so most people in the future will be programming and doing so in natural language. So does that mean we’re all just going to ChatGPT our programs into existence? Not exactly. Language is very useful for thinking, but writing prompts is hardly programming.</p><p>As folks who program regularly will tell you, programming is not about coding. It’s a way of thinking. My partner recently pointed out that when software engineers discuss problems with one another, they rarely talk about code. They speak more <strong>abstractly</strong> about the structure of the system and its behavior. They consider how to structure the <strong>semantics</strong> of the program and how that will translate into the program’s capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CLhYcB0NsxsCVYFKEtC5Aw.png" /></figure><p>Programming teaches you to think holistically about the <strong>system</strong> grappling with scale, recursion, network affordances, and emergent behaviors. It teaches <strong>problem solving</strong>, forcing someone to think critically, logically, and about operational flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rOLREqG-irWvJ3XmE5JbOw.png" /></figure><p>Prompting in contrast to programming today isn’t always the easiest way to think programmatically.</p><p>There’s a danger that as we move towards prompting that we’ll rely too heavily on computers to think on our behalf. Prompting is more about a wait and see what the computer does that’s more reactive, vs programming which is more about designing, orchestrating, thinking and creating. Prompting also doesn’t allow you to build the holistic system. You’re only ever working on segments. And you must bring those segments to a different environment to build anything more complex.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sxixRWqg-KUdmr-AUICRBA.png" /></figure><p>But there’s a great opportunity if we leverage natural language’s ease and flexibility with the more systemic capabilities of programming. My hope and aim is to help contribute to a best of both worlds scenario where we harness the capabilities of both to help people think at a higher level.</p><h4>Why we need a programming environment!</h4><p>But why do we need a programming environment if AI’s going to do all of the work anyway? Even if AI is going to do all the program writing on our behalf, we’ll still need a way to communicate with it, clarify our intentions, and hopefully design programs together <em>with</em> AI.</p><blockquote>We need an environment better than a chatbot to write natural language programs.</blockquote><p>My hope is that programming literacy is not about having AI think for you, but that it empowers many more people to think creatively in terms of holistic systems and interconnected programs. I’m bullish that we’ll need an environment better than a chatbot, beyond just language to help us become sophisticated programmers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>3 — What to Borrow from Code Editors.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>Luckily we’re not starting from scratch. There’s a great lineage of programming environments. Integrated development environments include a place to <em>make</em>, <em>see</em>, and <em>check</em> your program. Each of these capabilities is useful for an NLP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/18/1*Gtah_lYXglkRVwbSGr9gUA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*Z5atfCx0z87E9Yk_i08i1g.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/18/1*Gtah_lYXglkRVwbSGr9gUA.png" /><figcaption>Structure of P5.js Integrated Development Environment</figcaption></figure><ol><li><strong>Make. . . . . . . . . .[Code Editor]</strong><br><em>This is where you author and design a program. Where you actually write the code. It usually includes some form of syntax highlighting. This might also include tools like auto-complete to improve the ease of writing code. It also might include higher level maps that help you author the hierarchy and components of your program.</em></li><li><strong>See . . . . . . . . . . .[Playground]</strong><br><em>A way to run the programs to see how they behave, so you can actively test and iterate on your program. Some environments include a “playground” where you can “play” a version of your program.</em></li><li><strong>Check . . . . . . . . .[De-bugging]</strong><br><em>Tools for testing and understanding bugs and errors, including things like error logs.</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>4 — Emerging Experiments in Natural Language Programming</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h4>A Showcase of NLP-First Programming</h4><p>In 2021 <a href="https://openai.com/blog/openai-codex">OpenAI published CodeX</a>, an environment for natural language programming. Today the programming capabilities you access in ChatGPT are from this CodeX model. This environment demonstrates the ability to write a longer structured program by iteratively providing natural language instructions. I’m really encouraged by this environment. However, it also shows there’s definitely a ways to go to make something that can write sophisticated programs and that anyone can use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*EJ2ju3iwWE9aHcX_3ST3bQ.gif" /><figcaption>OpenAI CodeX — 2021</figcaption></figure><h4>NLP Environments Aren’t There Yet</h4><p><strong>Lacks the Natural Language <em>Program</em> </strong>— In the CodeX environment you never see more than a single prompt at a time which really limits your ability to control and design the program structure. What I’d consider your program code (all of the iterative prompts provided) isn’t viewable, scannable, editable. You’re always prompting one at a time one or two sentences. How can this become a designed and structured part of the interface?</p><p><strong>No Referencing Between Contexts</strong> — Additionally this environment makes it difficult to understand how the contexts interact. How does the prompt impacts portions of the formal code on the right? Where is “animate” in the formal code? And how does my prompt or elements in the formal code impact the behavior in the canvas (where the game is running)? Its hard to understand really how they all relate to one another beyond a general “they seem to relate”. In an environment that supports natural language which is inherently more abstract, it will be more important to be able to understand the relationships across these contexts.</p><h4>Other Emerging NLP Trends</h4><p><strong>Engineer-First NLP </strong>— Since CodeX, there’s been an explosion in AI coding Copilots: from <a href="https://github.com/features/copilot">Github Copilot</a>, <a href="https://aws.amazon.com/codewhisperer/?ref=codesubmit">Amazon AI Code Generator</a>, and <a href="https://about.sourcegraph.com/cody?utm_source=codesubmit">Sourcegraph</a>. These copilots allow you to type prompts in natural language, but they still return responses in formal code. For the time being, you’ll still need to be a software engineer or at least rather technical to use them.</p><p><strong>Novice Programming-NLP — </strong>Meanwhile, gaming companies like Roblox an Unity have started to advertise their aims of creating natural language programming capabilities natively inside of their development environments for novice programmers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*TFAsOUAl4EmPieXY-ozR2w.gif" /><figcaption>Draw-A-UI Demo — 2023</figcaption></figure><p><strong>Designer-first-NLP</strong> — Designer-first AI tools like <a href="https://www.draw-a-ui.com/">Draw-A-UI</a> could point to how future natural language environments could evolve in a way that leverages more native design skills like drawing. Here someone can draw shapes on a canvas and by adding labels to the elements the environment can intuit the program functionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GBNI_4fdcTU0LdRIZsBlCA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>5 —Provocations for Natural Language Programming.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>If we want a programming environment for natural language, it’ll need to be honed to the affordances of natural language. So how is natural language different from a formal language?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pWwYrf35vTBF5iPV98w-dg.png" /><figcaption>My Comparison of Formal and Natural Languages</figcaption></figure><h4>Ambiguity</h4><p><strong>Sequence Ambiguity — </strong>Today its hard to understand the order and sequence of natural language programs. To develop a complex program through prompt engineering, you’d have pages worth of instructions making it difficult to follow<strong> operational flow</strong>. It gets really muddy. A development environment should help reveal the inherent structure and sequence of the program. Perhaps like some visual programming environments attempt to do? This could facilitate a programmer’s ability to navigate and design that higher level structure of the program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFWtVR0dP4QD0nlHHAkn9Q.gif" /></figure><p><strong>Syntax Ambiguity </strong>— Natural language in contrast to structured language programming is far more flexible and ambiguous. It relies more heavily on the contextual semantics of the language and even the context of where and when the program takes place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T8l3W28PUuKaXFAmt47hJQ.png" /><figcaption>The same program demonstrated on the left as a Python program, and on the right as a natural language program.</figcaption></figure><p>So what might the syntax highlighting be in natural language? Perhaps its actually useful to reveal the grammar through highlighting each <strong>word class</strong>? Verbs <em>are</em> similar to functions and methods, whereas nouns <em>are</em> more like variables and objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mWfxGhkEE4zpKf4Ncbt-sg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*12Y9wXKgDDJIeuCxUF5zxA.gif" /><figcaption>Example of word class syntax highlighting to reveal the part-of-speech. Above experiment by Spatial Pixel.</figcaption></figure><p>But syntax highlighting that follows the grammar of natural language doesn’t quite feel useful because so much of the meaning is determined by the entire sentence from the semantics in the context of the whole sentence. Perhaps more appropriate than syntax highlighting would be a semantic highlighting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U8PlYNATHcANsIXfx7YjKQ.png" /><figcaption>Example of semantic highlighting to reveal whether instructions are a command, conditional or calculation.</figcaption></figure><p>But within natural language’s ambiguity, perhaps even more important is than highlighting is that in natural language programming, AI interprets <strong>intent</strong>, whereas in formal language the computer follows explicit instructions. , it’s more important to be able to continually control, check and account for that hidden context. I can give an abstract command like “make me something pretty” and AI will have to interpret what “pretty” is. Even for something that seems more concrete like “draw a line” the AI is still interpreting what I mean by “a line” and what I mean by “draw”.</p><p>In a natural language environment, it becomes more important to reveal what the program is inferring. We should make it easier to specify our intent by making the design and control its inference more explicit.</p><blockquote>Formal language: follows explicit instructions</blockquote><blockquote>Natural language: interprets ambiguous intent</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*9ct2p42SP9FtZvBr5hS2DA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*qAoZPD6YzsszFW23C_AAJA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*t69mCECZJouNnOvdnDyESg.gif" /><figcaption>Examples of inferring intent source: Spatial Pixel’s natural language program</figcaption></figure><p><strong>Inspecting hidden assumptions</strong>— This means future programs will need to be able to navigate between abstract and specific more quickly. Language is especially abstract in contrast to programs like C# which are highly specific. We’ll need to create better ways to help NL programmers navigate levels of specificity and abstraction. Consider a simple instruction for a robot “bring the cup downstairs” and how many assumptions (mini-decisions) the program would have to make. This designer of this program likely wants the ability to control more finite parameters of the program buried in the assumptions of that program. This might warrant <a href="https://www.telescopictext.org/text/KPx0nlXlKTciC">an interface that allows you to zoom into levels of specificity within the text</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*hpmtkhF1rhZ5DcBijSy-bw.gif" /></figure><p>This article has gotten so long that I’ve decided to create a separate one to to focus solely on mock-ups of what a natural language programming environment could be. That way I can explore more of these little ideas, and focus on the design itself.</p><p>Hopefully you have ideas too! I’m looking for more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p><em>Hey, you! Thanks for reading. <br>If you’re a researcher, designer, or just interested in natural language programming, </em><a href="https://www.linkedin.com/in/violetwhitney"><em>please reach out</em></a><em>! I’m actively looking for projects, case studies, research, and ideas for what a natural language code editor should be. If you have anything to share it’s much appreciated!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4dfd3efcc7f" width="1" height="1" alt=""><hr><p><a href="https://medium.spatialpixel.com/we-need-a-natural-language-programming-environment-c4dfd3efcc7f">We Need a Natural Language Programming Environment!</a> was originally published in <a href="https://medium.spatialpixel.com">Spatial Pixel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turning Your Hand Into a Keyboard]]></title>
            <link>https://medium.spatialpixel.com/turning-your-hand-into-a-keyboard-6b21d092cfd0?source=rss----9eb53d32da6b---4</link>
            <guid isPermaLink="false">https://medium.com/p/6b21d092cfd0</guid>
            <category><![CDATA[gesture-recognition]]></category>
            <category><![CDATA[natural-language-process]]></category>
            <category><![CDATA[keyboard]]></category>
            <category><![CDATA[lecture]]></category>
            <category><![CDATA[spatial-computing]]></category>
            <dc:creator><![CDATA[Violet Whitney]]></dc:creator>
            <pubDate>Tue, 12 Dec 2023 21:29:42 GMT</pubDate>
            <atom:updated>2023-12-13T15:51:13.612Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UmfJj5GebQZjQL3_gk0r8A.jpeg" /></figure><h4>Exploration of how bodies can create more natural interfaces</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*HlyOi0Tfq6ftd_8iXYIjHA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p><em>Sections of this article:</em></p><ol><li><strong>Precedents </strong>— Hands as controllers</li><li><strong>Why</strong> — Why do we need a hand keyboard?</li><li><strong>Natural Language Programming </strong>— The body in natural language.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>1 — Precedents — Hands as controllers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h4>My Student Gave Me This Idea</h4><p>Might I start by saying, this idea is not wholly my own, but rather a Frankenstein of ideas stolen from many creative people who’ve built up these ideas around me. Most importantly, I’ve had a student in my Physical Computing course in the Urban Technology program this semester who has been very keen on developing screen-less remotes by using key presses on the fingers, which planted this idea in my brain. He’s been focusing on this interaction as a sort of universal remote. <a href="https://youtu.be/YKOw4kwd42I">You can see his project here</a>. He also pointed out this technology called <a href="https://gest.co/">Gest</a> which was another point of inspiration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mgrHpoRpRUtaol3HNZ_1zw.jpeg" /><figcaption>Above: Technical Diagram and hand remote by Owen Sims (University of Michigan’s Urban Technologies).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*A49yxFO4Uip6mz6mr668dA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*F5V30FwCLtftIzYeXhZ2Uw.jpeg" /></figure><p>I‘ve been inspired by so many projects, some which have shown the potential for digital interactions through more natural ways of interacting with the hands, others which leverage computer vision and pose detection, such as in the image above. Projects such as <a href="https://www.youtube.com/watch?v=0QNiZfSsPc0&amp;ab_channel=GoogleATAP">Google ATAP’s Soli</a> pictured below show the potential for more natural microgetures without physically interacting with any hardware.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*s2F5brhEKpSceN7Fruds8Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*tT_lPn0tMpI79tFTKFMZwQ.gif" /></figure><p>Many tech companies have this same fascination and there are numerous <a href="https://www.google.com/search?q=patents+that+use+displays+on+hands&amp;tbm=isch&amp;ved=2ahUKEwjOw62cxKGCAxV0VjUKHQEsDH8Q2-cCegQIABAA&amp;oq=patents+that+use+displays+on+hands&amp;gs_lcp=CgNpbWcQAzoECCMQJ1DfBViMNmDbN2gAcAB4AIABRYgBnAuSAQIyNZgBAKABAaoBC2d3cy13aXotaW1nwAEB&amp;sclient=img&amp;ei=OZpBZY7DLPSs1QGB2LD4Bw&amp;bih=1169&amp;biw=2092">patents showing XR interfaces</a> on top of the hands such as the Apple patent drawing below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z2zOoLIR4lRFzrB9ktqsrQ.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*F5V30FwCLtftIzYeXhZ2Uw.jpeg" /></figure><p>If you need anymore convincing about why our hands are so special, just take a look at Bret Victor’s useful rant about hands in his article on <a href="http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign/">the future of interaction design</a>.</p><p>It got me to thinking about what else we can do with hands?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>2 — Why do we need a hand keyboard?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h4>FAQ</h4><p><strong>Q — Why don’t you just bring a phone or computer? </strong><br>A —I’ve been fascinated by the idea of not needing to carry a phone or a computer around to do computing tasks. I don’t want to bring my clunky keyboard. And I don’t really like the idea of carrying around a bunch of devices: computers, phones, or otherwise. What if we could rely on environmental computers and cloud-based software for interactions?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*OebfX_RVlOQ52QsT56_DKQ.gif" /></figure><p><strong>Q — Why don’t you just use speech?</strong><br>A — Speech is great, but most computer interactions and conversations are private. Either you’re looking something up, or texting someone. I’d prefer not to be that person at the cafe saying “Hey Siri, can you text my mom and ask her when we’re all getting together this year.”</p><p><strong>Q — Why don’t you just use sign language?</strong><br>A — That’s a great idea. The idea didn’t come across my mind until I wrote this article. Since I don’t sign, I’m not sure what the cons and benefits may be between these two systems. I’d love to hear from anyone more familiar here. I assume signing may not be as fast.</p><p><strong>Q — Doesn’t this hand keyboard thing exist already?</strong><br>A — That’s what I was wondering. Apparently it does, and <a href="https://androidcommunity.com/samsung-patent-rethinks-the-keyboard-for-augmented-reality-wearables-20140305/">Samsung patented it</a>. Amazing how quickly things get marked by large companies. But that’s a post for another time. In the mean time, I’ll let you enjoy their hilariously passive aggressive patent drawing below.</p><blockquote>At last you can harass your coworkers while away from your computer.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/0*eOA98_kr2xzO7qRI.jpg" /><figcaption>above: Samsung’s oddly passive aggressive patent drawing</figcaption></figure><p>But in all seriousness, this is a great idea.</p><blockquote>FUN FACT:</blockquote><blockquote>Did you know the QWERTY keyboard was invented in 1867 and gains its logic from the printing press where typesetters manually set individual characters?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R3qv-6kRvwoacnJ7H42OEA.jpeg" /></figure><blockquote>“Ok, you convinced me. Now I want a hand keyboard.</blockquote><blockquote>But how would I make one?” — You</blockquote><h4>It’s technically achievable.</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fwww.youtube.com%2Fembed%2F1QuBqLU0upw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fwww.youtube.com%2Fwatch%3Fv%3D1QuBqLU0upw&amp;image=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fi.ytimg.com%2Fvi%2F1QuBqLU0upw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/882247bd734d1eb15718a5983e5dc473/href">https://medium.com/media/882247bd734d1eb15718a5983e5dc473/href</a></iframe><p>The above is just a rough demo of how to use the position of the hand to recognize letters pressed. It is by no means a working keyboard. If you were building out a more robust system you’d instead use the skeleton structure of the hand. This is what we’re currently working on [end of this post] and what we hope to include in future posts along the way. If you’d like to see how to make this above simpler demo, its quite easy with Teachable Machine. <a href="https://spatialpixel.com/teachable-machine-tutorial-00437e01a793">You can find my tutorial for it here.</a></p><h4>Hands are great symbols.</h4><p>The hands are great symbols for, well, tons of things.</p><p>When I was working on <a href="https://spatialpixel.com/distant-prosthetics-851504f7680">this lecture</a>, I spent a bunch of time looking at these ancient drawings from witchcraft and alchemy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3_3191TsR-nLvFG2gT74jw.jpeg" /></figure><p>The more I perused the depth of the internet, I came across more and more examples where the hand is used as a symbol or for signs. It’s used for sign language, Palmistry, Mudras, cultural gestures like thumbs up, as <a href="https://www.fulltable.com/vts/m/memh/mh.htm">a mnemonic device</a>. The list could go on. So because we can map so many different recognizable meanings onto the hands, they become a really relevant interface for real world interactions.</p><p>This project is not really about turning your hand into a keyboard. Rather it’s an exploration of how to leverage our hands directly to interact with computers.</p><blockquote>It’s not just about typing, its about all the ways we use computers with our hands.</blockquote><p>Why should mice and keyboards confine our creativity? We should be able to tap into the flexibility of both our hands and of language to create our own controls for computers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>3 — Natural Language Programming</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>One idea we’ve been working on at Spatial Pixel is how to <a href="https://vimeo.com/893379256?share=copy">speak programs into the world around you like this working demo</a>.</p><p>In the future, we’ll be able to speak the changes we want in an interface and even speak changes we want to make to change how our hands behave as inputs, to swap our hands from a keyboard, to a paintbrush, or from a paintbrush to a puppet, or from a puppet to a music control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*oy_vHRSwvuctwMaiobGB9g.gif" /></figure><p>This “Speech to Hand” project above is about turning your hand into an interface. But its still in concept mode because there’s a lot to build! Each of these demos on the right is a working program with hand tracking. However, we’re still actively working on the speech functionality to translate programmed actions to the hand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pLPR-FL-1t4zWZoH7wBc5g.gif" /></figure><p>This includes inferring:</p><ol><li><strong>body identifier</strong>— what part of the hand the speaker is referring to: finger joints, hand, position of the whole hand or what part of each component (top, bottom, left, right)</li><li><strong>task identifier — </strong>what action the speaker is trying to map to the hand: pressing, tracking distance between fingers, position of the hand, shape or structure of hand which must be inferred from a higher level command such as “turn my hand into a paintbrush”. We’re still trying to sort out the best ways to break down and categorize motions with the hand, but we did come across this <a href="https://www.eng.yale.edu/grablab/pubs/Dollar_STAR14.pdf">amazing paper from the Yale Grab Lab</a>.</li></ol><p>If you have any ideas on how we might categorize tasks in terms of motions, please send your ideas, papers, or projects our way! There’s a lot we have to learn on converting language into actions that can be mapped to the hands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>By the way, I typed this whole article on my hand keyboard while lying under the canopy of a beautiful tree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*HlyOi0Tfq6ftd_8iXYIjHA.gif" /></figure><p>No, that was a lie.</p><p>I only made a demo version with a few letters.</p><p>But wouldn’t that have been cool?</p><p>Thanks for reading!</p><p>Now go type!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b21d092cfd0" width="1" height="1" alt=""><hr><p><a href="https://medium.spatialpixel.com/turning-your-hand-into-a-keyboard-6b21d092cfd0">Turning Your Hand Into a Keyboard</a> was originally published in <a href="https://medium.spatialpixel.com">Spatial Pixel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Teachable Machine Tutorial]]></title>
            <link>https://medium.spatialpixel.com/teachable-machine-tutorial-00437e01a793?source=rss----9eb53d32da6b---4</link>
            <guid isPermaLink="false">https://medium.com/p/00437e01a793</guid>
            <category><![CDATA[p5js]]></category>
            <category><![CDATA[teachable-machine]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[intro]]></category>
            <dc:creator><![CDATA[Violet Whitney]]></dc:creator>
            <pubDate>Tue, 12 Dec 2023 18:04:08 GMT</pubDate>
            <atom:updated>2023-12-12T18:05:07.964Z</atom:updated>
            <content:encoded><![CDATA[<h4>Recognizing key presses from hand gestures using Google’s Teachable Machine with the ml5 Library in P5.js</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>Tutorial — Teachable Machine + ml5 Library in P5.js</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>The following is just a rough demo of how to use the position of the hand to recognize letters pressed. It is by no means a working keyboard. If you were building out a more robust system you may choose instead to use the skeleton structure of the hand. This is what we’re currently working on and what we hope to include in future posts along the way. The following is just a quick way to play with recognizing a hand position from an image.</p><h4>Collect Data</h4><p>1 — Go to <a href="https://teachablemachine.withgoogle.com/">Teachable Machine</a>.</p><p>2 — Click on the “Get Started” button under “Image Project.”</p><p>3 — On the new page, you’ll see a live preview from your webcam (after granting access). You’ll also see buttons to add labels.</p><p>4 — Create a new label for each letter (or hand position) you want to recognize. I started with just some simple letters: “none”, “E”, and “R”.</p><p>5 — After creating a label, click on the “Webcam” button below the label to start capturing samples. You should try to get a diverse set of images, capturing various lighting conditions and angles. Ideally, get at least 100 samples per label, if not more. I found around 300 was good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hc6BKjuhOdoL0K_-szB-zQ.jpeg" /></figure><h4>Train</h4><p>6 — Once you’ve collected enough samples for each label, click the “Train Model” button. After training is complete, you’ll see a preview where you can test your model with a live webcam to see how well it performs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*s2F5brhEKpSceN7Fruds8Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*057lxLqsSW7bMWCfYl8L7Q.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*s2F5brhEKpSceN7Fruds8Q.jpeg" /></figure><p>I started with the simplest thing first: just one letter and “none”. Then I layered in more letters. This particular model did ok with 3 hand positions, but I noticed it got worse as I added letters. I imagine with enough training data you could train this on all letters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*s2F5brhEKpSceN7Fruds8Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*wbMyJZ2sWh3EPq3DSJzzjg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*s2F5brhEKpSceN7Fruds8Q.jpeg" /></figure><p>However, this points to a larger issue. For keyboards you need extreme accuracy. You can have the possibility of a failure at every keystroke. Alas, computer vision likely isn’t the right answer for the hand keyboard. But, hey, it’s still cool.</p><p>Now, lets get this trained model into P5 so we can do more fun stuff based on the “keys”.</p><h4>Using Your Trained Model in P5.js</h4><p>7 — If your model is working ok, click on the “Export Model” button. <br>Choose the “Tensorflow.js”, and then “Upload my model.” <br>This will upload the model files to Google Drive.</p><p>8 — Once the upload is complete, you’ll get shareable links for the model. Copy this. Make sure these files are publicly accessible when you want to use them in your p5.js project. <br>In the “Code snippet” section you’ll see prewritten code you can use in Javascript or directly in P5.js. <br>Click P5.js, then click “P5.js Web Editor”. This will bring you directly to a P5.js web editor page with your code!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*DiYDe3L7AldeHDsPVBTbEg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*kq2wiHX9VDfvkL8Uyn7TZw.gif" /><figcaption>Left: Step 7; Right: Steps 8</figcaption></figure><p>9 — Finally you’ll need to swap out the “imageModelURL” with your own model (the one you copied in the last step). This example below is just a model I trained on “Class 1: thumbs up” and “Class 2: thumbs down”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*-2aGxCdj_iHaDjF9ZwxdUg.gif" /></figure><h4>Adapt the P5.js Script To Your Own</h4><p>10 — Now that you have a model of your own in P5.js you can adapt it in any way you see fit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UjcHN9Utlz1QzJ39RFxOVQ.gif" /></figure><p>Below I used the text to speech capabilities to have the computer read the letters. I added this synth = window.speechSynthesis; into the setup function, then added a separate function to read the letter when the mouse is pressed:</p><pre>function mousePressed() {<br>  let utterance = new SpeechSynthesisUtterance(label+&quot;!&quot;);<br>  synth.speak(utterance);<br>}</pre><p>If you’d like to learn more about speech and audio in <a href="https://spatialpixel.com/sounds-bd05429aba38">P5.js you can find that tutorial here</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fwww.youtube.com%2Fembed%2F1QuBqLU0upw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fwww.youtube.com%2Fwatch%3Fv%3D1QuBqLU0upw&amp;image=https%3A%2F%2Fround-lake.dustinice.workers.dev%3A443%2Fhttps%2Fi.ytimg.com%2Fvi%2F1QuBqLU0upw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/882247bd734d1eb15718a5983e5dc473/href">https://medium.com/media/882247bd734d1eb15718a5983e5dc473/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=00437e01a793" width="1" height="1" alt=""><hr><p><a href="https://medium.spatialpixel.com/teachable-machine-tutorial-00437e01a793">Teachable Machine Tutorial</a> was originally published in <a href="https://medium.spatialpixel.com">Spatial Pixel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple API Calls]]></title>
            <link>https://medium.spatialpixel.com/simple-api-calls-f5bc0a9d579c?source=rss----9eb53d32da6b---4</link>
            <guid isPermaLink="false">https://medium.com/p/f5bc0a9d579c</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[intro]]></category>
            <category><![CDATA[p5js]]></category>
            <category><![CDATA[code]]></category>
            <dc:creator><![CDATA[Violet Whitney]]></dc:creator>
            <pubDate>Wed, 15 Nov 2023 00:36:22 GMT</pubDate>
            <atom:updated>2023-11-15T00:37:52.050Z</atom:updated>
            <content:encoded><![CDATA[<h4>Calling APIs with Javascript &amp; P5.js</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><p>In this tutorial you will learn the basics of APIs (where we can read information about the world). You will make an API call to a Weather API and read that information into P5.js to trigger an action.</p><ul><li>1— Understanding APIs</li><li>2 — API Keys &amp; Calls</li><li>3 — Draw something in P5.js</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>1 — APIs</h3><h4>What is an API</h4><p>An API, or an <em>Application Programming Interface, </em>is like a menu in a restaurant. It provides you with a list of operations you can request. Just as a menu offers different dishes you can order, an API provides different data or actions you can request from a software application or service. When you use an API, you’re essentially telling a program, “I would like this information or to perform this action,” and the API responds with the requested data or result, just like a waiter brings your order to the table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nhoRhavKuGtPVxX4lKw4wQ.jpeg" /></figure><p>Another way to think about an API is like <strong>doorways</strong>, that allow developers to interact with another application or microservice. It can also be thought of as a “<strong>contract</strong>” or set of clearly defined methods of communication between various services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5YNwU05IKALRtv-eZM2YcA.jpeg" /></figure><p>If a website or application has an API, they typically have a page on their website dedicated to describing their API and how to interact with it such as this: <a href="https://developers.google.com/maps/documentation">API Documentation</a>. Within the documentation you will often find pages within that page that describe the specific syntax in order to create <a href="https://developers.google.com/maps/documentation/streetview/request-streetview">API call/API requests</a> to their application.</p><h4>Why a direct API VS using IFTTT?</h4><p>If you’ve followed the IFTTT tutorials you are familiar with triggering actions. Writing information to the world from P5.js or IFTTT. IFTTT focuses on triggers: the <strong><em>writing side</em></strong>. This is a critical point. IFTTT does not focus on the reading side. Thus to read information into our sketch we will need to access APIs directly from specific websites that have the info we want in P5.js.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>2 — API Keys &amp; Calls</h3><p>We’ll fetch data from the OpenWeatherMap API using Javascript to access it. Once we have the weather data, we’ll use P5.js to visualize the data in our sketch.</p><ul><li>Step 1 — Sign up for <a href="https://openweathermap.org/api">OpenWeatherMap</a> + <a href="https://home.openweathermap.org/users/sign_up">Free Sign Up Here</a></li><li>Step 2 — Get an <a href="https://home.openweathermap.org/api_keys">API Key</a>, and copy it</li><li>Step 3 — At the top of your your P5.js sketch add your API key:</li></ul><pre>// Replace with your OpenWeatherMap API key<br>const openWeatherMapAPIKey = &#39;key&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>3 — Javascript Function</h3><p>Now lets create some Javascript that accesses the API. We’ll create a function called getWeather. Below I’m providing the whole block of code because its relatively straight forward.</p><pre>// Replace with your OpenWeatherMap API key<br>const openWeatherMapAPIKey = &#39;key&#39;;<br><br>// Convert Kelvin to Fahrenheit<br>function kelvinToFahrenheit(kelvin) {<br>  return (kelvin - 273.15) * 9/5 + 32;<br>}<br><br>let weatherData;<br><br>function getWeather(lat, lon) {<br>  let url = `https://api.openweathermap.org/data/2.5/weather?lat=${lat}&amp;lon=${lon}&amp;appid=${openWeatherMapAPIKey}`;<br><br>  fetch(url)<br>    .then(response =&gt; response.json())<br>    .then(data =&gt; {<br>      weatherData = data;<br>      console.log(weatherData);<br>    })<br>    .catch(err =&gt; console.error(err));<br>}<br><br>function setup() {<br>  createCanvas(500, 500);<br>  background(45, 0, 0);<br>  getWeather(&#39;39.9526&#39;, &#39;75.1652&#39;); // Example coordinates<br>}<br><br>function draw() {<br>  if (weatherData) {<br>    fill(255);<br>    textSize(26);<br>    text(`Temperature: ${kelvinToFahrenheit(weatherData.main.temp).toFixed(0)}°F`, 10, 50);<br>  }<br>}<br></pre><p>This description below shows each segment.</p><ul><li>API Key: Stores your unique key for OpenWeatherMap’s data.</li><li>function kelvinToFahrenheit: Converts temp from Kelvin to Fahrenheit.</li><li>weatherData: Holds the weather data fetched from the API.</li><li>getWeather : gets weather data from OpenWeatherMap with lat,lon.</li><li>function setup: creates the canvas and triggers the weather data fetch.</li><li>function draw: Continuously updates the canvas, displaying the converted temp if available.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><h3>4 — Bonus, Using Social Media Data from Reddit in P5</h3><p>You don’t always need an API key to get going. Sites like <a href="https://www.reddit.com/">Reddit</a> allow you to access information directly from the web and draw it in P5.js. The following code pulls data from the subreddit askscience/ parses it, and draws each item in the list in the draw loop.</p><pre>let posts = [];<br><br>function setup() {<br>  createCanvas(800, 600);<br>  loadJSON(&#39;https://www.reddit.com/r/askscience/new.json?limit=5&#39;, gotData);<br>}<br><br>function gotData(data) {<br>  // Extracting the titles from the fetched data<br>  posts = data.data.children.map(child =&gt; child.data.title);<br>}<br><br>function draw() {<br>  background(200);<br>  // Display each post title<br>  for (let i = 0; i &lt; posts.length; i++) {<br>    text(posts[i], 10, 20 + i * 20);<br>  }<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*vXEd1gEQI_mxVEIJCyhs3w.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5bc0a9d579c" width="1" height="1" alt=""><hr><p><a href="https://medium.spatialpixel.com/simple-api-calls-f5bc0a9d579c">Simple API Calls</a> was originally published in <a href="https://medium.spatialpixel.com">Spatial Pixel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ABCs of “How is this sign code?”]]></title>
            <link>https://medium.spatialpixel.com/abcs-of-how-is-this-sign-code-cd07ce48f326?source=rss----9eb53d32da6b---4</link>
            <guid isPermaLink="false">https://medium.com/p/cd07ce48f326</guid>
            <category><![CDATA[spatial-computing]]></category>
            <category><![CDATA[natural-language-process]]></category>
            <category><![CDATA[urbantech]]></category>
            <category><![CDATA[lecture]]></category>
            <category><![CDATA[natural-language]]></category>
            <dc:creator><![CDATA[Violet Whitney]]></dc:creator>
            <pubDate>Tue, 07 Nov 2023 21:15:56 GMT</pubDate>
            <atom:updated>2023-11-07T21:41:50.460Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using natural language programming for the “programs” of our world.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6DAyoYhsX7JNc9b5D_F2sw.jpeg" /></figure><blockquote><strong>CHATGPT PROMPT: <br></strong>Imagine you are a system that takes signs from the environment and executes them as actionable code that enforces the logic in the real world and in the city. Please give me a short summary of how this sign would be code, and give me a rough outline of how that program would look:</blockquote><p>I‘ve been thinking about how signs are basically “programs” that are instantiated in the real world. They have sets of rules that change behavior in the world. <a href="https://en.wikipedia.org/wiki/Speech_act"><strong>Speech Act Theory</strong></a> talks about the agency individuals have and how their speech enacts change directly on the world, like a program.</p><p>I was curious if I asked ChatGPT to read signage images as code, how it would do in turning a sign into a program. It was interesting to see how much context can be inferred from something as complex as a parking sign or a building permit.</p><p>The following A-Z signs I put through the same prompt. The image below describes the structure of the following A-Z images and captions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ENt7UicVrZps8DYoRFAoQ.jpeg" /></figure><h3>A</h3><h4>Access</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hxFUbBTXChTTkXg97oe6kA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*zfBU7hS3blxaKhT-0o55rg.jpeg" /><figcaption>The sign on the door provides a simple instruction for a behavior modification in the real world: it directs people to use a different entrance. As an automated system designed to interpret and execute such a directive, the logic would involve sensing when someone is at the wrong door and then providing a cue or instruction to redirect them. In a hypothetical smart city or smart building setup where doors are equipped with sensors and digital signage, the program could look something like this.</figcaption></figure><h3>B</h3><h4>Building Permit</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YKMdb2pjuf5uthln" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-L-Y-HJr6gXRhY3HcL2aww.jpeg" /><figcaption>BuildingPermit is a class that holds all the relevant information for a building permit. parse_permit_notice is a function that would take an image of a permit and extract the data into a BuildingPermit object. In reality, you would likely use an OCR library such as Tesseract and a text parsing process to extract the relevant data. process_permit is where the logic would be applied to determine if the permit meets the necessary criteria and to decide the next steps such as approval, further review, or update city records. meets_code_requirements, approve_permit, flag_for_review, and update_city_records are placeholder functions where more complex logic and database interactions would be implemented.</figcaption></figure><h3>C</h3><h4>Closed/Open</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GMp6-hFcLE8Z7JKAxAl3yQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*WXpJbqsPcuTMrzROtr6TIw.jpeg" /><figcaption>To represent the information from the “Open” sign with business hours in a code, you would create a data structure to hold the hours for each day and a function to check if the business is currently open. Below is a rough outline in Python code.</figcaption></figure><h3>D</h3><h4>Delivery Notice</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jJXBgkVvczY1blx-YCFEDw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*kLqs0Q2WERRKT86TD400Fg.jpeg" /><figcaption>If we were to conceptualize the logic of a UPS missed delivery notice as a piece of software code, it would involve several components, such as: DeliveryAttempt class to hold information about an attempt. DeliveryNotice class to generate a notice for the recipient. Package class to store package details. Recipient class to store recipient details and preferences. DeliveryService class to handle the delivery logic.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qmrxZQzn65B5rsl9JjIrNA.jpeg" /></figure><h3>E</h3><h4>Exit</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ci7tTCF8mLzernEg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*capURNvyQR90VYvw3E4eUA.jpeg" /><figcaption>If the sign in the image were to be represented as actionable code in a smart environment, the program would be responsible for guiding individuals towards the nearest exit in the event of an emergency or when they need to leave the building. The code would also likely interact with other systems, such as lighting and security, to ensure a safe and efficient evacuation if necessary.</figcaption></figure><h3>F</h3><h4>Fee</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PHWyevntB8bVoLBZ5Vs4pw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*slIz5zIG0FbCqP8Casd-Pg.jpeg" /><figcaption>A street parking sign with a reference to “MeterUP,” which is an app-based payment system for parking spaces. The sign includes a parking zone number (924017) that users can input into the app to pay for their parking session. As a city management system, the code would interface with parking meter management, handle payments, and enforce parking regulations based on the sign’s instructions.</figcaption></figure><h3>G</h3><h4>Ground water</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n31R8pN8-DhX1pWn8LophA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*SweEmj5rqLnAITZhhoAK4Q.jpeg" /><figcaption>We could envision a scenario where sensors detect the presence of a dog near the tree pit and issue an audio message or signal to the dog owner to discourage the behavior. In a real-world application, this would involve more sophisticated sensor technology capable of distinguishing between a dog and other moving objects.</figcaption></figure><h3>H</h3><h4>Historical marker</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3OMUz1IHAiaYC6WOStIxjg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*J0LtRYR_0-5LM9mhdi1lbA.jpeg" /><figcaption>If we were to create a digital interaction for this sign, it could be encoded with the following program logic in a real-world application:NFC/QR Code Interaction: A small NFC tag or QR code could be placed on the sign. When a user scans it with their smart device, it triggers an action. GPS Interaction: The user’s GPS coordinates are used to determine proximity to the sign and display the relevant historical information. Content Retrieval: The system fetches detailed content about Philadelphia’s history from a content management system (CMS) based on the interaction initiated by the user. User Interface: A mobile app or web interface presents the historical information in a user-friendly format, potentially with multimedia elements like images or audio.</figcaption></figure><h3>I</h3><h4>Informational</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*NwTpgJyJW3wMAHOtGuVtSw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*kLUhYhcP0cDSTOQrOjBlOA.jpeg" /><figcaption>This signpost could be encoded to interact with pedestrian’s smartphones or smart devices to provide directions, information, or even augmented reality experiences related to the points of interest. We would store the coordinates for each location and likely interact with a mapping service to provide navigation.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5qDoT45KRmpTujabyLm6EA.jpeg" /></figure><h3>J</h3><h4>Jay-walking</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TIjxLa6Yk8m4c-17HcEqJQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*fL9V6hb5hfWgmSvx6AOX1A.jpeg" /><figcaption>The picture shows a variety of traffic control signs and devices, including a pedestrian signal indicating “don’t walk” or wait, a traffic light on yellow (indicating caution and preparing to stop), and various parking signs including reserved parking and temporary police regulations. In a city management system, each of these signs would trigger different responses. Here’s a simplified outline of how the logic might look for each sign translated into actionable code.</figcaption></figure><h3>K</h3><h4>Kids at Play</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*iZRpiERc7aoxw_YHQ-Ed4Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_oa9FLfkxOhd-1LvfmwV2A.jpeg" /><figcaption>The sign in the image shows a fluorescent green, child-shaped figure with a flag on its head, holding a sign that reads “SLOW!” This is a visual prompt to remind drivers to reduce their speed, typically placed in areas where children may be present, like school zones or residential areas. In a system that could enforce the logic of this sign, the code would monitor vehicle speeds and issue warnings or penalties to drivers who do not comply.</figcaption></figure><h3>L</h3><h4>Loading</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ivghhoG_UXKmEsIVzmSf9Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5arHpZOuj8zlv3WJTmx6tA.jpeg" /><figcaption>To enforce the logic of this sign as a code in an automated system, the program would need to monitor the use of the loading zone and ensure that it is being used correctly. The actual implementation would likely involve sensors or cameras to detect when a vehicle enters and exits the loading zone, along with a system to measure the time each vehicle spends and to identify the vehicle, such as an automatic number plate recognition system.</figcaption></figure><h3>M</h3><h4>Money Exchange Sign</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*Xa8pxzhLq7KYK8iN9uWx3A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dCBJWFbMaVam8RPsXWJ9qA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iZcBKWyAlp8_ORn93kX6MA.jpeg" /></figure><h3>N</h3><h4>Nursing Room Sign</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IlLUjFj7hX4J5d7fM3gH1w.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*sqoJLgmd7u1OkNzLSdCzPA.jpeg" /><figcaption>To implement this system in the real world, there would need to be a secure way to identify nursing mothers, perhaps through an access card system or a digital check-in that verifies their status. Additionally, there could be an automated system to dial the provided number and report misuse when a violation is detected, possibly integrating with security cameras or other sensors to monitor the room’s use.</figcaption></figure><h3>O</h3><h4>Oversized Baggage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*flQ0YerRen4F6zKRuKdh9Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*vXPewa-d4zCBJCJ9Zz6Pww.jpeg" /><figcaption>A set of rules that determine whether a given piece of luggage is within the allowed size limits for carry-on baggage. This code would need to be integrated with sensors or input methods that could measure the physical dimensions of the luggage and feed those measurements into the Baggage object to automate the process in the real world.</figcaption></figure><h3>P</h3><h4>Parking</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t-rhNjrfZqAUhWopPb4mwg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*PdiJIf0QNtLDA_eKKJEgPA.jpeg" /></figure><pre>class ParkingSign:<br>    def __init__(self):<br>        # Define the restrictions for each rule<br>        self.tow_away_zone = {&quot;active&quot;: True, &quot;phone&quot;: &quot;888-591-3636&quot;}<br>        self.no_parking = {&quot;start_time&quot;: datetime.time(6, 0), &quot;end_time&quot;: datetime.time(16, 0), &quot;days&quot;: [&quot;Mon&quot;, &quot;Tue&quot;, &quot;Wed&quot;, &quot;Thu&quot;, &quot;Fri&quot;], &quot;permitted&quot;: &quot;registered_package_delivery&quot;}<br>        self.loading_only = {&quot;limit_minutes&quot;: 30, &quot;start_time&quot;: datetime.time(16, 0), &quot;end_time&quot;: datetime.time(18, 30), &quot;days&quot;: [&quot;Mon&quot;, &quot;Tue&quot;, &quot;Wed&quot;, &quot;Thu&quot;, &quot;Fri&quot;], &quot;start_time_sat&quot;: datetime.time(8, 0), &quot;end_time_sat&quot;: datetime.time(18, 30), &quot;day_sat&quot;: &quot;Sat&quot;}<br>        self.two_hr_parking = {&quot;limit_hours&quot;: 2, &quot;start_time&quot;: datetime.time(8, 0), &quot;end_time&quot;: datetime.time(18, 30), &quot;days&quot;: [&quot;Mon&quot;, &quot;Tue&quot;, &quot;Wed&quot;, &quot;Thu&quot;, &quot;Fri&quot;, &quot;Sat&quot;], &quot;paid&quot;: True}</pre><h3>Q</h3><h4>Quarantine Sign</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w1k-iN6-RHhjJ2PhGl8cCA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*3Y0OCP8hL0GhCwkQM772Wg.jpeg" /><figcaption>To implement this as actionable code in a real-world system, you would need a program that can monitor the occupancy of the bathroom and control access accordingly. In a practical application, sensors and an electronic lock might be used to detect occupancy and control access. The system would also include a display or indicator to show whether the bathroom is occupied or vacant. Additionally, the code could be connected to a central monitoring system to ensure compliance with the social distancing policy and to gather usage data for further analysis or resource planning.</figcaption></figure><h3>R</h3><h4>Recycling</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eazjVQzFVwv26KKMqxehvg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tloAHZAC3H3pWzHHq5eaCw.jpeg" /><figcaption>In a real-world application, is_recyclable might be connected to a database that includes information about what can be recycled, and place_in_bin and reject_item would be connected to the physical mechanisms of the recycling bin, such as a conveyor belt or a compartment that opens and closes. Sensors could be used to identify materials, like cameras with image recognition capabilities, weight sensors, or even scanners that could detect the composition of the items.</figcaption></figure><h3>S</h3><h4>Street</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u0zCVImcXPGKj0QJNQfvog.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*3fzzZ2ogxfd1l_RhX9ircw.jpeg" /><figcaption>The “3rd St” sign would primarily be used for navigational purposes within a traffic management system. When the system detects the “3rd St” sign, it could update the location data of the vehicle or the system, and use this information for providing directions, calculating routes, or dispatching emergency services. Here’s how you might extend the existing code to incorporate the “3rd St” street sign.</figcaption></figure><h3>T</h3><h4>Tattoo</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ycc_F4wCDtBSqcTdUQ-Ubw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*CGUwMJpMN9TLbLkWUPQhgw.jpeg" /><figcaption>The image shows a hand-drawn advertisement for “Cheap Tattoos” posted on a public post, which includes a drawing of a cartoon worm with a hat, and text that says “By Bone Apple #” along with “DM for inquiries”. There are also tear-off tabs at the bottom of the ad with what appears to be a social media handle to contact the advertiser. To create a digital version of this, we might think of a web page or a digital advertisement. Here’s a conceptual outline of how that might look in HTML and CSS.</figcaption></figure><h3>U</h3><h4>Unattended Luggage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*VoNAb_hEecaIXbk6fk2d2g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*bAPu77OvHXJkpzvFfc51dw.jpeg" /><figcaption>To turn this instruction into actionable code for a real-world automated system that enforces this rule, the system would need to include surveillance (camera-based) with object recognition capabilities to identify luggage and track if it is accompanied by a person. If luggage is detected without an accompanying person for a set duration, the system could alert security personnel.</figcaption></figure><h3>V</h3><h4>Violation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*su03qhIzb5A3M-kl-1pddg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*EKPkrvb3-API_x1ElmJTMw.jpeg" /><figcaption>a city management system designed to enforce the logic of such notices, there would be a program that takes the information from the notice and updates the status of the property in a database, initiates compliance checks, and schedules follow-up actions if necessary. This would involve the following steps: Identifying the property in question. Recording the nature of the violation. Outlining the required actions to be taken by the owner. Setting deadlines for compliance. Tracking the progress of compliance. Issuing fines or initiating further legal action if compliance is not met.</figcaption></figure><h3>W</h3><h4>Walk Countdown</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dfy3uJ_c0_rn9oGX2_kDuQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*TXGomJch7sqj-qN32UH5jQ.jpeg" /><figcaption>The “1” displayed on the traffic light is part of a countdown timer, which is commonly paired with pedestrian signals. It indicates the number of seconds remaining before the signal changes, and is intended for pedestrians to inform them of how much time they have left to safely cross the street. Since this countdown is at “1,” it implies that the pedestrian signal will soon change, likely to a “Don’t Walk” signal. This helps pedestrians gauge whether there is enough time to begin crossing or if they should wait for the next cycle. In the context of the city management system, the program would need to be updated to reflect this countdown state and to prepare for the impending signal change. Here’s how that might be implemented in the system.</figcaption></figure><h3>X</h3><h4>X-Don’t Block</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*NsJnrCcSXuMJfno5fD48fw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bg2ZW4eCrbOxdmIJuAfH9g.jpeg" /><figcaption>This sign reads “DON’T BLOCK THE BOX,” which is a traffic instruction to prevent vehicles from entering an intersection unless they can clear it without stopping. It’s typically used in busy intersections to prevent gridlock. To enforce this sign as a program in an intelligent traffic management system, it might look something like the following.</figcaption></figure><h3>Y</h3><h4>Yield</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*13drfmJrD3yTkidXbfxdPQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*to7PJSxqPHicnd9VfxTu-A.jpeg" /><figcaption>The “YIELD” sign is an instruction for drivers to give way to other vehicles before proceeding. This would involve monitoring incoming vehicles to the yield sign, checking for oncoming traffic, and only allowing the vehicle to proceed when it is safe to do so. In practice, the system would require real-time sensor data collection and processing to effectively control traffic.</figcaption></figure><h3>Z</h3><h4>Zone for Quiet</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*viFq_J431ck9KFFt8qXXgg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ng78hJFlR7bcvMJTpHw_gA.jpeg" /><figcaption>This “Silent Zone” sign indicates an area where noise should be minimized, and it mandates no talking and silencing of devices. If we were to translate this into an actionable code for a system enforcing this rule within a library or a similar environment, the code would manage the sound levels in the area and ensure compliance with the silence policy. This pseudocode represents a conceptual program where a system actively monitors noise levels in a designated silent zone and takes action when the noise exceeds a certain threshold. It could interface with real-world devices such as sound level meters and smart device management systems to enforce the rules depicted on the sign.</figcaption></figure><p>My goal is to show how naturally signs can be translated into real world programs with natural language. Even though this process was automated, to actually enact these programs should require humans in the loop!</p><p>I recognize turning signs into enforceable code implies a sort of police state of enforcement. We should be skeptical of cameras checking whether we’re all behaving, having violations sent automatically by mail. When developing natural language programs from signage, we should explicitly try to avoid such a future.</p><p>I am trying to show that many signs are already “programs” and can be more naturally translated to programmable rules that allow more people to enact “programs” on space.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd07ce48f326" width="1" height="1" alt=""><hr><p><a href="https://medium.spatialpixel.com/abcs-of-how-is-this-sign-code-cd07ce48f326">ABCs of “How is this sign code?”</a> was originally published in <a href="https://medium.spatialpixel.com">Spatial Pixel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>