I’m currently enrolled in an Informatics M.Sc. program at TUM. It’s now almost been two years since I applied for this program, I kinda wanted to look back on the essay we had to write for the application. We live in an interesting time, where there is a difference between a student essay from 2021 and 2023 in the fact that there now exists LLM tools like ChatGPT to write stuff for you. Stuff like gpt3 already existed back then, but the barriers of entry to use those models have clearly plummeted down Commomerating this nostalgia trip, I’ll be reposting my essay into this blog.

Even though I don’t think I’m a particularly good writer (yet?), I got accepted for the program. After some minor tweaking to fit this browser format, I hereby present a completely human-made essay:

Can Computers Think?

Intro

I am in my last online tutorial after a long day of online lectures, not the most exciting subject but important enough for more than a handful attending students. I can’t see the faces of the other attendees but I can imagine a wall of tired screen-lit faces waiting for the presenter to show the next slides - assuming they are even paying attention. The presentation is being halted because the presenter wanted to show new slides from another file. What he doesn’t know is that the software only supports pdf and not powerpoint. Unfortunately, this will take a few minutes to figure out and five more for him to fix. The phrase “stupid thing!” slips out of his mouth before finally resuming the tutorial. The ‘thing’ he is referring to could be either the computer or software. Don’t we all know that feeling? When we are tired and unmotivated to troubleshoot, that is when the desire for computers to think for themselves is the strongest. Just think like humans! Can’t computers do the same thing as humans? Or taking a step back - Can computers think?

Let’s take a few steps back

The roots of this question can be traced back to the father of theoretical computer science himself, Alan Turing. Like many others in his time, he did not have any luxury to complain about the shortcomings of computers. The prospect of machines being as intelligent as humans filled discussions with more fascination rather than impatience. However, the fundamental approach to thinking about this question has stayed the same. Discussing what it means to think is to inspect the origin of thought and how it behaves. To answer ‘Yes’ or ‘No’, we need some form of definition for thinking. Following Stuart Russel’s and Peter Norvig’s example, we will differentiate between human thinking and rational thinking.

Thinking like a computer

Seeing the construction of a computer just as another utility tool for humanity, we can view the concept of ‘thinking’ in a more general, some would say mathematical, domain. Rather than restricting the concept of thinking to a human discipline, we can generalize ‘thinking’ as ‘solving a problem’. This brings us to the concept of rational thinking. Given a proposed problem, finding an optimal solution to that problem would be analogous to ‘thinking’ itself. Modern applications like recommender systems solving the problem of ‘What is this person interested in given past interests’ open a huge industry of internet advertisement. Modern language models like GPT-3 are trying to solve the problem of ‘Given the input sequence of words, how would we complete the text’. Even more complex problems like playing human games such as Chess or GO has been proven to be doable by computers. Methods such as Backpropagation, which is the backbone of Machine Learning with Neural Network, shows that learning is just an optimization problem of a multi-dimensional function. Now, we have reasons to believe computers can indeed think - and that very well.

Thinking like a human

Intuitively however, thinking is more thought of as a human property. Any perception, reflection, decision-making that happens in our human self can be called thinking. In such a way, we can define thinking as performing the work of a human brain which implies ‘thinking’ also means ‘human thinking’. To compare the functionality of a brain and a computer, there are two different approaches to take, ‘intrinsic’ and ‘end-to-end’. I will now go into more detail on each comparison.

What I call intrinsic comparison means that we can measure how similar a computer thinks to a human by examining how similar the underlying structure is. This would require knowledge of the human brain. Neuroscience and biology has provided us with the structure of the nervous system by identifying parts like the hypothalamus or the cerebral cortex. To zoom further in, models like the Hodgkin-Huxley model describe how action potentials in neurons are propagated. A comparison with computer in this way means that we are more interested in the implementation of the computer program rather than the performance of it.

The Hodgkin-Huxley model describes how action potentials in neurons are initiated and propagated. This concept is abstracted in traditional Neural Networks in form of activation functions.

Thinking as a behaviour

If we are more interested in the behaviour of an entity to judge whether or not it can think then we can do an end-to-end approach. This is done by comparing how a human acts with how a computer acts without being bothered by the specifics of the implementation. The problem in this approach is that we can’t enumerate all the capabilities of the brain and just go through each point in the list to see whether computers can do the same, like going through a check list in a car safety examination. Any attempt to construct such an enumeration would lead us to a difficult path in defining each and every way a human can be creative. To circumvent this problem, we can rely on human intuition. The only judge whether or not something can think humanly is therefore another human being, which is the basic assumption of the Turing Test.

obligatory xkcd

Through a series of questions the computer then answers, a computer can pass if a human interrogator can’t distinguish between the responses of the computer or another human. Giving the term ‘human intuition’ more rigidity, an interrogator can look for the following properties: Can the computer communicate in natural language? Does it store given knowledge? Can it use given knowledge to answer questions and draw new conclusions? Can it adapt to new circumstance to detect and extrapolate patterns? A computer which passes the Turing Test is able to have the appearance of human intelligence, therefore also human thinking.

It’s been half a century. Has anything changed?

Many of the examinations on what thinking means has already been done in the past. The Dartmouth College Summer Workshop that seems to have kick-started the field of artificial intelligence is now more than half-a-century ago. However, the context around whether computers can think has changed quite a bit since Turing proposed it. Where has this led us? Considering the countless Hollywood movies portraying AIs and new startups advertising with words like ‘smart’, ‘AI’, ‘Neural Network’, the general populace might be quite comfortable with the idea of a thinking computer. This was also predicted by Turing. Unfortunately, there has been a dangerous misunderstanding revolving around this idea. The idea of a thinking computer might not be interesting in itself, even Turing called this discourse ‘too meaningless to deserve discussion’ with the lack of a clear single definition of thinking. It could be the fantasy of artificially creating an intellect which works just like a human that seems to drive the interest along this path. This is reflected in the language researchers use for the developed technology which can appear quite misleading to an uninformed observer. Current Neural Networks don’t work exactly like a real neuronal network of a human. Machine Learning doesn’t imply that computers learn the same way like a child does - in Deep Learning it often just means gradient descent. Even a more dated example like the General Problem Solver from 1963 used the word ‘general’ quite loosely. The field of artificial intelligence has survived periods of stagnation what some would call AI Winter. The cause of which is speculated to be the over-optimistic prognoses and unfulfilled promises for the performance of AI, though the validity of those speculations is also debated. Nevertheless, it is never a good idea to oversell a research achievement. It is better to judge it for its capabilities, present benchmark results, realize limitations, and staying away from terms like Artificial General Intelligence or Human Intelligence.

Concluding my rambling thoughts

It might come to no surprise that this essay won’t end with any definite conclusion. It depends on how one defines thinking. As it currently stands, we are far away from human level intelligence. We might not like it, but there is no hyper intelligent home-assistant baby sitter AI for us that can solve all of our problems. If we expect such a program when asking whether computers can think, we can cut this paragraph short just by saying ‘No’. However, if we see thinking in a more rational way, meaning problem solving, then we can consider to say ‘Yes’. Ultimately, instead of relying on computers with ‘human level’ intelligence to magically whisk away all of our problems, the grind of researching, developing, and troubleshooting is still ahead of us - And that might be a good thing.

So that was the essay. Added some pictures to make it more accessible. To be honest, most of my last blog post can be traced back on this essay. Going into the Master’s program I was quite excited to do AI, now I kinda changed direction. But that’s a story for another time. Anyways, if for some reason you make it this far, I would be happy to receive a comment. Finger-crossed the comment system is working.

That term sounds like a group of 5-digit-IQ’ed genius professors figured out how the actual human brain works, and now we can simulate it. I seem to remember my high school biology classes covering neurons to some degree. If we learn about it in high school, then it must’ve been fully figured out, right?

Well, in my 4 month journey of getting into Neural Networks for my bachelor’s thesis, I was quite surprised about how some of my initial misconceptions were quickly dispelled. In this post I want to document this experience and what I take out of it.

1. They lie to us!

The name is a lie! The first thing I did for research is watching the 3blue1brown video. And guess what? There is no simulation of neuron membrane voltages. There is no modelling of real biological synapses. There’s not even a “left-brain, right-brain” kind of architecture.

if you get into ML, you will inevitably see this xqcd

All I see is matrix multiplication and gradient descent, I thought neural nets are way cooler than that. I feel betrayed and it broke my heart.

2. Smart people have done most of the work already!

As soon as I recovered from a heart broken, I quickly realized that the barrier of entry for neural networks is surprisingly low. Tensorflow and pytorch - both are popular libraries that basically makes writing code to use neural nets super easy.

They figured out efficient matrix multiplication. A good working syntax for tensors. They implemented all the things you would find in your Introduction to Deep Learning course, like LSTM cells, the Adam optimizer, Regularization, Dropout, and many more…

Now, as end-users, we only need to learn how to load and augment our dataset, define our model, and train the model. Check out this python code that does exactly that with tensorflow:

First, importing everything you need.

import tensorflow as tf
import tensorflow_datasets as tfds

Yep, that’s all the libraries you need. Then, we gotta load our dataset…

# Loading dataset
ds = tfds.load( 'mnist', split='train', as_supervised=True )
# Batching dataset together into 128 big batches
ds = ds.batch(128)

Cool! We have loaded the mnist dataset in essentially two lines of code. Let’s define our model…

# Defining model
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(10),
  tf.keras.layers.Softmax(),
])
# Defining optimizer
model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["sparse_categorical_accuracy"],
)

With tf.keras.models.Sequential we have defined a simple feed-forwad neural network with 128 hidden neurons and 10 output neurons (with a softmax layer applied at the end). By using model.compile we define which additional parameter to train the model. Actually training the model is just a simple function call…

model.fit( ds, epochs=6 )

…and we are done! That took literally only five function calls. Congratulations, you can pat yourself in the back now because you just did Deep Learning.

(This code works but is useless because I skipped a lot of meaningful boilerplate. Check the official tensorflow keras example for more details)

3. Domain knowledge is very valuable

Because the barrier of entry is so low, it seems the main target audience for neural networks are non-IT people. A chemical-engineering friend of mine also used Deep Learning for her bachelor’s thesis. Linguist friends tell me how their colleagues are using NLP models. The AI focused hackathon of my university advertises itself for requiring “no programming skill”.

This only makes sense if I think about it. You won’t be able to create a neural network-based system for speech recognition, if you have exactly no idea about signal processing. The decision on which data to use is key to a successfull model training. This would explain why Deep Learning is so popular, it reaches a wide target audience.

4. Okay, okay, but what actually is intelligence?

It seems the word intelligence is used kinda loosely. Other than making me feel smart, it’s interesting to think about what we informally consider intelligence. This is something I got from twitter: Does Artificial General Intelligence really lie at the bottom of gradient descent?

When GPT-3 was trending on twitter, a lot of very interesting applications were shared that seemed to require some form of intelligence. It was exciting and I personally never seen NLP to that level before.

Learning about neural networks the first time you will realize it is purely a mathematical optimization problem. In the end, we are left with a function, an input, and an output.

Maybe this is what intelligence is - We receive our perception, short-term, and long-term memory at each timestep and output the next thoughts to occupy our minds with the next set of actuator commands for our muscles and pores. We might also add that this function is fuzzy and non-deterministic to account for randomness. Using the GPT-3 interface is very similar to this. But having experimented with it before, I realized (maybe something really obvious) that GPT-3 is not an AGI, in a way AGI’s are depicted in the movies. In fact, we are farther from hollywood AI than the 12-year old me would have expected.

This is just the rambling of a green-horn computer science student that just learned about Deep Learning. Alan Turing, a way more smarter person, thought about this question more than 70 years ago. Going into Deep Learning I didn’t realize how ambiguous the term “Intelligence” can be…

5. The community seems nice

The Deep Learning community is quite welcoming.

The semester before my bachelor’s thesis I was trying to go into cyber security. Specifically, I tried to get into the hardening of the linux kernel by using virtualization techniques in a seminar. I was a beginner to kernel-level development then too as I was a complete beginner to deep learning. By contrast, I could find overwhelmingly more accessible learning resources for deep learning than for virtualization of the linux memory allocator.

Many of my fellow students are getting into Deep Learning. Channels like 3blue1brown make it easy to get into the math. Big figures like Yann LeCun have active twitter pages you can follow. Youtube channels like Two Minute Papers or Yannic Kilcher make it quite easy to learn about some higher level concepts that I might have been too overwhelmed learning by myself. Tensorflow and pytorch are backed by Facebook and Google, so you can be damn sure there’s good documentation for it.

My verdic: The community seems nice.

Closing Words

I am by no means an expert on neural networks yet but this short but fruitful journey has opened the door to a big world. I hope this blog post helps anyone who was also a bit hesitant to dive into this.

Some other stuff

I currently kinda try to get into blogging and stuff, so any feedback is welcome, just contact me

Jekyll requires blog post files to be named according to the following format:

YEAR-MONTH-DAY-title.MARKUP

Where YEAR is a four-digit number, MONTH and DAY are both two-digit numbers, and MARKUP is the file extension representing the format used in the file. After that, include the necessary front matter. Take a look at the source for this post to get an idea about how it works.

Jekyll also offers powerful support for code snippets:

def print_hi(name)
  puts "Hi, #{name}"
end
print_hi('Tom')
#=> prints 'Hi, Tom' to STDOUT.

#include <stdio.h>

int main(int argc, char\*\* argv) {
    printf("Hello World");
}

Check out the Jekyll docs for more info on how to get the most out of Jekyll. File all bugs/feature requests at Jekyll’s GitHub repo. If you have questions, you can ask them on Jekyll Talk.