Module 1

Finding the link in the ‘Sciences’

2

4th Jan. 2024

 Recap
If eye ≠ just sight, then language is ≠ just communication

● The function of an object cannot be taken as the object itself.

● The eye is not its functions, it is also its anatomy, and the interactions with its
sub-parts, and its interaction with the world outside (through light rays).
 Recap
Language = Structure and Rules
● The idea of language as a system/’organ’ with primitive elements that combine according
to some fixed set of rules was proposed most prominently by the linguist Noam Chomsky.
● His theory of language is popularly known as Transformational Generative Grammar.

Noam Chomsky 35
On structure of language

● Take the following sentences

(1) John likes apples.
(2) Mary loves books.
(3) [John and Mary] love [apples and books].

● There is one common rule for these sentences (and most declaratives) in English

(1) Noun - Verb - Noun

36
On structure of language

● Take the Hindi sentences -

(5) John aam pasand-karta-he

(6) Mary kitaabe pasand-karti-he

(7) [John aur Mary] [aam aur kitaab] pasand-karte-hen.

● There is a common structure for these sentences (and other declaratives) in Hindi

(8) Noun - Noun - Verb

37
On structure of language

● Despite word-order differences in English and Hindi, both languages have a

common rule.

(9) A declarative will have noun(s) and a verb (or a verbal complex)

● Difference: the order in which we place the nouns and the verb vis-a-vis each other.
● Note: Studies done world-over have shown that there are three (most preferred)
orders: NP-NP-V (SOV), NP-V-NP (SVO), V-NP-NP (VSO).

38
Information/Meaning?

● Meaning or information is an essential function of language.

● Each sentence will have a meaning that has to be passed on to the listener; i.e.
the listener has to be informed about something that the speaker wants to
convey.

39
Structure-dependent meaning

● There is lexical information for each of these words - ‘John’, ‘Mary’, ‘like’, ‘love’.
● The meaning/information of the sentence however is ALSO dependent on the structure, i.e. the
position which the nouns occupy in a sentence.

(10) John likes Mary.

(liker: John, likee: Mary)

(11) Mary likes John.

(liker: Mary, likee: John)

40
Structure-dependent meaning

● There are also structure-dependent restrictions on what a noun/pronoun can stand

for.

(12) Trump likes him.

where ‘him’ can refer to Vivek Ramaswamy or some else.

41
Structure-dependent meaning

● But what about the following sentence? Can ‘him’ refer to Vivek Ramaswamy?
(13) Vivek Ramaswamy likes him.

No! The pronoun ‘him’ has to be someone else, which suggests

that ‘him’ has distributional restrictions dependent on the position
of the preceding noun.

42
Structure-dependent meaning

● But what about the following sentence? Can the two Vivek Ramaswamy’s be the
same?

(14) Vivek Ramaswamy likes Vivek Ramaswamy.

No! There is also a structural restriction on where a Proper Noun

with a specific reference can appear in a sentence.

43
Structure without meaning

● There are also constructions which prove that structure exists alone. Every sentence does not
need to have a meaning, i.e. there is no information conveyed by every sentence.

● Take this very famous sentence given by Noam Chomsky.

(15) Colourless green ideas sleep furiously. (grammatical sentence)

● Compare with:

(16) *Green sleep colourless ideas furiously. (ungrammatical sentences)

● Both sentences are meaningless, but (15) is grammatical; it has a structure.

44
Ambiguity

● Words are ambiguous

(17) Unlockable
Meanings: Not lockable, Able to unlock

● Sentences are ambiguous

(18) The boy killed the elephant in the pink pajamas.

45
Structural Ambiguity

● The information carried by such words or sentences can be inferred only with a structural
decoding/deconstruction.

(19) The boy killed the elephant in his pink pajamas.

[the boy in his pink pajamas]

(20) The boy killed the elephant in his pink pajamas.

[the elephant in his pink pajamas] 46

Just structure

● What we learnt so far:

(i) Information that a sentence carries is structure-dependent.

(ii) Language structure has to be/can be studied independent of meaning.

(iii) Language structure is, to some extent, universal/not dependent on specific languages.

(iv) Language is therefore not just information-carrying mechanism/not just

communication.

47
Conversation with Shannon

● If language is not just communication, then does Shannon’s Information Theory

present itself as the correct model for human language?

48
Shannon Information Theory_recap

49
Fundamentals of IT

a) Information is coded in discrete, binary units.

b) Information can be conveyed without errors over a noisy channel.
c) Messages are dependent on the previous messages to a large extent (probablistic).

50
Measuring Information

● Can we measure information?

(21) There is a reindeer on Ashoka Road.

(22) There is a reindeer on Ashoka Road near the New Parliament Building.

● There is more information in sentence (22).

51
Measuring Information

● But how do we measure semantic information carried by the following pair of

sentences?

(22) There is a reindeer on Ashoka Road near the New Parliament Building.

(23) There is a reindeer on Aurobindo Road near the Old AIIMS Building.

● It is difficult to state which sentence has more information.

52
Coding Information

● Information is coded in terms of letters.

(24) There is a reindeer on Ashoka Road.

(28 letters)
(25) There is a reindeer on Ashoka Road near the New Parliament Building. (56 letters)
(26) There is a reindeer on Aurobindo Road near the Old AIIMS Building. (54 letters)

● But there’s more to it!

53
Entropy

Shannon’s measure of information is the number of bits to

represent the amount of uncertainty (randomness) in a
data source, and is defined as entropy.

54
Essentials

● Higher the uncertainty (i.e. the number of questions/clarifications you need to ask),
the higher the entropy.
● The more the entropy, the more information the message carries.
● Therefore, a higher entropy message for formal messages and conversations (e.g.,
exams, emails to bosses).
● A lower entropy message for informal settings (e.g., removing redundancies in text
messages to friends).

55
Entropy, Redundancies and Information: Discussion:
https://www.youtube.com/watch?v=_PG-jJKB_do

56
Summary

● Things that we know about human language:

(i) information is structure dependent.

(ii) even information about words (John, him) is dependent on structure.

(iii) a word or array can have multiple meanings.

(iv) structure can exist independent of information.

● Question: Can human language be studied within an IT model?

● Answer: Studying information bits does not tell us much about language - it just
describes effective methods of conveying information. 57
Readings

Memory and the Computational Mind CR Gallistel and Adam Philip King –– Chapter 1 (Pages 1-25)

https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf (for more

mathematically oriented readers)

58