"from nlp import Lexicon, Rules, Grammar, ProbLexicon, ProbRules, ProbGrammar\n",

"from nlp import CYK_parse"

"S -> aSb | ε\n",

"That means `S` can be replaced by either `aSb` or `ε` (with `ε` we denote the empty string). The lexicon of the language is comprised of the terminals `a` and `b`, while with `S` we denote the non-terminal symbol. In general, non-terminals are capitalized while terminals are not, and we usually name the starting non-terminal `S`. The language generated by the above grammar is the language aⁿbⁿ for n greater or equal than 1."

"While a simple CFG can be very useful, we might want to know the chance of each rule occuring. Above, we do not know if `S` is more likely to be replaced by `aSb` or `ε`. **Probabilistic Context-Free Grammars (PCFG)** are built to fill exactly that need. Each rule has a probability, given in brackets, and the probabilities of a rule sum up to 1:\n",

"S -> aSb [0.7] | ε [0.3]\n",

"Now we know it is more likely for `S` to be replaced by `aSb` than by `e`.\n",

"\n",

"An issue with *PCFGs* is how we will assign the various probabilities to the rules. We could use our knowledge as humans to assign the probabilities, but that is a laborious and prone to error task. Instead, we can *learn* the probabilities from data. Data is categorized as labeled (with correctly parsed sentences, usually called a **treebank**) or unlabeled (given only lexical and syntactic category names).\n",

"\n",

"With labeled data, we can simply count the occurences. For the above grammar, if we have 100 `S` rules and 30 of them are of the form `S -> ε`, we assign a probability of 0.3 to the transformation.\n",

"\n",

"With unlabeled data we have to learn both the grammar rules and the probability of each rule. We can go with many approaches, one of them the **inside-outside** algorithm. It uses a dynamic programming approach, that first finds the probability of a substring being generated by each rule, and then estimates the probability of each rule."

},

{

"cell_type": "markdown",

"metadata": {},

"source": [

"## CYK PARSE\n",

"\n",

"### Overview\n",

"\n",

"Syntactic analysis (or **parsing**) of a sentence is the process of uncovering the phrase structure of the sentence according to the rules of a grammar. There are two main approaches to parsing. *Top-down*, start with the starting symbol and build a parse tree with the given words as its leaves, and *bottom-up*, where we start from the given words and build a tree that has the starting symbol as its root. Both approaches involve \"guessing\" ahead, so it is very possible it will take long to parse a sentence (wrong guess mean a lot of backtracking). Thankfully, a lot of effort is spent in analyzing already analyzed substrings, so we can follow a dynamic programming approach to store and reuse these parses instead of recomputing them. The *CYK Parsing Algorithm* (named after its inventors, Cocke, Younger and Kasami) utilizes this technique to parse sentences of a grammar in *Chomsky Normal Form*.\n",

"\n",

"The CYK algorithm returns an *M x N x N* array (named *P*), where *N* is the number of words in the sentence and *M* the number of non-terminal symbols in the grammar. Each element in this array shows the probability of a substring being transformed from a particular non-terminal. To find the most probable parse of the sentence, a search in the resulting array is required. Search heuristic algorithms work well in this space, and we can derive the heuristics from the properties of the grammar.\n",

"\n",

"The algorithm in short works like this: There is an external loop that determines the length of the substring. Then the algorithm loops through the words in the sentence. For each word, it again loops through all the words to its right up to the first-loop length. The substring it will work on in this iteration is the words from the second-loop word with first-loop length. Finally, it loops through all the rules in the grammar and updates the substring's probability for each right-hand side non-terminal."

]

},

{

"cell_type": "markdown",

"metadata": {},

"source": [

"### Implementation\n",

"\n",

"The implementation takes as input a list of words and a probabilistic grammar (from the `ProbGrammar` class detailed above) in CNF and returns the table/dictionary *P*. An item's key in *P* is a tuple in the form `(Non-terminal, start of substring, length of substring)`, and the value is a probability. For example, for the sentence \"the monkey is dancing\" and the substring \"the monkey\" an item can be `('NP', 0, 2): 0.5`, which means the first two words (the substring from index 0 and length 2) have a 0.5 probablity of coming from the `NP` terminal.\n",

"\n",

"Before we continue, you can take a look at the source code by running the cell below:"

]

},

{

"cell_type": "code",

"execution_count": 2,

"metadata": {

"collapsed": true

},

"outputs": [],

"source": [

"%psource CYK_parse"

]

},

{

"cell_type": "markdown",

"metadata": {},

"source": [

"When updating the probability of a substring, we pick the max of its current one and the probability of the substring broken into two parts: one from the second-loop word with third-loop length, and the other from the first part's end to the remainer of the first-loop length."

]

},

{

"cell_type": "markdown",

"metadata": {},

"source": [

"### Example\n",

"\n",

"Let's build a probabilistic grammar in CNF:"

]

},

{

"cell_type": "code",

"execution_count": 3,

"metadata": {

"collapsed": true

},

"outputs": [],

"source": [

"E_Prob_Chomsky = ProbGrammar('E_Prob_Chomsky', # A Probabilistic Grammar in CNF\n",

" ProbRules(\n",

" S='NP VP [1]',\n",

" NP='Article Noun [0.6] | Adjective Noun [0.4]',\n",

" VP='Verb NP [0.5] | Verb Adjective [0.5]',\n",

" ),\n",

" ProbLexicon(\n",

" Article='the [0.5] | a [0.25] | an [0.25]',\n",

" Noun='robot [0.4] | sheep [0.4] | fence [0.2]',\n",

" Adjective='good [0.5] | new [0.2] | sad [0.3]',\n",

" Verb='is [0.5] | say [0.3] | are [0.2]'\n",

" ))"

]

},

{

"cell_type": "markdown",

"metadata": {},

"source": [

"Now let's see the probabilities table for the sentence \"the robot is good\":"

]

},

{

"cell_type": "code",

"execution_count": 4,

"metadata": {},

"outputs": [

{

"name": "stdout",

"output_type": "stream",

"text": [

"defaultdict(<class 'float'>, {('Noun', 3, 1): 0.0, ('VP', 0, 3): 0.0, ('Article', 1, 1): 0.0, ('Adjective', 2, 1): 0.0, ('NP', 2, 2): 0.0, ('Adjective', 1, 3): 0.0, ('S', 0, 4): 0.015, ('NP', 1, 3): 0.0, ('VP', 1, 3): 0.0, ('VP', 3, 1): 0.0, ('Verb', 1, 1): 0.0, ('Adjective', 2, 2): 0.0, ('NP', 1, 1): 0.0, ('NP', 2, 1): 0.0, ('NP', 1, 2): 0.0, ('Adjective', 0, 3): 0.0, ('Noun', 2, 1): 0.0, ('Verb', 2, 1): 0.5, ('S', 2, 2): 0.0, ('Adjective', 0, 2): 0.0, ('Noun', 2, 2): 0.0, ('Adjective', 0, 1): 0.0, ('Adjective', 3, 1): 0.5, ('Article', 0, 3): 0.0, ('Article', 0, 1): 0.5, ('VP', 0, 2): 0.0, ('Article', 0, 2): 0.0, ('Noun', 1, 1): 0.4, ('VP', 1, 2): 0.0, ('VP', 0, 4): 0.0, ('Article', 1, 2): 0.0, ('S', 1, 3): 0.0, ('NP', 0, 1): 0.0, ('Verb', 0, 3): 0.0, ('Noun', 1, 3): 0.0, ('VP', 2, 2): 0.125, ('S', 1, 2): 0.0, ('NP', 0, 2): 0.12, ('Verb', 0, 2): 0.0, ('Noun', 1, 2): 0.0, ('VP', 2, 1): 0.0, ('NP', 0, 3): 0.0, ('Verb', 0, 1): 0.0, ('S', 0, 2): 0.0, ('VP', 1, 1): 0.0, ('NP', 0, 4): 0.0, ('Article', 2, 1): 0.0, ('NP', 3, 1): 0.0, ('Adjective', 1, 1): 0.0, ('S', 0, 3): 0.0, ('Adjective', 1, 2): 0.0, ('Verb', 1, 2): 0.0})\n"

]

}

],

"source": [

"words = ['the', 'robot', 'is', 'good']\n",

"grammar = E_Prob_Chomsky\n",

"\n",

"P = CYK_parse(words, grammar)\n",

"print(P)"

]

},

{

"cell_type": "markdown",

"metadata": {},

"source": [

"A `defaultdict` object is returned (`defaultdict` is basically a dictionary but with a default value/type). Keys are tuples in the form mentioned above and the values are the corresponding probabilities. Most of the items/parses have a probability of 0. Let's filter those out to take a better look at the parses that matter."

]

},

{

"cell_type": "code",

"execution_count": 7,

"metadata": {},

"outputs": [

{

"name": "stdout",

"output_type": "stream",

"text": [

"{('NP', 0, 2): 0.12, ('Adjective', 3, 1): 0.5, ('S', 0, 4): 0.015, ('Verb', 2, 1): 0.5, ('Article', 0, 1): 0.5, ('VP', 2, 2): 0.125, ('Noun', 1, 1): 0.4}\n"

]

}

],

"source": [

"parses = {k: p for k, p in P.items() if p >0}\n",

"\n",

"print(parses)"

]

},

{

"cell_type": "markdown",

"metadata": {},

"source": [

"The item `('Article', 0, 1): 0.5` means that the first item came from the `Article` non-terminal with a chance of 0.5. A more complicated item, one with two words, is `('NP', 0, 2): 0.12` which covers the first two words. The probability of the substring \"the robot\" coming from the `NP` non-terminal is 0.12. Let's try and follow the transformations from `NP` to the given words (top-down) to make sure this is indeed the case:\n",

"\n",

"1. The probability of `NP` transforming to `Article Noun` is 0.6.\n",

"\n",

"2. The probability of `Article` transforming to \"the\" is 0.5 (total probability = 0.6*0.5 = 0.3).\n",

"\n",

"3. The probability of `Noun` transforming to \"robot\" is 0.4 (total = 0.3*0.4 = 0.12).\n",

"\n",

"Thus, the total probability of the transformation is 0.12.\n",

"\n",

"Notice how the probability for the whole string (given by the key `('S', 0, 4)`) is 0.015. This means the most probable parsing of the sentence has a probability of 0.015."

]