LNFORMATION THEORY ie 2 INTRODUCTION Measure of INFORMATION ANERAGE INFORMATION EoNTENT OF SyWBoLs In tong INDEPENDENT SEQUENCES AVERAGE INFORMATLON SouTertT oF SYNBoLs IN Long DEPENDENT SEQUENCES Maek-orF STATISTIcAL MopeL FOR CINFORNATION Source ENTROPY AND INFORMATION RATE oF Mark -ofF sourceINTRODUCTION Commadicakion Involves enplicitly she beaneeat selon Gy nfeweation from one potat to another , “Haowgh a fuccesion . preceses - comrounvcation syste Te base clmente gen ane Tianamfitin charme) anal Receiver The functional bloc dragiarn ae digital Communication| Byphor CAnjamalion system) ts hoarn in diqune Lelow : information Source ‘source Electrical communication Anstog ina nar channel el Bin3r¥_ocirical anne sirens signal Sequence of symbols steam Source Channel Destination Je-4—| decor | decoder [@g —Loemredulator Figure 1.2 Functional blocks of a digital communication system. pytinn fe banemit The overall purpse 4 thie ‘ athe meweges C begpencer 4 Ayrnbele ) cooing oe a the owes tom Aeglnakion point ab o& high svobe and acceuray oe posible% The Yunctenal blows ag a digital Cornrountead orn sytem ane \. Information Source information sas % The meaning 4 the wad * Inarnelen in}amation the dy v4 menage a be an elechical menoge 2 intelligence % Tle menage may Bpeech mrenege & plelure < Fe ersogs i a fe memogy 21 music m ec - % A bounce Ubich produces there mettoges fe Called “Inamaton Source - % Tnjamalior dources can be clanifed tote 0 alae : a Analog Tnjororkon founces x Gi) Diserek Injareation ounces can be banglnmed G) An Analey netenad en frounce by the poss Yrlo a clinevele injarmation fource 4 Sarplag 4 agpantin fuch az safer phere a Tv camun seanning ounces > Analog Injamadi on peteteoh fy coin a Beene emit one BH me contaucus enplitud e pink Hme - electeical signalGi) Discrete igpemmation eee tee ah cue ae tele Lype A the numerical oulpot qa compiles conaiste $a Aegpence A Aiserete Sgrobele a lettns « Disereke injewation founces one chasactiniged by the dilloning poramebens : a. Source alphatek Ceynbole n Letline ) b- syrnbel nobe Cc. Sounce alphabe d. probabiliebe dependence 4° - mee . b probabilitves bol Tn + Far fource alphabet centide an example aglish alphabet telelype having 26 ledtens whe Engl plus Atm speccel chanachae ice the 4 3 ayrtel pounce alphabet qn this example consists charachns (comma , _ Fall slop eke ) wake of ie ne lees +6 special The symbol nate meen apo the Uhh the telelype produces Characters - Fe Exanple iy the telelype epuake ab the speed 4 to charactas [coe sthen the : Synt! act {s lo qrbele [se :=a] telelype ts producing metages fo English | sthen Beme letters appear mee geqpently whan othe lelter SE! appeane me open than elles “2°. These Alnchonol propebes 4 symbol Aegpences can be chosachatsed by Sndividual tyobols probability q oceumanee 4 ard by the cmdlitinal properties Cprbabiltie Fa Example es Fo cewenes 9 ayrnbele ahak the From whe figure, lee as anume nprmalion Aource fs = Atacvebe source exvitting dieeke metoge Ayrobel a si se, BY ail, prbabilibres Q ocemmmer given by py Pa oo Py areszectively - The fur gq all these probabilities vnust be ope to 4- PitP2 t+Ps alee PHF! qo preP)0 Source ENcoperR be a Let the input te de Source encoder éleing ay source ayrtele fon We Aounee alphabet 1805 oceuntog ak a nake 4 S = $s, 5s, --- “ag! ayrabele free Bs eer oe Coats Gr Ayobel Aegpence Gnlo a blran ‘ : ignin b ~ Beqpence 4 os 41S by omigring Code-whel to the aye - Binary code because a ite high -Yrrerey 3 bransmithion - There ane two Way q ony C1 Fnted length binary coding Hy bfnany coding codes : GH) variable leng The sienplest wey coding Sn te owge + fred length binevy codenad jo each ayreee! Sa athe impel Aepience . TRE coding i apfrcent oly individual Abele eacun pith ep! probabilidies ja a slatichedd ‘ndupendent Begpence - In mest prackeal gilualions the 4 bi, anegpal probebilities . The fource entocler Nanable length coda tele to these apabele - “The immpittant poramebens q a somee ercoden aon block aige, length 9 cade nade aver eclalu nah encoder efficzency - yy the hole cecur! Shen aign43B. TRansmitteR St Tranemittes couples the fopub -newage aigral +o the channe) 3 Sometimes it may be pewible yo couple input transducer alineclly do the channel - the mecenany Fa efferent bransminsion tb Ms ten ol she preex and modidy che inpuk Aig she Srgnel prcesing operak one performed ‘a 4 modulation. Nsanarnttbed Seles amplikcekinn dilkening 4. CANNEL ENcopeR 4 DE coPER Ee — ee % channel coding is a practeal meted 4 aealiaing hegh beansrniMion reliability and eypteney . channel encoder one * The impatant parametene q* the method 4 coding _ take mn fidence 4 the coder ent conhel capabilibes and complenity 4 Futthermine | the ane is casupted by unvanted unprediclable Aiqrel called noite . The effect gem Te te Convent fore gq the OS into Ys and Some 4] the 1's who os. The signal ie hen fad to be caaupted ty noise - + Denop erates Lik sheam of ike apa evejere suitable Jn kansmistion over the emenmunicakion channel - wed mining fo makeh the 6. Mopvtator % The modulalo aceeph: a and coments ik te an eleckical 8 ae Modulabn can be efieckrely the epee 9 channe] notee | speck the lranamitted aiqrel chanackuelret , to prwele athe Cape! mony Signal and he overcome ome eqptpmes limidatioae - ) pile channe bilty 4 enultiplex t% The exkachen 4 the memoge jem she ingame Son beaning wavedem produced by the modulata i# Accomplithed by the Aemodulatsr - & Fn a given type 4 modulation the mast ienpattank Parametin q the demodulahn 4 He method % Aemedulah'en . 7. Décoper +4 DESTINATION The founce decoder converte the binary oukpuke @ the channel dlecoder whe = aye Sequence + is x The decoder Jar a dined -lengih code wade quite simple Luk the decoder for aythern wring Voniable Length codewads is Very complex - * The qunckon 4, de Achives ( Deshnah'en) te to idenkgy the sgrbel Aeqpence and mokeh iF with the chrect AegpenceExplanation oF Glock DIAgRAM oF AN IINForMATION SYSTEM % Fin communication the basic input should be inferoation ounce | Which can be analog a Avsevete . Here ve This ° and % consider it as fincrete infeomark'on fourte . injsmadion WIN be fn She dam 4g Aynbole posed te dhe source encoden - # Source Encoder will compen the Sige 4 Infaimmation ee ee ee eer et oe Beapence and then posed “eke channel enceder . 4 Th he channel encoder the neise meg at added. SE he noite get added af the last bike one Called as cheoneume - > channel ig a path to hansen the data fom pounce encoder to modulala and the ol 4 channel encoder vill be fn binany Begpence ° Medulelien fe We pres 3 ying the ingparnaion nal and the olp % Aignal wink comet Bg a the demedulbr Se elechraol igre) and is ar via elechical communiadten Channel. A Demoduladren {6 whe prees gp geting back the binesy Aepence Jon the clechical tignals elie gyro + channe) decoder -Channel deceden f¢ use fo delecking and ctrachog he esse |rofte - ot danes place fn 3 skips: i) Ewa debechon Cit) pews bi kty 4 en casecken and Litt) Ewe caelion - Now the ouput will be ‘nm the dam q binagy ] bik seqnence ard if paiteal +2 he Fomrce decoder . Source decoder fe uted fo convent the injamnadion hich fe fn the fam 4 binary J bit Iolo the miginel signal be Ayrebele do We deskroten * his Bigral 1s veh fi receiver, £0 Bent % aeceive+ 2- Measore oF INFoRMATION tg ee % nade to Know and Compare athe | Ingrmatren content’ . 4 Vanious meboges prduced by an injomelven founce Mecwtany to knew Hak injernation content | a Measure 1% te bakvely The oukpuk goa discrete infamalron Aounee ba ~ meoge that consist, 4+ Acgpence 4 Aymbole “The Ayrbole ane emitted in a Sanden qdashien henee the ounce oukpub is mondleled az a cliterebe aandom ventable © - inite hich duces on Aymbole five * grued + alphabet Was es prbabilehes pls= su) = Pe Kee, This Bet aq probabiltes neha Condi: on ket SR t! ko Tt vor “Shannen” who published “the mathe maken! sheay 4 cammuntcabien in Abich be aslabliched haste thetrien! bounde dr he peyomance® F Communi cation systime .¥ Shannon’ theay fe based on probabilishe models Ja Injamation Sources and eommunicabion channels. thus om an Intuthive basis the amount infamadien recived frm the knovoledy « 4 ip nelabed We He he event” cccutence og an event probability 9 oceumence Ff lren than other Contain m®e informe A Some Metage mMetoger ¥ The mesoges aspociated| Hite an event least lieely information : emiiz one 4 ‘yy with 4 cceun Contains most 2 Suppote ar Infermodion founce Powtible memages prbabiliker 4 cecumence mg a-- ym mma 8 OY woe Py Pi, Po Po) Pit Pat Pat --- + Pye! Sabuikien the afematten content : asthe it mes one | arelate d * Acending te out B the amount 4 in}eroabieon in ented by (med, math be invensely 4o Pe > Also, T Comune) ™ oppeaches | nen vat apprach o ot Pe tent Lm) must be mation + % Fushermae , the injarmadt on Con ee eee een Neqahive since each meweg Ab wast Lene) mag be aqpal te zer0Summrnantging these aepinements Limn) must salighy : , % Lew probable an event is the mare tqferrnakion He goin when TE eccuns CR 2 ce) ie es ee % xp ue ane absolutely contain | the outcome 2, Qon event even befate ft occurs | dthene "7° infarmabion guned =) —> © => ITcmi > ° os Pu The cccumence 4 “> event mem ethos provide fame & no Iafemalien bub even brings Pees a lows 5 Injamakion . > rl) =e hen of Pe £) — © * I,m, and my ote gtaksteally indapend ent > xm anday) & x (mes) atm) EE) — @© the * (kh. cockinuss Juncker q Pu conshaints apecited tn eapobons da can Aepred ~ leqeatiimie dunelien and Be measure infemateen Era +* os* The base fn the logaaithens i eqpation ws) dolemines the unit axigned te the iqbamaticn content . —> Th the nalusal legen’ tren eee anes then de unit fs nat Thothe base fs to then athe unik fs Hartley © Deeit . Ty the bare fe den the anit 4 an abbieviatten él lathy leg expreni on % Logarithmic enpretsion 14 chooten fai measuring injermaleon because q the qollewing mearons « + The infomation content a elf information MY wera cannot be negehee : Each coed mus fain centain amount 3 Ingromalion - + The lowest pewibte self infermatien fu Oceans Ja a fume event Aince Pol: Mae infmadion ’% cated by a lens Tecely mettoge, - len independent symbole ane Iranemitted | the bola sel - infamection erusk be egal te the fom 4 individual Belt - infemnakion :Examples 4. A source oub, me F 5 posible menager ding each memiage inkaval . The probabilities a these Treiegee ane Preh, Reva Pst Ve Pat ‘Ne, Pet Wg Find the injamadion cevtent q cach q Hes mtg! Tom) = log, = = leg, P= -teg,(%) + thi Z(m.y = - legs, = —I9g,(/e) = 2 bits Itm3) = -leog,Ps = = log, (Ys) : 3 bike Tima) = -t03, Pa = - besa (yw) - 4 bike bits Zims) = - leaPs = —tog s (Yu) a Censiden a Afserete sneronnylen Aource (Sou ce) hak oukpub, 2 bite ak a hme - and 8), each 2 the Bounces combi buding tome bit. the tee Lnany Aource jstthin “He fous € are Sodependent hak ts athe fnfimation content g each oukpuk the founce © 2 This Source Comprises shoo binary ounces C Source A oloitene > Since each Aource Cm bibubing too bite they vill be eqpi probable play = pla) = 4 = oF ome. bit out opSs There ane 4 penible oukcomes {oo ol to "4 Per ee occa Honea probability Pe) = pCa? Poe) = Cos)? Co-s) = 0-25 Since the Bouncer ante Independent . “The infamadion content q each oukpuk fom the fource ec 15 ler = — leg, Poe) = - leg Co.2s) ce) 2 bits oceun with probability 3. Th binany pem ‘o Ya and ‘1’ occur with probability 3)4, sen caleulake arneunt 4 ieqotrontion conveyed ba each unit . Solution plo) = Va pod = ala a bik Ilo) = log, +— ara % P lo) . TO) = leg 1 = otf ee ~ puy Andependent A Ty there ane M eqpally lncely and dep metages then prove hak amount inference canted by each menage wit be T= nw bite, there He 2% 4 N ie an fnkages |pom Since all he M mesages are eqpaly Ieely and independ ent : Probability 4 eccumence J each menege will be POM) = oF tet ha, i) Tet loam = log 2 Hence proved - Th, fe He ta|ncredins canted by memoge m, 4 2, is the Tnpoonah en Comted Seda ™ injareation cased shen prove hak the arnount 4 te L, wath Comparitely due to om, ard ma Sorution WetA. AVERAGE INFORMATION CoNTENT OF inrareeea ees SymBors In Leong TINDEPENDENT SEQUENCES s in 72 ee lee Memages produced by Infermnadkion Bounce consist A eqpences symbole - fokenested mainly Sn wade | LUlhexear the Communicahion 4ys wihy individual leHens m1 Ayrrbole - “The neceiven end “4 phrases and sentences: fem hes +o deal The mehic dev measusing the Tnjarmaction content to be fransmitted wus developed by shannen , sucha mebic HW called tte Exbopy 4 < rnemcge Bourte - ZERO- MEMORY SOURCE ¢ > 4b aepresente a model 4 Atacrebe injorabon ounce erntlting ee Beqpence 4 sgrabels yor & dined finite source alphabek $= fy, See, Suh do some dined Symbols ane Selected aandonl, probebility and ane Stahkalenlly independent a called one another - Such Ayre 4 Bounces ane “ Bources « * merntrylen ” av Zeno - memhy . te “these {a no connection betaen any te Ayriclt and thal dhe Zource has no mery Such tyre q ounce ase Called “ memryleu” 8 “ Zeno-memby” Sources -ye Lek us Consider a Zeno-membty source that emits ome a ON pewsible symbole S152) ee, Smt" a Skakisheally ‘independent seapence . we the probability 4 cccanance panenlan Apatel “enna * Ayre fatinval cloee oot Aepend the Aynbels emitted ty He Aounee using the preceding Aymeet fakewvals . Lee Py Pe ---- Po be tHe prbabilitves cceunence the M ayree ouspectively ” containing N syoieb she Now, in a long memege 4 the Aynbel 5, all sh oceut PN Himes ad in ge , wid occur PLN Hmes - the syne si Treating Individual fyenbels as moamages 4 oa oe 4 ee ene, the infamakeo conkent 9 athe 1 Ayr can be dapined a log, ‘Je: bite: D> lence the PN Ceeutinces F si Conbibukes OM injerena bien content GQ PLN og, + bre. The total tnjnmahion Content gq He metage | q she Coohibulten due to cach q He HH ayenbele 4 the four alphabet and tt gee by Zital = 2 NP: leg, (+) bike t=)> Eapation @ wa wed by banner * Some Properties OF ENTROPY The Average Tnjamalion Per symbel can be oblained athe total ‘nfamation conkint a He by dividing mremage by the number 4 Ayrbele qn the memage 5 ~ a ee eee (Fi) vite | aqabel N ° ” 4 Zz Pi log, ( point fe hie augine! pres entation 4 the mathematel thedy 4 Camrmunicaken - Content per syrte! % The Average Tngewnakon alto called dhe " Souret Enbory ounce whete 04®@ Contider a diserebe Zeno-memby watremalical model is defined by eqpakions below : s- {si se os -- Suh = 2 ’ - P(s2sx) = S* 20,4, 777 ey ee wet bhre = Pe =! wes: a source ‘6 hounded os dollous: The Enbopy HS) Auch oz H¢s) & (9,4iy and only yo the probability Pu >! dh some K, and dhe semaining probabitives ‘o the Ack ane all zero. This lower bound on Entepy Cbsusponde +o no ancertainity - H(s) = tog m , it and oly if Pr * Ye qaal Cie all symbole fa the alphabet 5 ate epiprbctle) . This upper bound on Enbopy cdtespord +o with an alghabet 4 % In general, ja a Source 4 syrbe le He maximum enborg is attained her the ayrtet probabilities ane a We, when Pye Pe ec-c7 = Pw = Mn and Horan 4 quay 2 tog." bits | symbel Han R Soorce INFORMATION RATE — ———-~_~ ~_—_— emitted by * Source . abo + Ty te Ayonbole ane ee Aounte infinmatien sake “Rin as the poduck g he aveugt ingrmation pew syeobel and the Ayre aoke As” R= ast bib fiec Whae ag 2 syrobole cee = bete |tqenbel R= bik [sec = As HH symbols [tee - The aveneg® bite |see can be dafined contentProglems 1. A discrele founce enh, ome g fe Ayrrbele ence every millisecond . The Agrobel probabilikes ane ‘bh va, Ve, de * Ye mapectvely . Final the Source Enbery and Injawroation mate - SoLut ion . Ss 1 Enbopy hoe oP: ba, oy cel A leg, Ar. Lu 1 + 4 log, a + > 7S + la, © my) GQ s Ce) ‘ \ + bg + + 7 2 ¢ a ra Ge) + Pa + + leg, (+) +2 tog, (#) + LL be LL tog Le te tes, tied + ag baat) =| os + oS FORTIS FOR +038 "Woe Lets Lit| abel Tnjamation sake R= ag*h = (1000) * ae RB 2 1a te bits [wc ee hak emite ome B symbole A, & ard cin a Atakabienlly independent nhc Bepnes wilh pobatilcker Ya and Ya meepeehedSoLutz0N Enhopy He \ 2 \ og. (2) + + founce emit one bog *) tog, (4) +L tog, (4) a OS +05 +0'S cs | syrabel Bin 4Y mbols BA Atscreke once evay msec - The fAymbe! prbabiltker 76 “uo . Ye Va, Ye, Yu, a a aespeckvely _ Ria she source enbepy andl Tapamahion aake Sototsen ‘ aed 2° toa 3 Neg, aay ae) . A to9, aw” * sa. hy * ee) ‘ SL tog Xe Ie Uy +s pe ie) Be - ) eel fe A ig, ¥ a er Weg 4 FE 3. 1 tog, ae Hu [Hoe ans oe Trjamabion aoke RF # Pilioco ee eee, R = ast bits|seo Lh teg oF + WG - 32 of 40-8 + 0-318 + ps x 0. 1SG2S + ONSERE 2431s bits | genb Ae + 0 sis)A. Consider a discrete memaylen Aounce nits Source alphabet s+ | So, $1,525 and Source Atodishes Jot 0 is ots h. Coleulate the Enhopy 4 ae founce - SOLUTION Enbopy W = Pr log, “or bite | eqnbed log = 0-1 log, tA + 0-18 tog, tons 09 os = O.3bor FO-FlOS + O-AI0S “ \ei2 fs | eqenbel bite . 5+ Find melabionship between Hartley , mot and bits Solution Wek-T I Le ike x = les, + & G1 Vanes — nate rrelabronship Frown eg La) = 1 Hantley e e910 F 2 loge @ - ly F | = log to le n todo F leg 6-3 SL Heatley = 109,%0 sake teg 2° tee) hk \ Hatley = 2-303 4!Gi) Hastley - bik relationship \ fom ep%tay Ts tog, 7p Horta fiom ter I = leg, -s bibs \ Mantle = x. lea F Con) 1 Reakl, 47 Gi) Nats - bite arladieas hip Frona og Cb) Ie 09 66. The output oq an infamalion Source consists 1So ayrobels 32 4 Lhieh occur With ow probability 4 Yea and the remaining wih a probability bole [See 4 'Yasg - the Zource emite 2000 #4) Arsurning that the symbole ate chosen independently ind the avenage gyration poke this Bounce - SoLutiton Ris aH (150-32) WOT Average injnmabion smoke 1. leq at | x Ne 1 1. log 6* [ Ge J nse + ern 3 43-44'3 We cA qs x32 + 003340 *NE 2 0-043 Woe 66-9413 bike | syrabel syrabee | tee Given As > 2000 R z=ae* 4 = 2000 %* 6.9413 \ e+ 13, 882°6 bits [ee 1. The iwhamalvonal entice code wee o Aeyteree dole ard dashes fo Wwanewit (etbess english alphabek . he dash is Aapetented by + cumunt pulte thet hee a duaakon dosh a dunakion q 3 unit and y Vunik - The probabil q octusence oe 4 dot . is hog we probability F ocewmence go(a) coleulate the Injewnation content gq a dob + dash. (6) Caleulate the avenage Infomation fn the dot - dash code - (e) Ansume thak the dot laste meee, hich fs the Aame -kme inknval as the pause belween syrobole . Ree dhe average nok. g iqfamakien brane mixtion , Solution doi and clasher ane prtent > Since onl. Poot + Pdash = Paw = Plot 2 Pdok * 3 Pdash ay Poot + Pdok = 1 7 Ss Wes Eta i Fa every a ayobele . Pdosk = log 4 = 0.4150 bike [leg Cars) # 23 ——_—$—_—__— log f= 2. bike 2% sleylt 3 log A leg + soe BPI, Zrastta 4, rial) = 0.35276 + os = o- euare bits] yonte ee Tegrabl © a dete dash spaces Ts (Sxl mice + UV esmsee T 4 ximsce) = lo msce As = A Symbe ls |r see Aoo symbole Jece Injarection mabe R= 1s * 4 = hoo € 0. GN27R = 324-5112 bib icc. Consider a telegraph founce having 2 Agrnbela alot and dash. The dot dunakon {4 oatee and the clash dunakion 1h 5 -hmee g the det dloraton. Te Prbaliltly gq de dob cccusing ‘a befee shok 4 clash ancl time between Aymbole ie Ovatec - coluslake she infamahen nate % the telegraph, Source Producing the ashing 200 syrnbole - Solution Lek the probably 9 dash be oF and tHe probability 9q dot be ‘2p and pr>ps! Ean pecan arcs Vs Plo, Par 215] °F Te caleulate Enbopy (¥) He Llog 3 + & bog, (3h) -0s 282 0-476 = 0.4183 bits | ymbel ee % Te calealake sqrbel_ sake (A) 3 Tat 2o2f, Tdesk * 3 (tat) Time bebocen synbele = 0-28 B(o-2s) =o ee - Ts Tak * Tdesh + Livoo + “egerbat ) Number 9 doe = ZH 1200 = #°° > Number gy dosh = fu woe * 4°° “ T2 [800 40.2] +[400 0-6] + [i200 #023 - loo+ 2404240 2 6406eCas 120% 2 Leis symbols fice e4g 2 Ne #H = eis # O-NES an Tn permation robe RK bits [see Ro = [7218125 A. A black and uhh Tv piche conith y S28 lines each line consists 525 4 pickin infdimadkion . Assuring pichne element having ase brightnes levele andl the pichna ate mupeated ak the eke 4 aolice « Calculate dhe overag © injarmotoe rate emveyed by a Ty see to a Deceiver - Solution 62s line One picture Contains g2s picture | Each line contiatz ements Pichwes ane Aenk of a nate 9 pichse element nake Ca] = 528 piohee elements} tine x 525 Vine | prehwe ¥% 5° pichue [sec 30] ice pichue zlemenk, fic 4 2 g-26 87S AD pe 2 log a choos 8 log 2 leg,» aga 6 g- 2681 AIO pr 66 Is Mbps chee clemenh hee #8 bie nc\o- A cand te drawn Jrom a deck gq playing conde. (a) You ane infeed that the cand qe draw is a Spade _ how much inferrnedion did tr” Meceive - (Cb) How much inkamadion do ye secive, Yo Yor OF told that the cond that ae drew fe an ace? (Ce) How mach infarmad'er do ia Sucve YY yer oid thak He cand ou daeve fo an ace 4 spade? (A) Le the injamation obtained tm Ce) te the Bum 4 infomation conkewh 9 the metages “spade” and 30": Solution 3 Spode cands tn adeek qe) (a) Since there ore Pspede = 2 038 s 2 leg = bib Sel] infamation OL spade (b) Sine Hore ate 4 aces in adece 4 se Conds Poce = # >t sz 18 © Sey iqjewken Laer = 9 67 Paw Cod Since there is oly one ace - y - 4f % Se wade Poe q - spade “52 \ = tog $2 -S-7bi leg — 2. Selj nptrtion Et spade : Pag ipa(A) Yee, the infamation obtained in Ce) f4 the Bure J injamab'on Content: gq the mesoge4 Spader and ace - Since doaering a spades poner drawing an dea‘ cand ane toe independent events and hotel Bel} infomation must be eyel + individued Sel} ingewnation L > Tye + Tad ate -Q - < a 2 6-7 bike = 2431 51} ed ancl Aignal is bard limited +o S004® and. Aampled ak” Nyqystt Rake "+ The amples one pentsed tnbo 4 levele . the Quantigahen levele are ee yablikes Pye ED the Source - is be indupendent and occur with po Pa 2 ty = 3+ Find he injnmabion sake 4 Solutton HOS = og bits | yrobet es i tog 59 ] + [3 toa, (£) J - 2 (0.375) +2 (0-306 #4) 2 OS + 0613 4 = 1-33 bils| tyrnbed *) Is ust aake Ginee the signal is sampled ok NYY ieee ae2re] oa yee 500 \ _ goo gpalce + Tnfamadtien pee enea emia ons, wes) _ pooo # VF 1Slobib [ee a 3 = 1Blo bile Re =wil, thak 1p. A Source Wwansmits two independ ent memeg pebabilly qf and (i-p) aapechvely - pore che Enbopy 1s manimum when beth the memoyes ate plot ‘he Enbopy % Bounce Var kut egpaly Kixcely _ Probability oe Sowtlien ae “ y 4 +d wis) 2 & Pr leg, (+;) bite feyrnbel 2 metre tyra ten Fay boo mess ae, . His) = 2 PS eg. (=) 2) 2 P, tog, at % lea, 5 a probabilitres pac) “To memogesé om, amy saspectively wots = P leg (+) * (ir) '3, ta cn %loge pe % when poo the Enbopy HWUs) =?) , x20 X when pet the enbepg PEPE © + Te Enhopy Bs) attane 16 maine value ce whe - \ brk whee = Uiep) tt Yeast = Ayenbols | and o one opel probe ble -Proof For Louler 4 LppeR Boontos oN Ha) Mae Ket Hos) = pleg, (4) + Ci-P> leg (=) To find marimum Nimit dus. de daned 2 Q [r te9, (5) Lice) 993 (te) J dp ae ; -P) 2 AL Pepi ep - Gee) 92 O 4 [- ris, og UTP) = = -\ - pA log i? + 129, og, P i ° ber) leg, OP) w “t leg MF log % leg ,% » — ee tog tog," AGL) ~ tog ppd Mek + tog, CrP) = ap ° a AF tog > : (i-p) A leg iPTe find masimum value d ues) 20 ap lo = leq Cir) af 9 PF IAP > pit Vo Thos when Pe%) licpr= YL the HUs) is moasimum . itt be lence suleny create ere exquiproballe En bepy oa masimum - naar] : ence eHts) Slegh WUs) cannot be negate, henes le £ ae | plok 4 Hs) Nensus probability r(eR) A binaay ounce is emitting an indepe 4 o's and vs with Probabilities pa inh ™pe ndent 4 Avely plst the enkropy gq He founce versus P Solution Enhopy % binasy fowce “8 ae >) H(s) = SP beg Tr Pa Faget nlak WO 2 phy t +e P) Arp) 20 bik leg 2 0ae ably oO: 1 proe Hls) soaleg 1 to log a a sar bitsy oe os PEO3 BCs) 20-3 leq at bot beg 20.88 bits } 0 a 20.4! bik [yr i} bib [4-0 Lek pso w (s) Fo. leg ine ° Leb pso-l Wis) = on eg 2 0-4 0:1 peo-4 HLS) 20-4 log +, to-6 Wg 20S s \ 6 leg P Wes) = os leg 35 tors beg te 2 0-4 bits |445 pro wWls) o-1 tog 4. 408 log 1, = 0-881 bits bys $4 os hk h peo-e wes) 0-8 beg tere ley gg * O° 7 is By “20-469 bib fey Hee) 2 04 logee fob ley 5 L eee (s) = Vlog L +o oy poo 6 HU) Foe leg ae +0-4 log coer o ° bits Jey sab pez}t ' ' 1 ‘ ' ’ O11 0 oO OW 0% Ob DA OL OM I Hene ane 3 13. Fa a diserele mmerotiylen Bounce Ayobole hth probabilibes P, 2% ana P, = P3° Dekrmine the enbepy gq the Source and sxetch th Voasnlahion Ja iperent values J %° Sovut1e0N ~ B prbabilha are AP + Ps KT » pe Rrh = ! Arr, *«(st ) ba (ED (Sh (E wate x) leg, (4) HenoryLess Source of A D1scRETE ae blocks rather than “individual Extension ~ # ae te sel +e econsicles Ayrbels wlth each blocs consiskon qe Baceas ve Bounce. Byrnbels : berg prdsced 4y abet 5” dhok- yk cach fubbleen 1s atsumed an entended gounce with a Fore al leg has ur Aishnck bleee there te the suenben ch Aisne synbls tn the ee alphabet s 4 the digi? SourceH Tn te cate gq a Aserete memayles Source, he Bounce symbole ane stociseally ‘ndapend ak Hence , dhe pobabilty qa ome Aymtel in 6” te egpol to dhe pubalilities predict 4 the 9 Source aprtele he Enhopy q the extended source H C5") is eps n times BUS) - 14. consider a Atscrete renngie Bounce with Source alphabet 5 = f S051, S25 ith pobabilikes p= fhe pel suspectivdy © find he Enbopy 4 Yak ade source and Beend Ader source HS) - Sowwtten o ‘ wis) = oP leg \ beg, Ar a eg * ( o-s xors a0 = x order Extension pe Sounce " % Uh the fource Ie S consiatiog 4 Aye di Bounce Ld He source alphabek s*q te extende: mine tymbo le.¥ Lee the nine & SoS, 15,50/5,5,/ 5,52 1 a 1 1 u 2 1 | @ wet) 2 Sore) ba re) i2o > ayia ce) + a Noy, C16) > 7" eg, Ce) ¥p leg, 6) + aes, G6) + 1 10g, (8) ea 129, (eek log, (2) + L9,4) > ous xots to vyar tone > oS + sane gant boast wenet H¢st) 2 n BG) eras = 3 bite + Prove Hs") = n Hs) Considen the syrabee se Ss sly aith pbs Perit, P a) . ote) 2 & Ps Mey FF bi bebe + = plea. 49 8 ao) 24 nder ertengion 4 sh beg fosnce Wid PAE eatingion . Ce [a] 724 number q syratele ane byvith probabilities PP, = P7 Pipa BPs rd ackensi Sum gall Probabilitee g the * ader extnsion Bounce is a . preann res (he) * The Enhopy 4 a4 mde fource | ave A ‘ \ Pf log Hise) = 2B ae: jefe! P92 (&) RES] te = ¢, log, (5+ ate (4 je Ble. acs Pa log Se \ tog, 2 + Ps leq, + PLR eB Pres tot P,P, 4, ee ‘ air ‘ L ap 4 a Bales, Joe aR es a Py PP Sf po leg, tog 2P, Leen] leg, es + PL Lari) 7 com Binte Pia PS 2) . n ee = aeleg at Ah 92 Pe 2 1 jg bea, 2] 2 2 Leh, ate PL [aed- 2 Ws) — © had ee ee 4d indey extineion vill Fete 3 Srevilealy65,5) $,)8)52 5525) 5, S25 6.5151 S255) S,5, 52 52 52 So uith pobabiltes pS p> Pa , 2 mo . L : Pres PPP Po, Peet Poh, pe sapechwely “sum g all prbabilites g the 5° eddended brety Bounce if Lo 8 s pir sen tanh + Pe = (rer) =! vow (eys aut Ta grovel mY extension gq the ° symbole and Aource will have 2 ath eukinded four. is giver by binary basic the sober 1} wes") = on HOS) Properties OF ENTRoP wate ENTROPY % Fn a Source alphabet s= $5152 --- Sq} tt Probabilives p= LP Pa --- Py where 9+ een 9 Aounce Aymbols % The Enbopy dueckon i gen by Hs) = 2 65 Ie a 2 FIA 6, ic) Z te Tey% The properties q the Enbopy are listed belo + © We Eabepy Fircton fs Conknuoue der €¥eU{ independ the wwhaval (0,9 te Y Pe veniet ctunekion Nasiable Pye in continuously fen of 1, $ @ The Eabepy Jerekion ko AY ongimenti: be H [ee on red] 24 Loe red Pe] eee ye oY amehical qunetion gp Mt Ce the value gq Hs) Semaine he fame imupeckve Q de locati ens q the probatilidies « Thus the sources Sa Sp F Se vit prbatilikes. Pa = 4h, Pa bed : fear { Pa Pay ey metmnts Seg ite u(Sa) = 4 (Se) = (Se) ® ExtRenal property * fame source Sith Y > Leb us consider athe F abiltkes sy} with pe b Symbols 525. S2,--- The Enhopy at 54 eo by y ep Ze \ 2) Hts) 2 2 PL eg> Lek us now prve dhak the Enbopy Hos) has an * upper bound ” rt considering a gpantty Lisgy - 8c) [4t ie obvieus that the lover bound fer HOS) is Zeno] peg, Me leg g - BOS) = Leilgy - I ie) S

Ze Leavrd cae = bon bea ve ved toq.* “leq a - Hes) = tage 2 Pi day Pi 3 t \ 3 Shaignt line fi fom (yee Tf Logasatennie Tv. dy cunve 2 ine ie te evidert that tHe shraight > Fen the above rr aritaentc CURVE line ye te) abicegs Ves above te log yebrx exept ak we! - Thus he sheaight line gerene the “daagent bo He cuwe ok 121een : ok Stmilealy bg -\ he g enn 2I-% > det 2i-% —- © me . 36 i ee ™ ari” 4 ha Pi 4 then dune Z fa eld ceed, We get ear dn op, = Zr b-s] Similaaly binaag Sounce Wd es by lg, ¢ manSee ¥ Aw tine S Pi bye, 2 leg & 2 ja ier Fron eg” C1 4 RWS te chong ° 7 leg a His) < teg y - #OS rat — © 20 The Eapality higr holds ge pheo P; aa 2 fy al) values 9 it 4» 2 @ ged then LUhes dhe Condon q eg?le) i* hocks the enhoyy betwee sraninuen , ger *@ * Wa) = Neg bt [renege gratd > O We Enbopy alteang a maxima value” when o!) the Source Byrnbol « beceme * equiprbable Prepekty of APDITLVITy we aplit the Aymbol sy indo 9 Bub symbole Suppose bid, Auch thot Sq ee San ae Pao Buch occuning ib th Probabitltes Py, Pye -- > ab Pay = Pay,t Past Pay ~--+ Pyn x 2 Z Py; log a Pa a + 5 Pa, le a fae aby; (Fem €970)' Hs) = n(s) + 4 porte yoy Binks . Pag A Py fa all 5 J. ies) 2 HGs) be parkkening 4 Apel decrease the Enbopy - into Sab- tye conn © Sovrce epescsency Source Effreiengy is denoted by 1 red 402 Bes) (s) maw ae “source Redundan Redundaney te denoted by Ra. s St oe qe y e -i-¥" s Nowe > MX Ra can alse be exprused on ch 4. AVERAGE INFORMATION CoNTENT Of SYMBOLS ene eae In Leon4 DEPENDENT SEQUENCES shabishaly The fourcer ane Laid to be producing indapendenk Keaencet | iy the oceumence 4, intemal _ doe nel aymbe ls a panticulan Ayenbet using a syd altin the pobabikly 9 sccumnces 9 ; \ dung any obben fpabel ey ee na all prachkeal feunee emite Aapences F symbole dak ore Braksbielly dupenslent lie Sources have memary )H The Seqpencer g letters Mumenals anel Apectal characters however ane not compktely mandem To general hey fem Bentences and have Alotistcal shuchune This Btadslreal dependence * Bhuchie seduces the amount gq infetmation Coming ouk gq such 2 Aounce compared te crew 4 infra ben coming ot g e a" emilting the fame et q Symbols i indapenclen Beqpences . % ence He need + develop a Alalsbed enodel dh sources emailing sppnele in d@pendent Acguenea and then we should ose this model ce define dle Eohopy LS) and the ifeination nok d9 Se Mark - OFF STATISTICAL MopeLl For INFoRMATLON Source ioe A Atbume the Discrete Tnjnmation Sounce emife & Aymbel ence evey Th Seeonde (sampling dusabion) « Since dhe founce is diterehe, 1 colt 9 init number Q Ayenbele hich is ar Cankain probabilities .HA physical system a Bhatigheal model q+ systin that produces such a Sequence % symbole gowwed by a seb 4 pobabilikes ig Knonm ac Slochaske 2) Random” proce - —— — # The discrete gource can be freaked as a srancleen voduce a proces . Conversely oy Randen proces sthot Pp’ aisciete, Aeqpence g Ayenbele chosen dom 4 hoite et moy be Considered a4 4 cliccrete SOC - ule can Atabiatenlly node! the Byrnhel Sequences emitted by the discrete source by claw g randow Prcenes Known as Diserebe stokmary Manegf Pranes | Marx- off Mapel LIAR Aaheheal model Yo discrete ae Mann-off model is a # deat: Sovtce groduces fegunce q Symbele which eae dupe ‘a’ poi ble shakes 42,-- 7 \. The ounce if tn one go 0 ak dhe begining each Ayrabel jntenval - The Source, Changes thake ome clung each Ayrbol snkenved Lo Lboj. The probability gq dhie transition i4 Puy | uhich dapends ony an che inthal sbake Cand te cf flake 5 uk dae not dependl on the bes : : bel intowele. The dang omy ge preceding yo eg 2 hrantilion pobaittkhe Py (Sena Tl Ae” ) rrenain coitant at che pran progaates in tme -th emits As the Source changes state prom V5 a Bymb - The ponticulas Aymbel emo emitted dipencde ™ the “nikal abobe 7 and he bvrontition fo43.- the alphabel = Sy be the Bymbele g B- Leb Si Sa - ard kee os Xe --- bem Beyisnee 4 Aanden vaaiable, Where Xe represents the 1% syeabel dhe Aoune ‘im A Begpence emitted y Then the probability heb the uth Ayrabel emitted a dipend on the prevrous Ayrobole 1 ke, a emitted hy emitted by the Aounetr that ie Sy + vik the conditional prbebi ely the fours P (%* = Sy\x te, --- au-1) he om ue him ok the she re A. The Injluence F athe stake 4 the 44 vet Aymbed sateval « freuen ayrbel ve aeperented by beginning Q the ply gq escamees Asi ng the eh Aye snbiwal Apede only om she slate q the sytlen ak he beginning 4 dhe synb sabeval P Can? S4| Aken When Se is a disceuke the Ahk 9 she Apem ak te beginning