1 One-hot representations of words

• Motivation: 将词汇用向量来表示
• Problem

2 Distributed Representations of words

• Motivation: You shall know a word by the company it keeps - Firth, J. R. 1957:11
• Solution: co-occurence matrix
  • A co-occurrence matrix is a terms × terms matrix which captures the number of times a term appears in the context of another term
  • The context is defined as a window of k words around the terms
• Problem
  • Stop words (a, the, for, etc.) are very frequent —> these counts will be very high
    • Solution 1: Ignore very frequent words
    • Solution 2: Use a threshold t
    • Solution 3： Use PMI

• Very high dimensional (|V|)
• Very sparse
• Grows with the size of the vocabulary
  • Solution: dimensionality reduction (SVD)

    3 SVD for learning word representation

    

• ， SVD gives the best rank-k approximation of the original data(X)
• SVD theorem tells us that u1,v1 and σ1 store the most information in X
• Each subsequent term (σ2u2v2T, σ3u3v3T, : : : ) stores less and less important information
• The ij-th entry of thus (roughly) captures the cosine similarity between wordi; wordj
• Notice that the dot product between the rows of the the matrix is the same as the dot product between the rows of

%5Cleft(U%20%5CSigma%20V%5E%7BT%7D%5Cright)%5E%7BT%7D%20%5C%5C%0A%26amp%3B%3D%5Cleft(U%20%5CSigma%20V%5E%7BT%7D%5Cright)%5Cleft(V%20%5CSigma%20U%5E%7BT%7D%5Cright)%20%5C%5C%0A%26amp%3B%3DU%20%5CSigma%20%5CSigma%5E%7BT%7D%20U%5E%7BT%7D%20%5Cquad%5Cleft(%5Cbecause%20V%5E%7BT%7D%20V%3DI%5Cright)%20%5C%5C%0A%26amp%3B%3DU%20%5CSigma(U%20%5CSigma)%5E%7BT%7D%3DW%7B%5Ctext%20%7Bword%20%7D%7D%20W%7B%5Ctext%20%7Bword%20%7D%7D%5E%7BT%7D%0A%5Cend%7Baligned%7D%3C%2Ftitle%3E%0A%3Cdefs%20aria-hidden%3D%22true%22%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMATHI-58%22%20d%3D%22M42%200H40Q26%200%2026%2011Q26%2015%2029%2027Q33%2041%2036%2043T55%2046Q141%2049%20190%2098Q200%20108%20306%20224T411%20342Q302%20620%20297%20625Q288%20636%20234%20637H206Q200%20643%20200%20645T202%20664Q206%20677%20212%20683H226Q260%20681%20347%20681Q380%20681%20408%20681T453%20682T473%20682Q490%20682%20490%20671Q490%20670%20488%20658Q484%20643%20481%20640T465%20637Q434%20634%20411%20620L488%20426L541%20485Q646%20598%20646%20610Q646%20628%20622%20635Q617%20635%20609%20637Q594%20637%20594%20648Q594%20650%20596%20664Q600%20677%20606%20683H618Q619%20683%20643%20683T697%20681T738%20680Q828%20680%20837%20683H845Q852%20676%20852%20672Q850%20647%20840%20637H824Q790%20636%20763%20628T722%20611T698%20593L687%20584Q687%20585%20592%20480L505%20384Q505%20383%20536%20304T601%20142T638%2056Q648%2047%20699%2046Q734%2046%20734%2037Q734%2035%20732%2023Q728%207%20725%204T711%201Q708%201%20678%201T589%202Q528%202%20496%202T461%201Q444%201%20444%2010Q444%2011%20446%2025Q448%2035%20450%2039T455%2044T464%2046T480%2047T506%2054Q523%2062%20523%2064Q522%2064%20476%20181L429%20299Q241%2095%20236%2084Q232%2076%20232%2072Q232%2053%20261%2047Q262%2047%20267%2047T273%2046Q276%2046%20277%2046T280%2045T283%2042T284%2035Q284%2026%20282%2019Q279%206%20276%204T261%201Q258%201%20243%201T201%202T142%202Q64%202%2042%200Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMAIN-5E%22%20d%3D%22M112%20560L249%20694L257%20686Q387%20562%20387%20560L361%20531Q359%20532%20303%20581L250%20627L195%20580Q182%20569%20169%20557T148%20538L140%20532Q138%20530%20125%20546L112%20560Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMATHI-54%22%20d%3D%22M40%20437Q21%20437%2021%20445Q21%20450%2037%20501T71%20602L88%20651Q93%20669%20101%20677H569H659Q691%20677%20697%20676T704%20667Q704%20661%20687%20553T668%20444Q668%20437%20649%20437Q640%20437%20637%20437T631%20442L629%20445Q629%20451%20635%20490T641%20551Q641%20586%20628%20604T573%20629Q568%20630%20515%20631Q469%20631%20457%20630T439%20622Q438%20621%20368%20343T298%2060Q298%2048%20386%2046Q418%2046%20427%2045T436%2036Q436%2031%20433%2022Q429%204%20424%201L422%200Q419%200%20415%200Q410%200%20363%201T228%202Q99%202%2064%200H49Q43%206%2043%209T45%2027Q49%2040%2055%2046H83H94Q174%2046%20189%2055Q190%2056%20191%2056Q196%2059%20201%2076T241%20233Q258%20301%20269%20344Q339%20619%20339%20625Q339%20630%20310%20630H279Q212%20630%20191%20624Q146%20614%20121%20583T67%20467Q60%20445%2057%20441T43%20437H40Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMAIN-3D%22%20d%3D%22M56%20347Q56%20360%2070%20367H707Q722%20359%20722%20347Q722%20336%20708%20328L390%20327H72Q56%20332%2056%20347ZM56%20153Q56%20168%2072%20173H708Q722%20163%20722%20153Q722%20140%20707%20133H70Q56%20140%2056%20153Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMAIN-28%22%20d%3D%22M94%20250Q94%20319%20104%20381T127%20488T164%20576T202%20643T244%20695T277%20729T302%20750H315H319Q333%20750%20333%20741Q333%20738%20316%20720T275%20667T226%20581T184%20443T167%20250T184%2058T225%20-81T274%20-167T316%20-220T333%20-241Q333%20-250%20318%20-250H315H302L274%20-226Q180%20-141%20137%20-14T94%20250Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMATHI-55%22%20d%3D%22M107%20637Q73%20637%2071%20641Q70%20643%2070%20649Q70%20673%2081%20682Q83%20683%2098%20683Q139%20681%20234%20681Q268%20681%20297%20681T342%20682T362%20682Q378%20682%20378%20672Q378%20670%20376%20658Q371%20641%20366%20638H364Q362%20638%20359%20638T352%20638T343%20637T334%20637Q295%20636%20284%20634T266%20623Q265%20621%20238%20518T184%20302T154%20169Q152%20155%20152%20140Q152%2086%20183%2055T269%2024Q336%2024%20403%2069T501%20205L552%20406Q599%20598%20599%20606Q599%20633%20535%20637Q511%20637%20511%20648Q511%20650%20513%20660Q517%20676%20519%20679T529%20683Q532%20683%20561%20682T645%20680Q696%20680%20723%20681T752%20682Q767%20682%20767%20672Q767%20650%20759%20642Q756%20637%20737%20637Q666%20633%20648%20597Q646%20592%20598%20404Q557%20235%20548%20205Q515%20105%20433%2042T263%20-22Q171%20-22%20116%2034T60%20167V183Q60%20201%20115%20421Q164%20622%20164%20628Q164%20635%20107%20637Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMAIN-3A3%22%20d%3D%22M666%20247Q664%20244%20652%20126T638%204V0H351Q131%200%2095%200T57%205V6Q54%2012%2057%2017L73%2036Q89%2054%20121%2090T182%20159L305%20299L56%20644L55%20658Q55%20677%2060%20681Q63%20683%20351%20683H638V679Q640%20674%20652%20564T666%20447V443H626V447Q618%20505%20604%20543T559%20605Q529%20626%20478%20631T333%20637H294H189L293%20494Q314%20465%20345%20422Q400%20346%20400%20340Q400%20338%20399%20337L154%2057Q407%2057%20428%2058Q476%2060%20508%2068T551%2083T575%20103Q595%20125%20608%20162T624%20225L626%20251H666V247Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMATHI-56%22%20d%3D%22M52%20648Q52%20670%2065%20683H76Q118%20680%20181%20680Q299%20680%20320%20683H330Q336%20677%20336%20674T334%20656Q329%20641%20325%20637H304Q282%20635%20274%20635Q245%20630%20242%20620Q242%20618%20271%20369T301%20118L374%20235Q447%20352%20520%20471T595%20594Q599%20601%20599%20609Q599%20633%20555%20637Q537%20637%20537%20648Q537%20649%20539%20661Q542%20675%20545%20679T558%20683Q560%20683%20570%20683T604%20682T668%20681Q737%20681%20755%20683H762Q769%20676%20769%20672Q769%20655%20760%20640Q757%20637%20743%20637Q730%20636%20719%20635T698%20630T682%20623T670%20615T660%20608T652%20599T645%20592L452%20282Q272%20-9%20266%20-16Q263%20-18%20259%20-21L241%20-22H234Q216%20-22%20216%20-15Q213%20-9%20177%20305Q139%20623%20138%20626Q133%20637%2076%20637H59Q52%20642%2052%20648Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMAIN-29%22%20d%3D%22M60%20749L64%20750Q69%20750%2074%20750H86L114%20726Q208%20641%20251%20514T294%20250Q294%20182%20284%20119T261%2012T224%20-76T186%20-143T145%20-194T113%20-227T90%20-246Q87%20-249%2086%20-250H74Q66%20-250%2063%20-250T58%20-247T55%20-238Q56%20-237%2066%20-225Q221%20-64%20221%20250T66%20725Q56%20737%2055%20738Q55%20746%2060%20749Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJSZ1-28%22%20d%3D%22M152%20251Q152%20646%20388%20850H416Q422%20844%20422%20841Q422%20837%20403%20816T357%20753T302%20649T255%20482T236%20250Q236%20124%20255%2019T301%20-147T356%20-251T403%20-315T422%20-340Q422%20-343%20416%20-349H388Q359%20-325%20332%20-296T271%20-213T212%20-97T170%2056T152%20251Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJSZ1-29%22%20d%3D%22M305%20251Q305%20-145%2069%20-349H56Q43%20-349%2039%20-347T35%20-338Q37%20-333%2060%20-307T108%20-239T160%20-136T204%2027T221%20250T204%20473T160%20636T108%20740T60%20807T35%20839Q35%20850%2050%20850H56H69Q197%20743%20256%20566Q305%20425%20305%20251Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJAMS-2235%22%20d%3D%22M23%20411Q23%20437%2041%20454T84%20471Q108%20471%20125%20454T143%20411T126%20368T83%20351Q57%20351%2040%20368T23%20411ZM523%20411Q523%20437%20541%20454T584%20471Q608%20471%20625%20454T643%20411T626%20368T583%20351Q557%20351%20540%20368T523%20411ZM274%20-22Q274%204%20291%2021T334%2038Q356%2038%20374%2022T392%20-22T375%20-65T333%20-82Q307%20-82%20291%20-65T274%20-22Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMATHI-49%22%20d%3D%22M43%201Q26%201%2026%2010Q26%2012%2029%2024Q34%2043%2039%2045Q42%2046%2054%2046H60Q120%2046%20136%2053Q137%2053%20138%2054Q143%2056%20149%2077T198%20273Q210%20318%20216%20344Q286%20624%20286%20626Q284%20630%20284%20631Q274%20637%20213%20637H193Q184%20643%20189%20662Q193%20677%20195%20680T209%20683H213Q285%20681%20359%20681Q481%20681%20487%20683H497Q504%20676%20504%20672T501%20655T494%20639Q491%20637%20471%20637Q440%20637%20407%20634Q393%20631%20388%20623Q381%20609%20337%20432Q326%20385%20315%20341Q245%2065%20245%2059Q245%2052%20255%2050T307%2046H339Q345%2038%20345%2037T342%2019Q338%206%20332%200H316Q279%202%20179%202Q143%202%20113%202T65%202T43%201Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMATHI-57%22%20d%3D%22M436%20683Q450%20683%20486%20682T553%20680Q604%20680%20638%20681T677%20682Q695%20682%20695%20674Q695%20670%20692%20659Q687%20641%20683%20639T661%20637Q636%20636%20621%20632T600%20624T597%20615Q597%20603%20613%20377T629%20138L631%20141Q633%20144%20637%20151T649%20170T666%20200T690%20241T720%20295T759%20362Q863%20546%20877%20572T892%20604Q892%20619%20873%20628T831%20637Q817%20637%20817%20647Q817%20650%20819%20660Q823%20676%20825%20679T839%20682Q842%20682%20856%20682T895%20682T949%20681Q1015%20681%201034%20683Q1048%20683%201048%20672Q1048%20666%201045%20655T1038%20640T1028%20637Q1006%20637%20988%20631T958%20617T939%20600T927%20584L923%20578L754%20282Q586%20-14%20585%20-15Q579%20-22%20561%20-22Q546%20-22%20542%20-17Q539%20-14%20523%20229T506%20480L494%20462Q472%20425%20366%20239Q222%20-13%20220%20-15T215%20-19Q210%20-22%20197%20-22Q178%20-22%20176%20-15Q176%20-12%20154%20304T131%20622Q129%20631%20121%20633T82%20637H58Q51%20644%2051%20648Q52%20671%2064%20683H76Q118%20680%20176%20680Q301%20680%20313%20683H323Q329%20677%20329%20674T327%20656Q322%20641%20318%20637H297Q236%20634%20232%20620Q262%20160%20266%20136L501%20550L499%20587Q496%20629%20489%20632Q483%20636%20447%20637Q428%20637%20422%20639T416%20648Q416%20650%20418%20660Q419%20664%20420%20669T421%20676T424%20680T428%20682T436%20683Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMAIN-77%22%20d%3D%22M90%20368Q84%20378%2076%20380T40%20385H18V431H24L43%20430Q62%20430%2084%20429T116%20428Q206%20428%20221%20431H229V385H215Q177%20383%20177%20368Q177%20367%20221%20239L265%20113L339%20328L333%20345Q323%20374%20316%20379Q308%20384%20278%20385H258V431H264Q270%20428%20348%20428Q439%20428%20454%20431H461V385H452Q404%20385%20404%20369Q404%20366%20418%20324T449%20234T481%20143L496%20100L537%20219Q579%20341%20579%20347Q579%20363%20564%20373T530%20385H522V431H529Q541%20428%20624%20428Q692%20428%20698%20431H703V385H697Q696%20385%20691%20385T682%20384Q635%20377%20619%20334L559%20161Q546%20124%20528%2071Q508%2012%20503%201T487%20-11H479Q460%20-11%20456%20-4Q455%20-3%20407%20133L361%20267Q359%20263%20266%20-4Q261%20-11%20243%20-11H238Q225%20-11%20220%20-3L90%20368Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMAIN-6F%22%20d%3D%22M28%20214Q28%20309%2093%20378T250%20448Q340%20448%20405%20380T471%20215Q471%20120%20407%2055T250%20-10Q153%20-10%2091%2057T28%20214ZM250%2030Q372%2030%20372%20193V225V250Q372%20272%20371%20288T364%20326T348%20362T317%20390T268%20410Q263%20411%20252%20411Q222%20411%20195%20399Q152%20377%20139%20338T126%20246V226Q126%20130%20145%2091Q177%2030%20250%2030Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMAIN-72%22%20d%3D%22M36%2046H50Q89%2046%2097%2060V68Q97%2077%2097%2091T98%20122T98%20161T98%20203Q98%20234%2098%20269T98%20328L97%20351Q94%20370%2083%20376T38%20385H20V408Q20%20431%2022%20431L32%20432Q42%20433%2060%20434T96%20436Q112%20437%20131%20438T160%20441T171%20442H174V373Q213%20441%20271%20441H277Q322%20441%20343%20419T364%20373Q364%20352%20351%20337T313%20322Q288%20322%20276%20338T263%20372Q263%20381%20265%20388T270%20400T273%20405Q271%20407%20250%20401Q234%20393%20226%20386Q179%20341%20179%20207V154Q179%20141%20179%20127T179%20101T180%2081T180%2066V61Q181%2059%20183%2057T188%2054T193%2051T200%2049T207%2048T216%2047T225%2047T235%2046T245%2046H276V0H267Q249%203%20140%203Q37%203%2028%200H20V46H36Z%22%3E%3C%2Fpath%3E%0A%3Cpath%20stroke-width%3D%221%22%20id%3D%22E2-MJMAIN-64%22%20d%3D%22M376%20495Q376%20511%20376%20535T377%20568Q377%20613%20367%20624T316%20637H298V660Q298%20683%20300%20683L310%20684Q320%20685%20339%20686T376%20688Q393%20689%20413%20690T443%20693T454%20694H457V390Q457%2084%20458%2081Q461%2061%20472%2055T517%2046H535V0Q533%200%20459%20-5T380%20-11H373V44L365%2037Q307%20-11%20235%20-11Q158%20-11%2096%2050T34%20215Q34%20315%2097%20378T244%20442Q319%20442%20376%20393V495ZM373%20342Q328%20405%20260%20405Q211%20405%20173%20369Q146%20341%20139%20305T131%20211Q131%20155%20138%20120T173%2059Q203%2026%20251%2026Q322%2026%20373%20103V342Z%22%3E%3C%2Fpath%3E%0A%3C%2Fdefs%3E%0A%3Cg%20stroke%3D%22currentColor%22%20fill%3D%22currentColor%22%20stroke-width%3D%220%22%20transform%3D%22matrix(1%200%200%20-1%200%200)%22%20aria-hidden%3D%22true%22%3E%0A%3Cg%20transform%3D%22translate(167%2C0)%22%3E%0A%3Cg%20transform%3D%22translate(-11%2C0)%22%3E%0A%3Cg%20transform%3D%22translate(0%2C2150)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-58%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-5E%22%20x%3D%22274%22%20y%3D%22261%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(852%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-58%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-5E%22%20x%3D%22274%22%20y%3D%22261%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%221205%22%20y%3D%22966%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3Cg%20transform%3D%22translate(2292%2C0)%22%3E%0A%3Cg%20transform%3D%22translate(0%2C2150)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3D%22%20x%3D%22277%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(1334%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJSZ1-28%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-55%22%20x%3D%22458%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3A3%22%20x%3D%221226%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(1948%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-56%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%221167%22%20y%3D%22583%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJSZ1-29%22%20x%3D%223372%22%20y%3D%22-1%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%3Cg%20transform%3D%22translate(5331%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJSZ1-28%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-55%22%20x%3D%22458%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3A3%22%20x%3D%221226%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(1948%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-56%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%221167%22%20y%3D%22583%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJSZ1-29%22%20x%3D%223372%22%20y%3D%22-1%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%225417%22%20y%3D%22875%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3Cg%20transform%3D%22translate(0%2C607)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3D%22%20x%3D%22277%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(1334%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJSZ1-28%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-55%22%20x%3D%22458%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3A3%22%20x%3D%221226%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(1948%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-56%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%221167%22%20y%3D%22583%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJSZ1-29%22%20x%3D%223372%22%20y%3D%22-1%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%3Cg%20transform%3D%22translate(5331%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJSZ1-28%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-56%22%20x%3D%22458%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3A3%22%20x%3D%221228%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(1950%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-55%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%221121%22%20y%3D%22583%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJSZ1-29%22%20x%3D%223341%22%20y%3D%22-1%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3Cg%20transform%3D%22translate(0%2C-935)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3D%22%20x%3D%22277%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-55%22%20x%3D%221334%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3A3%22%20x%3D%222101%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(2824%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3A3%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%221021%22%20y%3D%22583%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%3Cg%20transform%3D%22translate(4144%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-55%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%221121%22%20y%3D%22583%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%3Cg%20transform%3D%22translate(6702%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJSZ1-28%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJAMS-2235%22%20x%3D%22458%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(1403%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-56%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%221167%22%20y%3D%22583%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-56%22%20x%3D%222827%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3D%22%20x%3D%223874%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-49%22%20x%3D%224930%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJSZ1-29%22%20x%3D%225435%22%20y%3D%22-1%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3Cg%20transform%3D%22translate(0%2C-2478)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3D%22%20x%3D%22277%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-55%22%20x%3D%221334%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3A3%22%20x%3D%222101%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-28%22%20x%3D%222824%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-55%22%20x%3D%223213%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3A3%22%20x%3D%223981%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(4703%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-29%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%22550%22%20y%3D%22583%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMAIN-3D%22%20x%3D%225968%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(7025%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-57%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(944%2C-150)%22%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMAIN-77%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMAIN-6F%22%20x%3D%22722%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMAIN-72%22%20x%3D%221223%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMAIN-64%22%20x%3D%221615%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3Cg%20transform%3D%22translate(9855%2C0)%22%3E%0A%20%3Cuse%20xlink%3Ahref%3D%22%23E2-MJMATHI-57%22%20x%3D%220%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMATHI-54%22%20x%3D%221526%22%20y%3D%22488%22%3E%3C%2Fuse%3E%0A%3Cg%20transform%3D%22translate(944%2C-328)%22%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMAIN-77%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMAIN-6F%22%20x%3D%22722%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMAIN-72%22%20x%3D%221223%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%20%3Cuse%20transform%3D%22scale(0.707)%22%20xlink%3Ahref%3D%22%23E2-MJMAIN-64%22%20x%3D%221615%22%20y%3D%220%22%3E%3C%2Fuse%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3C%2Fsvg%3E#card=math&code=%5Cbegin%7Baligned%7D%0A%5Chat%7BX%7D%20%5Chat%7BX%7D%5E%7BT%7D%20%26%3D%5Cleft%28U%20%5CSigma%20V%5E%7BT%7D%5Cright%29%5Cleft%28U%20%5CSigma%20V%5E%7BT%7D%5Cright%29%5E%7BT%7D%20%5C%5C%0A%26%3D%5Cleft%28U%20%5CSigma%20V%5E%7BT%7D%5Cright%29%5Cleft%28V%20%5CSigma%20U%5E%7BT%7D%5Cright%29%20%5C%5C%0A%26%3DU%20%5CSigma%20%5CSigma%5E%7BT%7D%20U%5E%7BT%7D%20%5Cquad%5Cleft%28%5Cbecause%20V%5E%7BT%7D%20V%3DI%5Cright%29%20%5C%5C%0A%26%3DU%20%5CSigma%28U%20%5CSigma%29%5E%7BT%7D%3DW%7B%5Ctext%20%7Bword%20%7D%7D%20W%7B%5Ctext%20%7Bword%20%7D%7D%5E%7BT%7D%0A%5Cend%7Baligned%7D&id=QO9eY)
conventionnally,
and

4 CBOW（it predicts an output word give a bag of context words

• the i-th column of W**context** as the representation of context i
• the i-th column of W**word** as the representation of word i
• output function: softmax
• loss function: cross entropy
  
• how to train: backpropagation

update rule:

• Problem: the softmax function at the output is computationally very expensive.

  5 Skip-gram（it predicts context words given an input word

  

  6 Negative Sampling

  —>

—>

7 分层softmax

哈夫曼树： 具有最小带权路径长度的二叉树（也称最优树） 构造哈夫曼树的原则：权值越大（小）的叶节点越靠近（远离）根节点 哈夫曼编码：左子树为0，右子树为1

—>—>