• John Koleszar's avatar
    Reduce size of tokenizer tables · 147b125b
    John Koleszar authored
    This patch reduces the size of the global tables maintained by the
    tokenizer to 16k from 80k-96k. See issue #177.
    
    Change-Id: If0275d5f28389af11ac83c5d929d1157cde90fbe
    147b125b
tokenize.c 13.8 KB