Commits

cnu  committed 1492c58

Fixed decimal number problem in tokenizing

  • Participants
  • Parent commits a43f000

Comments (0)

Files changed (1)

File MontyTokenizer.py

         dirname=0
         contents_arr=re.compile('([?!]+|[.][.]+)$')
         more1=re.compile('([.])$')
-
+        num_re = re.compile(r'(\d+\.\d+\.?)+')
+        
         while dirname<len(input):
+            if num_re.search(input[dirname]):
+                # input contains "number.number.number" eg: "It is 2.8 inch or 127.0.0.1"
+                dirname += 1
+                continue
+            
             info_cleaned=contents_arr.search(input[dirname])
 
             if info_cleaned: