autohotkey.py version6

Anonymous avatarAnonymous created an issue
  1. test output at: http://www.autohotkey.net/~tinku99/lexer/test.html
  1. more test output at: http://www.autohotkey.net/~tinku99/lexer/test2.html
class AutoHotkeyLexer(RegexLexer):
    """
    For AutoHotkey source code with preprocessor directives.
    """
    name = 'autohotkey'
    aliases = ['ahk']
    filenames = ['*.ahk', '*.ahkl']
    flags = re.IGNORECASE
    #: optional Comment or Whitespace
    _ws = r'(?:\\s|;.*?\\n|/[*].*?[*]/)+'

    tokens = {
        'whitespace': [
            (r'^\\s*#if\\s+0', Comment.Preproc, 'if0'),
            (r'^\\s*#', Comment.Preproc, 'macro'),
            (r'\\n', Text),
            (r'\\s+', Text),
             (r';(\\n|(.|\\n)*?[^\\\\]\\n)', Comment),
            (r'/(\\\\\\n)?[*](.|\\n)*?[*](\\\\\\n)?/', Comment),
        ],
        'statements': [
            (r'L?"', String, 'string'),
            (r"L?'(\\\\.|\\\\[0-7]{1,3}|\\\\x[a-fA-F0-9]{1,2}|[^\\\\\\'\\n])'", String.Char),
            (r'(\\d+\\.\\d*|\\.\\d+|\\d+)[eE][+-]?\\d+[lL]?', Number.Float),
            (r'(\\d+\\.\\d*|\\.\\d+|\\d+[fF])[fF]?', Number.Float),
            (r'0x[0-9a-fA-F]+[Ll]?', Number.Hex),
            (r'\\d+[Ll]?', Number.Integer),
            (r'[~!%^&*+=|?:<>/-]', Operator),
            (r'[()\\[\\],.]', Punctuation),
            (r'\\b(case)(.+?)(:)', bygroups(Keyword, using(this), Text)),
            (r'(auto|break|case|const|continue|default|do|else|enum|extern|'
             r'%.*%'
             r'RTrim|JoinLow|BelowNormal|Normal|AboveNormal|'
             r'High|Realtimeahk_id|ahk_pid|ahk_class|ahk_groupBetween|'
             r'Contains|In|Is|Integer|Float|'
             r'IntegerFast|FloatFast|Number|Digit|Xdigit|'
             r'Alpha|Upper|Lower|Alnum|Time|'
             r'DateNot|Or|AndAlwaysOnTop|Topmost|Top|'
             r'Bottom|Transparent|TransColor|Redraw|Region|'
             r'ID|IDLast|ProcessName|MinMax|ControlList|'
             r'Count|List|Capacity|StatusCD|Eject|'
             r'Lock|Unlock|Label|FileSystem|Label|'
             r'SetLabel|Serial|Type|Statusstatic|global|'
             r'local|ByRefSeconds|Minutes|Hours|DaysRead|'
             r'ParseLogoff|Close|Error|SingleTray|Add|'
             r'Rename|Check|UnCheck|ToggleCheck|Enable|'
             r'Disable|ToggleEnable|Default|NoDefault|Standard|'
             r'NoStandard|Color|Delete|DeleteAll|Icon|'
             r'NoIcon|Tip|Click|Show|MainWindow|'
             r'NoMainWindow|UseErrorLevelText|Picture|Pic|GroupBox|'
             r'Button|Checkbox|Radio|DropDownList|DDL|'
             r'ComboBox|ListBox|ListView|DateTime|MonthCal|'
             r'Slider|StatusBar|Tab|Tab2|TreeView|'
             r'UpDownIconSmall|Tile|Report|SortDesc|NoSort|'
             r'NoSortHdr|Grid|Hdr|AutoSize|Rangexm|'
             r'ym|ys|xs|xp|ypFont|'
             r'Resize|Owner|Submit|NoHide|Minimize|'
             r'Maximize|Restore|NoActivate|NA|Cancel|'
             r'Destroy|CenterMargin|MaxSize|MinSize|OwnDialogs|'
             r'GuiEscape|GuiClose|GuiSize|GuiContextMenu|GuiDropFilesTabStop|'
             r'Section|AltSubmit|Wrap|HScroll|VScroll|'
             r'Border|Top|Bottom|Buttons|Expand|'
             r'First|ImageList|Lines|WantCtrlA|WantF2|'
             r'Vis|VisFirst|Number|Uppercase|Lowercase|'
             r'Limit|Password|Multi|WantReturn|Group|'
             r'Background|bold|italic|strike|underline|'
             r'norm|BackgroundTrans|Theme|Caption|Delimiter|'
             r'MinimizeBox|MaximizeBox|SysMenu|ToolWindow|Flash|'
             r'Style|ExStyle|Check3|Checked|CheckedGray|'
             r'ReadOnly|Password|Hidden|Left|Right|'
             r'Center|NoTab|Section|Move|Focus|'
             r'Hide|Choose|ChooseString|Text|Pos|'
             r'Enabled|Disabled|Visible|LastFound|LastFoundExistAltTab|'
             r'ShiftAltTab|AltTabMenu|AltTabAndMenu|AltTabMenuDismissNoTimers|Interrupt|'
             r'Priority|WaitClose|Wait|Exist|Close{Blind}|'
             r'Mod|Pow|Exp|Sqrt|Log|'
             r'Ln|Round|Ceil|Floor|Abs|'
             r'Sin|Cos|Tan|ASin|ACos|'
             r'ATan|BitNot|BitAnd|BitOr|BitXOr|'
             r'BitShiftLeft|BitShiftRightYes|No|Ok|Cancel|'
             r'Abort|Retry|Ignore|TryAgainOn|Off|'
             r'for|goto|if|register|restricted|return|sizeof|static|struct|'
             r'switch|typedef|union|volatile|virtual|while)\\b', Keyword),
            (r'(global|static|local)\\b',
             Keyword.Type),
            (r'(_{0,2}inline|naked|restrict|thread|typename)\\b', Keyword.Reserved),
            (r'__(asm|int8|based|except|int16|stdcall|cdecl|fastcall|int32|'
             r'true|false|NULL|try|leave)\\b', Keyword.Reserved),
            (r'(true|false|NULL)\\b'
             r'BlockInput|Break|Click|ClipWait|Continue|'
             r'Control|ControlClick|ControlFocus|ControlGet|ControlGetFocus|'
             r'ControlGetPos|ControlGetText|ControlMove|ControlSend|ControlSendRaw|'
             r'ControlSetText|CoordMode|Critical|DetectHiddenText|DetectHiddenWindows|'
             r'Drive|DriveGet|DriveSpaceFree|Edit|Else|'
             r'EnvAdd|EnvDiv|EnvGet|EnvMult|EnvSet|'
             r'EnvSub|EnvUpdate|Exit|ExitApp|FileAppend|'
             r'FileCopy|FileCopyDir|FileCreateDir|FileCreateShortcut|FileDelete|'
             r'FileGetAttrib|FileGetShortcut|FileGetSize|FileGetTime|FileGetVersion|'
             r'FileInstall|FileMove|FileMoveDir|FileRead|FileReadLine|'
             r'FileRecycle|FileRecycleEmpty|FileRemoveDir|FileSelectFile|FileSelectFolder|'
             r'FileSetAttrib|FileSetTime|FormatTime|GetKeyState|Gosub|'
             r'Goto|GroupActivate|GroupAdd|GroupClose|GroupDeactivate|'
             r'Gui|GuiControl|GuiControlGet|Hotkey|If Else|'
             r'IfExist|IfGreater|IfGreaterOrEqual|IfInString|IfLess|'
             r'IfLessOrEqual|IfMsgBox|IfNotEqual|IfNotExist|IfNotInString|'
             r'IfWinActive|IfWinExist|IfWinNotActive|IfWinNotExist|ImageSearch|'
             r'IniDelete|IniRead|IniWrite|Input|InputBox|'
             r'KeyHistory|KeyWait|ListHotkeys|ListLines|ListVars|'
             r'Loop|Loop, FilePattern|Loop, Parse|Loop, Read|Loop, Reg|'
             r'Menu|MouseClick|MouseClickDrag|MouseGetPos|MouseMove|'
             r'MsgBox|OnExit|OutputDebug|Pause|PixelGetColor|OnMessage|'
             r'PixelSearch|PostMessage|Process|Progress|Random|'
             r'RegExMatch|RegExReplace|IsFunc|DllCall|'
             r'RegDelete|RegRead|RegWrite|Reload|Repeat|'
             r'Return|Run|RunAs|RunWait|Send|'
             r'SendEvent|SendInput|SendMessage|SendMode|SendPlay|'
             r'SendRaw|SetBatchLines|SetCapslockState|SetControlDelay|SetDefaultMouseSpeed|'
             r'SetEnv|SetFormat|SetKeyDelay|SetMouseDelay|SetNumlockState|'
             r'SetScrollLockState|SetStoreCapslockMode|SetTimer|SetTitleMatchMode|SetWinDelay|'
             r'SetWorkingDir|Shutdown|Sleep|Sort|SoundBeep|'
             r'SoundGet|SoundGetWaveVolume|SoundPlay|SoundSet|SoundSetWaveVolume|'
             r'SplashImage|SplashTextOff|SplashTextOn|SplitPath|StatusBarGetText|'
             r'StatusBarWait|StringCaseSense|StringGetPos|StringLeft|StringLen|'
             r'StringLower|StringMid|StringReplace|StringRight|StringSplit|'
             r'StringTrimLeft|StringTrimRight|StringUpper|Suspend|SysGet|'
             r'Thread|ToolTip|Transform|TrayTip|URLDownloadToFile|'
             r'While|WinActivate|WinActivateBottom|WinClose|WinGet|'
             r'WinGetActiveStats|WinGetActiveTitle|WinGetClass|WinGetPos|WinGetText|'
             r'WinGetTitle|WinHide|WinKill|WinMaximize|WinMenuSelectItem|'
             r'WinMinimize|WinMinimizeAll|WinMinimizeAllUndo|WinMove|WinRestore|'
             r'WinSet|WinSetTitle|WinShow|WinWait|WinWaitActive|'
             r'WinWaitClose|WinWaitNotActive', Name.Builtin),
            ('[a-zA-Z_][a-zA-Z0-9_]*:(?!:)', Name.Label),
            (r'[a-zA-Z_][a-zA-Z0-9_]*::(?!:)', Name.Label),
            (r'::[a-zA-Z_][a-zA-Z0-9_]*::(?!:)', Name.Label),
            ('[a-zA-Z_][a-zA-Z0-9_]*', Name),
        ],
        'root': [
            include('whitespace'),
            # functions
            (r'((?:[a-zA-Z0-9_*\\s])+?(?:\\s|[*]))'    # return arguments
             r'([a-zA-Z_][a-zA-Z0-9_]*)'             # method name
             r'(\\s*\\([^;]*?\\))'                      # signature
             r'(' + _ws + r')({)',
             bygroups(using(this), Name.Function, using(this), using(this),
                      Punctuation),
             'function'),
            # function declarations
            (r'((?:[a-zA-Z0-9_*\\s])+?(?:\\s|[*]))'    # return arguments
             r'([a-zA-Z_][a-zA-Z0-9_]*)'             # method name
             r'(\\s*\\([^;]*?\\))'                      # signature
             r'(' + _ws + r')(;)',
             bygroups(using(this), Name.Function, using(this), using(this),
                      Punctuation)),
            ('', Text, 'statement'),
        ],
        'statement' : [
            include('whitespace'),
            include('statements'),
            ('[{}]', Punctuation),
            (';', Punctuation, '#pop'),
        ],
        'function': [
            include('whitespace'),
            include('statements'),
            ('{', Punctuation, '#push'),
            ('}', Punctuation, '#pop'),
        ],
        'string': [
            (r'"', String, '#pop'),
            (r'\\\\([\\\\abfnrtv"\\']|x[a-fA-F0-9]{2,4}|[0-7]{1,3})', String.Escape),
            (r'[^`"\\n]+', String), # all other characters
            (r'\\\\', String), # stray backslash
        ],
        'macro': [
            (r'[^/\\n]+', Comment.Preproc),
            (r'/[*](.|\\n)*?[*]/', Comment),
            (r'//.*?\\n', Comment, '#pop'),
            (r'/', Comment.Preproc),
            (r'(?<=\\\\)\\n', Comment.Preproc),
            (r'\\n', Comment.Preproc, '#pop'),
        ],
        'if0': [
            (r'^\\s*#if.*?(?<!\\\\)\\n', Comment, '#push'),
            (r'^\\s*#el(?:se|if).*\\n', Comment.Preproc, '#pop'),
            (r'^\\s*#endif.*?(?<!\\\\)\\n', Comment, '#pop'),
            (r'.*?\\n', Comment),
        ]
    }

    stdlib_types = ['global', 'local', 'static']

    def __init__(self, **options):
        self.stdlibhighlighting = get_bool_opt(options,
                'stdlibhighlighting', True)
        self.c99highlighting = get_bool_opt(options,
                'c99highlighting', True)
        RegexLexer.__init__(self, **options)

    def get_tokens_unprocessed(self, text):
        for index, token, value in \\
            RegexLexer.get_tokens_unprocessed(self, text):
            if token is Name:
                if self.stdlibhighlighting and value in self.stdlib_types:
                    token = Keyword.Type
            yield index, token, value

Reported by guest

Comments (16)

  1. Anonymous

    Please let me know if there is something else I can do for this autohotkey lexer I submitted. Thanks.

  2. thatch

    Note for next time: avoid putting your code in the description itself, attach a patch and include a number like "ver2" so we know which is newer.

    First, don't take the lack of response as a desire to not include this. Autohotkey does seem like a lexer which we'd like to have in Pygments. A few problems with your lexer keep it from being accepted, however, and I'd like your help to fix them.

    1. Lines are too long, it doesn't fit our Python style guidelines (PEP-8). Please reduce lengths to <80 characters, and use 'make check' to verify. 2. Your own example files should pass with no red (error) characters. 3. If `;` introduces a comment, it shouldn't end a statement. I think this comes from basing on the C lexer, maybe go through and rip out parts that aren't needed. 4. `%ab%` parses incorrectly, because `%` matches as an operator. Where `%.*%` is defined, I don't think that's supposed to be greedy or combined with the following pattern. 5. `_{0,2}inline` is strange, can you check that `inline`, `_inline`, and `inline` are all supposed to match, and that the underscore shouldn't apply to the following words? 6. Decide whether `true` and `false` are builtins or reserved. 7. `(true|false|NULL)
    bBlockInput` can never match. 8. Remove the overridden functions for dealing with `stdlibhighlighting` altogether, that's not necessary since those words are matched earlier, and c99highlighting isn't used either. 9. We need to make an FAQ entry about these, but your pattern for Name.Builtin needs to end with `
    b` for `ControlClick` to match properly. 10. Slightly more complex, `(Loop|Loop, Blah)
    b` needs the shorter match put at the end so the string `Loop, Blah` doesn't take the first match, `Loop
    b`. This is more complex because there's punctuation in the words.

    Once you get those taken care of, attach a unified diff and I'll give it another review. If you have questions about how to accomplish them, feel free to ask specific questions on the mailing list (easy questions generally get answered quickly).

  3. Anonymous

    Replying to [comment:3 thatch]:

    Autohotkey does seem like a lexer which we'd like to have in Pygments.

    I am glad to hear that.

    A few problems with your lexer keep it from being accepted, however, and I'd like your help to fix them. Once you get those taken care of, attach a unified diff and I'll give it another review.

    I think I have fixed all the issues you listed. See attachment ver2. You can view the samle output [http://www.autohotkey.net/~tinku99/lexer/test.html here] for [http://www.autohotkey.net/~tinku99/lexer/test.ahk sample_script]

    Thanks.

  4. thatch

    I've attached v3, mostly minor cleanups (whitespace, added a docstring, etc). Please also try with the following snippet, which I believe to be valid ahk.

    indexdir = %A_ScriptDir%\\%dir%
    FileCreateDir, % indexdir
    fileindex = %indexdir%\\_files
    
    • The variables like `%dir%` aren't working
    • Backslashes aren't working
    • In your test.ahk there are two errors on the line where `FileSelectFile` is used

    And this one, although it appears to use a module of some sort, also has a funky backslash.

    ; viper
    !i::     
    viper("iviper")   ; stdlib
    Return
    
    #If WinExist("iviper")
    i::viper_off() 
    #If
    
    ;; keybindings
    #If WinExist("iviper") and WinActive("ahk_class Emacs")
    i::viper_off()
    a::send, ^a
    m::send, ^{space}  ;; set mark
    e::send, ^e
    j::
    n::
      send, ^n
      return
      
    p::
    k::
    send, ^p
    return
    ,::send, +!,  ;; beginning of page
    .::send, +!.  ;; end of page
    [::send, !a
    ]::send, !e  
    d:: ^k  ;; kill line
    x:: send ^d
    \\:: ^!k  ;; kill next word or sexp
    
  5. Anonymous
    • The variables like `%dir%` aren't working

    I am not sure that these need to be highlighted.

    • Backslashes aren't working

    fixed: made them recognized as text

    • In your test.ahk there are two errors on the line where `FileSelectFile` is used

    fixed: require a
    n or
    s before ';' for comments

    And this one, although it appears to use a module of some sort, also has a funky backslash.

    Fixed: allow [

    .,]etc. in hotkey names by the way, the script does not require a module. If WinExist("iviper") refers to presence of a window (named iviper) for context sensitive hotkeys.

    I have separated the autohotkey lexer from compiled.py for easier maintenance and diffing...

    I have also started working on a new lexer from scratch, for better lexing of some rarely used syntaxes. Autohotkey has a fairly complex, context sensitive syntax. I believe this version is reasonably usable.

  6. Anonymous

    updated test script and test output files: added the two example scripts that you asked for.

  7. Anonymous

    wrote AutoHotkeyV5.py from scratch. changes: variable highlighting is working heredocs are working

    issues: probably some redundant includes in there... number highlighting is not working.

  8. Anonymous

    version 6: [http://dev.pocoo.org/projects/pygments/attachment/ticket/417/autohotkey_v6.py autohotkey_v6.py]BR 1. fixed number matches using python example.BR 2. reordered literals to match after numbers BR 3. added support for keynames in brackets. Example: [pgdn]BR 4. removed duplicate string match in expressions. (use literals which already contains strings)BR 5. reordered continuation sections to work better.BR new example: [http://dev.pocoo.org/projects/pygments/attachment/ticket/417/test_v6.html test_v6.html]

  9. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.