1. abudden
  2. TagHighlight

Wiki

Clone wiki

TagHighlight / IntegratingMakeHLTags

Summary

This page is a work-in-progress attempt to document the operation of Alexey Radkov's VimMakeHlTags script, which uses TagHighlight's python scripts to manage tags and types highlighting for the projects on which he works. The aim is to integrate the functionality of VimMakeHlTags into TagHighlight (albeit in a manner consistent with TagHighlight's design) so that the need for an external wrapper is no longer there.

The Scripts

vim-myprojects.vim

Works on premise that all tags are stored in a single standard location, typically ~/opt/tags/. Stored in $TAGSDIR and g:MprTagsdir.

There is a major variable called g:MprTags which defines parameters of one or more projects. It contains a list of elements, each element contains 6 fields:

  1. A directory pattern for an autocmd - BufReadPre,BufNewFile and VimEnter. VimEnter configures tags (used as an alternative to setlocal it seems), others configure general variables for scripts I think. Yes, there are several buffer-wise variables that provide specific info for each open buffer for making updates of highlighted words and also prevent highlighting in buffers with filetypes that must be skipped. So, VimEnter configures databases (ctags and cscope) for jumping around, BufReadPre and BufNewFile configure buffer variables for making correct updates of jumps and highlights in all buffers (imagine that you open in same vim instance some unrelated file: its state - jumps database and highlights, must stay unchanged). There is another autocmd for FileType that really does tags highlights.
  2. The name of the project and it's dependencies.
  3. Language (or empty). Passed to TagHighlight.py as value of option --include-language
  4. File-types to skip for this project. If TagHighlight could be configured for showing tags only for related language (e.g. when i build tags for C++ i want to see highlights only in cpp files, not in perl or python files as well), then there would be no need for this option.
  5. Project root (used for tags/types generation)
  6. Options for builder? In particular, this includes keywords that shouldn't be highlighted. Yes, this is simply a fragment for insertion into MakeVimHlTags command-line args.

autocmd for FileType set for all files (SetHighlight) - uses the above set variables.

BuildTags

This is a bash script. Takes options -p - prefix for output file names, -d destination dir, -c ctags program and then a list of directories in which to look for source code. Looks for .h, .x, .c, .cc and .icc files in each source directory and builds a cscope DB with cscope -u -b -q -i ${filelist} -f ${outputfile}. It then builds tags with ctags -R --c++-kinds=+p --fields=+iaS --extra=+q -f ${out} ${dirs}.

MakeVimHlTags

This is a perl script. Options -r - recurse, -l include locals. -e excluded tags - uses . as a * wildcard. Also allows skipping tags for a subset of programming languages. I think that this is doing things like skipping 'bool' in C++ if you include -e@cpp. Interesting concept... -a add symbols for other projects to result file. I guess this is how the user libraries are generated. Rest of the options are fairly standard. Beware: option -a is no more used and in principle can be removed: it adds tags from other projects into the resulting file, but now i replaced this concept by expanding the list b:TagHighlightSettings['UserLibraries'] (see the end of function SetHighlights()).

Integration with TagHighlight

In order to integrate this with TagHighlight, a number of new options / methods will need to be introduced:

  1. Ability to generate cscope databases. This was present in ctags_highlighting (before the 'rebrand'/rewrite, but was removed due to a lot of problems with file-locking conflicts. I think this was mainly due to my habit of doing :UpdateTypesFile and then switching to a console window and doing bzr st - Bazaar had a habit of looking at cscope.db (even though it was in the ignore list) while cscope was working and then cscope wouldn't be able to overwrite it. This may be better with mercurial (which I now use), but it may affect other people as well. Needs a little thought.
  2. Need a method of configuring TagHighlight to work with a central types/tags repository. This would be the equivalent of Alexey's ~/opt/tags/ directory in which he stores the tags files and types_*.taghl files. Obviously with this approach there would need to be a naming convention for the files, but if it's intended to work with project-style settings, it could simply be ${PROJECT_NAME}_tags etc.
  3. Need a method of defining a 'project' in the sense of g:MprTags. I suspect a lot of this can be done with a taghl_config.txt, but it may be worthwhile to extend this to a set of 'project options' that are enabled for the buffer with an autocmd. This could take the form (update 2012-05-18 - now a dictionary):
let g:TagHighlightSettings['Projects'] = 
    \ {
    \     'MyProject': {
    \         'SourceDir': '/path/to/source/code',
    \         'Dependencies': ['UserLib1', 'SystemLib2'],
    \         'AutocmdPattern': '/usr/*/geant4/*/source/*',
    \         'AnyTagHighlightOption': 'AppropriateValue',
    \         'SkipPatterns': ['G4.*',],
    \     },
    \     'UserLib1': {
    \     ...
    \     },
    \ ]

In principle, the only option that is required is the name (the dictionary key), although it wouldn't do much without some other options being set. Unless I'm missing something, 'Dependencies' is a more advanced version of UserLibraries that allows TagHighlight to do the generation of the tags/types files automatically and uses the library name to reference another entry in the 'Projects' array. It could be made to work either with an entry in the 'Projects' array or an absolute path to a types highlighter file. If 'SourceDir' is set for the project and the dependencies, then there can be an option (e.g. 'UpdateDependencies' or something like that) that makes the tags/types for the dependencies be updated at the same time as the project ones. (I do not think so, as I see dependencies as dependencies for showing rather than for making tags. Originally dependency concept was involved for getting rid of long updates of everything, but just a small part of the project. This concept assumes that one makes single highlights database for a 3rd party library just once and further attach it to other projects that he may update partially very often). Alternatively, there could be some new commands, e.g. UpdateProjectTypes, UpdateProjectDepTypes or something like that. The naming can come later...

In practice, the autocmd for the buffer would only have to:

let b:TagHighlightSettings['CurrentProject'] = 'MyCustomProject'

everything else could be read directly from the 'Projects' option. The autocmd would, in most cases, be unnecessary anyway as you could just create a taghl_config.txt looking like this:

CurrentProject:MyCustomProject

but I guess Alexey is trying to avoid having files that aren't related to the source code in the source code directory. I guess this would save me having to add my standard

tags
types_*.taghl
taghl_config.txt

to .hgignore for every project!

If the autocmd were used rather than the taghl_config.txt file, it may be necessary to make 'SourceDir' a mandatory requirement: in nested projects it can be difficult to know where the source code starts and if the tags/types files are elsewhere and there is no taghl_config.txt then we lose any method of determining this.

Discussion

I'm adding this section on the end so that there can be some order to further development discussions in some specific categories. Could probably be done by email, but I thought this might be an interesting experiment into using a wiki (I particularly like the way you can hg clone it and edit it with vim and vim-creole) and might keep old discussions more easily available when I actually come to code some of this stuff!

Standalone Scripts

Alexey 2012-05-18

One important remark: it is crucial to implement BuildTags and MakeVimHlTags as standalone scripts like TagHighlight.py (e.g. not to move implementation to VimL) as far as it is sometimes very convenient to use them from command-line, not from vim (for example for generation 3rd party libraries tags).

Al 2012-05-18

Okay, I hadn't taken that into account: thank you. I'll probably make everything into python modules for consistency with the rest of TagHighlight (two languages is quite enough!). I imagine I'll then make a couple of very simple wrapper python scripts that give a command line interface to those python modules. This isn't that different to how TagHighlight works anyway: there are two wrappers around the python modules - one for vim's python interface and one for command-line calls.

Alexey 2012-05-19

Making script using python is really good: i just do not know this language well and implemented it using sh and perl.

Al 2012-05-19

No problem: I had imagined that I would be the one writing the python code!


Third Party Library Tags

Al 2012-05-18

In the case of 3rd party library tag generation, I'd like to have an interface in Vim that is something along the lines of:

:UpdateProjectTypes <projname>

e.g.

:UpdateProjectTypes geant4

The intention is that this would ignore all the buffer variables and go and regenerate the tags and types files for the geant4 entry in g:TagHighlightSettings['Projects'] without changing anything in the working files (except maybe calling ReadTypes. I can, however, see the benefit of being able to do this from the command line.


Autocmds

Al 2012-05-18

I don't know how to quote in creole's wiki syntax, so I'll just bung the text under discussion in here:

Al 2012-05-18

A directory pattern for an autocmd - BufReadPre,BufNewFile and VimEnter. VimEnter configures tags (used as an alternative to setlocal it seems), others configure general variables for scripts I think.

Alexey 2012-05-18

Yes, there are several buffer-wise variables that provide specific info for each open buffer for making updates of highlighted words and also prevent highlighting in buffers with filetypes that must be skipped. So, VimEnter configures databases (ctags and cscope) for jumping around, BufReadPre and BufNewFile configure buffer variables for making correct updates of jumps and highlights in all buffers (imagine that you open in same vim instance some unrelated file: its state - jumps database and highlights, must stay unchanged). There is another autocmd for FileType that really does tags highlights.

Al 2012-05-18

I'm still not completely convinced of the need for a VimEnter autocmd here. As far as I can see it's just configuring buffer-specific settings. However, these can be done with buffer local variables (b:) and buffer local settings (setlocal). Then VimEnter becomes redundant I think? Cscope may break that though: it depends whether cs add and the like have buffer local versions: if not, we'll need VimEnter for cscope. Actually, we'll probably need BufEnter: VimEnter is something completely different to what I was thinking of - did you mean VimEnter or BufEnter (:help VimEnter)?

Hopefully the FileType autocmd will be soaked up by TagHighlight's existing autocmds.

Alexey 2012-05-19

About VimEnter: when i did that i supposed by some reason that i would make tags for jumps "static" in contrast to tags for highlights. I mean that when i open a single instance of vim i configure jump tags once at this time, i.e. if i open file from a "project" i setup global "set tags" and "cs add" for ever during this session instance: if i would open another file from another "project" or not from a project at all the jump tags won't change and you would still be able to jump to tags from original "project" even from unrelated buffers. In contrast, tags "highlights" is fully buffer-location aware: if i open unrelated file then original highlights are unknown for this buffer, moreover if new open buffer is from another "project", the tags from that project will be highlighted. I decided that this is safe enough (do not switch jump tags) and does not mix tags "highlights" too. Probably i just did not think well that i can use setlocal for jumps tags thus making them dependent on current buffer: in this case you are right and i did oversimplified implementation of jumps tags.

Al 2012-05-19

That makes sense: thanks for the explanation.

Alexey 2012-05-19

By the way, i think it is very rare when a user opens files from more than 1 project in same vim instance. If he does that then it is even more rare if he expects to jump to tags of another project in open buffer (probably i am not right). It much worse when he visually sees wrong tags highlights in another project - but this is covered by correct 'highlight' tags tracking. Probably it makes sense to leave it as is (with VimEnter autocmd): or just let user choose an option): i am afraid that switching tags databases may decrease performance of vim (probably i am not right again).

Al 2012-05-19

I'll look into any performance hit. I, for one, often edit files from more than one project in the same vim instance. In general this is when I'm (e.g.) editing a microcontroller application and the bootloader for that application (a completely independent project that happens to run on the same microcontroller) and also sometimes some python test GUIs etc: they're all related but they have separate code bases and hence separate tags / types.

I don't think that switching tags databases will slow things down much: I'm fairly, but not completely sure that when you do Ctrl-] or whatever, vim opens the tags files and searches, rather than holding anything in memory. I'll see whether I can verify this though. As for the cscope database though, I've no idea and it could well incur a performance hit. I'll see what I can discover; thanks for the note.


Skipped File-Types

Al 2012-05-18

Another quote from above:

File-types to skip for this project.

Alexey 2012-05-18

If TagHighlight could be configured for showing tags only for related language (e.g. when i build tags for C++ i want to see highlights only in cpp files, not in perl or python files as well), then there would be no need for this option.

Al 2012-05-18

I'm very intrigued by this: it sounds like a serious bug in TagHighlight: if you build tags for C++ I wouldn't expect highlights in perl or whatever, especially if you use the 'Languages' option. Please tell me more!

Alexey 2012-05-19

I experienced that always. Probably i missed something when tags are created, but i always thought that this is how TagHighlight implemented. I will look again. To make it clear: i mean situations when i created tags using TagHighlight for C/C++ then i open a perl file for example and see the highlights there.

Al 2012-05-19

Okay, I've raised this as an 'Urgent' bug in my bug tracker and will look at it as soon as I can.


Dependencies

Al 2012-05-18

And another quote:

If 'SourceDir' is set for the project and the dependencies, then there can be an option (e.g. 'UpdateDependencies' or something like that) that makes the tags/types for the dependencies be updated at the same time as the project ones.

Alexey 2012-05-18

(I do not think so, as I see dependencies as dependencies for showing rather than for making tags. Originally dependency concept was involved for getting rid of long updates of everything, but just a small part of the project.

Al 2012-05-18

Ah, okay: that was more in line with how I understood dependencies. I guess I misread your code and thought it was doing the updates as well. Given that I'm thinking of adding a method of updating another project's tags/types it would probably only take three or four lines of code to have a UpdateProjectAndDependencies* command that actually updates the tags and types for the dependencies as well as the project, but in principle it shouldn't really be used very much.

* Don't worry, I'm going to do some serious thinking about command names before implementing any of this!).

Alexey 2012-05-19

No, I never meant to update dependencies upon update of tags of a current project. I think implementation of updating dependencies is hard: there should be implemented a mechanism for tracking parameters of the projects through all lines of "g:MprTags". And i do not really understand reason for that: the full tags are comprised from independent parts: they make "dependency" only for showing tags, not making them. For example if you are updating some code in a project then there is no reason to update its dependencies (3rd party libraries): they did not really change. Result will always be correct as far as "full" tags database is a simple aggregation of different tags.

Al 2012-05-19

I think for 99% of cases I agree with you. Experience with a couple of projects at my work has, however, suggested that there would be some use cases where updating 'dependency' tags at the same time could be useful. We (as a company) have recently (basically since I joined the company) started trying to do a lot more code re-use. The only software we produce is (small) embedded microcontroller code. In the past there were only a very small number of microcontrollers, each with completely independent code bases. Now, there are lots of microcontrollers with a lot of common code (communication stacks etc). In the process of moving across to this common code structure, one of the engineers would typically start a new project and a new 'library' module (dependency) at the same time. They would be developed and tested in parallel and after a while the library code would start being used in other projects and probably be frozen. Once frozen, it would work much like a normal 3rd party library in terms of highlighting/tags etc, but during the initial development it would be very likely that you'd want to update types/tags for both the project and the library at the same time.

So there you: that's the use case. As you've suggested it's definitely the exception rather than the rule, but I can see that it would be useful to support this mode of operation.


vim:ft=creole:tw=80

Updated