1. Victor Stinner
  2. hachoir

Commits

jm  committed 8bfaf70
  • Participants
  • Branches default

Comments (0)

Files changed (233)

File hachoir-core/AUTHORS

View file
+ * Team
+
+Julien Muchembled <jm AT jm10 DOT no-ip DOT com>
+Victor Stinner aka haypo <victor DOT stinner AT haypocalc DOT com>
+
+ * Packagers
+
+Arnaud Pithon aka bildoon <apithon AT free DOT fr> - ArchLinux package (v0.5.2 and svn)
+Emmanuel GARETTE aka GnunuX <gnunux AT laposte DOT net> - ArchLinux package (v0.5.2)
+Michael Scherer aka misc <misc AT mandriva DOT org> - Mandriva package (v0.5.2)
+Michel Casabona aka plumbear <michel DOT casabona AT free DOT fr> - Debian package (v0.5.2)
+Richard DEMONGEOT <richard AT demongeot DOT biz> - Debian package (v0.5.2)
+Thomas de Grenier de Latour TGL <degrenier AT easyconnect DOT fr> - Gentoo ebuild
+

File hachoir-core/COPYING

View file
+		    GNU GENERAL PUBLIC LICENSE
+		       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.
+                       51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+			    Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Library General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+		    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+			    NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+		     END OF TERMS AND CONDITIONS
+
+	    How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) year name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Library General
+Public License instead of this License.

File hachoir-core/Makefile

View file
+.PHONY: clean doc checkroot
+
+PYTHONVER=2.4
+PYTHON=python$(PYTHONVER)
+REST2HTML=rst2html   # included in Python docutils
+export PYTHON PYTHONVER
+XGETTEXT=xgettext
+MSGFMT=msgfmt
+
+DOC_SOURCES=$(wildcard doc/*.txt)
+DOC_HTML=$(patsubst %.txt,%.html,$(DOC_SOURCES))
+CSS="rest.css"
+VERSION=$(shell $(PYTHON) ./get_version.py)
+PYLINT_OPT =--function-rgx="_?[a-z][a-zA-Z0-9]+"
+PYLINT_OPT+=--method-rgx="_{0,2}[a-z][a-zA-Z0-9]+"
+PYLINT_OPT+=--class-rgx="[A-Z][a-zA-Z0-9]+"
+
+# --no-classattr: No "No class attribute (XXX) found"
+# --exec: Warn about use of --exec
+# --stdlib: Ignore stdlib errors
+# --no-returnvalues: Ignore "Function return types are inconsistent"
+PYCHECKER_OPT=--no-classattr --exec --stdlib --no-returnvalues
+
+ROOT_DIR=$(PWD)
+EPYDOC_OPT=--html -n "Hachoir" -u "http://hachoir.org/"
+API_DIR=$(shell cd doc; pwd)/api
+
+# Translation (i18n)
+PACKAGE=hachoir-core
+POT=locale/$(PACKAGE).pot
+LANGUAGES=$(patsubst %.po,%,$(shell cd locale; ls *.po))
+SOURCES=$(wildcard hachoir_core/*.py hachoir_core/stream/*.py hachoir_core/parser/program/python.py)
+
+WIKI2TXT= s/\r$$//;			\
+if (/^{{{/) {				\
+	$$indent = "\t";		\
+} elsif (/^}}}/) {			\
+	$$indent = "";			\
+}					\
+if (/^(  )* 1. /) {			\
+	my $$j = $$_;			\
+	$$j =~ s/^((  )*).*/$$1/;	\
+	$$j = length($$j) >> 1;		\
+	@i = @i[0..$$j];		\
+	$$i[$$j]++;			\
+	s/^((  )*) 1. /$$1 $$i[$$j]. /;	\
+} else {				\
+	@i = 0;				\
+}					\
+s/'\'\'\''?|{{{|}}}//g;			\
+s/!([A-Z]\w+)/$$1/g;			\
+					\
+s/^(.)/$$indent$$1/;
+
+.PHONY: INSTALL
+
+all:
+	@echo "Hachoir version $(VERSION)"
+	@echo
+	@echo "[compile & install]"
+	@echo "  make install - install hachoir-core"
+	@echo "  make doc - create HTML documentation (from reST text)"
+	@echo "  make api - create API HTML documentation (using epydoc)"
+	@echo "  make egg - build an egg (then list dist/ directory)"
+	@echo "  make win32 - build win32 package (then list dist/ directory)"
+	@echo "  make uninstall - (try to) uninstall hachoir-core"
+	@echo "  make clean - remove all useless files"
+	@echo
+	@echo "[tests]"
+	@echo "  make test - run all tests"
+	@echo "  make test-unittest - run unit tests"
+	@echo "  make test-doc - run documentation tests"
+	@echo "  make test-setup - test setup.py"
+	@echo
+	@echo "[static code analys]"
+	@echo "  make pylint - run pylint on source code"
+	@echo
+	@echo "[i18n]"
+	@echo "  make pot - create .po template"
+	@echo "  make mo  - compile .po files into .mo and create locale/ tree"
+
+checkroot:
+	@if test $(shell id -u) -ne 0; then \
+		echo "You have to be root (use su or sudo)"; \
+		exit 1; \
+	fi
+
+prepare-install: INSTALL
+install: prepare-install checkroot
+	./setup.py install
+
+uninstall: checkroot
+	@set -x; \
+    rm -rf /usr/lib/python$(PYTHONVER)/site-packages/hachoir-core-$(VERSION)-py2.4.egg;
+
+doc: $(DOC_HTML)
+	@echo "Documentation updated."
+
+%.html: %.txt
+	@echo "Compile documentation $<"
+	@$(REST2HTML) <$< >$@ --stylesheet=$(CSS) --link-stylesheet || { rm -f $@; exit 1; }
+
+egg: prepare-install
+	./setup.py sdist bdist_egg
+
+win32: prepare-install
+	#./setup.py bdist_wininst
+	#./setup.py bdist --format=wininst
+	./setup.py bdist_msi
+
+test: test-unittest test-doc test-setup
+	@echo "End of tests."
+
+test-unittest:
+	@echo "[run unit tests]"
+	@$(PYTHON) run_unit_test.py && echo
+
+test-doc:
+	@echo "[test using documentation]"
+	@$(PYTHON) test_doc.py && echo
+
+test-setup:
+	@echo "[test setup.py]"
+	@$(PYTHON) setup.py test && echo
+
+pylint:
+	@echo "Run pylint on hachoir_core/*.py"
+	@PYTHONPATH=$(ROOT_DIR) pylint $(PYLINT_OPT) $(shell find hachoir_core/ -name "*.py" ! -name "__init__.py")
+
+pychecker: pychecker-root pychecker-field pychecker-stream pychecker-parser
+pychecker-root:
+	@echo "Run pychecker on hachoir_core/*.py"
+	@(cd hachoir_core; \
+		PYTHONPATH=$(ROOT_DIR) pychecker $(PYCHECKER_OPT) $(shell ls *.py|grep -v compatibility))
+pychecker-field:
+	@echo "Run pychecker on hachoir_core/field/*.py"
+	@(cd hachoir_core/field; \
+		PYTHONPATH=$(ROOT_DIR) pychecker $(PYCHECKER_OPT) *.py)
+pychecker-parser:
+	@echo "Run pychecker on parsers"
+	@(cd hachoir_core/parser; \
+		PYTHONPATH=$(ROOT_DIR) pychecker $(PYCHECKER_OPT) $(shell cd $(ROOT_DIR)/hachoir_core/parser; find -name "*.py"|grep -v init|grep -v template))
+pychecker-stream:
+	@echo "Run pychecker on hachoir_core/stream/*.py"
+	@(cd hachoir_core/stream; \
+		PYTHONPATH=$(ROOT_DIR) pychecker $(PYCHECKER_OPT) *.py)
+
+clean:
+	find \( -name "*.pyc" -o -name "*.pyo" -o -name "*.class" -o -name "*.orig" -o -name "*.rej" \) -print0 | xargs -0 rm -f
+	find doc -name "*.html" -print0 | xargs -0 rm -f
+	rm -rf dist build hachoir_core.egg-info doc/api $(POT) INSTALL
+	rm -rf {gettext,distutils,textwrap,optparse,datetime}.py # downloaded for compatibility
+	(cd locale && rm -rf $(LANGUAGES))
+
+api:
+	test -e $(API_DIR) || mkdir -p $(API_DIR)
+	(cd $(ROOT_DIR) && epydoc $(EPYDOC_OPT) -o $(API_DIR) hachoir_core)
+
+mo:
+	@for lang in $(LANGUAGES); do \
+	    echo "Compile language $$lang"; \
+		LOCALE=locale/$$lang/LC_MESSAGES; \
+		mkdir -p $$LOCALE && $(MSGFMT) locale/$$lang.po -o $$LOCALE/$(PACKAGE).mo; \
+	done
+
+pot: $(POT)
+$(POT): $(SOURCES)
+	$(XGETTEXT) -o $(POT) --keyword=_u --keyword=ungettext --language=Python $(SOURCES)
+
+INSTALL:
+	@echo -n "Create INSTALL: "
+	@wget -qO - 'http://hachoir.org/wiki/Install?format=txt' | \
+		perl -pe '$(WIKI2TXT)' | fmt -sw80 > INSTALL
+	@echo "ok"
+

File hachoir-core/README

View file
+Hachoir library
+===============
+
+Hachoir is a Python library used to represent of a binary file as a tree of
+Python objects. Each object has a type, a value, an address, etc. The goal is
+to be able to know the meaning of each bit in a file.
+
+Why using slow Python code instead of fast hardcoded C code? Hachoir has many
+interresting features:
+
+ * Autofix: Hachoir is able to open invalid / truncated files
+ * Lazy: Open a file is very fast since no information is read from file,
+   data are read and/or computed when the user ask for it
+ * Types: Hachoir has many predefined field types (integer, bit, string, etc.)
+   and supports string with charset (ISO-8859-1, UTF-8, UTF-16, ...)
+ * Addresses and sizes are stored in bit, so flags are stored as classic fields
+ * Endian: You have to set endian once, and then number are converted in the
+   right endian
+ * Editor: Using Hachoir reprensentation of data, you can edit, insert, remove
+   data and then save in a new file.
+
+Website: http://hachoir.org/
+
+Installation
+============
+
+For the installation: see INSTALL file or http://hachoir.org/wiki/Install
+
+What's new in hachoir-core 0.7.1?
+=================================
+
+ * New field type Float80 (80-bit flotting point number)
+ * New field type CompressedField (for compressed content)
+ * Create utility function createDict()
+ * Remove (old and unused) unit tests
+
+What's new in hachoir-core 0.7?
+===============================
+
+Important changes:
+
+ * Rename the component "hachoir" to "hachoir-core"
+ * Editor supports Float32, Float64 and Character types
+ * Floats are now field set: it's possible
+   to read sign, exponent and mantissa
+ * New type EncodedFile to parse "encoded" subfile:
+   compressed, crypted, encoded in base64, etc.
+ * New types GenericVector and UserVector to parse vectors
+ * Raw display is now closer to hexadecimal reprensentation for many types
+ * Rewrite documentation (hachoir-api.txt)
+ * Parser: don't have mime_type or tags attributes anymore
+ * Cleanup some Field methods: getOriginalDisplay() becomes raw_display
+   property, methods _createValue() and _createDescription() become public,
+   etc.
+ * FileInputStream() and FileOutputStream() now have optional "real_filename"
+   to accept invalid unicode filename
+
+Minor changes:
+
+ * Add __repr__() and __unicode__() methods to Field
+ * Use cache for array() method
+ * New types Int24 and UInt24
+ * Field value is now read-only
+ * FieldSet.seekBit()/seekByte() has null optional argument to create
+   nul padding
+ * createPaddingField() raise an exception on invalid size
+   instead of using assertion
+ * str2hex() always returns Unicode string
+ * Remove file export_xml.py (moved to hachoir-console component)
+
+Bugfixes:
+
+ * Use fstat() to get input size instead of seek()+tell() since unusual
+   files on /proc on Linux returns invalid size
+ * GenericString: fix UnixLine, fix UTF-16 with BOM,
+   remove ISO-8859-12 charset (doesn't exist)
+ * Catch seek() error in InputStream and raise InputStreamError()
+ * Fix _fixFieldSize() method for field set with nul size
+ * Fix UnicodeStdout on 8-bit terminal (eg. MS-Dos terminal on Windows)
+ * Fix makePrintable(): quote quote character if needed
+
+what's new in hachoir 0.6.1?
+============================
+
+Bugfixes:
+
+ * Fix GenericString length attribute: wasn't initialized for UTF-* strings
+ * Fix and improve FakeArray (created with fieldset.array("name"))
+
+Improvements:
+
+ * Add text_handler optional argument to Bits and RawBits
+ * On name duplicate error (a field was same name already exists), add "[]"
+   to its name instead and display an error raising an exception
+
+Minor changes:
+
+ * Add class documentation to all types (PaddingBits, Float32, ...)
+ * Fix reversed() code and __all__ constant in hachoir.compatiblity
+ * Add available_types variable to hachoir.field (used by hachoir-wx)
+
+What's new in Hachoir 0.6?
+==========================
+
+First of all, Hachoir projet is splitted in backend: hachoir core and
+hachoir-parser, and in frontends: hachoir-console, hachoir-urwid,
+hachoir-metadata, etc. Changes listed here are only about hachoir core.
+
+ * Hachoir is able to edit a file: edit field value, delete field, insert
+   new field
+ * Suppport of piped input: use small data cache and try to do the most
+   without knowing stream size
+ * Better autofix feature to be able to open invalid / truncated files
+ * Support charset UTF-16 (and UTF-32) for strings
+ * Use Unicode strings everywhere (but only for text, not for binary data)
+ * Use gettext to translate messages => but disabled by default since
+   setup.py is not able to compile .po file to .mo...
+
+Changes for developers:
+
+ * Add new field types: NullBits, NullBytes and SubFile
+ * Create array() method for field set: self["name[%u]" % index] is now
+   the same then self.array("name")[index]
+

File hachoir-core/compatibility/check_interpreter.py

View file
+#!/usr/bin/env python
+import sys
+
+# TODO: decorator
+# TODO: reversed
+
+def checkBoolean():
+    try:
+        x = True
+        x = False
+    except:
+        return "Don't support boolean (True, False)"
+    return "ok"
+
+def checkImportParenthesis():
+    try:
+        code = compile("from sys import (version, hexversion)", "", "exec")
+    except SyntaxError:
+        return "Import doesn't support parenthesis"
+    return "ok"
+
+def checkListGenerator():
+    try:
+        if eval("[ x*2 for x in range(3) ]") != [0, 2, 4]:
+            return 0
+    except SyntaxError:
+        return "List generator doesn't work: [ x*2 for x in range(3) ]"
+    return "ok"
+
+def checkGenerator():
+    try:
+        if eval("list( (x*2 for x in range(3)) )") != [0, 2, 4]:
+            return 0
+    except SyntaxError:
+        return "Creation of generator doesn't work: ( x*2 for x in range(3) )"
+    return "ok"
+
+def checkStruct():
+    try:
+        import struct
+    except ImportError:
+        return "Module struct is missing"
+
+    try:
+        # Copy/paste from hachoir_core.float
+        assert struct.calcsize("f") == 4
+        assert struct.calcsize("d") == 8
+    except (AssertionError, struct.error):
+        return "Wrong struct format size (SEND US A BUG REPORT PLEASE!)"
+
+    try:
+        # Copy/paste from hachoir_core.float
+        assert struct.unpack("<d", "\x1f\x85\xebQ\xb8\x1e\t@")[0] == 3.14
+        assert struct.unpack(">d", "\xc0\0\0\0\0\0\0\0")[0] == -2.0
+    except (AssertionError, OverflowError):
+        return "struct.unpack() error"
+    return "ok"
+
+def checkYield():
+    try:
+        eval(compile("""
+from __future__ import generators
+
+def gen():
+    yield 0
+    yield 2
+    yield 4
+
+if list(gen()) != [0, 2, 4]:
+    raise KeyError("42")
+""", "<string>", "exec"))
+    except (KeyError, SyntaxError):
+        return "Use yield in a function doesn't work"
+    return "ok"
+
+def checkUnicode():
+    tests = (
+        ('a', 'ASCII', 1),
+        ('\xe9', 'ISO-8859-1', 1),
+        ('\xc2\xae', 'UTF-8', 1),
+        ('\xe9\x00', 'UTF-16-LE', 1),
+        ('\x00\xe9', 'UTF-16-BE', 1),
+    )
+    for test in tests:
+        data, charset, size = test
+        try:
+            if eval("len(unicode(data, charset))") != size:
+                return 0
+        except (SyntaxError, NameError):
+            return "Unicode support is missing"
+        except LookupError:
+            return "Charset '%s' is not supported" % charset
+        return "ok"
+
+def checkCallFunctionArgs():
+    try:
+        eval(compile("""
+def foo(*bar):
+    return bar
+
+if foo(0, 2, 4) != (0, 2, 4):
+    raise KeyError("42")
+""", "<string>", "exec"))
+    except (KeyError, SyntaxError):
+        return "Calling a function with foo(*bar) doesn't work"
+    return "ok"
+
+def checkIntLong():
+    try:
+        x = sys.maxint
+        x = x * 8
+    except OverflowError:
+        return "Switch from int type to long is not automatic (sys.maxint*8)"
+    return "ok"
+
+def checkAddEqual():
+    try:
+        eval(compile("x=4; x += 1", "<string>", "exec"))
+    except SyntaxError:
+        return "Syntax x += y is not supported"
+    return "ok"
+
+def checkAll(verbose=0):
+    """
+    Check all Python features needed by Hachoir.
+    Returns a list of error as string. If the list is empty, you're able to
+    use Hachoir :-)
+    """
+    msg = []
+    tests = (
+        ("Types", (checkBoolean, checkUnicode, checkIntLong)),
+        ("Syntax", (checkImportParenthesis, checkAddEqual, checkCallFunctionArgs)),
+        ("Language feature", (checkListGenerator, checkYield, checkGenerator)),
+        ("Modules", (checkStruct,)),
+    )
+
+    for group in tests:
+        print "=== %s ===" % group[0]
+        for test in group[1]:
+            sys.stdout.write("%s(): " % test.__name__)
+            sys.stdout.flush()
+            result = test()
+            print result
+            if result != "ok":
+                msg.append(result)
+        print
+    return msg
+
+def main():
+    print "Run tests."
+    print
+    result = checkAll(1)
+    print "=== Results ==="
+    for error in result:
+        print "ERROR: %s" % error
+    if not result:
+        print "=> Ok, your Python interpreter can run Hachoir."
+        sys.exit(0)
+    else:
+        print "=> You're NOT ABLE to run Hachoir!"
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()
+

File hachoir-core/compatibility/jython.patch

View file
+Index: run_testcase.py
+===================================================================
+--- run_testcase.py	(révision 916)
++++ run_testcase.py	(copie de travail)
+@@ -214,7 +214,7 @@
+     return True
+
+ def testFiles(directory, url_prefix):
+-    if not os.path.exists(directory):
++    if not os.path.exists(str(directory)):
+         try:
+             os.mkdir(directory)
+         except OSError, err:
+@@ -224,7 +224,7 @@
+     for item in files:
+         # Download file if it doesn't exist
+         filename, filesize, md5sum, check_parser, check_metadata = item
+-        fullname = os.path.join(directory, filename)
++        fullname = os.path.join(str(directory), str(filename))
+         try:
+             # Check file MD5
+             read = os.stat(fullname)[stat.ST_SIZE]
+@@ -249,7 +249,7 @@
+     if len(sys.argv) > 1:
+         directory = sys.argv[1]
+     else:
+-        directory = os.path.join(os.path.dirname(__file__), "testcase")
++        directory = "testcase"
+     charset = getTerminalCharset()
+     directory = unicode(directory, charset)
+     print "Start testcase."
+@@ -271,6 +271,5 @@
+         print
+         sys.exit(1)
+
+-if __name__ == "__main__":
+-    main()
++main()
+
+Index: hachoir/plugin.py
+===================================================================
+--- hachoir/plugin.py	(révision 918)
++++ hachoir/plugin.py	(copie de travail)
+@@ -129,6 +129,7 @@
+
+ def guessParser(stream, filename=None, force_mime=None):
+     # Extract file extension
++    filename = str(filename)
+     filename = os.path.basename(filename)
+     filename = filename.split(".")
+
+Index: hachoir/tools.py
+===================================================================
+--- hachoir/tools.py	(révision 935)
++++ hachoir/tools.py	(copie de travail)
+@@ -249,7 +249,7 @@
+             data = ''.join((quote, data, quote))
+     elif quote:
+         data = "(empty)"
+-    data = data.encode(charset, "backslashreplace")
++    data = data.encode(charset, "replace")
+     if to_unicode:
+         data = unicode(data, charset)
+     return data
+Index: hachoir/stream/input.py
+===================================================================
+--- hachoir/stream/input.py	(révision 918)
++++ hachoir/stream/input.py	(copie de travail)
+@@ -44,11 +44,11 @@
+         assert endian in (BIG_ENDIAN, LITTLE_ENDIAN)
+
+         shift, data = self.read(address, nbits)
+-        value = str2long(data, endian)
++        value = int(str2long(data, endian))
+         if endian is BIG_ENDIAN:
+-            value >>= len(data) * 8 - shift - nbits
++            value >>= int(len(data) * 8 - shift - nbits)
+         else:
+-            value >>= shift
++            value >>= int(shift)
+         return value & (1 << nbits) - 1
+
+     def readInteger(self, address, signed, nbits, endian):
+@@ -227,7 +227,7 @@
+         address, shift = divmod(address, 8)
+         self._input.seek(address)
+         size = (size + shift + 7) / 8
+-        data = self._input.read(size)
++        data = self._input.read(int(size))
+         if len(data) != size:
+             raise InputStreamError(
+                 _("Can't read %u bytes at address %u (got %u bytes)!")
+@@ -241,8 +241,8 @@
+
+
+ def FileInputStream(filename):
+-    assert isinstance(filename, unicode)
+     try:
++        filename = str(filename)
+         inputio = open(filename, 'rb')
+     except IOError, err:
+         charset = getTerminalCharset()
+Index: hachoir/i18n.py
+===================================================================
+--- hachoir/i18n.py	(révision 939)
++++ hachoir/i18n.py	(copie de travail)
+@@ -31,7 +31,7 @@
+         charset = locale.getpreferredencoding()
+         if charset:
+             return charset
+-    except (locale.Error, AttributeError):
++    except AttributeError:
+         pass
+
+     # (2) Try locale.nl_langinfo(CODESET)
+@@ -39,7 +39,7 @@
+         charset = locale.nl_langinfo(locale.CODESET)
+         if charset:
+             return charset
+-    except (locale.Error, AttributeError):
++    except AttributeError:
+         pass
+
+     # (3) Try sys.stdout.encoding
+@@ -91,7 +91,7 @@
+     # Setup locales
+     try:
+         locale.setlocale(locale.LC_ALL, "")
+-    except (locale.Error, IOError):
++    except (IOError, AttributeError):
+         pass
+
+     charset = getTerminalCharset()

File hachoir-core/compatibility/jython_gettext.patch

View file
+--- gettext.py.old	2006-09-07 16:12:29.000000000 +0200
++++ gettext.py	2006-09-07 16:12:41.000000000 +0200
+@@ -126,8 +126,11 @@
+ 
+ 
+ def _expand_lang(locale):
+-    from locale import normalize
+-    locale = normalize(locale)
++    try:
++        from locale import normalize
++        locale = normalize(locale)
++    except ImportError:
++        pass
+     COMPONENT_CODESET   = 1 << 0
+     COMPONENT_TERRITORY = 1 << 1
+     COMPONENT_MODIFIER  = 1 << 2

File hachoir-core/compatibility/pypy.patch

View file
+Index: run_testcase.py
+===================================================================
+--- run_testcase.py	(révision 916)
++++ run_testcase.py	(copie de travail)
+@@ -17,7 +17,6 @@
+ import stat
+ import sys
+ from hachoir.i18n import getTerminalCharset
+-from urllib2 import Request, urlopen, URLError, HTTPError
+ from unit_test.valid_parser import (checkFile, stringMD5, checkFileMD5,
+     checkValue, checkNames, checkDisplay, checkAttr)
+
+@@ -33,7 +32,7 @@
+
+ def checkLogoUbuntuMeta(metadata):
+     ok  = checkAttr(metadata, "bits_per_pixel", 8)
+-    ok &= checkAttr(metadata, "creation_date", ["2006-05-26 09:41:46"])
++#    ok &= checkAttr(metadata, "creation_date", ["2006-05-26 09:41:46"])
+     ok &= checkAttr(metadata, "mime_type", "image/png")
+     return ok
+
+@@ -52,7 +51,7 @@
+ def checkGzipMeta(metadata):
+     ok  = checkAttr(metadata, "file_size", 99)
+     ok &= checkAttr(metadata, "compr_size", 90)
+-    ok &= checkAttr(metadata, "last_modification", ["2006-07-29 14:20:44"])
++#    ok &= checkAttr(metadata, "last_modification", ["2006-07-29 14:20:44"])
+     ok &= checkAttr(metadata, "compression", "deflate")
+     return ok
+
+@@ -64,7 +63,7 @@
+
+ def checkPng331_90_8Meta(metadata):
+     ok  = checkAttr(metadata, "width", 331)
+-    ok &= checkAttr(metadata, "creation_date", [u"2006-05-26 09:41:46"])
++#    ok &= checkAttr(metadata, "creation_date", [u"2006-05-26 09:41:46"])
+     ok &= checkAttr(metadata, "mime_type", "image/png")
+     ok &= checkAttr(metadata, "endian", u"Big endian")
+     return ok
+Index: hachoir/text_handler.py
+===================================================================
+--- hachoir/text_handler.py	(révision 918)
++++ hachoir/text_handler.py	(copie de travail)
+@@ -102,8 +102,7 @@
+     """
+     assert hasattr(field, "value") and hasattr(field, "size")
+     assert field.size == 32
+-    timestamp = datetime.datetime.fromtimestamp(field.value)
+-    return unicode(timestamp)
++    return unicode(field.value)
+
+ def timestampMac(field):
+     """

File hachoir-core/compatibility/revert_all.sh

View file
+#!/bin/sh
+
+cd $(dirname $0)/..
+
+M=$(svn stat|grep ^M|cut -d' ' -f7|egrep -v '(revert_all|compatibility|\.patch$)')
+if test "x$M" = "x"; then
+    echo "Repository is clean (no modification, but may add and remove)"
+    exit 0
+fi
+
+if test "x$1" = "xREVERTALL"; then
+    echo "*** REVERT ALL ***" 1>&2
+    svn revert $M
+    echo "Revert: done"
+    exit 0
+else
+    echo "Repository is not clean. Clean it using:"
+    echo "$0 REVERTALL"
+    exit 1
+fi
+

File hachoir-core/compatibility/to_jython.sh

View file
+#!/bin/sh
+# Convert Hachoir source code to Python for Jython 2.2
+
+PYPY_PYTHONLIB_SVN=https://codespeak.net/viewvc/pypy/dist/lib-python/2.4.1
+COMPAT=$(cd $(dirname $0); pwd)
+ROOT=$(cd $COMPAT/..; pwd)
+RM=`which rm`
+WGET=`which wget`
+
+for MODULE in gettext optparse textwrap; do
+    if ! test -e $ROOT/$MODULE.py; then
+      echo "Download $MODULE.py from Python library of PyPy"
+      $WGET -O $ROOT/$MODULE.py $PYPY_PYTHONLIB_SVN/$MODULE.py?revision=HEAD || { $RM $ROOT/$MODULE.py; exit 1; }
+      if test "x$MODULE" = "xgettext"; then
+          (cd $ROOT; patch -p0 < $COMPAT/jython_gettext.patch)
+      fi
+    fi
+done
+
+$COMPAT/revert_all.sh || exit $?
+
+cd $ROOT
+patch -p0 < $COMPAT/jython.patch
+$COMPAT/to_python23.sh NOCHECK jython
+

File hachoir-core/compatibility/to_python22.patch

View file
+Index: run_testcase.py
+===================================================================
+--- run_testcase.py	(révision 916)
++++ run_testcase.py	(copie de travail)
+@@ -249,7 +249,7 @@
+     if len(sys.argv) > 1:
+         directory = sys.argv[1]
+     else:
+-        directory = os.path.join(os.path.dirname(__file__), "testcase")
++        directory = "testcase"
+     charset = getTerminalCharset()
+     directory = unicode(directory, charset)
+     print "Start testcase."
+Index: hachoir/field/integer.py
+===================================================================
+--- hachoir/field/integer.py	(révision 904)
++++ hachoir/field/integer.py	(copie de travail)
+@@ -50,7 +50,6 @@
+                 self.getOriginalDisplay = self._getDisplay
+                 self._getDisplay = lambda: text_handler(self)
+     cls = Integer
+-    cls.__name__ = name
+     return cls
+
+ UInt8 = integerFactory("UInt8", False, 8)
+Index: hachoir/field/string_field.py
+===================================================================
+--- hachoir/field/string_field.py	(révision 919)
++++ hachoir/field/string_field.py	(copie de travail)
+@@ -247,7 +247,6 @@
+             GenericString.__init__(self, parent, name, format, description,
+             strip=strip, charset=charset)
+     cls = NewString
+-    cls.__name__ = name
+     return cls
+
+ # String which ends with nul byte ("\0")

File hachoir-core/compatibility/to_python22.sh

View file
+#!/bin/bash
+
+MV=`which mv`
+RM=`which rm`
+PYTHON=`which python2.4`
+WGET=`which wget`
+COMPAT=$(cd $(dirname $0); pwd)
+ROOT=$(cd $COMPAT/..; pwd)
+PYPY_DATETIME=https://codespeak.net/viewvc/pypy/dist/pypy/lib/datetime.py?revision=HEAD
+
+MODULE=datetime.py
+if ! test -e $ROOT/$MODULE; then
+  echo "Download $MODULE from PyPy"
+  $WGET -O $ROOT/$MODULE $PYPY_DATETIME || { $RM $ROOT/$MODULE; exit 1; }
+fi
+
+if test "$1x" != "NOCHECKx"; then
+    $COMPAT/revert_all.sh || exit $?
+fi
+
+cd $ROOT
+patch -p0 < $COMPAT/to_python22.patch
+$COMPAT/to_python23.sh NOCHECK
+
+cd $ROOT
+echo "*** ADD sum() function to all Python scripts (at the second line) ***"
+for file in $(grep -l sum $(ls script/* $(find -name "*.py"))|egrep -v '(check_interpreter|datetime|compatibility)'); do
+    head -n 1 "$file" >"$file.new"
+    cat >>"$file.new" <<EOF
+import operator
+def sum(datas):
+    return reduce(operator.add, datas)
+EOF
+    tail -n +2 "$file" >>"$file.new"
+    $MV "$file.new" "$file"
+done
+

File hachoir-core/compatibility/to_python23.patch

View file
+Index: hachoir/parser/program/python.py
+===================================================================
+--- hachoir/parser/program/python.py	(révision 882)
++++ hachoir/parser/program/python.py	(copie de travail)
+@@ -238,9 +238,9 @@
+ 
+     # Dictionnary which associate the pyc signature (4-byte long string)
+     # to a Python version string (eg. "m\xf2\r\n" => "2.4b1")
+-    STR_MAGIC = dict( \
++    STR_MAGIC = dict([ \
+         (long2raw(magic | (ord('\r')<<16) | (ord('\n')<<24), LITTLE_ENDIAN), value[0]) \
+-        for magic, value in MAGIC.iteritems())
++        for magic, value in MAGIC.iteritems()])
+ 
+     def validate(self):
+         signature = self.stream.readBits(0, 16, self.endian)
+Index: hachoir/parser/audio/au.py
+===================================================================
+--- hachoir/parser/audio/au.py	(révision 923)
++++ hachoir/parser/audio/au.py	(copie de travail)
+@@ -31,8 +31,8 @@
+     }
+ 
+     # Create bit rate and codec name dictionnaries
+-    bit_rate = dict( (key, value[0]) for key,value in codec_info.iteritems() )
+-    codec_name = dict( (key, value[1]) for key,value in codec_info.iteritems() )
++    bit_rate = dict([ (key, value[0]) for key,value in codec_info.iteritems() ])
++    codec_name = dict([ (key, value[1]) for key,value in codec_info.iteritems() ])
+ 
+     def validate(self):
+         if self.stream.readBytes(0, 4) != ".snd":
+Index: hachoir/field/static_field_set.py
+===================================================================
+--- hachoir/field/static_field_set.py	(révision 872)
++++ hachoir/field/static_field_set.py	(copie de travail)
+@@ -23,7 +23,6 @@
+             cls.static_size = cls._computeStaticSize()
+         return object.__new__(cls, *args, **kw)
+ 
+-    @staticmethod
+     def _computeItemSize(item):
+         item_class = item[0]
+         if item_class.static_size is None:
+@@ -37,6 +36,7 @@
+         else:
+             assert isinstance(item_class.static_size, (int, long))
+             return item_class.static_size
++    _computeItemSize = staticmethod(_computeItemSize)
+ 
+     def createFields(self):
+         for item in self.format:
+@@ -45,7 +45,7 @@
+             else:
+                 yield item[0](self, *item[1:])
+ 
+-    @classmethod
+     def _computeStaticSize(cls):
+-        return sum(cls._computeItemSize(item) for item in cls.format)
++        return sum([cls._computeItemSize(item) for item in cls.format])
++    _computeStaticSize = classmethod(_computeStaticSize)
+ 
+Index: hachoir/parser/__init__.py
+===================================================================
+--- hachoir/parser/__init__.py	(révision 961)
++++ hachoir/parser/__init__.py	(copie de travail)
+@@ -1,3 +1,3 @@
+ from hachoir.parser import (archive, audio, container,
+-    file_system, image, misc, office, program, video)
++    file_system, image, misc, program, video)
+ 

File hachoir-core/compatibility/to_python23.py

View file
+import re
+import sys
+
+def replaceImport(regs):
+    sys.stderr.write("[+] Import: %s\n" % regs.group(0))
+    text = regs.group(1).split("\n")
+    text = " \\\n".join(text)
+    return "import %s" % text
+
+def replaceGen(regs):
+    sys.stderr.write("[+] Generator: %s\n" % regs.group(0))
+    return "[%s]" % regs.group(1)
+
+def replaceCallGen(regs):
+    sys.stderr.write("[+] Call with generator: %s\n" % regs.group(0))
+    return "%s([%s])" % (regs.group(1), regs.group(2))
+
+#def replaceReversed(regs):
+#    return "list(%s)[::-1]" % regs.group(1)
+
+default = "python23"
+if 1 < len(sys.argv):
+    convert_to = sys.argv[1]
+    if convert_to not in ("python23", "nofuture"):
+        convert_to = default
+else:
+    convert_to = default
+
+error = False
+orig_content = sys.stdin.read()
+try:
+    expr_exclude = "][()"
+    expr_a = r"\([^%s]*\)" % expr_exclude
+    expr_aa = r"\([^%s]*\([^%s]*\)[^%s]*\)" % (expr_exclude, expr_exclude, expr_exclude)
+    expr_b = r"\[[^%s]*\]" % expr_exclude
+    expr_c = r"[^%s]" % expr_exclude
+    expr = r"(?:%s|%s|%s|%s)+" % (expr_a, expr_aa, expr_b, expr_c)
+    regex_import = re.compile(r"import \(([a-zA-Z0-9_,%s ]+)\)" % "\n")
+    regex_gen = re.compile("\((%s for %s in %s)\)" % (expr, expr, expr))
+#    regex_reversed = re.compile("reversed\((%s)\)" % expr)
+    regex_call_gen = re.compile(
+        "([a-zA-Z][a-zA-Z0-9_]*)\((%s for %s in %s)\)"
+        % (expr, expr, expr))
+
+    content = orig_content
+    content = regex_import.sub(replaceImport, content)
+    content = regex_call_gen.sub(replaceCallGen, content)
+    content = regex_gen.sub(replaceGen, content)
+    content = content.replace(" 1 <<", " 1L <<")
+#    content = regex_reversed.sub(replaceReversed, content)
+    if convert_to != "nofuture" and "yield" in content:
+        content = "from __future__ import generators\n%s" % content
+except KeyboardInterrupt:
+    sys.stderr.write("INTERRUPTED.\n")
+    error = True
+    content = orig_content
+except Exception, err:
+    sys.stderr.write("ERROR: %s\n" % err)
+    error = True
+    content = orig_content
+sys.stdout.write(content)
+sys.exit(int(error))
+

File hachoir-core/compatibility/to_python23.sh

View file
+#!/bin/bash
+
+MV=/bin/mv
+PYTHON=`which python2.4`
+COMPAT=$(cd $(dirname $0); pwd)
+ROOT=$(cd $COMPAT/..; pwd)
+CONVERTER=$COMPAT/to_python23.py
+
+if test "$1x" != "NOCHECKx"; then
+    CONVERT_TO=$1
+    $COMPAT/revert_all.sh || exit $?
+else
+    CONVERT_TO=$2
+fi
+
+cd $ROOT
+patch -p0 < $COMPAT/to_python23.patch
+for file in $(ls script/* $(find . -name "*.py")\
+|egrep -v '(check_interpreter|datetime|compatibility)'); do
+    $PYTHON $CONVERTER <"$file" >"$file.new" $CONVERT_TO || exit $?
+    $MV "$file.new" "$file"
+done
+

File hachoir-core/doc/adding_fields.graphviz

View file
+digraph GenericFieldSet {
+	rankdir=LR;
+	{
+		node [shape=box];
+		StopIteration [ color=green ];
+		FieldError [ color=red, rank=max ];
+		ParserError -> FieldError;
+	};
+
+	_addField -> { _setUniqFieldName _fixFieldSize };
+	readFirstFields -> readMoreFields;
+	{ _stopFeeding _fixFeedError } -> _fixLastField -> _deleteField;
+	{ __iter__ _feedAll readMoreFields _feedUntil }
+	-> _addField;
+	{ __iter__ _feedAll readMoreFields _feedUntil }
+	-> _stopFeeding [ color=green ];
+	{ __iter__ _feedAll readMoreFields _feedUntil }
+	-> { _fixFeedError FieldError } [ color=red ];
+
+	{ _addField _stopFeeding } -> ParserError;
+	_fixFieldSize -> StopIteration;
+
+	{ __len__ _getSize } -> _feedAll;
+	_getField -> _feedUntil;
+}

File hachoir-core/doc/default.css

View file
+/*
+:Author: David Goodger
+:Contact: goodger@users.sourceforge.net
+:date: $Date: 2003/08/28 22:51:30 $
+:version: $Revision: 1.1 $
+:copyright: This stylesheet has been placed in the public domain.
+
+Default cascading style sheet for the HTML output of Docutils.
+*/
+
+.first {
+  margin-top: 0 }
+
+.last {
+  margin-bottom: 0 }
+
+a.toc-backref {
+  text-decoration: none ;
+  color: black }
+
+dd {
+  margin-bottom: 0.5em }
+
+div.abstract {
+  margin: 2em 5em }
+
+div.abstract p.topic-title {
+  font-weight: bold ;
+  text-align: center }
+
+div.attention, div.caution, div.danger, div.error, div.hint,
+div.important, div.note, div.tip, div.warning, div.admonition {
+  margin: 2em ;
+  border: medium outset ;
+  padding: 1em }
+
+div.attention p.admonition-title, div.caution p.admonition-title,
+div.danger p.admonition-title, div.error p.admonition-title,
+div.warning p.admonition-title {
+  color: red ;
+  font-weight: bold ;
+  font-family: sans-serif }
+
+div.hint p.admonition-title, div.important p.admonition-title,
+div.note p.admonition-title, div.tip p.admonition-title,
+div.admonition p.admonition-title {
+  font-weight: bold ;
+  font-family: sans-serif }
+
+div.dedication {
+  margin: 2em 5em ;
+  text-align: center ;
+  font-style: italic }
+
+div.dedication p.topic-title {
+  font-weight: bold ;
+  font-style: normal }
+
+div.figure {
+  margin-left: 2em }
+
+div.footer, div.header {
+  font-size: smaller }
+
+div.sidebar {
+  margin-left: 1em ;
+  border: medium outset ;
+  padding: 0em 1em ;
+  background-color: #ffffee ;
+  width: 40% ;
+  float: right ;
+  clear: right }
+
+div.sidebar p.rubric {
+  font-family: sans-serif ;
+  font-size: medium }
+
+div.system-messages {
+  margin: 5em }
+
+div.system-messages h1 {
+  color: red }
+
+div.system-message {
+  border: medium outset ;
+  padding: 1em }
+
+div.system-message p.system-message-title {
+  color: red ;
+  font-weight: bold }
+
+div.topic {
+  margin: 2em }
+
+h1.title {
+  text-align: center }
+
+h2.subtitle {
+  text-align: center }
+
+hr {
+  width: 75% }
+
+ol.simple, ul.simple {
+  margin-bottom: 1em }
+
+ol.arabic {
+  list-style: decimal }
+
+ol.loweralpha {
+  list-style: lower-alpha }
+
+ol.upperalpha {
+  list-style: upper-alpha }
+
+ol.lowerroman {
+  list-style: lower-roman }
+
+ol.upperroman {
+  list-style: upper-roman }
+
+p.attribution {
+  text-align: right ;
+  margin-left: 50% }
+
+p.caption {
+  font-style: italic }
+
+p.credits {
+  font-style: italic ;
+  font-size: smaller }
+
+p.label {
+  white-space: nowrap }
+
+p.rubric {
+  font-weight: bold ;
+  font-size: larger ;
+  color: darkred ;
+  text-align: center }
+
+p.sidebar-title {
+  font-family: sans-serif ;
+  font-weight: bold ;
+  font-size: larger }
+
+p.sidebar-subtitle {
+  font-family: sans-serif ;
+  font-weight: bold }
+
+p.topic-title {
+  font-weight: bold }
+
+pre.address {
+  margin-bottom: 0 ;
+  margin-top: 0 ;
+  font-family: serif ;
+  font-size: 100% }
+
+pre.line-block {
+  font-family: serif ;
+  font-size: 100% }
+
+pre.literal-block, pre.doctest-block {
+  margin-left: 2em ;
+  margin-right: 2em ;
+  background-color: #eeeeee }
+
+span.classifier {
+  font-family: sans-serif ;
+  font-style: oblique }
+
+span.classifier-delimiter {
+  font-family: sans-serif ;
+  font-weight: bold }
+
+span.interpreted {
+  font-family: sans-serif }
+
+span.option {
+  white-space: nowrap }
+
+span.option-argument {
+  font-style: italic }
+
+span.pre {
+  white-space: pre }
+
+span.problematic {
+  color: red }
+
+table {
+  margin-top: 0.5em ;
+  margin-bottom: 0.5em }
+
+table.citation {
+  border-left: solid thin gray ;
+  padding-left: 0.5ex }
+
+table.docinfo {
+  margin: 2em 4em }
+
+table.footnote {
+  border-left: solid thin black ;
+  padding-left: 0.5ex }
+
+td, th {
+  padding-left: 0.5em ;
+  padding-right: 0.5em ;
+  vertical-align: top }
+
+th.docinfo-name, th.field-name {
+  font-weight: bold ;
+  text-align: left ;
+  white-space: nowrap }
+
+h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt {
+  font-size: 100% }
+
+tt {
+  background-color: #eeeeee }
+
+ul.auto-toc {
+  list-style-type: none }

File hachoir-core/doc/hachoir-api.rst

View file
+Hachoir is the French name for a mincer: a tool used by butchers to cut meat.
+Hachoir is also a tool written for hackers to cut a file or any binary stream.
+A file is split in a tree of fields where the smallest field can be a
+bit. There are various field types: integer, string, bits, padding, sub file,
+etc.
+
+This document is a presentation of Hachoir API. It tries to show most
+the interesting part of this tool, but is not exhaustive. Ok, let's start!
+
+hachoir.stream: Stream manipulation
+===================================
+
+To split data we first need is to get data :-) So this section presents the
+"hachoir.stream" API.
+
+In most cases we work on files using the FileInputStream function. This function
+takes one argument: a Unicode filename. But for pratical reasons
+we will use StringInputStream function in this documentation.
+
+   >>> data = "point\0\3\0\2\0"
+   >>> from hachoir_core.stream import StringInputStream, LITTLE_ENDIAN
+   >>> stream = StringInputStream(data)
+   >>> stream.source
+   '<string>'
+   >>> len(data), stream.size
+   (10, 80)
+   >>> data[1:6], stream.readBytes(8, 5)
+   ('oint\x00', 'oint\x00')
+   >>> data[6:8], stream.readBits(6*8, 16, LITTLE_ENDIAN)
+   ('\x03\x00', 3)
+   >>> data[8:10], stream.readBits(8*8, 16, LITTLE_ENDIAN)
+   ('\x02\x00', 2)
+
+First big difference between a string and a Hachoir stream is that sizes
+and addresses are written in bits and not bytes. The difference is a factor
+of eight, that's why we write "6*8" to get the sixth byte for example. You
+don't need to know anything else to use Hachoir, so let's play with fields!
+
+hachoir.field: Field manipulation
+=================================
+
+Basic parser
+------------
+
+We will parse the data used in the last section.
+
+   >>> from hachoir_core.field import Parser, CString, UInt16
+   >>> class Point(Parser):
+   ...     endian = LITTLE_ENDIAN
+   ...     def createFields(self):
+   ...         yield CString(self, "name", "Point name")
+   ...         yield UInt16(self, "x", "X coordonate")
+   ...         yield UInt16(self, "y", "Y coordonate")
+   ...
+   >>> point = Point(stream)
+   >>> for field in point:
+   ...     print "%s) %s=%s" % (field.address, field.name, field.display)
+   ...
+   0) name="point"
+   48) x=3
+   64) y=2
+
+`point` is a the root of our field tree. This tree is really simple, it just
+has one level and three fields: name, x and y. Hachoir stores a lot of
+information in each field. In this example we just show the address, name and
+display attributes. But a field has more attributes:
+
+   >>> x = point["x"]
+   >>> "%s = %s" % (x.path, x.value)
+   '/x = 3'
+   >>> x.parent == point
+   True
+   >>> x.description
+   'X coordonate'
+   >>> x.index
+   1
+   >>> x.address, x.absolute_address
+   (48, 48)
+
+The index is not the index of a field in a parent field list, '1' means that it's
+the second since the index starts at zero.
+
+Parser with sub-field sets
+--------------------------
+
+After learning basic API, let's see aa more complex parser: parser with
+sub-field sets.
+
+   >>> from hachoir_core.field import FieldSet, UInt8, Character, String
+   >>> class Entry(FieldSet):
+   ...     def createFields(self):
+   ...         yield Character(self, "letter")
+   ...         yield UInt8(self, "code")
+   ...
+   >>> class MyFormat(Parser):
+   ...     endian = LITTLE_ENDIAN
+   ...     def createFields(self):
+   ...         yield String(self, "signature", 3, charset="ASCII")
+   ...         yield UInt8(self, "count")
+   ...         for index in xrange(self["count"].value):
+   ...             yield Entry(self, "point[]")
+   ...
+   >>> data = "MYF\3a\0b\2c\0"
+   >>> stream = StringInputStream(data)
+   >>> root = MyFormat(stream)
+
+This example present many interresting features of Hachoir. First of all, you
+can see that you can have two or more levels of fields. Here we have a tree
+with two levels:
+
+   >>> def displayTree(parent):
+   ...     for field in parent:
+   ...         print field.path
+   ...         if field.is_field_set: displayTree(field)
+   ...
+   >>> displayTree(root)
+   /signature
+   /count
+   /point[0]
+   /point[0]/letter
+   /point[0]/code
+   /point[1]
+   /point[1]/letter
+   /point[1]/code
+   /point[2]
+   /point[2]/letter
+   /point[2]/code
+
+A field set is also a field, so it has the same attributes than another field
+(name, address, size, path, etc.) but has some new attributes like stream or
+root.
+
+Lazy feature
+------------
+
+Hachoir is written in Python so it should be slow and eat a lot of CPU and
+memory, and it does. But in most cases, you don't need to explore an entire
+field set and read all values; you just need to read some values of some
+specific fields. Hachoir is really lazy: no field is parsed before you ask for
+it, no value is read fr