Anonymous avatar Anonymous committed b4d0f8d

The new fetch_data_files.py downloads all the input data files
used by encoding tests. Fiddled the Windows buildbot helper
scripts to invoke this if needed. Note that this isn't needed
on the trunk (the encoding tests download input files automatically
in 2.5).

Comments (0)

Files changed (4)

Tools/buildbot/README.txt

+Helpers used by buildbot-driven core Python testing.
+
+external.bat
+build.bat
+test.bat
+clean.bat
+    On Windows, these scripts are executed by the code sent
+    from the buildbot master to the slaves.
+
+fetch_data_files.py
+    Download all the input files various encoding tests want.  This is
+    used by build.bat on Windows (but could be used on any platform).
+    Note that in Python >= 2.5, the encoding tests download input files
+    automatically.

Tools/buildbot/build.bat

 @rem Used by the buildbot "compile" step.
 cmd /c Tools\buildbot\external.bat
 call "%VS71COMNTOOLS%vsvars32.bat"
-devenv.com /useenv /build Debug PCbuild\pcbuild.sln
+cd PCbuild
+devenv.com /useenv /build Debug pcbuild.sln
+@rem Fetch encoding test files.  Note that python_d needs to be built first.
+if not exist BIG5.TXT python_d.exe ..\Tools\buildbot\fetch_data_files.py

Tools/buildbot/external.bat

 cd ..
 
 @rem bzip
-if not exist bzip2-1.0.3 svn export http://svn.python.org/projects/external/bzip2-1.0.3
-
+if not exist bzip2-1.0.3 svn export http://svn.python.org/projects/external/bzip2-1.0.3

Tools/buildbot/fetch_data_files.py

+"""A helper to download input files needed by assorted encoding tests.
+
+fetch_data_files.py [directory]
+
+Files are downloaded to directory `directory`.  If a directory isn't given,
+it defaults to the current directory (.).
+"""
+
+DATA_URLS = """
+    http://people.freebsd.org/~perky/i18n/BIG5HKSCS.TXT
+    http://people.freebsd.org/~perky/i18n/EUC-CN.TXT
+    http://people.freebsd.org/~perky/i18n/EUC-JISX0213.TXT
+    http://people.freebsd.org/~perky/i18n/EUC-JP.TXT
+    http://people.freebsd.org/~perky/i18n/EUC-KR.TXT
+    http://people.freebsd.org/~perky/i18n/SHIFT_JISX0213.TXT
+
+    http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.TXT
+    http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP936.TXT
+    http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP949.TXT
+    http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP950.TXT
+
+    http://www.unicode.org/Public/3.2-Update/NormalizationTest-3.2.0.txt
+
+    http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/SHIFTJIS.TXT
+    http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/KSC/JOHAB.TXT
+    http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/OTHER/BIG5.TXT
+"""
+
+# Adapted from test_support.open_urlresource() in Python 2.5.
+# Fetch the file give by `url` off the web, and store it in directory
+# `directory`.  The file name is extracted from the last URL component.
+# If the file already exists, it's not fetched again.
+def fetch_file_from_url(url, directory):
+    import urllib, urlparse
+    import os.path
+
+    filename = urlparse.urlparse(url)[2].split('/')[-1] # '/': it's a URL!
+    target = os.path.join(directory, filename)
+    if os.path.exists(target):
+        print "\tskipping %r -- already exists" % target
+    else:
+        print "\tfetching %s ..." % url
+        urllib.urlretrieve(url, target)
+
+def main(urls, directory):
+    print "Downloading data files to %r" % directory
+    for url in urls.split():
+        fetch_file_from_url(url, directory)
+
+if __name__ == "__main__":
+    import sys
+
+    n = len(sys.argv)
+    if n == 1:
+        directory = "."
+    elif n == 2:
+        directory = sys.argv[1]
+    else:
+        raise ValueError("no more than one argument allowed")
+
+    main(DATA_URLS, directory)
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.