Lars Yencken avatar Lars Yencken committed bb53325

Basic utilities for drake.

Comments (0)

Files changed (3)

+.DS_Store
+*.pyc
+*.pyo
+*.orig
+*.bak
+# drakeutil
+
+Utilities for making life easier in Python with [Drake](https://github.com/Factual/drake) workflows.
+
+## Installing
+
+Run `pip install drakeutil`, then for Python steps inside your workflow include:
+
+```
+somefile.out <- somefile.in [python]
+    from drakeutil import *
+```

drakeutil/__init__.py

+# -*- coding: utf-8 -*-
+#
+#  __init__.py
+#  drakeutil
+#
+
+"""
+Helpers for Drake workflows.
+"""
+
+from shutil import copy, move, copytree, rmtree  # noqa
+from os import path, rename, stat  # noqa
+from datetime_tz import datetime_tz
+
+import subprocess
+
+
+def hdfs_timestamp(filename):
+    "When was this file last modified?"
+    stdout, stderr = subprocess.Popen(['hadoop', 'fs', '-stat', filename],
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
+    if not stdout and 'No such' in stderr:
+        return None
+
+    return datetime_tz.smartparse(stdout)
+
+
+def hdfs_exists(filename):
+    "Does this file or directory exist?"
+    p = subprocess.Popen(['hadoop', 'fs', '-test', '-e', filename])
+    p.communicate()
+    return p.returncode == 0
+
+
+def file_timestamp(filename):
+    "When was this file last modified?"
+    try:
+        return datetime_tz.utcfromtimestamp(stat(filename).st_mtime)
+    except OSError:
+        return
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.