Clone wiki

pypes / User_Guide

User Guide


This guide assumes you have installed pypes properly and have a instance of it up and running somewhere. If this is not the case, please see the Install Guide for details on installing pypes.


This section identifies and explains some of the configuration settings used to control pypes. In general, you shouldn't have change any settings since the defaults should fine in most cases.

We'll list a few of the settings that might be of relevance for experienced users who want more control over the application.


This sections contains some parameters that help control email settings for the application. When pypes needs to send out information, these settings control how that mail is sent,

  • email_to - Specifies the email address where system alerts should be sent.
  • smtp_server - The default mail server to use for when sending alerts
  • error_email_from - The return address used on alerts

server:main Section

This section contains information regarding the host and port settings for the application. The application can tied to a specific hostname or IP but by default, it binds to every network interface on the machine. You can also specify the port number which defaults to 5000.

  • host - The host name/IP the application should bind to (defaults to which means all local interfaces)
  • port - The default port to bind the application (defaults to port 5000)

app:main Section

This section contains settings to control multiprocessing, authentication, and debug mode.

The core setting controls the number of (internal) instances to generate. When you create a project, pypes will place one instance on every CPU and/or core so that content can be processed in parallel. This setting helps pypes determine the number of CPUs/cores to use. You don't have to use every core. There are reasons why you might not want pypes to utilize every CPU/core on the system. For instance, you might be running other CPU intensive applications on the system in which case you don't want to create contention.

The other relevant setting is the which specifies the user credentials for logging into the system. The format is username and password separated by a colon (i.e., admin:123). The password is (obviously) stored in plain text.

  • cores - The number of CPUs and/or cores on the machine (defaults to 1).
  • - The user credentials necessary to log into the application (defaults to admin:123)

For Power Users

The set debug parameter can be used to invoke an interactive debug shell directly in the browser. This can be extremely useful if you're a developer or power user debugging an issue.

handler_file Section

This section allows you control log settings. More specifically, you can change the log level here to provide more verbose output.

  • level - The log level of the application (defaults to INFO but valid choices are INFO, WARN, ERROR, and DEBUG]

Getting Started

Pypes, or more specifically the Visual Design Studio, uses the latest web technologies to provide a rich user experience directly from within your web browser.

In doing so, we've taken advantage of several advanced features of the web, namely the HTML5 canvas tag. Most modern web browsers support HTML5 (i.e., Firefox, Safari, Chrome, and Opera). Unfortunately, Microsoft's Internet Explorer still lags behind the standards so pypes will not function properly if you're an IE user. You can download any of the supported browsers free of charge and they all support Windows.

Once you've got a supported browser installed, you're ready to begin using pypes. Start by visiting the following URL:


You will be presented with the login screen. The default username is "admin" and the password is "123" (omit any quotes).

We've made a few assumptions here. First, we assume you've installed pypes on your local computer. If you've installed pypes on a different computer, you can access the interface using the computers network name or IP address.

Secondly, we're assuming you're running pypes on the standard port of 5000. This can be changed in the pypes configuration file in case you want to run it on a different port (see Configuration). If you have changed the port number, then replace 5000 with the proper port number.

Adding Components To Your Project

The panel on the left side of the application window contains several components that ship with pypes. Others can be downloaded and added separately.

There are two ways to add a component to your project, you can simply drag the component name from the left pane onto the canvas, or you can double click the component name. Both methods will create a new instance of the component on the project canvas.

Removing Components From Your Project

You may want to delete a component from your project. You can do by clicking the small pink "x" in the upper right hand corner of the component. This will remove the component from the project canvas.

Configuring Components

When a component is developed, the developer can chose to expose certain configuration settings to users in order to make it more customizable for different scenarios. A user can access configuration settings by clicking on the small blue "+" sign in the upper right hand corner of the component.

If the component offers any type of runtime configuration settings, a dialog will appear where settings can be provided. If the component doesn't offer any customizable settings, a dialog will still appear notifying the user that the current component doesn't offer any settings.

Connecting Components

The concept behind pypes is to allow components to be connected to form a workflow. The workflow represents a graph like structure whereby data will flow from one component to the next in a directional manner. The idea is to form more complex applications from smaller, more manageable pieces that are easily testable.

To form this workflow you must connect your components together. Data always flow from the Adapters down through any transformations, and then out through the publisher.

Connecting components is a simple matter of placing your mouse on the output port of one component and dragging it to the input port of another component. A blue "pipe" will be displayed to show you the two components are now connected.

Component ports are displayed as circular areas on the top and bottom of the component itself. Input ports are display on the top of the component while output ports are displayed on the bottom. Some ports may have multiple input and/or output ports, depending on it functionality.

At this time, port ordering is random but this is something we're working on.

Disconnecting Components

At times you may want to disconnect two previously connected components. You can do so by placing your mouse over the blue pipe indicating the connection. A small scissors will appear. Clicking on this scissors will disconnect the pipe and remove it from the project. The two components are now disconnected.

Saving Your Work

Once you've designed your workflow and configured all of your components, simply select the "save" button from the toolbar to save your work. A small dialog should display indicating your project was successfully saved. If there were problems saving your work, an error dialog will be displayed informing you there was a problem.

Anytime your project changes because you've added or removed a component, changed any connections, or modified a components configuration, you must save your project for those changes to be reflected.

Exporting Your Project

Pypes provides a mechanism for exporting your work. You may want to send it to someone, applying it on another machine, or simply back it up. You can do so by selecting the export button in the application toolbar. A dialog containing the configuration will be displayed.

The dialog contains a link which allows you to easily create an email with the body of the email being containing the configuration. You can also copy and paste the configuration.

The configuration is nothing more than a JSON serialization and it can be safely edited by hand if necessary.

Importing Your Project

Pypes doesn't currently offer any graphical import mechanism. To import a project into pypes, go to the root of the application (i.e., top folder) and locate the "projects" folder. This folder will likely contain a file named "default.txt" which represents the default or current project. Rename this file to something else such as "current_project.txt" and name the file to be imported to "default.txt".

Restart the application and your recently imported project should become the current project. We're working on providing a graphical import tool that can manage multiple project files.

Submitting Content

Pypes was designed to be a push oriented system. What this means is that typically one writes some sort of data pump to generates data that gets sent to pypes to be transformed. There are several reasons for designing the system this way but the overall idea is to submit content to the system rather than having the system poll for content.

Like any web based application, content can easily be submitted through the UI. To submit a document, select the submit button from the application toolbar. A small file upload dialog will appear, allowing you to select a file from your computer, to be uploaded. This document will be submitted to the system and, depending on the publisher/s, output in some fashion.

This submission mechanism works well for submitting singe documents but doesn't scale well for working will sets of files.

For any type of real data feed, pypes provides a fully REST compliant interface. Content can be submitted using an HTTP POST using multipart/form-data. Pypes provides a FileCrawler utility under its bin/ directory which can be used to submit entire directories of files to pypes.

Run bin/FileCrawler -h for a list of options