Search with Design

Yahoo! Pipes is a great GUI interface for something that has previously been done only by developers. The ability to filter, sequence, add, combine and translate RSS feed content any way you like is a great tool to have; especially for bloggers, avid RSS readers and website owners.

Although it’s getting more user-friendly all the time it can still be quite an inapproachable application for the beginner. Inspired by GrayWolf’s blog post on Yahoo! Pipes, I’ve written this guide, designed to help you get a basic grasp of Pipes so that you can start experimenting yourself. It’s set up in two parts, a ‘Quick Start Guide’ and the ‘Quick Reference’

Quick Start Guide

What is Pipes?

Yahoo! Pipes is composed of 4 items: Modules, Pipes, Pipe Output, and Input/Outputs spots. Modules take RSS information through their inputs, operate on the data, and export the data to their outputs, where they are delivered, by a pipe, to the pipe output.

The easiest way to think of Pipes is to imagine each Pipe as a sentence: modules are the verbs, RSS feeds the nouns, the output box is a period, and the pipes are the “and, but, so, then”; the conjunction words.

For instance, the sentence “Fetch the RSS feeds for Google Base and Craigslist, using a user-inputted location, and limit the results to apartments.” could be made into a pipe using 2 fetch modules, a user-input module, and a filter module.

Here is a quick example of how you can use Yahoo! pipes. Spring is just around the corner, and it’s time to find a new bike. This pipe will take all the bike listings in a Craigslist feed, analyze them for a certain style or brand of bike, and output the results to a new RSS feed.

“Search Craigslist for My Favorite Bike Brand.”

This pipe makes use of 3 modules: Fetch, User Input, and Filter. Fetch gets the craiglist URL for the SFBay Bikes section, the User Input asks for the keyword (bike brand, type, etc.), and Filter allows only those items with the keyword in the title to be passed to the final feed.

You can try out the pipe for yourself here: http://pipes.yahoo.com/pipes/Wl07ZY_H2xGsZ4mql7okhQ/

How to build the pipe from the example:

1. Start a new pipe.

2. You will need to get 3 modules for this Pipe:

a. Open ‘Sources’ and drag the ‘Fetch’ module out to work area.

b. Open ‘User Inputs’ and drag ‘Text Input’ to the work area.

c. Open ‘Operators’ and get the ‘Filter’ module.

3. Go to Craigslist.org, find URL for the ‘Bikes’ section in your area.

4. Copy this URL into the Field in the Fetch module.

5. In the User-Input module, add some appropriate text for the Prompt and Name boxes, you should also add a keyword about bikes to the ‘default’ and ‘debug’ sections.

6. Change the fields in the Filter module to read ‘Permit’ ‘any’, and then change the first pull down box to ‘dc:title’ and leave the second one set to ‘Contains’

7. Now connect the pipes as I’ve done in the above picture:

a. Fetch > Filter

b. User-Input > keyword field in ‘Filter’

c. Filter > Pipe Output

And now you have a personalized RSS feed to find the bike you’re looking for on Craigslist (Please note: does not apply to the droids you’re looking for).

Click ‘More’ to see the Quick Reference…

Quick Reference

The basics:

The work area: A grid-lined backdrop which is semi-scrollable. It automatically expands to fit new modules that you place outside your current borders.

Pipes: Look more like wires, but to each his own. They can be dragged from either output or input spots to other output/input spots. Basically these are the universal connectors, they represent information flowing from one module to the next, so think of them as “and’s”, “but’s”, “or’s”, and “then’s” in sentence structures. For example “Search Yahoo! then Find items called [Penelope Cruz]”

The ‘Pipe Output’ box: An ever-present module that only accepts 1 input, and has no outputs. This represents the final result of your work which will be displayed on your pipe’s finished page.

Modules: Are the actual operators of the pipes application. They accept inputs, or create input from outside sources (Yahoo Search, Technorati, etc.), or they analyze feeds, or they extract information, or any combination of the above. Think of modules as the verbs in a sentence “Search” “Find” “Get” etc.

Input spots: Input spots can be large buttons located at the top of a module, or smaller spots located near fields inside of modules. Input spots take the results of other modules and can insert them into the production of the current module.

Larger inputs form the basic operations of a module, as in the case of the ‘Union’ piece, which has multiple inputs and one output.

Smaller inputs (or ‘inline inputs’) accept data and incorporate it into a larger function. For instance, you can use an inline input spot to take the result of a user-input and add it to a search query. Input boxes are like blank spaces where nouns go; so a Yahoo! search module is the sentence “Search Yahoo! for ______” and an input box can fill in that blank.

Output spots: >Are the large, glassy buttons on the bottom of modules which allow you to take the data that has been worked on in the module’s process, and transport it to a new module, or to the Pipe Output box. To connect Pipes, you click & drag from an output spot to an input spot.

The Modules (Operators, Blocks, Boxes, etc.):

Sources - Outside information gatherers

Yahoo! Search - Pulls in an RSS feed of the Yahoo! search results for a given term.

Yahoo! Local - Returns an RSS feed of the Yahoo! local results for a given search term and location.

Fetch - Fetches an RSS feed at the specific URL or URLs.

Google Base - Returns an RSS feed of results from Google Base given a keyword, (optional) location and valid Base API key.

Flickr - Returns an RSS feed of pictures from Flickr given a keyword (tag) and (optional) location.

User Inputs

Inputs passed into the Pipe through a form at the start of the pipe running. Also can be passed in through a URL variable. The ‘Position’ field indicates which input should come first, second, third, etc. at the user-prompt page.
URL Input - Prompts the user for a URL at the start of the pipe running.

Number Input - Prompts the user for a numeric value.

Location Input - Prompts the user for input in the form of a physical location.

Text Input - Prompts the user for input in text format (good for keywords!)

Date Input - Prompts the user for input in date format.

Operators - Modules that change, analyze, or combine data.

For Each: Annotate - One of two modules that require submodules. Submodules allow you to drag any module inside of the ‘For Each’ module. This function appends each item in a feed with the value calculated in the submodule, adding the new data to each feed item using the field name specified in the ‘Attribute’ field. For instance, you could append a Spanish translation of each item’s title in the field ‘espanol’.

For Each: Replace - The second of the modules that run submodules, this function replaces the specified attribute value with the value calculated in the submodule for each item in th input feed.

Rename - Renames (multiple) attribute names or copies them as is.

Count - Counts the number of items in the input feed. Outputs a number.

Filter - Filters (positive or negative) items based on text in a given attribute field.

Truncate - Cuts off the number of items in a feed at a given number.

Content Analysis - Searches each feed item for significant keywords and then appends an attribute called ‘Content_analysis’ which contains the keywords.

Sort - Sorts the feed items by a given attribute, in ascending or descending order.

Babelfish - Translates the feed from one language to another using the Babelfish translation service, given From and To languages.

Location Extractor - Searches the feed items for location information, including addresses, location names or map service URLs and places the results in a “Location” attribute.

Union - Joins up to 5 feeds together into one feed.

Unique - Combines duplicate items into one listing based on matching values in the specified attribute.

URL - Contains only the URL builder - a function which builds a URL based on a root domain, and adding in path elements (folder/file names) and query parameters (”?query=parameter”).

String - Contains only String Concatenate - joins each each specified attribute value into a string.

Date - Contains only the Date Formatter - which takes a date in one format and reformats it to the specified format.

My Pipes - Contains links to your other saved Pipes, allowing you to bring in commonly used functions that you’ve built.

Now that you know a little bit about Pipes, go forth, and create! If you have more questions about Pipes, post them in the comments and I’ll do what I can to answer them. Also feel free to post other Pipes resources.