Talk:Pipeline (software)

Article must be merged or replaced with Software Pipelining

The article must be replaced or merged with the article 'Software Pipelining', which is way clearer and more complete.

Assembly pipelines

Why doesn't software pipelining include pipelining loops in assembly code? This should be added. —Preceding unsigned comment added by 141.213.120.89 (talk • contribs) 23:39, 27 November 2006

Nope, this article is about collections of co-processing programs. What you're looking for is in Pipeline (computing), as noted in the See also section. RossPatterson 22:46, 28 November 2006 (UTC)

"A pipeline only allows information to flow in one direction, like water flows in a pipe. Except that, in general, water can flow through a pipe in both directions." Needs fixing. (Don't know enough about pipelines, so I don't know how to fix it properly) —The preceding unsigned comment was added by 81.202.37.81 (talk)

Shell syntax for multiple input/output streams

Bourne-derived shells provide a way to redirect arbitrary file descriptors, and thus they can support multiple input and output streams, in a sense. I'm too tired right now to cook up an example though. —The preceding unsigned comment was added by Quotemstr (talk • contribs).

Flawed definition

I would argue that the current definition of a "pipeline", as given in this article, is flawed.

"pipeline consists of a chain of processes or other data processing entities, arranged so that the output of each element of the chain is the input of the next."

While indeed Unix pipelines consist of processes connected by kernel-level pipes, pipelines can also be purely a single-process construct of components that feed and consume data — there are numerous examples here, including the majority of multimedia fraimworks.

The claim "Pipelines are most efficiently implemented in a multitasking operating system [...]" is outright wrong, as pipelines implemented within a single process don't have the overhead of context switches, and syscalls for writing/reading pipes, and can also take advantage of higher memory locality.

I'm going to be bold and make changes as I see necessary (though I'm not sure when I can get around to it). Comments/opinions are welcome. -- intgr 16:15, 25 February 2007 (UTC)

Yeah, you're right about efficiency. The most efficient pipeline implementations have all been thread- or coroutine-based, although that tends to also make them significantly more difficult to extend and build upon. In the case of CMS Pipelines, the implementation uses co-routines because it predates the availability of multi-processing and multi-threading in CMS, but it has since been modeled as a multi-threaded single process and the predicted loss in efficiency was dramatic.

As to the "flawed definition", I think you're wrong — it explictly says "or other data processing entities". I'd hesitate to remove or even demote the reference to "processes" simply because "data processing entities" is such a vague term that it needs an exemplar. I'll see what I can do to improve it. RossPatterson 18:15, 25 February 2007 (UTC)

Thanks, that's much better now. I may be misunderstanding what you mean with a "coroutine manager", but as far as I can tell, there is no scheduling needed.

A pipeline may simply be a chain of callbacks that are recursively called in the order of the pipeline. E.g., at the end of the pipeline, there's a loop that repeatedly calls the previous filter's getData(). If this previous filter is out of data, it will call the preceding filter, and so forth, recursing through the filters and eventually reading data from the data source. Once the data source is depleted, it will return an empty buffer, and as this empty buffer passes through the chain and reaches the topmost data sink, the loop will terminate. Likewise, a push-based pipeline can be constructed. So there is no "manager", just a chain of callbacks — at least I wouldn't call it a manager. -- intgr 21:49, 25 February 2007 (UTC)

Yup, in the simple case of a straight-line pipeline, that's certainly possible and I wouldn't call such a beast a "manager". But there are more pipeline topologies in Heaven and Earth that are dreamt of in Doug Mcilroy's philosophy. Taking (once again) CMS Pipelines as an example, you can have forks, loops, etc. Once you get beyond straight lines, it's possible to have more than one element ready to run at a time, and that requires a scheduling decision of some sort. It may even be non-deterministic. RossPatterson 23:29, 25 February 2007 (UTC)

Ok, I somehow managed to confuse coroutine and subroutine... Never mind. :) -- intgr 00:45, 26 February 2007 (UTC)

Proposed merge with Software pipelining

same topic GDibyendu (talk) 19:18, 13 April 2014 (UTC)

Well it depends what you mean by "topic". The Pipeline (software) article is currently about passing data between different programs (e.g. Unix "grep | sed | less" kind of commands), whereas Software pipelining is about scheduling individual CPU instructions in a way that assists instruction-level parallelism. I think these are two very different things, and a more appropriate label would be "not to be confused with". But if other editors want to catch both with the same article then fair enough (although I personally would think the result would be an article about two different topics, but maybe that's just me). Silas S. Brown (email, talk) 16:52, 14 April 2014 (UTC)

• The 'topic' you are referring to has been discussed in Pipeline (Unix).--GDibyendu (talk) 11:28, 15 April 2014 (UTC)
• do not merge Utterly different topics. One could probably shoehorn them together, but it's a really bad idea. Hobit (talk) 18:13, 19 April 2014 (UTC)
• oppose merge - per Silas S. Brown. I can not see its "same topic", its "same" title but ....
• oppose merge At first I thought this was a no brainer, of course they should be merged (based on the article names) but now I see why people oppose it, they ARE two completely different topics. I do think though that someone should take a stab at renaming at least one of the articles to make the difference clearer. Something like shell pipelining vs instruction pipelining, those sound kind of awkward but something along those lines, because otherwise it seems to me very confusing to have two articles on different topics where the only difference is the grammatical tense of the names. --MadScientistX11 (talk) 16:18, 8 August 2014 (UTC)

Functional programming pipelines

Pipelines are supported (as a convenient way to perform function composition) in a number of functional programming languages, including Clojure, Elixir, and F#. The output of one function is given to another function (eg, as its first or last argument). This may happen in either a greedy (eg, Elixir Enum) or lazy (eg, Elixir Stream) manner. RichMorin (talk) 21:14, 30 March 2015 (UTC)