Do not specify UPC support for UPC threads implemented as OS threads

[Former user]

Originally reported on Google Code with ID 22 ``` This informative note is responsive to language issues 12 and 14.

12. Are the pthreads−based implementations of UPC truly C99/POSIX/UPC conformant? Can/should the language specification speak to this?

14. Shall Berkeley’s "#pragma upc upc_code" and "#pragma upc c_code", "#pragma upc detect_upc suspend_insertion" and "#pragma upc detect_upc resume_insertion" be defined in the specification? Note that these are mainly used to deal with the difficulties of the pthreads implementation.

Some UPC implementations support the mapping of UPC threads to the underlying operating system supported thread of execution (often implemented as a POSIX thread or "pthread"). However, to do so, the UPC compiler must rewrite static file scoped declarations into declarations of similarly named variables that have per- OS thread persistence and locality. This works as expected as long as all source files that declare static file scoped data are compiled with the UPC compiler invoked in this specific "UPC-thread-as-pthread" mode.

The Berkeley compiler implements a tool called "detect-upc" that passes over the preprocessed input source code to determine which file scope static variables are defined in system headers and which are defined else where in application code. It annotates the UPC-specific sections with "#pragma upc upc_code", "#pragma upc c_code" and "#pragma upc detect_upc resume_insertion".

These pragmas and the implementation of UPC threads as OS supported threads are by their nature extensions to UPC and these extensions are not strictly UPC conforming, because UPC is defined as a dialect of C and linking of a UPC program to a separately compiled C library must preserve the semantics of the C library. If the C library defines file scoped static variables, these will not be processed by "detect_upc", will not be annotated with the special UPC pragmas, and therefore will likely not work as expected when called from separate UPC threads.

A case in point is the 'rand' function. The rand function is not inherently thread reentrant. The Berkeley compiler and GUPC deal with this by defining rand as a preprocessor macro. For example, here is the GUPC definition.

1. ifdef UPC_USES_PTHREADS /* Pthreads implementation uses per thread random seed */
2. define rand upc_rand
3. define srand upc_srand extern int upc_rand (void); extern void upc_srand (unsigned int seed);
4. endif

This use of preprocessor macros fixes issues that relate to 'rand' and 'srand' when compiled in "pthreads mode", but is obviously not a general solution for the "C library problem".

There is some precedent for using macros as rewrites of references to values that are globally scoped but that require pthreads-specific persistence -- the 'errno' variable for example. There is an open problem however with rewriting global names into pthread-specific values: a thourogh analysis of the POSIX standard would be required to ensure that most of the relevant system-defined entities are rewritten.

For two current implementations of UPC threads as OS supported threads (Berkeley UPC and GNU UPC) there is one particularly notable lack of per-pthread instantiation: stdo and its predefined file handles: stdin, stdout and stderr. Thus, if individual UPC threads attempt to re-define the mapping of these file handles, the current pthreads based implementations will not behave as expected.

```

Reported by `gary.funck` on 2012-05-21 21:51:27

[Former user]

``` Berkeley UPC has historically supported UPC-threads-as-pthreads as the most effective way to make use of a multi-core (originally just "smp") compute node. Getting a default behavior that works 98% of the time has been (and continues to be) a non-trivial amount of work. The fact that it works as well as it does often surprises me. We are actively trying to improve the portability of our intra-node shared memory support in each new release to eventually render the pthreads-based implementation irrelevant.

Based on our experience implementing it, I don't see a manner in which Berkeley UPC can resolve (in an automatic may) the remaining "C library problems" or related problems that come from linking to separately-compiled C codes. HOWEVER, I would not go so far as to make the claim that it is not possible to do (especially in the case where the C and UPC compilers are under common control).

So, I think it is premature for the language spec to address this question.

I know IBM's xlupc has a pthreads mode as well. So, perhaps Gheorghe would like to comment on its "compliance"? ```

Reported by `[e-mail]` on 2012-05-21 23:57:30

[Former user]

```

These pragmas and the implementation of UPC threads as OS supported threads are by

their nature extensions to UPC and these extensions are not strictly UPC conforming,

because UPC is defined as a dialect of C and linking of a UPC program to a separately compiled C library must preserve the semantics of the C library.

I disagree with this statement for several reasons. First, pragmas are by their nature a hook for implementation-defined behavior, as defined by ISO C 6.5.10: A preprocessing directive of the form: # pragma pp-tokensopt new-line ... causes the implementation to behave in an implementation-defined manner. ... Any such pragma that is not recognized by the implementation is ignored.

Syntactically pragmas are not an extension to UPC, they are standard-conformant C99. They are by their very nature implementation-dependent, and therefore would be inappropriate for inclusion in the language specification (at least in that syntactic form).

In regards to the claim that a UPC-over-pthreads compiler is somehow non-conformant because it can produce unexpected behavior when its output is linked with library objects built using a *different* compiler, that simply makes no sense. The C99 and UPC specs say nothing at all about the output ABI format or interoperability between conforming implementations. Indeed there are several examples where two C99-compliant C compilers on the same architecture produce objects that don't work correctly together. All of the issues mentioned above are a result of such "mixing" where some objects were compiled by a UPC compiler for a pthreads "ABI" and others were compiled by a C compiler for a serial ABI, which is technically erroneous and all bets are off in regards to spec conformance.

All that being said, it would be *nice* if separately-compiled C libraries simply worked as expected when linked with BUPC-compiled pthreads objects, and we strive to make that work when possible, especially in the case of the C99 libraries which are technically "part" of BUPC. However any failing in that effort does not represent a non-conformance, nor does it seem to motivate any change to the language specification. In the extreme case users should re-compile their libraries using entirely one compiler (UPC) and get the expected behavior.

```

Reported by `danbonachea` on 2012-05-31 07:59:37

[Former user]

``` Issue 23 has been merged into this issue. ```

Reported by `danbonachea` on 2012-05-31 08:10:44

[Former user]

``` The motivation for breaking this related group of open UPC language issues into separate tickets was to make it possible to review each (related) ticket on its own. Now that they have been merged into this single issue, I have changed the subject to the core topic vis-a-vis the UPC language specification update process.

The motivation for this topic was to highlight that UPC-threads-as-OS-threads implementations are extensions to UPC in that at least two current implementations use one or more of the following techniques: (1) A special front-end pre-processing tool that intersperses

1. pragma's which de-mark "C" code that requires special handling of file scope static variables, (2) #define's which rewrite to specific "C" library routines into calls to UPC runtime implemented routines that ensure that the routines are pthread-reentrant (3) detect declarations of file scope static data within "system headers" (generally, those included via <...>) and re-write those declarations into per-threads (Threads Local Storage, or TLS) data declarations.

It is important to highlight both the fact that UPC-threads-as-OS-threads are implementation-defined extensions and that there are corner cases that will not work as a UPC programmer might expect, if the programmer is unaware of the mechanisms used to achieve UPC-threads-as-pthreads and the limitations of those mechanisms.

Most UPC programmers/users are unaware of the underlying mechanisms used to achieve UPC-threads-as-pthreads because the UPC compilers do a good job of masking any inherent limitations. As Paul indicated, it can be surprising how well these implementation-defined extensions work in practice.

I agree that the choice of the word "conformant" in the original subject line (and related issues) was incorrect. Perhaps better, would have been "clarification: pthreads based implementations are implementation-defined extensions".

The questions and point of the related proposals is this:

1. Is there agreement that UPC-threads-as-OS-threads (in this context "OS threads", "POSIX threads" and "pthreads" are used interchangeably) are implementation-defined extensions to the current (1.2) version of the UPC language specification? [yes]

2. Should the UPC specification codify support for UPC-threads-as-OS-threads? [no]

3. Should the UPC specification define the special #pragma directives currently used by BUPC (and perhaps other compilers) to de-mark regions of "C" source code that require special handling? [no]

I have placed the proposed answers to the above questions in square brackets.

```

Reported by `gary.funck` on 2012-05-31 14:59:41

[Former user]

``` Gary Wrote:

start quote

The questions and point of the related proposals is this:

1. Is there agreement that UPC-threads-as-OS-threads (in this context "OS threads", "POSIX threads" and "pthreads" are used interchangeably) are implementation-defined extensions to the current (1.2) version of the UPC language specification? [yes]

2. Should the UPC specification codify support for UPC-threads-as-OS-threads? [no]

3. Should the UPC specification define the special #pragma directives currently used by BUPC (and perhaps other compilers) to de-mark regions of "C" source code that require special handling? [no]

I have placed the proposed answers to the above questions in square brackets. <<<end quote

I think I agree: 1. pthreads is EITHER an extension or an implementation "mechanism" - either way it should not appear in the spec 2. spec should NOT codify anything regarding pthreads as UPC threads (neither support NOR prohibit) 3. spec should NOT codify the pragmas

So, I believe that with regards to the spec: NO SPEC CHANGES ARE REQUIRED / DESIRED Agree or disagree?

-Paul ```

Reported by `[e-mail]` on 2012-05-31 16:43:29

[Former user]

``` Gary and I seem to disagree as to whether pthreads-based implementations are standards-conformant.

Either way there seems to be agreement that any explanatory matter about interoperability limitations when linking objects compiled with the system C compiler does not belong in the UPC spec and instead belongs in the implementation documentation (which BUPC provides, in great length).

Closing this issue. ```

Reported by `danbonachea` on 2012-06-14 22:43:53 - Status changed: `WontFix`

[Former user]

``` I agree that the spec should remain silent on implementation mechanisms. I don't want to reopen the issue, but just to state my opinion on the underlying question:

I am not 100% certain if Dan or Gary is correct for any existing instance of UPC-threads-as-pthreads. I suspect that despite our best efforts, a determined adversary could find some aspect of C99 to which Berkeley UPC's pthreads-based implementation does not conform. However, we could/would then implement the necessary work-arounds. So, it is my belief that a phreads-based implementation COULD be made compliant given sufficient engineering efforts.

My view is that, at a minimum, every function defined in the ISO C99 standard library must behave as documented in that specification. There is an additional user expectation (NOT a requirement for conformance) that POSIX-defined library calls will behave are documented in that specification. I believe that for at least the stdio library defined in ISO C99, the Berkeley UPC implementation using pthreads does NOT behave the same as when using processes. However, I cannot off-hand point to differences that would be violations of the C99 spec. ```

Reported by `[e-mail]` on 2012-06-14 22:59:14

[Former user]

``` I agree that the issue should be closed, and that no UPC specification changes are required.

As noted above, my motivation for filing this issue (and two other related issue tickets) is to gain consensus that UPC-threads-as-pthreads is (as Paul stated) a vendor-specific extension or implementation artifact for which the UPC specification should not define/require specific pragams, constraints, or other mechanisms. There appears to be a consensus on that point.

In passing - my use of "conforming" above might have been more accurately stated as "strictly conforming". Per Section 4, par. 5 of the C99 specification:

A strictly conforming program shall use only those features of the language and library specified in this International Standard.2) It shall not produce output dependent on any unspecified, undefined, or implementation-defined behavior, and shall not exceed any minimum implementation limit.

Thus, to the degree that for correct operation a program requires specific pragmas that are not called out in the UPC specification (BUPC), or specific treatment of "system header" files (GUPC) in order for otherwise standard conforming UPC programs to generate correct, expected, results the program is not strictly standard conforming.

```

Reported by `gary.funck` on 2012-06-14 23:17:23

[Former user]

``` Mass update to closed issue status ```

Reported by `danbonachea` on 2012-06-15 18:40:08 - Status changed: `Rejected`

[Former user]

Reported by `gary.funck` on 2012-07-06 21:04:02 - Labels added: Milestone-Spec-1.3