Output file from perl script in CCSID 1208 and this causes failure with subsequent CPYFRMSTMF command

[Keith Pryke]

Is there anyway you can control the CCSID of an output file from a perl script?

‌

As far as I can tell all files are written in the CCSID (1208) of the perl executable. We previously ran a GNU (?) version of perl on another IBM i server and that executable was CCSID 819 and the output files from there were in CCSID 819.

‌

As part of our software build process we transfer files from a windows server to the IBM i IFS. These files are CCSID 819. Some files are directly copied to physical files; a mixture of data and source types, without any other processing before the CPYFRMSTMF command is issued, and these copy across OK.

‌

We have one file which we process via a perl script that only writes out “header” information to a new file in the IFS and we issue the CPYFRMSTMF command from there. This file is created by the perl script in CCSID 1208 and the CPYFRMSTMF command fails. I’ve reviewed the output file from the perl script and cannot find any characters that would prevent the command from working; i.e. all rows contain valid text characters. Issuing a CHGATR command before issuing the CPYFRMSTMF command to change the CCSID from 1208 to 819 enables the CPYFRMSTMF to behave as expected.

‌

I previously raised this as a PMR with IBM Support. If you have access to that the PMR reference is PMR 77549,999,866. They suggested raising an issue with the perl we are now running - the open source one from Access Client Solutions.

[Kevin Adler]

Hi Keith, this issue doesn’t really have anything to do with Perl.

Perl runs in PASE, just like most other open source software nowadays (Node.js, Python, Java, etc). All files created when running in PASE will be created using the PASE CCSID for that job. The PASE CCSID is set at PASE startup time based on the ILE job LANGID and CNTRYID values. These values are looked up in a table and the equivalent PASE locale and CCSIDs are set. For many locales, this was a single byte CCSID such as 819, 912, 913, etc. while other locales used 1208 (UTF-8).

In 7.4, to better enhance compatibility with modern programs, the default PASE CCSID for all locales was changed to 1208 (UTF-8). This is why you’re seeing a difference between 7.2 and 7.4 (nothing to do with Perl). You can go back to the old behavior by setting environment variable PASE_DEFAULT_UTF8=N before calling a PASE program via QSH, QP2TERM, etc… This was supposed to be documented in the 7.4 MTU, but due to an oversight it was unfortunately missed in the initial version at 7.4 GA (It has since been fixed - see section “Default PASE CCSID and Locale Changed to UTF-8”).

In your specific situation, using CPYFRMSTMF to a CCSID 65535 and wanting conversion is problematic, because 65535 is not a valid conversion target. However, I believe in this case for single-byte ASCII CCSIDs, there’s special behavior that uses a default mapping to the equivalent single byte EBCDIC CCSID that is used. So CPYFRMSMTF 819 → 65535 becomes 819 → 37. However, there is no default mapping for UTF-8, since there is no single byte (or doublebyte) EBCDIC CCSID which can map all the Unicode code points.

‌

You have a couple options:

1. Use the workaround to use previous PASE CCSID behavior noted above
2. Use your current workaround of CHGATR to change the CCSID to 819 once it’s created
3. Set the target CCSID explicitly instead of using DBFCCSID(*FILE), eg. `DBFCCSID(37)`

I’m sorry you did not get routed to the correct team to solve your problem via the PMR as the customer support team should have been able to give you all of this info. I will follow up with them for the future.

[Keith Pryke]

Thanks Kevin. I’ve set PASE_DEFAULT_UTF8=N as a *SYS environment variable and that works.

‌

I couldn’t dictate the CCSID of the target file since this is just a CRTPF FILE(MYLIB/MYFILE) RCDLEN(132) where the CCSID is always 65535.

[Kevin Adler]

I couldn’t dictate the CCSID of the target file since this is just a CRTPF FILE(MYLIB/MYFILE) RCDLEN(132) where the CCSID is always 65535.

Right, but you can dictate which target CCSID is used on CPYFRMSTMF, though.