ckermit90.txt 95 KB

  1. [1]The Columbia Crown The Kermit Project | Columbia University
  2. 612 West 115th Street, New York NY 10025 USA o [2]
  3. ...since 1981
  4. [3]Home [4]Kermit 95 [5]C-Kermit [6]Scripts [7]Current [8]New [9]FAQ
  5. [10]Support
  6. C-Kermit 9.0 Update Notes
  7. Note: C-Kermit 9.0.301 contains a correction that applies only to
  8. Solaris 10 and 11.
  9. C-Kermit 9.0.302 contains corrections that apply only to FreeBSD 8
  10. and 9.
  11. * [15]Large Files
  12. * [16]How to Test Large-File Transfer
  13. * [17]Arithmetic with Large Integers
  14. * [18]FORCE-3 Packet Protocol
  15. * [19]Variable Evaluation
  16. * [20]The RENAME Command You Always Wanted
  17. * [21]Other New Features
  18. * [22]Incompatibilities
  19. * [23]What's Not In C-Kermit 9.0
  20. * [24]And a Loose End
  21. * [25]Demonstration: Secure POP mail fetcher
  22. * [26]Demonstration: HP Switch Configuration Backup
  23. * [27]Demonstration: HP iLO Blade Configuration
  24. * [28]Demonstration: IBM/Rolm/Siemens CBX Management
  25. * [29]Demonstration: CSV and TSV Files
  26. * [30]Demonstration Scripts for Webmasters
  27. This is the third supplement to [31]Using C-Kermit, Second Edition. I
  28. apologize for the scattered nature of the information and I hope I can
  29. organize it and gather it all into one place for easy and definitive
  30. reference some day. It's a big job so it depends on the demand. For the
  31. time being the definitive reference and introduction is the book (which
  32. is now available also in a [32]Kindle Edition), plus the [33]C-Kermit
  33. 7.0 update, [34]C-Kermit 8.0 update, and now this one. Plus tons of
  34. other web pages on this site, sample script programs, and so on.
  35. In version 6.0, C-Kermit was a pretty powerful and flexible
  36. communication program with scripting capabilities. By version 9.0, I'd
  37. like to think of it more as a scripting language with built-in
  38. communications. You can get an idea of the kinds of programs you can
  39. write in Kermit language [35]here. You can develop programs quickly
  40. because it's an interactive program, not a compiler. The scripting
  41. language is the command language. Kind of like the Unix shell but
  42. "somewhat" less cryptic, including concepts not only from C but from
  43. PL/I, Snobol, LISP, Bliss, and Smalltalk. The language itself is built
  44. upon the command language of the much-loved [36]DECSYSTEM-20 from the
  45. 1970s and 80s, the Clipper Ship of the Text Era. (Text is not a bad
  46. word. Those of us who can touch-type and who are proficient in
  47. text-based computing environments like Unix shell or VMS DCL are likely
  48. to be orders of magnitude more productive than users of GUIs.)
  49. Thanks to (at least) Jeff Altman, William Bader, Ian Beckwith, Nelson
  50. Beebe, Gerry Belanger, Joop Boonen, Rob Brown, Christian Corti, Alexey
  51. Dokuchaev, John Dunlap, Peter Eichhorn, Carl Friedberg, Terry Kennedy,
  52. Günter Knauf, Jason Lehr, Arthur Marsh, Lewis McCarthy, Gary Mills, Ed
  53. Ravin, Jonathan Reams, Mike Rechtman, Mark Sapiro, Steven Schweda
  54. (SMS), Kinjal Shah, Michael Sokolov, Andy Tanenbaum, Seth Theriault,
  55. Zach A. Thomas, Martin Vorländer, and Eric Weaver for assistance, and
  56. to Hewlett-Packard Company for support.
  57. - Frank da Cruz [37], 30 June 2011
  58. P.S. It occurred to me just before the end of the day that maybe I
  59. should back up the Kermit website on DVD, just in case. Using
  60. [38]Kermit 95 on the desktop over an SSH connection to the Unix file
  61. system where the website resides, I made a fresh directory on the PC,
  62. CD'd to it, and on Unix cd'd to the Website directory, and told
  63. C-Kermit 9.0 to:
  64. C-Kermit> send /recursive /dotfiles /nobackup *
  65. and it re-created the website directory tree in the PC directory, text
  66. files correctly converted to Windows format and binary files correctly
  67. left as-is. The /dotfiles switch means to include files such as
  68. .htaccess whose names start with a dot (period), and the /nobackup
  69. switch means to skip backup files created by EMACs (such as
  70. index.html.~243~). And then I did the same with the FTP sites, about
  71. 8GB in all. Watching the file-transfer display was kind of like having
  72. 30 years of my life flash before my eyes in a few minutes. Then I
  73. copied the two directories to DVD (the FTP site had to be split over 2
  74. DVDs). The whole operation took under half an hour. The directory tree
  75. on the CD is directly usable in Windows, Unix, or any other operating
  76. system (unlike if I had transferred the files all in binary mode or all
  77. in text mode, or if I had made, say, a gzipped tar archive or a zip
  78. archive). I believe that, to this day, Kermit is the only software that
  79. can do this. If someday I have to upload from these DVDs to Unix, VMS,
  80. or any other operating system, it can be done exactly the same way,
  81. with any necessary conversions on text files done automatically, and
  82. binary files left intact, recursively through a whole very large
  83. directory tree.
  84. What's New in General
  85. Very briefly, the major items:
  86. * [39]Open Source license.
  87. * [40]64-bit file access and transfer and 64-bit integer arithmetic
  88. on most common platforms.
  89. * Support for recent releases of Linux, Mac OS X, *BSD, etc ([41]see
  90. table).
  91. * Support for newer OpenSSL releases up to and including 1.0.0d
  92. ([42]see table).
  93. * [43]Strengthened error checking for file transfer under extremely
  94. harsh conditions.
  95. * [44]Simplified semantics for variables used in scripts.
  96. * Super-handy [45]extensions to the RENAME command.
  97. * Other scripting improvements including support for reading and
  98. writing [46]CSV and TSV files.
  99. * [47]MIME character-set names are now recognized.
  100. * Improved logging and debugging (see demo [48]here).
  101. * Lots more described or listed below, and [49]here.
  102. Open Source License
  103. C-Kermit 9.0 has the [50]Revised 3-Clause BSD License, an open source
  104. license approved by OSI, the [51]Open Source Initiative.
  105. Large Files
  106. Kermit is, first and foremost, a file-transfer program. One might
  107. expect it to be able to transfer any kind of file, but that has been
  108. decreasingly the case as file sizes began to cross the 2 gigabyte
  109. threshold.
  110. The biggest change since C-Kermit 8.0.211 is support for large files on
  111. platforms that support them. A "large file" is one whose size is
  112. greater than 2^31-1 (2,147,483,647) bytes (2GB-1); that is, one whose
  113. size requires more than 31 bits to represent. Before now, Kermit was
  114. able to access such files only on 100% 64-bit platforms such as Digital
  115. Unix, later known as Tru64 Unix. In the new release, Kermit takes
  116. advantage of the X/Open Single UNIX Specification Version 2 (UNIX 98)
  117. Large File Support (LFS) specification, which allows 32-bit platforms
  118. to create, access, and manage files larger than 2GB.
  119. Accommodating large files required code changes in many modules,
  120. affecting not only file transfer, but also file management functions
  121. from directory listings to local file manipulation, plus the user
  122. interface itself to allow entry and display of large numbers. All this
  123. had to be done in a way that would not affect pure 32-bit builds on
  124. platforms that do not support large files. Large file support is
  125. summarized in the [52]Table of Platforms; entries in Yellow (32-bit
  126. builds that support 64-bit integers) and Green (64-bit builds) support
  127. large files.
  128. Note that VMS C-Kermit and Kermit 95 for Windows have always been able
  129. to transfer large files. However their user interface used 32-bit
  130. integers for statistics and the file transfer display. In C-Kermit 9.0
  131. Alpha.03, VMS C-Kermit on 64-bit platforms (Alpha and Itanium) should
  132. now give correct statistics and progress displays. (We'll see about
  133. Kermit 95 later.)
  134. How to Test Large-File Transfer
  135. Several methods are available for testing large-file transfers:
  136. * By transferring a real file that is more than 2147483648 bytes long
  137. (a file whose length requires more than 31 bits to express); or to
  138. be totally sure, that is longer than 4294967296 bytes (32 bits or
  139. more). Or to be double super sure, longer than 8589934592 (33
  140. bits).
  141. * If you don't have such a file or there is not sufficient disk space
  142. for such a file, you can create a special kind of file that takes
  143. up one block on the disk but appears to be 4.3GB long by compiling
  144. and running [53]THIS C PROGRAM on Linux, Solaris, HP-UX, or other
  145. Unix platform that supports large files. Kermit or FTP or any other
  146. file transfer program will transfer the result (BIGFILE) in such a
  147. way as to actually put 4.3GB (or other desired size; see source) on
  148. the wire.
  149. * You can use Kermit's CALIBRATE feature to transfer a large file
  150. that doesn't exist. At the receiver, use RECEIVE /CALIBRATE. At the
  151. sender, use SEND /CALIBRATE:length, e.g.:
  152. (At remote kermit...)
  153. $ kermit -Y
  154. C-Kermit> receive /calibrate
  155. (Return to local kermit...)
  156. Ctrl-\c
  157. C-Kermit> send /calibrate:4300000000
  158. This sends a simulated file 4.3GB in length, that does not exist on
  159. the sender and will not take up any disk space on the receiver.
  160. SEND /CALIBRATE: accepts big numbers only in Kermit versions that
  161. support them (this does not include Kermit 95 on Windows). This
  162. method tests only Kermit's ability to express and understand large
  163. file sizes, but does not test Kermit's file-system interface, since
  164. no files are involved.
  165. Arithmetic with Large Integers
  166. Because large file support requires the availability of a 64-bit signed
  167. integer data type, other aspects of C-Kermit were adapted to use it
  168. too, most notably Kermit's algebraic expression evaluator and its
  169. [54]S-Expression interpreter, on all platforms that support large files
  170. (those listed as 64 or 32/64 in the Word column of the [55]table). In
  171. fact, every Kermit command that parses a number in any field can now
  172. parse a large number on those platforms.
  173. S-Expressions can now be forced to operate with integers only, without
  174. floating-point conversion or having to explicitly truncate each result;
  175. as an example. see the revised [56]Easter date calculation script.
  176. FORCE-3 Packet Protocol
  177. The Kermit protocol has proven itself over the past 30 years to be
  178. robust in terms of surviving harsh transmission environments and
  179. delivering the data correctly and completely. In these times of
  180. Internet everywhere and error-correcting modems in the few places where
  181. the Internet isn't, few people even recall the kinds of difficult
  182. conditions that were common when the Kermit protocol was first
  183. developed: noisy telephone lines, serial interfaces that drop
  184. characters, lack of transparency to control or 8-bit characters,
  185. absence of flow control, "bare" modems without error correction.
  186. But the Internet is not everywhere, and not all modems are
  187. error-correcting. Perhaps the most difficult trial so far for Kermit or
  188. any other protocol is the [57]EM-APEX project, in which floats are
  189. dropped into the ocean from an aircraft into the path of a hurricane;
  190. these floats dive into the water measuring current, temperature, and
  191. salinity at different depths and then surface to phone home, sending
  192. the data to land stations using Kermit protocol over
  193. non-error-correcting 300bps [58]Iridium satellite modems, with high
  194. seas and winds battering the floats and heavy ([59]sometimes
  195. electrical) storms between the modem and the satellite.
  196. Because of the transmission speed and long distances involved, the
  197. transfers were very slow. The Kermit software in the floats is
  198. [60]Embedded Kermit, which did not implement sliding windows, which
  199. would have sped up the flow considerably. John Dunlap, engineer at the
  200. University of Washington's Applied Physics Laboratory, undertook the
  201. task of adding sliding windows to E-Kermit. For testing, he rigged up a
  202. [61]simulator in which Kermit transfers take place over a connection
  203. with different amounts of noise and delay. He found that occasionally,
  204. a transfer would appear to succeed, but the received file would be
  205. corrupt.
  206. According to the Kermit protocol definition, the first packet always
  207. has block-check type 1, a 6-bit checksum, which is the only block check
  208. type that all Kermit implementations are required to support; thus any
  209. Kermit partner can process this packet. This packet itself can
  210. negotiate a higher level of checking, such that subsequent packets have
  211. (say) block-check type 3, a 16-bit cyclic redundancy check (CRC)
  212. encoded as three printable 7-bit ASCII characters. The 16-bit CRC can
  213. catch all errors of certain kinds (single-bit, double-bit, bursts of 16
  214. bits or less), and more than 99.9984741210937% of all other possible
  215. errors.
  216. John's simulations revealed that file corruption could occur undetected
  217. when the initial packet was corrupted in such a way that a parameter or
  218. capability byte was changed and the checksum also changed to make the
  219. packet appear to be correct, thus allowing the transfer to proceed with
  220. the two Kermit partners out of sync as to packet encoding and
  221. interpretation (the chances of two such errors producing a seemingly
  222. valid packet are about 1 in 6000 when using the 6-bit checksum). For
  223. example, the compression technique might be misnegotiated and then the
  224. receiver might store incoming data without decompressing it.
  225. The solution is a new option, selected by:
  227. to require a type 3 block check (16-bit CRC) on every packet, including
  228. the initial ones, thus reducing the probability of a misnegotiation by
  229. many orders of magnitude. THIS PARAMETER CAN NOT BE NEGOTIATED. Each
  230. Kermit program must be given the "set block 5" command prior to
  231. transfer. That's because normally every Kermit program expects the
  232. first packet to have a 6-bit checksum, and if the first packet has a
  233. 3-byte, 16-bit CRC, the packet receiver will think it is corrupted.
  234. In practice, however, it is possible to code the packet receiver
  235. "cheat" by reading the packet data before verifying the block check.
  236. Thus when the receiver is C-Kermit 9.0 or later or E-Kermit 1.7 or
  237. later, it is only necessary to give the "set block 5" command to the
  238. file sender, and the receiver will check for a FORCE-3 first packet. If
  239. the receiver does not support this feature, however, the initial packet
  240. will be be rejected (after several retries) and the file transfer will
  241. not take place. There is no attempt to "back off" to normal behavior.
  242. CAPTION: Table 4. Kermit Protocol Packet Block Check Types
  243. Type Command Bytes Status Explanation
  244. 1 SET BLOCK 1 1 Required in all Kermit implementations. Negotiated.
  245. 6-bit checksum, suitable for good connections.
  246. 2 SET BLOCK 2 2 Optional, negotiated. 12-bit checksum. 64 times
  247. stronger than type 1.
  248. 3 SET BLOCK 3 3 Optional, negotiated. 16-bit CRC.
  249. BLANK-FREE-2 SET BLOCK 4 2 Optional, negotiated. 12-bit checksum, two
  250. nonblank bytes.
  251. FORCE-3 SET BLOCK 5 3 Optional, not negotiated. 16-bit CRC forced all
  252. packets.
  253. BLANK-FREE-2 is for environments where Kermit packets are treated as
  254. lines of text, and in which trailing blanks can be stripped; for
  255. example, when transferring files with an IBM mainframe through a 3270
  256. protocol converter.
  257. [62]E-Kermit 1.7
  258. Variable Evaluation
  259. Does the strange behavior of Kermit's \%x variables puzzle or annoy
  260. you?
  261. Kermit software development has been a collaborative project over the
  262. years, with contributions coming in from almost every country and every
  263. sector of the economy - academic, corporate, government. Thus not all
  264. versions, and not all features of a given version, are a product of
  265. systematic design.
  266. One example was the introduction of variables for text substitution,
  267. first in a version of MS-DOS Kermit that was sent in by someone
  268. somewhere (I could look it up, but no time...) Although the design of
  269. the notation for variable names (table below) is mine, the underlying
  270. code was contributed. In that code there was only one kind of variable,
  271. and if I recall correctly the variable name was a backslash followed by
  272. a single letter, for example \a, \b, etc. The contributed code
  273. evaluated these variables recursively, meaning if the definition of a
  274. variable contained variable references, then these were resolved when
  275. dereferencing the variable, and the process would continue as deep down
  276. as necessary to resolve the thing fully.
  277. This was sometimes handy, but it had one severe drawback: There was no
  278. way to use variables in a straightforward way to represent strings that
  279. contained literal backslashes; for example, DOS or Windows pathnames.
  280. This gave rise to all kinds of quoting rules and conventions (e.g.
  281. doubling backslashes or forcing single-level evaluation with
  282. \\fcontents()), and also to the introduction of other kinds of
  283. variables that were evaluated one level deep, rather than recursively.
  284. To accommodate coexistence of different kinds of variables as well as
  285. "escape sequences" for representing control and 8-bit characters, the
  286. syntax for variable names was extended to include three elements: the
  287. leading backslash, then a single character indicating the type of
  288. variable, and then the name of the variable in a format corresponding
  289. to the type designator, as shown in this somewhat simplified table:
  290. CAPTION: Table 1. Variable-name Syntax in Kermit
  291. Notation Meaning
  292. \000 - \255 8-bit character constant (decimal)
  293. \d000 - \d255 Alternative notation for 8-bit character (byte) constant
  294. (decimal)
  295. \o000 - \o377 8-bit character constant (octal)
  296. \x00 - \xff 8-bit character constant (hexadecimal)
  297. \%a - \%z Scalar variable, evaluated recursively.
  298. \%0 - \%9 Macro argument, scalar, evaluated recursively.
  299. \&a - \%& Array name
  300. \&a[x] Array reference, evaluated recursively (x is any constant or
  301. variable)
  302. \v(name) Built-in scalar variable, evaluated one level deep.
  303. \m(name) User-defined scalar variable, evaluated one level deep.
  304. \$(name) An environment variable, evaluated one level deep.
  305. \s(name[n:m]) Compact substring notation, evaluated one level deep.
  306. \fname(args...) Built-in function with zero or more arguments.
  307. \\ Literal backslash
  308. \N OUTPUT command only: NUL, ASCII 0
  309. \B OUTPUT command only: BREAK (250ms, for serial connections)
  310. \L OUTPUT command only: Long BREAK (1.5sec, ditto)
  311. Variable names in Kermit are case-independent. The simplifications in
  312. the table are that the notation for decimal and octal bytes can have
  313. from one to three digits, and can include braces to separate them from
  314. text digits, e.g. \7, \{123}, \o{50}. Hex bytes too, except they must
  315. always have exactly two hex digits, 0-9a-f. Array indices must be, or
  316. must evaluate to, numbers (floating point numbers are truncated).
  317. Associative arrays are also available (dynamic arrays with arbitrary
  318. text as subscript), but they are really just a variation on \m()
  319. variables (read about associative arrays [63]here). Also, there are
  320. some alternative notations for compact substring notation.
  321. We didn't want to have lots of "distinguished" characters, as the UNIX
  322. shell does; one is enough, clarity over brevity. Although the notation
  323. can be a bit cumbersome, we can use the \m(name) form to circumvent the
  324. overevaluation in most contexts. But macro arguments are always
  325. assigned to the \%0-9 variables, and thus always evaluated recursively,
  326. making it difficult and confusing to pass (e.g.) Windows pathnames as
  327. arguments to macros. The same is true for array elements, especially in
  328. contexts where they are used to return results from built-in functions
  329. (for example, \fsplit() used to return the elements of a
  330. [64]comma-separated value list if any of the values contained
  331. backslashes). An even worse scenario is when macro arguments are passed
  332. from one macro to another; for some graphic illustrations see
  333. [65]Taming the Wild Backslash - Part Deux from the [66]C-Kermit 7.0
  334. Update Notes.
  335. We can't just change how variables are evaluated because that would
  336. break existing scripts. But we can always add Yet Another SET Command:
  338. This applies only to \%a-z and \%0-9 variables and to \&a-z[] arrays
  339. (since all other kinds of variables are evaluated only one level deep).
  340. The default, of course, for backwards compatibility, is RECURSIVE.
  341. SIMPLE forces the evaluation of these variables to return their literal
  342. contents, without further evaluation:
  343. * An exception is made in the case of array subscripts, because
  344. changing how they are evaluated could break a lot of scripts, and
  345. anyway there should never be any harm in evaluating them
  346. recursively because their final value is always (or should be)
  347. numeric, not some string that might contain backslashes.
  348. * The VARIABLE-EVALUATION setting is on the command stack. Thus you
  349. can give this command in a macro, command file, or user-defined
  350. function without affecting the calling environment.
  351. * The new \frecurse() function forces recursive evaluation of its
  352. argument regardless of the VARIABLE-EVALUATION setting. The
  353. argument can be any string (or nothing at all); all the variables
  354. in the string, even \m() ones, are evaluated recursively:
  355. def \%a 1 \%b 3
  356. def \%b 2
  357. def xx easy as \%a
  358. show mac xx
  359. echo \frecurse(\m(xx))
  360. easy as 1 2 3
  361. echo \frecurse(it's as easy as \m(xx))
  362. it's as easy as easy as 1 2 3
  363. * The new \v(vareval) built-in variable contains the current setting
  364. (recursive or simple) at the current command-stack level.
  365. Here's a short script for illustration:
  366. define path c:\users\fdc\somefile.txt
  367. define test1 { # Normal recursive argument evaluation
  368. echo \%0: arg=\%1
  369. }
  370. define test2 { # Simple argument evaluation
  371. set var simple
  372. echo \%0: arg=\%1
  373. }
  374. test1 \m(path)
  375. test2 \m(path)
  376. exit
  377. And here's the result:
  378. ?<ERROR:NO_SUCH_FUNCTION:\fdc\somefile.txt()>
  379. test2: arg=c:\users\fdc\somefile.txt
  380. The first line might seem surprising, but under the normal rules (see
  381. table above) \f indicates a function call, with the letters following
  382. the 'f' being the name of the function. But there is no function by
  383. that name... and if there were, you probably didn't intend to call it!
  384. SET COMMAND VARIABLE-EVALUATION SIMPLE has no effect on constants, only
  385. on variables. Note how \m(path) is defined. The DEFINE command assigns
  386. the literal value of its argument to the named variable (see Table 3
  387. below), thus in this case no special syntax is needed. But in other
  388. contexts, you must double the backslashes or use the \fliteral()
  389. function to use literal backslashes in data:
  390. test2 c:\\users\\fdc\\somefile.txt
  391. test2 \fliteral(c:\users\fdc\somefile.txt)
  392. C-Kermit 9.0 adds a new notation for \fliteral() which also has certain
  393. advantages over it: \q(string):
  394. test2 \q(c:\users\fdc\somefile.txt)
  395. Since \fliteral() is a function, its argument list (the text within
  396. parentheses) has special syntax of its own, in which commas and braces
  397. are treated specially and introduce another set of quoting problems.
  398. \q(string) doesn't have these problems. The only consideration is that
  399. parentheses must be balanced or else quoted (preceded by backslash), or
  400. represented as numeric character entities (left paren = \40, (right
  401. paren = \41).
  402. Or else hold the value in a simple variable as we did with \\m(path)
  403. above.
  404. SET COMMAND VARIABLE-EVALUATION SIMPLE is a big change and might have
  405. repercussions that didn't show up in the initial tests; a lot more
  406. testing is needed.
  407. On the topic of variables, let's summarize in one place the ways in
  408. which values can be explicitly assigned to variables. There is nothing
  409. new here except the table itself:
  410. CAPTION: Table 2. Variable Assignment in Kermit
  411. Command Shorthand Explanation
  412. DEFINE name value .name = value The literal value becomes the contents
  413. of the named variable; variables names in the value are copied without
  414. evaluation. This command is for defining macros that take parameters,
  415. as well as for defining simple variables, especially if the values
  416. contain backslashes.
  417. _DEFINE name value Like DEFINE but the name is evaluated before use.
  418. ASSIGN name value .name := value The value is evaluated and the result
  419. becomes the contents of the named variable.
  420. _ASSIGN name value Like ASSIGN but the name is evaluated before use.
  421. EVALUATE name expression .name ::= value The expression (in regular
  422. algebraic notation) is evaluated arithmetically and the result becomes
  423. the contents of the named variable. If the expression contains any
  424. variables they are evaluated first.
  425. _EVALUATE name expression Like EVALUATE but the name is evaluated
  426. before use.
  427. INCREMENT name expression Evaluates the variables in the expression,
  428. then evaluates the expression arithmetically, and then adds the value
  429. to the contents of the named variable, which must be a number or an
  430. algebraic expression. If the expression is empty, a value of 1 is used.
  431. _INCREMENT name expression Like INCREMENT but the name is evaluated
  432. before use.
  433. DECREMENT name expression Evaluates the variables in the expression,
  434. then evaluates the expression arithmetically, and then subtracts the
  435. value from the contents of the named variable, which must be a number
  436. or an algebraic expression. If the expression is empty, a value of 1 is
  437. used.
  438. _DECREMENT name expression Like DECREMENT but the name is evaluated
  439. before use.
  440. DECLARE name = list An array declaration can include an initializer
  441. list; items in the list are evaluated before assignment. This can be
  442. defeated by doubling any backslashes or enclosing individual arguments
  443. in \fliteral().
  444. DO name arguments name arguments When invoking a macro with a DO
  445. command (or an implied one), the arguments are evaluated, then assigned
  446. to \%1, \%2, etc, and the macro's name to \%0.
  447. (SETQ name value) Kermit also includes a mini-[67]LISP interpreter
  448. Variables are evaluated automatically in Kermit commands simply by
  449. referencing them, according to rules given in Table 1. The following
  450. functions can be used to change how a a particular variable is
  451. evaluated:
  452. CAPTION: Table 3. Kermit Functions for Evaluating Variables
  453. Function Argument Description
  454. \fcontents() \%x or \&x[y] Evaluates the variable or array element
  455. (which normally would be evaluated recursively) one level deep.
  456. \fdefinition() name If the argument is a \%x variable or an array
  457. element, it is evaluated to get the name; otherwise the argument is the
  458. name. Its definition is returned with no recursion.
  459. \m() name Equivalent to \fdefinition().
  460. \frecurse() \m(name) Forces recursive evaluation of a macro definition
  461. (a.k.a. long variable name). NOTE: \frecurse() can operate on any kind
  462. of variable as well as on any string containing any mixture of
  463. variables.
  464. C-Kermit's RENAME Command
  465. C-Kermit's RENAME command, which is used for changing the names of
  466. local files or for moving files locally, has two basic forms:
  467. RENAME [ optional-switches ] oldfilename newfilename
  468. This form lets you change the name of a single file from
  469. oldfilename to newfilename. Example:
  470. rename thismonth.log lastmonth.log
  471. RENAME [ optional-switches ] filespec directoryname
  472. This form lets you move (without renaming) one or more files
  473. (all the files that match the filespec, which may contain
  474. wildcard characters such as "*") to the given directory.
  475. Example:
  476. rename *.txt ~/textfiles/
  477. Traditionally, the optional switches have been:
  478. RENAME /LIST oldname newname
  479. Display the old and new name for each file while renaming.
  480. Synonyms: /LOG, /VERBOSE. Example:
  481. rename /list *.txt ~/textfiles/
  482. RENAME /NOLIST oldname newname
  483. Don't display the old and new name for each file while renaming.
  484. This is the default behavior. Synonyms: /NOLOG, /QUIET. Example:
  485. rename /nolist *.txt ~/textfiles/
  486. Reminder: Every switch starts with a slash (/) and must be preceded by
  487. a space.
  488. New RENAME Features for C-Kermit 9.0
  489. A series of new options (switches) have been added to let you change
  490. the names of multiple files at once by case conversion, string
  491. substitution, or character-set conversion, and optionally also move
  492. them to a different directory:
  493. /LOWER: Convert the filename to lowercase
  494. /UPPER: Convert the filename to uppercase
  495. /CONVERT: Change the filename's character encoding
  496. /REPLACE: Do string substitutions on the filename
  497. If the source-file specification includes a path or directory, any
  498. changes are applied to the filenames only, not to the directory or path
  499. specification.
  500. Since name changes, when applied to many files at once, can have
  501. consequences that are not easily undone, there are also some new
  502. controls, safeguards, and conveniences:
  504. This switch tells Kermit to show you what the RENAME command
  505. would do without actually doing it. /SIMULATE implies /LIST.
  507. This switch governs Kermit's behavior when renaming multiple
  508. files, and any of the names would collide with the name of a
  509. file that already exists. The default, for compatibility with
  510. earlier releases of C-Kermit, is OVERWRITE, i.e. write over the
  511. existing file. The other two protect existing files. SKIP means
  512. to skip (not rename) the file that would cause the collision,
  513. and proceed to the next file, if any. FAIL means that no files
  514. will be renamed if there would be any collisions; for this
  515. Kermit makes two passes, checking each new name it constructs
  516. for existence before starting the second pass (however, there is
  517. no guarantee that in the second pass, it won't create the same
  518. new name for more than one file; in that case, it will stop
  519. before executing the second rename). Example:
  520. rename /simulate /collision:proceed * ~/tmp/
  521. Reminder: In switches such as /COLLISION that take arguments
  522. (operands), the switch name and its argument(s) are separated by a
  523. colon (:) with no intervening spaces. Also remember that Kermit
  524. keywords can always be abbreviated by leaving off characters from the
  525. right, as long as the result is still unique in its context. Thus "ren
  526. /col:f" would be equivalent to "rename /collision:fail".
  527. You can change the following preferences for the RENAME command with
  528. the new SET RENAME command:
  530. Tells the RENAME command whether to list its actions if you
  531. don't include a /LIST or /NOLIST or equivalent switch.
  533. Tells the RENAME command how to handle filename collisions in
  534. the absence of a /COLLISION switch. That is, it replaces the
  535. default action of OVERWRITE with action of your choosing, which
  536. is then used in any RENAME command that does not include an
  537. explicit /COLLISION switch.
  539. Displays the current SET RENAME settings.
  540. Changing the Case of Filenames
  541. RENAME /UPPER:{ALL,LOWER} filespec [ directory ]
  542. RENAME /LOWER:{ALL,UPPER} filespec [ directory ]
  543. These switches let you change the alphabetic case of letters in
  544. all the files whose names match the filespec. If a directory
  545. name is given after the filespec, then the files are also moved
  546. to the given directory.
  547. By default, all files that match the given filespec have their names
  548. changed (if necessary). This is what the ALL argument means, e.g.:
  550. RENAME /LOWER *
  551. You can use either form: RENAME /LOWER is equivalent to RENAME
  552. /LOWER:ALL. The other argument (/LOWER:UPPER or /UPPER:LOWER) means to
  553. leave mixed-case filenames alone, and rename only those files whose
  554. names contain letters of only the given case. Examples:
  556. Changes the filename to FOO.BAR.
  558. Same as "rename /upper:all".
  559. RENAME /UPPER ~/old/
  560. Renames to FOO.BAR and moves it to the user's old
  561. directory (Unix).
  562. RENAME /LOWER *
  563. Changes the names of all files to have only lowercase letters.
  565. Changes the names of only those files whose names contain no
  566. lowercase letters to have only lowercase letters. For example,
  567. FOO.BAR would be changed, Foo.Bar would not be changed.
  568. would not be changed either because it's already all lowercase.
  569. RENAME /LOWER:UPPER * ~/new/
  570. Same as the previous example, but also moves each file to the
  571. user's new directory (whether it was renamed or not).
  572. Case conversion works reliably for ASCII characters only. Kermit uses
  573. the C library for this, which on any given platform might or might not
  574. handle non-ASCII letters, and if it does, then how it works would
  575. normally depend on your locale definitions (the LC_CTYPE and/or LANG
  576. environment variable in Unix). When non-ASCII letters are not handled
  577. by the C library, the RENAME command does change their case. For
  578. example, Olga_Tañón.txt might become OLGA_TAñóN.TXT.
  579. String Replacement in Filenames
  580. The RENAME command also lets you change filenames by string
  581. substitution.
  582. RENAME /FIXSPACES[:String] filespec [ directory ]
  583. Replaces all spaces in each matching filename by the given
  584. string, if any, or if none is given, by underscore. Examples:
  585. RENAME /FIX *
  588. RENAME /FIXSPACES:<040> *
  589. The first two are equivalent, replacing each space with
  590. underscore; a file called "My Favorite Photo.jpg" becomes
  591. "My_Favorite_Photo.jpg". The third example removes all spaces
  592. ("MyFavoritePhoto.jpg"). The fourth replaces each space with the
  593. string "<040>" ("My<040>Favorite<040>Photo.jpg").
  594. RENAME /REPLACE:{{String1}{String2}} filespec [ directory ]
  595. Renames each matching file by changing occurrences of String1 in
  596. its name to String2. If a directory specification is included,
  597. the file is also moved to the given directory (even if the name
  598. was not changed). Note that in this case, the curly braces are
  599. part of the command. Example:
  600. RENAME /REPLACE:{{.jpeg}{.jpg}} *
  601. changes all *.jpeg files to *.jpg.
  602. By default, RENAME /REPLACE changes all occurrences of String1 in each
  603. filename to String2 so, for example, if you had a file called
  604. abcjpegxyz.jpeg, the command just shown would change its name to
  605. abcjpgxyz.jpg.
  606. For greater control and flexibility, the /REPLACE: switch argument can
  607. take several distinct forms:
  608. RENAME /REPLACE:String1 filespec [ directory ]
  609. This means to remove all occurrences of String1 from the given
  610. filenames name. It is equivalent to /REPLACE:{{String1}{}}. A
  611. handy use for this option is to remove spaces from filenames.
  613. As already noted, this replaces every occurrence of String1 with
  614. String2 in each filename. Alphabetic case in string matching is
  615. done according to the current SET CASE setting.
  616. RENAME /REPLACE:{{ }{_}} filespec [ directory ]
  617. This replaces all spaces in the given filenames with underscore,
  618. equivalent to RENAME /FIXSPACES.
  619. RENAME /REPLACE:{{String1}{String2}{Options}} filespec [ directory ]
  620. Options can be included that add more control to the process.
  621. The option string is a sequence of characters; each character in
  622. the string is an option. The choices are:
  623. A String matching is to be case-sensitive, regardless of SET CASE.
  624. a String matching is to be case-independent, regardless of SET CASE.
  625. ^ String replacement will occur only at the beginning of the filename.
  626. $ String replacement will occur only at the end of the filename.
  627. 1 Only the first occurrence of the string will be replaced.
  628. 2 Only the second occurrence of the string will be replaced.
  629. 3 4 5 6 7 8 ...
  630. 9 Only the ninth occurrence of the string will be replaced.
  631. - (hyphen, minus sign) Before a digit: occurrences will be counted from
  632. the right.
  633. ~ (tilde) Before digit or minus sign: all occurrences but the given one
  634. will be replaced.
  635. The tilde modifier works only with single-byte character sets such as
  636. ASCII, CP437, ISO 8859-1, etc, but not with multibyte character sets
  637. such as UCS2, UTF8, or any of the Japanese Kanji sets.
  638. Here are some examples showing how to use the /REPLACE options:
  639. RENAME /REPLACE:{{foo}{bar}{^}} *
  640. For all files whose names start with "foo", replaces the "foo"
  641. at the beginning with "bar".
  642. RENAME /REPLACE:{{}{New-}{^}} *
  643. Prepends "New-" to the name of each file.
  644. RENAME /REPLACE:{{.jpeg}{.jpg}{$}} *
  645. Replaces ".jpeg" at the end of each filename with ".jpg".
  646. RENAME /REPLACE:{{}{-Old}{$}} *
  647. Appends "-Old" to the name of each file.
  648. RENAME /REPLACE:{{foo}{bar}{a}} *
  649. Replaces "foo", "FOO", "Foo", "fOO", etc, with "bar" in each
  650. filename.
  651. RENAME /REPLACE:{{foo}{bar}{A}} *
  652. Replaces only (lowercase) "foo" in filenames with "bar".
  653. RENAME /REPLACE:{{a}{XX}} *
  654. Changes every "a" to "XX". For example a file called "a.a.a.a"
  655. would become "XX.XX.XX.XX".
  656. RENAME /REPLACE:{{a}{X}{2}}
  657. Changes only the second "a" to "X". For example a file called
  658. "a.a.a.a" would become "a.X.a.a".
  659. RENAME /REPLACE:{{a}{X}{-1}}
  660. Changes only the final "a" in the filename (it doesn't have to
  661. be at the end) to "X". For example a file called "a.b.a.c.a.d"
  662. would become "a.b.a.c.X.d".
  663. RENAME /REPLACE:{{foo}{NOTFOO}{-2}}
  664. Changes the second-to-last "foo" (if any) in the filename to
  665. "NOTFOO".
  666. RENAME /REPLACE:{{foo}{}{-2}}
  667. Deletes the second-to-last "foo" (if any) from the filename.
  668. RENAME /REPLACE:{{.}{_}{~1}}
  669. Changes all but the first period to an underscore; for example,
  670. "a.b.c.d.e" would become "a.b_c_d_e".
  671. RENAME /REPLACE:{{.}{_}{~-1}}
  672. Changes all but the final period to an underscore; for example,
  673. "a.b.c.d.e" would become "a_b_c_d.e".
  674. In the Options field, digits (and their modifiers), ^, and $ are
  675. mutually exclusive. If you include more than one of these in the option
  676. string, only the last one is used. Similarly for 'a' and 'A':
  677. RENAME /REPLACE:{{foo}{bar}{Aa2$^}} *
  678. This replaces "foo" with "bar" no matter what combination of
  679. upper and lower case letters are used in "foo" ('a' overrides
  680. 'A' in the option string), but only if "foo" is at the beginning
  681. of the filename ('^' overrides '$' and '2').
  682. If you give an /UPPER or /LOWER switch and a /REPLACE switch in the
  683. same RENAME command, the /REPLACE action occurs first, then the case
  684. conversion:
  685. RENAME /REPLACE:{{foo}{bar}} /UPPER * /tmp
  686. For each file: changes all occurrences of "foo" in the name to
  687. "bar", then converts the result to uppercase, and then moves the
  688. file to the /tmp directory. So (for example) "foot.txt" would
  689. become "/tmp/BART.TXT".
  690. Changing the Character Encoding of Filenames
  691. As you know, text is represented on the computer as a series of
  692. numbers, with a given number corresponding to a given character
  693. according to some convention or standard. Filenames are represented the
  694. same way. The trouble is, different computers, or even different
  695. applications on the same computer, might use different standards or
  696. conventions ("character sets") for representing the same characters.
  697. Usually ASCII is safe, but anything beyond that -- non-ASCII characters
  698. such as accented or non-Roman letters -- is likely to vary. Sometimes
  699. you have text that's in the "wrong" character set and you need to
  700. convert it to something you can can use. Kermit has always been able to
  701. handle this as part of file transfer and terminal emulation, as well as
  702. being able to convert text files locally with its TRANSLATE command.
  703. Now there's a way to convert filenames too, for example after copying
  704. files from a CD that uses a different encoding:
  705. RENAME /CONVERT:charset1:charset2 filespec [ directory ]
  706. Converts filenames from the first character set to the second
  707. one. The two character sets can be chosen from the SET FILE
  708. CHARACTER-SET list; for complete details see [68]this page. For
  709. example suppose you have a file called "Olga_Tañón.txt" on a
  710. computer where ISO 8859-1 Latin Alphabet 1 is used, and you have
  711. transported it (e.g. on CDROM) to another computer where the
  712. text encoding is UTF8. Maybe you also have a lot of other files
  713. with similar names in the same directory. You can convert the
  714. filenames to UTF8 like this:
  715. RENAME /CONVERT:latin1:utf8 *
  716. /CONVERT can not be combined with /UPPER, /LOWER, or /REPLACE.
  717. You should NOT use UCS2 for filenames since this encoding is not
  718. compatible with C strings used in Unix and elsewhere.
  719. RENAME /CONVERT affects only the filename, not the file's contents. You
  720. can use the TRANSLATE command to convert the encoding of the contents
  721. of a text file.
  722. Other New Features
  723. See the [69]C-Kermit Daily Builds page for details. Very briefly:
  724. * Perhaps most important, modernized makefile targets for the major
  725. Unix platforms: Linux, Mac OS X, AIX, Solaris, etc. These are
  726. somewhat automated; not autoconf exactly, but they cut down
  727. significantly on redundant targets. For example, one single "linux"
  728. target works on many (hopefully all) different Linux
  729. configurations, where before different targets were required for
  730. different combinations of (e.g.) curses / ncurses / no curses;
  731. 32-bit / 64-bit; different feature sets and library locations.
  732. (Separate targets are still required for Kerberos and/or SSL
  733. builds, but they are "subroutinized".)
  734. * Bigger buffers, more storage for commands, macros, scripts,
  735. strings, and filename expansion in 64-bit versions and in 32-bit
  736. versions that support large files.
  737. * User-settable FTP timeout, works on both the data and control
  738. connection.
  739. * FTP access to ports higher than 16383.
  740. * Built-in FTP client for VMS. This is the [70]same FTP client Unix
  741. C-Kermit has had since version 8.0, minimally adapted to VMS by
  742. SMS, supporting binary and Stream_LF file transfer only (in other
  743. words, nothing to handle RMS files), but otherwise fully functional
  744. (and scriptable) and theoretically capable of making connections
  745. secured by SSL (at least it compiles and links OK with SSL - HP SSL
  746. 1.3 in this case).
  747. * Large file support in VMS, also by SMS. Alpha and Itanium only (not
  748. VAX). VMS C-Kermit was already able to transfer large files, but
  749. the file-transfer display (numbers and progress bar) and statistics
  750. were wrong because they used ints. In the present Alpha test
  751. release, this is an optional feature requested by including the "f"
  752. option in P1.
  753. * New PUTENV command that allows Kermit to pass environment variables
  754. to subprocesses (Unix only, "help putenv").
  755. * New TOUCH command, many file selection options ("help touch").
  756. * New DIRECTORY command options and switches (/TOP, /COUNT;
  757. HDIRECTORY, WDIRECTORY...). To see the ten biggest files in the
  758. current directory: "dir /top:10 /sort:size /reverse *" or
  759. equivalently, "hdir /top:10 *". WDIR lists files in reverse
  760. chronological order, shorthand for "dir /sort:date /reverse".
  761. * New command FSEEK /FIND:string-or-pattern, seeks to the first line
  762. in an FOPEN'd file that contains the given string or matches the
  763. given pattern. Example: Suppose you have a file of lines like this:
  764. quantity description...
  765. in which the first "word" is a number, followed by a description
  766. (for example, the name of an item). Here is how to use FSEEK to
  767. quickly get the total quantity of any given item, which is passed
  768. as a parameter (either a literal string or a pattern) on the
  769. command line:
  770. #!/usr/local/bin/kermit +
  771. if not def \%1 exit 1 Usage: \fbasename(\%0) string-or-pattern
  772. .filename = /usr/local/data/items.log # Substitute the actual filename
  773. set case off # Searches are case-independent
  774. fopen /read \%c \m(filename) # Open the file
  775. if fail exit 1 "\m(filename): \v(errstring)" # Fail: exit with error message
  776. .total = 0 # OK: Initialize the total
  777. echo Searching "\%1"...
  778. while true {
  779. fseek /line /relative /find:\%1 \%c 0 # Get next line that has target
  780. if fail break # Failure indicates EOF
  781. fread /line \%c line # Read it
  782. if fail break # (shouldn't happen)
  783. increment total \fword(\m(line),1) # Increment the total
  784. }
  785. fclose \%c # Close the file
  786. echo Total for "\%1" : \m(total) # Print the result
  787. exit 0
  788. The syntax of the FSEEK command in this example indicates that each
  789. search should start relative to the current file line. Since Kermit
  790. is an interpretive language, FSEEK is a lot faster than FREAD'ing
  791. each line and checking it for the target, especially for big files.
  792. An especially handy use for FSEEK is for use with potentially huge
  793. sequentially timestamped logs, to seek directly to the date-time
  794. where you want to start processing. Some other improvements for the
  795. FOPEN/FREAD/FWRITE/FCLOSE family of commands are included also
  796. (performance, bug fixes, convenience features), listed in the
  797. [71]change log. (Prior to 9.0.299 Alpha.02, the FSEEK /FIND:
  798. command always started from the top.)
  799. * MIME synonyms for character-set names: A new equivalence between
  800. MIME names and Kermit names for character sets, with a new table
  801. showing the supported sets [72]HERE (this feature is also
  802. illustrated in the [73]Weblog script).
  803. * Unix C-Kermit SET TERMINAL TYPE now passes its arguments to
  804. subprocesses as an environment variable.
  805. * SET SESSION-LOG TEXT now strips out ANSI escape sequences from the
  806. session log.
  807. * For interacting with POP servers over clear-text or SSL-secured
  808. connections:
  809. + New SSL and TLS "raw" connections (no Telnet protocol).
  810. + New INPUT command options for reading and capturing (perhaps
  811. while scanning) continuous incoming text, such as INPUT
  812. /NOWRAP (explained [74]HERE).
  813. + New \femailaddress() command to extract the e-mail address
  814. from an Internet mail message To: or From: line, used in
  815. fetching mail from POP servers.
  816. + Improved date parsing commands and functions for parsing the
  817. different date formats that can appear in e-mail.
  818. + Production scripts for fetching mail from a secure POP server,
  819. available [75]HERE.
  820. * Various features added to make Kermit more useful for writing CGI
  821. scripts such as INPUT /COUNT:n to INPUT exactly n characters
  822. (useful for reading form data).
  823. * New \fpictureinfo() function for getting orientation and dimensions
  824. of JPG and GIF images, described [76]HERE.
  825. * New \fgetpidinfo() function for testing whether a given process
  826. exists.
  827. * \fkwdvalue() function fixed to allow multiword values.
  828. * New function \fcount(s1,s2) to tell the number of occurrences of s1
  829. in s2.
  830. * New \flopx() function returns rightmost field from string (such as
  831. a file's extension).
  832. * New function \ffunction(s1) to tell whether a built-in s1 function
  833. exists.
  834. * New \fsqueeze(s1) function removes leading and trailing whitespace
  835. from string s1, changes tabs to spaces, squeezing each run of
  836. repeated whitespace characters to a single space.
  837. * Compact substring notation: \s(somestring[12:18]) is the same as
  838. \fsubstring(\m(somestring),12,18), i.e. the substring starting at
  839. position 12, 18 characters long. \s(somestring[12_18]) means
  840. characters 12 through 18 of the string (7 characters). Also,
  841. \s(somestring[17.]) returns character number 17 of somestring.
  842. * The string indexing functions now accept an optional trailing
  843. argument specifying the occurrence number of the target string.
  844. Likewise, \fword() can fetch words from the right as well as the
  845. left.
  846. * The COPY command in Unix C-Kermit has a new /PRESERVE switch,
  847. equivalent to Unix "cp -p".
  848. * ASKQ /ECHO:c can be used to make the characters the user types echo
  849. as the character c, e.g. asterisk when typing a password.
  850. * IF LINK filename to test if the filename is a symlink.
  851. * Ctrl-K, when typed at the command parser, replaces itself with most
  852. recently entered file specification.
  853. * In Unix, the ability to log a terminal session to a serial port,
  854. for use with speaking devices or serial printers; described
  855. [77]HERE. Also for the same purpose, SET SESSION-LOG
  856. NULL-PADDED-LINES for a speech synthesizer than needed this.
  857. * Adaptation to OpenSSL 0.9.8 and 1.0.0.
  858. * Lifted the restriction on having a remote Kermit program send
  859. REMOTE commands to the local. A very big ex-client needed to be
  860. able to do this (branches would connect to headquarters and upload
  861. files; HQ would then download patches, a REMOTE HOST command was
  862. necessary to allow the remote headquarters machines to install the
  863. patches on the local client; of course the client first has to
  864. ENABLE HOST because this is a risky scenario). The reason for the
  865. restriction was that the server, upon receiving any REMOTE command
  866. would send the results (output) back to the client as a file
  867. transfer with "destination screen", but of course the remote has no
  868. screen.
  869. * Added XMESSAGE, which is to [78]MESSAGE as XECHO is ECHO: it
  870. outputs a string with no line terminator DEBUG MESSAGE is ON.
  871. * Fixed \frecurse() to not dump core when invoked with no arguments.
  872. * Improved text for HELP FUNCTION SPLIT and HELP FUNCTION WORD.
  873. * Patches for Debian 6.0 "Squeeze" from Ian Beckwith.
  874. * \fcontents(\&a[3]) got an error if the array was declared but its
  875. dimension was less than 3. Now it simply returns and empty string.
  876. * \fsplit(), when parsing lines from CSV and TSV files, was treating
  877. backslash in the data the same way it treats backslash in Kermit
  878. commands. This was fixed to treat backslash like any other
  879. character.
  880. * Builds for Solaris 9 and later now use streams ptys rather then the
  881. old BSD-style ptys. Thanks to Gary Mills for this one, who noticed
  882. that he couldn't have more than 48 C-Kermit SSH sessions going at
  883. once and figured out why.
  884. * As noted [79]below DES encryption is being retired from many
  885. platforms and libraries that once used it. I changed the Solaris
  886. and Linux OpenSSL builds to account for this by testing for it. I
  887. probably should also add a OMITDES option to omit DES even if it is
  888. installed, but "KFLAGS=-UCK_DES" seems to do the job for now.
  889. * I changed the Linux build to test for the OpenSSL version (like the
  890. Solaris version already did), rather than assuming OpenSSL 0.9.7.
  891. * A couple minor changes for Tru64 Unix 5.1B from Steven Schweda but
  892. we still have some trouble on that platform. As a workaround "make
  893. osf1" can be used there.
  894. * Unix makefile and man page are now included in the Zip
  895. distribution.
  896. * \fjoin(), which is the inverse function of fsplit() now accepts CSV
  897. and TSV as a second argument, to transform an array into a
  898. comma-separated or tab-separated value list, as described [80]HERE.
  899. * Even in 2010, Unix distributions continue to change their UUCP
  900. lockfile conventions. C-Kermit 9.0 contains support from Joop
  901. Boonen for OpenSuSE >= 11.3 and recent Debian, which no longer have
  902. baudboy.h, which first appeared in Red Hat 7.2 in 2003.
  903. * From Lewis McCarthy:
  904. Based on code inspection, C-Kermit appears to have an SSL-related
  905. security vulnerability analogous to that identified as CVE-2009-3767
  906. (see e.g.
  907. [81]
  908. I'm attaching a patch for this issue relative to the revision of
  909. ck_ssl.c obtained from a copy of
  910. [82] downloaded on
  911. 2010/07/30, which I believe is the latest.
  912. When this flaw was first widely publicized at last year's Black Hat
  913. conference, it was claimed that some public certificate authorities
  914. had indeed issued certificates that could be used to exploit this
  915. class of vulnerability. As far as I know they have not revealed
  916. specifically which public CA(s) had been found issuing such
  917. certificates. Some references:
  918. + [83]
  919. + [84]
  920. ificate/
  921. * Peter Eichhorn reported that "RENAME ../x ." didn't work; fixed
  922. now.
  923. * If only one file is FOPEN'd, FCLOSE given with no arguments would
  924. close it; this was a "convenience feature" that turned out to be
  925. dangerous. For safety FCLOSE has to require a specific channel
  926. number or the word ALL.
  927. * Added \fstrcmp(s1,s2,case,start,length), which has the advantage
  928. over IF EQU,LGT,LLT that case sensitivity can be specified as a
  929. function arg, and also substrings can be specified.
  930. * New built-in functions:
  931. \fcvtcsets(string,cs1,cs2)
  932. Function to convert a string from one character set to
  933. another.
  934. \fdecodehex(string[,prefix])
  935. Function to decode a string containing hex escapes.
  936. \fstringtype(string)
  937. Function to tell whether a string is 7-bit, 8-bit, or
  938. UTF-8.
  939. For the motivation for these features and an application that uses
  940. them to analyze web logs, see the Weblog script below.
  941. *
  942. Lazy IF Conditions: Now you can do this:
  943. define foo some number
  944. if foo command
  945. instead of this:
  946. define foo some number
  947. if \m(foo) command
  948. Of course the old way still works too. But watch out because if the
  949. variable name is the same as a symbolic IF condition (for example
  950. COUNT), it won't do what you expected. (IF COUNT was used for loop
  951. control in early versions of MS-DOS Kermit, before it got real FOR
  952. and WHILE loops; it was added to C-Kermit for compatibility, and it
  953. can't be removed because that could break existing scripts).
  954. * Escape sequences are now stripped from text-mode session logs not
  955. only in CONNECT sessions but also in whatever is logged by the
  956. INPUT command; described in the [85]next section.
  957. * New commands for selectively issuing progress or debugging messages
  958. from scripts, also described in the next section.
  959. * Fix from [86]John Dunlap to prevent the fixed packet-timeout
  960. interval from going to an unexpected value.
  961. * Alpha.04 fixes a problem with FTP connections made from 64-bit Unix
  962. platforms. All the other changes in this section were to Alpha.03.
  963. * Relaunching a closed SSH connection with the CONNECT command is now
  964. possible, as it always has been with Telnet and other connection
  965. types; suggested by Peter Eichhorn (needs testing).
  966. * A symbol conflict fixed that prevented successful build on
  967. [87]FreeBSD 8.0.
  968. * Fixes from Christian Corti for building on SunOS 4.1.
  969. * New aixg target for building on AIX with gcc.
  970. * New aix+ibmssl target. This is nice because the IBM-supplied SSL
  971. libraries and header files are in a known location; no need to
  972. [88]set environment variables giving their locations.
  973. * "Large File Support" is now included by default on Alpha and IA64
  974. hardware on VMS 7.3 and later, and it should work much better than
  975. before.
  976. * Kermit's internal FTP client is now included by default in any
  977. build that also includes TCP/IP networking. At present, the FTP
  978. client seems to work well for binary-mode transfers; text (ASCII)
  979. mode transfers still need some work. In builds that also include
  980. Secure Sockets Layer (SSL) security (next item) the FTP client
  981. should be able to make securely authenticated and encrypted
  982. connections.
  983. * In network builds that request OpenSSL support, e.g.:
  984. $ @ckvker "" "" "CK_SSL"
  985. the OpenSSL version is detected automatically and the appropriate
  986. compile-time options are emitted (such as
  988. * Preliminary / limited support for the ODS-5 file system on VMS 7.2
  989. and later, Alpha and Itanium only (needs testing): Filenames can be
  990. mixed case and can be longer.
  991. * Support for older and older VMS versions.
  992. * In the VMS build procedure, CKVKER.COM, the "i" option in P1 now
  993. means don't include the internal FTP client, and the "f" option
  994. means do not include "Large File" support. Large File support in
  995. VMS really only applies to the file-transfer display and
  996. statistics, which would go out of whack as soon as the byte count
  997. overflowed 31 bits because this is C-Kermit, built with the C
  998. compiler and the C library (runtime system), which did not support
  999. long integers until VMS 7.3.
  1000. * The [89]LISP Operator ROUND now takes an optional second argument
  1001. that specifies the number of places to round to, e.g.
  1002. (ROUND dollars 2) rounds dollars to 2 decimal places.
  1003. * Improved pattern matching in many commands for both strings and
  1004. filenames.
  1005. * Various minor new features, plus numerous bug fixes and speedups.
  1006. Incompatibilities
  1007. A top priority for new Kermit software releases has always been
  1008. backwards compatibility. A script written for a previous Kermit release
  1009. should run the same way in the new release.
  1010. There's one exception this time. The [90]\fsplit() function is
  1011. incredibly handy, it can do almost anything, up to and including
  1012. parsing a LISP program (the underlying code is the basis of the
  1013. [91]S-Expression interpreter). But did you ever try to use it to parse
  1014. (say) a Tab-Separated-List (TSV file) or Comma-Separated-List (CSV)? It
  1015. works as expected as long as the data contains only 7-bit characters.
  1016. But if your data contains (say) Spanish or German or Russian text
  1017. written in an 8-bit character set such as ISO 8859-1, every 8-bit
  1018. character (any value 128-255) is treated as a break character. This is
  1019. fixed in C-Kermit 9.0 by treating all 8-bit bytes as "include"
  1020. characters rather than break characters, a total reversal of past
  1021. behavior. I don't think it will affect anyone though, because if this
  1022. had happened to anyone, I would have heard about it!
  1023. Since most standard 8-bit character sets have control characters in
  1024. positions 128-160, it might have made sense to keep 128-160 in the
  1025. break set, but with the proliferation of Microsoft Windows code pages,
  1026. there is no telling which 8-bit character is likely to be some kind of
  1027. text, e.g. "smart quotes" or East European or Turkish accented letters.
  1028. What's Not In C-Kermit 9.0
  1029. Some large projects that were contemplated have not been done,
  1030. including:
  1031. * IPv6. Honestly, there has been zero demand for this, and it would
  1032. be a lot of work and disruption to the code base. Volunteers
  1033. welcome, I guess. It could be a CS project.
  1034. * A database interface - MySQL or ODBC. For this one, there is some
  1035. demand but I haven't had a chance to even look into it.
  1036. * There's a looming issue with DES encryption; major vendors are
  1037. removing it from their platforms, starting with Apple in Mac OS X
  1038. 10.6, with Microsoft to follow suit. A secure version of Kermit can
  1039. be built without DES, but in limited testing successful connections
  1040. were spotty (e.g. with Kerberos 5).
  1041. * Cleaning up the Unix makefile. It has 25 years' worth of targets in
  1042. it. It is very likely safe to remove most of them, since (a) most
  1043. old platforms have gone away by now, or have been upgraded, due to
  1044. hacking vulnerabilities; (b) the market has consolidated
  1045. considerably; and (c) most of the new features of C-Kermit 9.0,
  1046. such as large files, won't be of any use on older platforms and
  1047. previous C-Kermit versions will remain available.
  1048. * Packages. Everybody wants an install package custom made for their
  1049. own computer, Linux RPMs being the prime example but far from the
  1050. only one. These will come, I suppose (especially with some Linux
  1051. sites having a policy against installing any application that does
  1052. not come as an RPM). In the meantime, here's a page that describes
  1053. some Kermit-specific issues in package construction:
  1054. [92]ckpackages.html.
  1055. And a Loose End...
  1056. Using External File-Transfer Protocols on Secure Connections
  1057. After C-Kermit 8.0.212 Dev.27 (2006/12/22), I spent a big chunk of time
  1058. trying to solve a particular problem that some of you have complained
  1059. about and others might be familiar with: If you use C-Kermit to make a
  1060. secure Telnet connection to another host (e.g. with Telnet SSL/TLS,
  1061. Kerberos, or SRP) and then attempt to transfer a file using an external
  1062. protocol such as Zmodem, it doesn't work.
  1063. That's because as coded (through 8.0.211), C-Kermit simply starts the
  1064. external protocol in a fork with its standard i/o redirected to the
  1065. connection. This completely bypasses the encryption and decryption that
  1066. is done by C-Kermit itself, and of course it doesn't work. The same
  1067. thing occurs if you use the REDIRECT command. The routine that handles
  1068. this is ttruncmd() in ckutio.c.
  1069. In order to allow (say) Zmodem transfers on secure connections, it is
  1070. necessary for C-Kermit to interpose itself between the external Zmodem
  1071. program and the connection, decrypting the incoming stream before
  1072. feeding it to Zmodem and encrypting Zmodem's output before sending out
  1073. the connection.
  1074. In principal, this is simple enough. We open a pseudoterminal pair
  1075. ("master" and "slave") for Zmodem's i/o and we create a fork and start
  1076. Zmodem in it; we read from the fork pty's standard output, encrypt, and
  1077. send to the net; we read from the net, decrypt, and write to the fork
  1078. pty's standard input.
  1079. In practice, it's not so simple. First of all, pseudoterminals (ptys)
  1080. don't seem to interface correctly with certain crucial APIs, at least
  1081. not in the OS's I have tried (Mac OS X, Linux, NetBSD, etc), such as
  1082. select(). And i/o with the pty often - perhaps always - fails to
  1083. indicate errors when they occur; for example, when the fork has exited.
  1084. But, even after coding around the apparent uselessness of select() for
  1085. multiplexing pty and net, and using various tricks to detect when the
  1086. external protocol exits and what its exit status is, I'm still left
  1087. with a show-stopping problem: I just simply can not download (receive)
  1088. a file with Zmodem, which is the main thing that people would probably
  1089. want to do. I can send files just fine, but not receive. The incoming
  1090. stream is delivered to Zmodem (to the pty slave) but upon arrival at
  1091. the Zmodem process itself, pieces are always missing and/or corrupt.
  1092. Yet I can receive files just fine if I use Kermit itself (C-Kermit or
  1093. G-Kermit) as the external protocol, rather than Zmodem.
  1094. I can think of two reasons why this might be the case:
  1095. 1. Zmodem sends all 8-bit bytes and control codes in the clear, and
  1096. maybe the pty is choking on them because it thinks it is a real
  1097. terminal.
  1098. But Zmodem puts its controlling terminal into raw mode. And C-Kermit
  1099. puts the pty into raw mode too, just for good measure. If any 0xFF
  1100. codes are in the Zmodem data stream, and it's a Telnet session, Kermit
  1101. does any needed byte stuffing/unstuffing automatically. Anyway, if I
  1102. tell Zmodem to prefix everything, it makes no difference.
  1103. 2. Zmodem is a streaming protocol and perhaps the pty driver can't
  1104. keep up with a sustained stream of input at network speeds. What
  1105. would be the method of flow control?
  1106. I can vary the size of the i/o buffers used for writing to the pty, and
  1107. get different effects, but I am not able to get a clean download, no
  1108. matter what buffer size I use. write()'ing to the pty does not return
  1109. an error, and I can't see the errors because they happen on the master
  1110. side. It's as if the path between the pty slave and master lacks flow
  1111. control; I deliver a valid data stream to the pty slave and the master
  1112. gets bits and pieces. This impression is bolstered somewhat by the
  1113. "[93]man 7 pty" page in HP-UX, which talks about some special modes for
  1114. ptys that turn off all termio processing and guarantee a
  1115. flow-controlled reliable stream of bytes in both directions - a feature
  1116. that seems to be specific to HP-UX, and exactly the one we need
  1117. everywhere.
  1118. Well, in Pass One I used C-Kermit's existing pty routines from
  1119. ckupty.[ch], which are well-proven in terms of portability and of
  1120. actually working. They are currently used by SET HOST /PTY for making
  1121. terminal connections to external processes. But these routines are
  1122. written on the assumption that the pty is to be accessed interactively,
  1123. and maybe they are setting the fork/pty arrangement up in such a way
  1124. that that's not suitable for file transfer. The Pass One routine is
  1125. called xttptycmd() in ckutio.c.
  1126. So in Pass Two I made a second copy of the routine, yttptycmd(), that
  1127. manages the pty and fork itself, so all the code is in one place and
  1128. it's simple and understandable. But it still doesn't work for Zmodem
  1129. downloads. In this routine, I use openpty() to get the pty pair, which
  1130. is not portable, so I can have access to both the master and slave pty
  1131. file descriptors. This version can be used only a platforms that have
  1132. openpty(): Linux, Mac OS X, NetBSD, etc.
  1133. In Pass Three, zttptycmd(), I tried using pipes instead of ptys, in
  1134. case ptys are simply not up to this task (but that can't be true
  1135. because if I make a Telnet or SSH connection into a host, I can send
  1136. files to it with Zmodem, and the remote Zmodem receiver is, indeed,
  1137. running on a pty). But pipes didn't work either.
  1138. In Pass Four, I extracted the relevant routines into a standalone
  1139. program based on yttptycmd() (the openpty() version, for simplicity),
  1140. which I tested on Mac OS X, the idea being to rule out any
  1141. "environmental" effects of running inside the C-Kermit process. There
  1142. was no difference -- Kermit transfers (with C-Kermit itself as the
  1143. external protocol) worked; Zmodem transfers (neither sz or lsz) did
  1144. not.
  1145. Well, it's a much longer story. As the external protocol, I've tried
  1146. rzsz, crzsz, and lrzsz. We know that some of these have quirks
  1147. regarding standard i/o, etc, which is one of the reasons for using ptys
  1148. in the first place, and i/o does work - just not reliably. Anyway, the
  1149. 1100 lines or so of [94]ckc299.txt, starting just below where it says
  1150. "--- Dev.27 ---" tell the full story. At this point I have to give up
  1151. and move on; it might be more productive to let somebody else who has
  1152. more experience with ptys take a look at it - if indeed anyone still
  1153. cares about being able to do Zmodem transfers over secure Telnet
  1154. connections.
  1155. C-Kermit 9.0 contains the three new routines (and some auxiliary ones),
  1156. but they are not compiled or called unless you build it specially:
  1157. make targetname KFLAGS=-DXTTPTYCMD (builds with xttptycmd())
  1158. make targetname KFLAGS=-DYTTPTYCMD (builds with yttptycmd())
  1159. make targetname KFLAGS=-DZTTPTYCMD (builds with zttptycmd())
  1160. These are all in [95]ckutio.c. As noted, the second one works only for
  1161. Linux, FreeBSD, NetBSD, and Mac OS X, because it uses non-POSIX,
  1162. non-portable openpty(). If you want to try it on some other platform
  1163. that has openpty(), you can build it like this:
  1164. make targetname "KFLAGS=-DYTTPTYCMD -DHAVE_OPENPTY"
  1165. (and let me know, so I can have HAVE_OPENPTY predefined for that
  1166. platform too). The best strategy to get this working, I think, would be
  1167. to concentrate on yttptycmd(), which is the simpler of the two
  1168. pty-based routines. If it can be made to work, then we'll see if we can
  1169. retrofit it to use the ckupty.c routines so it will be portable to
  1170. non-BSD platforms.
  1171. By the way, if you build with any of [XYZ]TTPTYCMD defined, then the
  1172. selected routine will always be used in place of ttruncmd(). This is to
  1173. allow testing on all kinds of connections, not just secure ones, in
  1174. both local and remote mode. Once the thing works, if it ever does, I'll
  1175. add the appropriate tests and/or commands.
  1176. By default, in the initial test release, C-Kermit 9.0 uses ttruncmd()
  1177. on serial connections and ttyptycmd() on network connections. Even when
  1178. a network connection is not encrypted, Kermit still needs to handle the
  1179. network protocol, e.g. the quoting of 0xff bytes on Telnet connections.
  1180. Demonstration: Fetch Mail from POP Server Secured by SSL
  1181. pop.ksc is a fully elaborated production script for fetching one's
  1182. mail from a POP3 server over a connection secured by SSL. For
  1183. explanation and documentation, mailcheck is a wrapper for the pop.ksc
  1184. script, which collects your password one time, and then checks for new
  1185. mail every 5 minutes (or other selected interval) and calls pop.ksc to
  1186. fetch it if there is any.
  1187. Demonstration: HP Switch Configuration Backup
  1188. A common use for Kermit software is to make automated backups of the
  1189. configuration of network switches and routers, such as those made by
  1190. Cisco or Hewlett-Packard (although [99]tftp can be used for this, it is
  1191. not available in all such devices; Kermit, however, works with those
  1192. that have tftp as well as those that don't).
  1193. Typically a backup can be done by making a Telnet, SSH, or serial
  1194. connection to the device with Kermit and giving a command such as "show
  1195. config" at the command-line prompt of the device with Kermit's session
  1196. log activated. The result is a list of the commands that were used to
  1197. establish the current configuration, suitable for feeding back to the
  1198. device's console (e.g. with C-Kermit's TRANSMIT command) to reestablish
  1199. the same configuration or to duplicate it on another device.
  1200. At an HP installation it was noted, however, that while the HP switches
  1201. (various ProCurve models) produced the desired list of commands, they
  1202. were interspersed with escape sequences for special effects, thus
  1203. rendering the recorded sessions unsuitable for feeding back into the
  1204. switches.
  1205. C-Kermit 9.0 introduces a new feature to strip the offending sequences
  1206. out of a session log, leaving just the text. The command SET
  1207. SESSION-LOG TEXT activates this feature. In C-Kermit 9.0 Alpha.02 and
  1208. earlier, escape sequence stripping occurred only while logging
  1209. interactive (CONNECT) sessions; beginning with Alpha.03 it is done also
  1210. for data that is read by INPUT commands and therefore works for scripts
  1211. too.
  1212. A sample HP Switch Configuration Backup script is [100]HERE, and its
  1213. data file is [101]HERE. This script also illustrates some other new
  1214. features of Alpha.03:
  1215. MESSAGE text
  1216. This lets you put debugging messages in your script that can be
  1217. displayed or not, according to SET DEBUG MESSAGE (below). This
  1218. way you don't have to change your script for debugging. Hint:
  1219. In Unix, invoke the script like this:
  1220. $ DEBUG=1 scriptname arg1 arg2...
  1221. and then include the following command in your script:
  1222. if defined \$(DEBUG) set debug message on
  1223. XMESSAGE text
  1224. Like MESSAGE but prints the text with no line terminator, so it
  1225. can be continued by subsequent messages.
  1227. ON means MESSAGE commands should print to standard output; OFF
  1228. means they shouldn't print anything; STDERR means the messages
  1229. should be printed to [102]stderr. DEBUG MESSAGE is OFF by
  1230. default, i.e. unless you SET it to ON or STDERR.
  1231. IF DEBUG command
  1232. Executes the command if SET DEBUG MESSAGE is not OFF.
  1233. The \v(lastcommand) variable
  1234. This variable contains the previous command. You can use it in
  1235. debugging and error message to show (for example) exactly what
  1236. the command was that just failed, without having to make a copy
  1237. of the command:
  1238. set host
  1239. if fail exit 1 "FATAL - \v(lastcommand)"
  1240. which, if the SET HOST command fails, prints "FATAL - set host
  1241." and then exits with status 1 (which
  1242. normally indicates failure).
  1243. Demonstration: HP iLO Blade Configuration
  1244. [103]THIS DOCUMENT describes a script in production use at Columbia
  1245. University for configuring and deploying racks full of HP blade servers
  1246. through their "integrated Lights Out" (iLO) management interface,
  1247. bypassing the tedious and error-prone process of configuring the
  1248. servers one by one through the vendor-provided point-and-click Web
  1249. interface, which is ill-suited to configuring large numbers of blades.
  1250. The script illustrates some of C-Kermit 9.0's new features; source code
  1251. is available through the link. The code is apt to change from time to
  1252. time as new requirements surface.
  1253. Demonstration: IBM/Rolm/Siemens CBX Management
  1254. [104]THIS DOCUMENT describes a suite of scripts (some in production,
  1255. some in development) used to manage the Columbia campus 20,000-line
  1256. main telephone switch, along with about 10 satellite switches at
  1257. off-campus locations. These switches are 1980s technology*, their
  1258. management consoles are serial ports. Access is via Telnet to reverse
  1259. terminal servers. The scripts allow for interactive sessions as well as
  1260. automatic production (and in some cases formatting) of different
  1261. reports required by different groups at different intervals. These
  1262. scripts replace a whole assortment of ad-hoc ProComm ASPECT scripts
  1263. that were scattered all over the place, with passwords embedded. The
  1264. new scripts are intended to be run from a centralized server where
  1265. there is a single well-secured configuration file, and where they can
  1266. be used on demand, or in cron jobs. They are modular so code
  1267. duplication is minimal.
  1268. __________________________
  1269. * Of course the University is deploying new technology but the but the
  1270. old system will be used in parallel for some time to come.
  1271. Demonstration: CSV and TSV Files
  1272. Contents
  1273. * [105]Reading a CSV or TSV Record and Converting it to an Array
  1274. * [106]Using \fjoin() to create a Comma- or Tab-Separated Value List
  1275. from an Array
  1276. * [107]Using CSV or TSV Files
  1277. Comma-Separated Value (CSV) format is commonly output by spreadsheets
  1278. and databases when exporting data into plain-text files for import into
  1279. other applications. Here are the details:
  1280. Comma-Separated List Syntax
  1281. 1. Each record is a series of fields.
  1282. 2. Records are in whatever format is used by the underlying file
  1283. system for lines of text.
  1284. 3. Fields within records are separated by commas, with zero or more
  1285. whitespace characters (space or tab) before and/or after the comma;
  1286. such whitespace is considered part of the separator.
  1287. 4. Fields with embedded commas must be enclosed in ASCII doublequote
  1288. characters.
  1289. 5. Fields with leading or trailing spaces must be enclosed in ASCII
  1290. doublequotes.
  1291. 6. Any field may be enclosed in ASCII doublequotes.
  1292. 7. Fields with embedded doublequotes must be enclosed in doublequotes
  1293. and each interior doublequote is doubled.
  1294. Here is an example:
  1295. aaa, bbb, has spaces,,"ddd,eee,fff", " has spaces ","Muhammad ""The Greatest"" A
  1296. li"
  1297. The first two are regular fields. The second is a field that has an
  1298. embedded space but in which any leading or trailing spaces are to be
  1299. ignored. The fourth is an empty field, but still a field. The fifth is
  1300. a field that contains embedded commas. The sixth has leading and
  1301. trailing spaces. The last field has embedded quotation marks.
  1302. Prior to C-Kermit 9.0 Alpha.06, C-Kermit did not handle CSV files
  1303. according to the specification above. Most seriously, there was no
  1304. provision for a separator to be surrounded by whitespace that was to be
  1305. considered part of the separator. Also there was no provision for
  1306. quoting doublequotes inside of a quoted string.
  1307. Reading a CSV record
  1308. Now the \fsplit() function can handle any CSV-format string if you
  1309. include the symbolic include set "CSV" as the 4th parameter. To
  1310. illustrate, this program:
  1311. def xx {
  1312. echo [\fcontents(\%1)]
  1313. .\%9 := \fsplit(\fcontents(\%1), &a, \44, CSV)
  1314. for \%i 1 \%9 1 { echo "\flpad(\%i,3). [\&a[\%i]]" }
  1315. echo "-----------"
  1316. }
  1317. xx {a,b,c}
  1318. xx { a , b , c }
  1319. xx { aaa,,ccc," with spaces ",zzz }
  1320. xx { "1","2","3","","5" }
  1321. xx { this is a single field }
  1322. xx { this is one field, " and this is another " }
  1323. xx { name,"Mohammad ""The Greatest"" Ali", age, 67 }
  1324. xx { """field enclosed in doublequotes""" }
  1325. exit
  1326. gives the following results:
  1327. [a,b,c]
  1328. 1. [a]
  1329. 2. [b]
  1330. 3. [c]
  1331. -----------
  1332. [ a , b , c ]
  1333. 1. [a]
  1334. 2. [b]
  1335. 3. [c]
  1336. -----------
  1337. [ aaa,,ccc," with spaces ",zzz ]
  1338. 1. [aaa]
  1339. 2. []
  1340. 3. [ccc]
  1341. 4. [ with spaces ]
  1342. 5. [zzz]
  1343. -----------
  1344. [ "1","2","3","","5" ]
  1345. 1. [1]
  1346. 2. [2]
  1347. 3. [3]
  1348. 4. []
  1349. 5. [5]
  1350. -----------
  1351. [ this is a single field ]
  1352. 1. [this is a single field]
  1353. -----------
  1354. [ this is one field, " and this is another " ]
  1355. 1. [this is one field]
  1356. 2. [ and this is another ]
  1357. -----------
  1358. [ name,"Mohammad ""The Greatest"" Ali", age, 67 ]
  1359. 1. [name]
  1360. 2. [Mohammad "The Greatest" Ali]
  1361. 3. [age]
  1362. 4. [67]
  1364. [ """field enclosed in doublequotes""" ]
  1365. 1. ["field enclosed in doublequotes"]
  1366. -----------
  1367. The separator \44 (comma) must still be specified as the break set (3rd
  1368. \fsplit() parameter). When "CSV" is specified as the include set:
  1369. * The Grouping Mask is automatically set to 1 (which specifies that
  1370. the ASCII doublequote character (") is used for grouping;
  1371. * The Separator Flag is automatically set to 1 so that adjacent field
  1372. separators will not be collapsed;
  1373. * All bytes (values 0 through 255) other than the break character are
  1374. added to the include set;
  1375. * Any leading whitespace is stripped from the first element unless it
  1376. is enclosed in doublequotes;
  1377. * Any trailing whitespace is trimmed from the end of the last element
  1378. unless it is enclosed in doublequotes;
  1379. * If the separator character has any spaces or tabs preceding it or
  1380. following it, they are ignored and discarded;
  1381. * The separator character is treated as an ordinary data character if
  1382. it appears in a quoted field;
  1383. * A sequence of two doublequote characters ("") within a quoted field
  1384. is converted to a single doublequote.
  1385. There is also a new TSV symbolic include set, which is like CSV except
  1386. without the quoting rules or the stripping of whitespace around the
  1387. separator because, by definition, TSV fields do not contain tabs.
  1388. Of course you can specify any separator(s) you want with either the
  1389. CSV, TSV, or ALL symbolic include sets. For example, if you have a TSV
  1390. file in which you want the spaces around each Tab to be discarded, you
  1391. can use:
  1392. \fsplit(variable, &a, \9, CSV)
  1393. \9 is Tab.
  1394. The new symbolic include sets can also be used with \fword(), which is
  1395. just like \fsplit() except that it retrieves the nth word from the
  1396. argument string, rather than an array of all the words. In C-Kermit you
  1397. can get information about these or any other functions with the HELP
  1398. FUNCTION command, e.g.:
  1399. C-Kermit> help func word
  1400. Function \fword(s1,n1,s2,s3,n2,n3) - Extracts a word from a string.
  1401. s1 = source string.
  1402. n1 = word number (1-based) counting from left; if negative, from right.
  1403. s2 = optional break set.
  1404. s3 = optional include set (or ALL, CSV, or TSV).
  1405. n2 = optional grouping mask.
  1406. n3 = optional separator flag:
  1407. 0 = collapse adjacent separators;
  1408. 1 = don't collapse adjacent separators.
  1409. \fword() returns the n1th "word" of the string s1, according to the
  1410. criteria specified by the other parameters.
  1411. The BREAK SET is the set of all characters that separate words. The
  1412. default break set is all characters except ASCII letters and digits.
  1413. ASCII (C0) control characters are treated as break characters by default,
  1414. as are spacing and punctuation characters, brackets, and so on, and
  1415. all 8-bit characters.
  1416. The INCLUDE SET is the set of characters that are to be treated as
  1417. parts of words even though they normally would be separators. The
  1418. default include set is empty. Three special symbolic include sets are
  1419. also allowed:
  1420. ALL (meaning include all bytes that are not in the break set)
  1421. CSV (special treatment for Comma-Separated-Value records)
  1422. TSV (special treatment for Tab-Separated-Value records)
  1423. For operating on 8-bit character sets, the include set should be ALL.
  1424. If the GROUPING MASK is given and is nonzero, words can be grouped by
  1425. quotes or brackets selected by the sum of the following:
  1426. 1 = doublequotes: "a b c"
  1427. 2 = braces: {a b c}
  1428. 4 = apostrophes: 'a b c'
  1429. 8 = parentheses: (a b c)
  1430. 16 = square brackets: [a b c]
  1431. 32 = angle brackets: <a b c>
  1432. Nesting is possible with {}()[]<> but not with quotes or apostrophes.
  1433. Returns string:
  1434. Word number n1, if there is one, otherwise an empty string.
  1435. Also see:
  1437. C-Kermit>
  1438. Using \fjoin() to create Comma- or Tab-Separated Value Lists from Arrays
  1439. In C-Kermit 9.0, \fsplit()'s inverse function, [108]\fjoin() received
  1440. the capability of converting an array into a comma-separated or a
  1441. tab-separated value list. Thus, given a CSV, if you split it into an
  1442. array with \fsplit() and then join the array with \fjoin(), giving each
  1443. function the new CSV parameter in the appropriate argument position,
  1444. the result will be will be equivalent to the original, according to the
  1445. CSV definition. It might not be identical, because if the result had
  1446. extraneous spaces before or after the separating commas, these are
  1447. discarded, but that does not affect the elements themselves. The new
  1448. syntax for \fjoin() is:
  1449. \fjoin(&a,CSV)
  1450. Given the array \&a[] or any other valid array designator, joins
  1451. its elements into a comma-separated list according to the
  1452. [109]rules listed above.
  1453. \fjoin(&a,TSV)
  1454. Joins the elements of the given array into a tab-separated list,
  1455. also described above.
  1456. [110]Previous calling conventions for \fjoin() are undisturbed,
  1457. including the ability to specify a portion of an array, rather than the
  1458. whole array:
  1459. declare \&a[] = 1 2 3 4 5 6 7 8 9
  1460. echo \fjoin(&a[3:7],CSV)
  1461. 3,4,5,6,7
  1462. Using \fsplit() and \fjoin() it is now possible to convert a
  1463. comma-separated value list into a tab-separated value list, and vice
  1464. versa (which is not a simple matter of changing commas to tabs or vice
  1465. versa).
  1466. Applications for CSV Files
  1467. Databases such as MS Access or MySQL can export tables or reports in
  1468. CSV format, and then Kermit can read the resulting CSV file and do
  1469. whatever you like with it; typically something that could not be done
  1470. with the database query language itself (or that you didn't know how to
  1471. do that way): create reports or datasets based on complex criteria or
  1472. procedures, edit or modify some fields, etc, and then use \fjoin() to
  1473. put each record back in CSV form so it can be reimported into a
  1474. spreadsheet or database.
  1475. Here is a simple example in which we purge all records of customers who
  1476. have two or more unpaid bills. The file is sorted so that each license
  1477. purchase record is followed by its annual maintenance payment records
  1478. in chronological order.
  1479. #!/usr/local/bin/kermit
  1480. .filename = somefile.csv # Input file in CSV format
  1481. fopen /read \%c \m(filename) # Open it
  1482. if fail exit # Don't go on if open failed
  1483. copy \m(filename) ./new # Make a copy of the file
  1484. .oldserial = 00000000000 # Multiple records for each serial number
  1485. .zeros = 0 # Unpaid bill counter
  1486. while true { # Loop
  1487. fread /line \%c line # Get a record
  1488. if fail exit # End of file
  1489. .n := \fsplit(\m(line),&a,\44,CSV) # Split the fields into an array
  1490. if not equ "\m(oldserial)" "\&a[6]" { # Have new serial number?
  1491. # Remove all records for previous serial number
  1492. # if two or more bills were not paid...
  1493. if > \m(zeros) 1 {
  1494. grep /nomatch \m(oldserial) /output:./new2 ./new
  1495. rename ./new2 ./new
  1496. }
  1497. .oldserial := \&a[6] # To detect next time serial number changes
  1498. .zeros = 0 # Reset unpaid bill counter
  1499. }
  1500. if equ "\&a[5]" "$0.00" { # Element 5 is amount paid
  1501. increment zeros # If it's zero, count it.
  1502. }
  1503. }
  1504. fclose \%c
  1505. Rewriting the file multiple times is inelegant, but this is a quick and
  1506. dirty use-once-and-discard script, so elegance doesn't count. The
  1507. example is interesting in that it purges certain records based on the
  1508. contents of other records. Maybe there is a way to do this directly
  1509. with SQL, but why use SQL when you can use Kermit?
  1510. Here is the same task but this time no shelling out, and this time we
  1511. do change and add some fields and then join the result back into a CSV
  1512. record and write it out to a new file. The object is to create a record
  1513. for each license that shows not only the date and purchase price of the
  1514. license but also the date and amount of the last maintenance payment,
  1515. and to add new fields for sorting by anniversary (month and day):
  1516. #!usr/local/bin/kermit +
  1517. cd ~/somedirectory # CD to appropriate directory
  1518. if fail exit 1 # Make sure we did
  1519. .filename := \%1 # Filename from command line
  1520. if not def filename { # If none give usage message
  1521. exit 1 "Usage: \%0: infile [ outfile ]"
  1522. }
  1523. fopen /read \%c \m(filename) # Open the input CSV file
  1524. if fail exit # Make sure we did
  1525. .output := \%2 # Output filename from command line
  1526. if not def output { # Supply one if not given
  1527. .output := New_\m(filename)
  1528. }
  1529. fopen /write \%o \m(output) # Open output file
  1530. if fail exit # Check that we did
  1531. .serial = 00000000000 # Initialize serial number
  1532. .licenses = 0 # and license counter
  1533. fread /line \%c line # First line is column labels
  1534. if fail exit # Check
  1535. fwrite /line \%o "\m(line),AMM_DD,AYYYY" # Write new labels line
  1536. # Remaining lines are license purchases (K95B) followed by zero or more
  1537. # maintenance invoices (K95BM) for each license.
  1538. .datepaid = 00/00/0000 # Initialize last maint payment date
  1539. .amtpaid = $0.00 # Initialize last maint payment amount
  1540. set flag off # For remembering we're at end of file
  1541. while not flag { # Loop to read all records
  1542. fread /line \%c line # Read a record
  1543. if fail set flag on # If EOF set flag for later
  1544. .n := \fsplit(\m(line),&a,\44,CSV) # Break record into array
  1545. if ( flag || equ "\&a[3]" "K95B" ) { # License or EOF
  1546. if fail exit 1 "FAILED: \v(lastcommand)"
  1547. if licenses { # If this is not the first license
  1548. .\&x[5] := \m(amtpaid) # Substitute most recent amount paid
  1549. .\&x[21] := \m(datepaid) # Substitute most recent date paid
  1550. void \fsplit(\&x[18],&d,/) # Break up original (anniversary) date
  1551. # and put mm_dd and yyyy in separate fields for sorting...
  1552. fwrite /line \%o "\fjoin(&x,CSV),\flpad(\&d[1],2,0)_\flpad(\&d[2],2,
  1553. 0),\&d[3]"
  1554. if fail exit 1 WRITE # Check for error
  1555. xecho . # Show progress as one dot per record
  1556. }
  1557. if flag break # We're at EOF so we're finished
  1558. increment licenses # New license - count it
  1559. array copy &a &x # Keep this record while reading next
  1560. .serial := \&a[6] # Remember serial number
  1561. .datepaid = 00/00/0000 # Initial maintenance payment date
  1562. .amtpaid = $0.00 # and amount
  1563. continue # and go back to read next record
  1564. }
  1565. if not eq "\m(serial)" "\&a[6]" { # Catch out-of-sequence record
  1566. echo
  1567. echo "SEQUENCE: \m(serial)..\&a[6]: \&a[7] [\&a[1]]"
  1568. continue
  1569. }
  1570. if equ "\&a[5]" "" .\&a[5] = $0.00 # If amount is empty make it $0.00
  1571. if not equ "\&a[5]" "$0.00" { # If amount is not $0.00
  1572. .datepaid := \&a[21] # remember date paid
  1573. .amtpaid := \&a[5] # and amount paid
  1574. }
  1575. }
  1576. fclose ALL # Done - close all files and exit
  1577. exit 0 Done.
  1578. The result imports back into Excel, where it can be sorted, formatted,
  1579. or otherwise manipulated as desired.
  1580. Using CSV Files: Extending Kermit's Data Structures
  1581. Now that we can parse a CSV record, what would we do with a CSV file -
  1582. that is, a sequence of records? If we needed all the data available at
  1583. once, we would want to load it into a matrix of (row,column) values.
  1584. But Kermit doesn't have matrices. Or does it?
  1585. Kermit has several built-in data types, but you can invent your own
  1586. data types as needed using Kermit's macro feature:
  1587. define variablename value
  1588. For example:
  1589. define alphabet abcdefghijklmnopqrstuvwxyz
  1590. This defines a macro named alphabet and gives it the value
  1591. abcdefghijklmnopqrstuvwxyz. A more convenient notation (added in
  1592. C-Kermit 7.0, see [111]Table 2) for this is:
  1593. .alphabet = abcdefghijklmnopqrstuvwxyz
  1594. The two are exactly equivalent: they make a literal copy the "right
  1595. hand side" as the value of the macro. Then you can refer to the macro
  1596. anywhere in a Kermit command as "\m(macroname)":
  1597. echo "Alphabet = \m(alphabet)"
  1598. There is a second way to define a macro, which is like the first except
  1599. that the right-hand side is evaluated first; that is, any variable
  1600. references or function calls in the right-hand side are replaced by
  1601. their values before the result is assigned to the macro. The command
  1602. for this is ASSIGN rather than DEFINE:
  1603. define alphabet abcdefghijklmnopqrstuvwxyz
  1604. assign backwards \freverse(\m(alphabet))
  1605. echo "Alphabet backwards = \m(backwards)"
  1606. which prints:
  1607. Alphabet backwards = zyxwvutsrqponmlkjihgfedcba
  1608. This kind of assignment can also be done like this:
  1609. .alphabet = abcdefghijklmnopqrstuvwxyz
  1610. .backwards := \freverse(\m(alphabet))
  1611. [112]Any command starting with a period is an assignment, and the
  1612. operator (= or :=) tells what to do with the right-hand side before
  1613. making the assignment.
  1614. In both the DEFINE and ASSIGN commands, the variable name itself is
  1615. taken literally. It is also possible, however, to have Kermit compute
  1616. the variable name. This is done (as described in [113]Using C-Kermit,
  1617. 2nd Ed., p.457), using parallel commands that start with underscore:
  1618. _DEFINE and _ASSIGN (alias _DEF and _ASG). These are just like DEFINE
  1619. and ASSIGN except they evaluate the variable name before making the
  1620. assignment. For example:
  1621. define \%a one
  1622. _define \%a\%a\%a 111
  1623. would create a macro named ONEONEONE with a value of 111, and:
  1624. define \%a one
  1625. define number 111
  1626. _assign \%a\%a\%a \m(number)
  1627. would create the same macro with the same value, but:
  1628. define \%a one
  1629. define number 111
  1630. _define \%a\%a\%a \m(number)
  1631. would give the macro a value of "\m(number)".
  1632. You can use the _ASSIGN command to create any kind of data structure
  1633. you want; you can find some examples in the [114]Object-Oriented
  1634. Programming section of the [115]Kermit Script Library. In the following
  1635. program we use this capability to create a two-dimensional array, or
  1636. matrix, to hold the all the elements of the CSV file, and then to
  1637. display the matrix:
  1638. fopen /read \%c data.csv # Open CSV file
  1639. if fail exit 1
  1640. .\%r = 0 # Row
  1641. .\%m = 0 # Maximum columns
  1642. while true {
  1643. fread /line \%c line # Read a record
  1644. if fail break # End of file
  1645. .\%n := \fsplit(\m(line),&a,\44,CSV) # Split record into items
  1646. incr \%r # Count this row
  1647. for \%i 1 \%n 1 { # Assign items to this row of matrix
  1648. _asg a[\%r][\%i] \&a[\%i]
  1649. }
  1650. if > \%i \%m { .\%m := \%i } # Remember width of widest row
  1651. }
  1652. fclose \%c # Close CSV file
  1653. decrement \%m # (because of how FOR loop works)
  1654. echo MATRIX A ROWS: \%r COLUMNS: \%m # Show the matrix
  1655. for \%i 1 \%r 1 { # Loop through rows
  1656. for \%j 1 \%m 1 { # Loop through columns of each row
  1657. xecho "\flpad(\m(a[\%i][\%j]),6)"
  1658. }
  1659. echo
  1660. }
  1661. exit 0
  1662. The matrix is called a and its elements are a[1][1], a[1][2], a[1][3],
  1663. ... a[2][1], etc, and you can treat this data structure exactly like a
  1664. two-dimensional array, in which you can refer to any element by its "X
  1665. and Y coordinates". For example, if the CSV file contained numeric data
  1666. you could compute row and column sums using simple FOR loops and
  1667. Kermit's built-in one-dimensional array data type:
  1668. declare \&r[\%r] # Make an array for the row sums
  1669. declare \&c[\%m] # Make an array for the column sums
  1670. for \%i 1 \%r 1 { # Loop through rows
  1672. increment \&r[\%i] \m(a[\%i][\%j]) # Accumulate row sum
  1673. increment \&c[\%j] \m(a[\%i][\%j]) # Accumulate column sum
  1674. }
  1675. }
  1676. Note that the sum arrays don't have to be initialized to zero because
  1677. Kermit's INCREMENT command treats empty definitions as zero.
  1678. Demonstration Scripts for Webmasters
  1679. These scripts all use new features of C-Kermit 9.0.
  1680. [116]ksitemap
  1681. A C-Kermit 9.0 script to build sitemap.xml for a website,
  1682. complete with Google image extensions (this is the file used by
  1683. webmasters to get their sites crawled and indexed optimally).
  1684. [117]The Weblog Script
  1685. Reads a web log, extracts the Google searches, normalizes the
  1686. search strings, and prints the top 20 searches, along with their
  1687. counts.
  1688. [118]The Amazon Script
  1689. Reads an Amazon Associate orders report and lists the products
  1690. according to the number of orders for each, or the number of
  1691. clicks on each.
  1692. [119]Photoalbum
  1693. Makes a website from a collection of JPG images.
  1694. [120]Home [121]Kermit 95 [122]C-Kermit [123]Scripts [124]Current
  1695. [125]New [126]FAQ [127]Support
  1696. C-Kermit 9.0 / [128]The Kermit Project / [129]Columbia University /
  1697. [130] / [131]validate