String Class: long strings plus manipulation functions

[DemonWasp]

It occurred to me that having a limit of 255 characters to a string (presumably because Holtsoft's implementation uses a char [256] in the background) is quite restrictive. Even worse, Turing strings have minimal methods associated with them. So, if your project happens to require a more powerful string implementation, here's one. It's based on the Java String API, but unlike Java strings, this one is mutable, which means you can change its contents (as can any function you pass it into).

Documentation is pretty sparse, but most things are similar to the Java String API. I'd be happy to add more, but I'm off shortly. You can run my simple smoke-test application by running StringTest.t . To use String.t in your own programs, just have

code:

include "String.t"

at the top of your file; that tells Turing to pretend that the contents of String.t are right there at the top of the file.

Suggestions, criticisms, comments all welcome.

[andrew.]

Very nice!! It looks great. One problem though, I get an error when it gets to the stress test.

It opens your String.t file and points to line 370. The error is I/O attempted on unopened stream number -10. Open on line 255........

[DemonWasp]

That's because I didn't include test.txt. Just put whatever you want in there; I took the UTF-8 article from Wikipedia, copy-pasted it into Notepad, then saved in ANSI format (it'll warn you about some characters not being available in ANSI, ignore it. Turing doesn't appear to handle anything other than ANSI).

[SNIPERDUDE]

That is a very useful tool to have. Thank you for your share.

I created a simple function about a year ago that mimics VB's Tokenize function. I've certainly found it useful.

Here it is:

Turing:

type Tokenizer_Prop :     record         OldSent : string         TokCurs : int     end record var Tokenizer : Tokenizer_Prop fcn Tokenize (Sent, token : string) : string     if Sent ~= Tokenizer.OldSent then         Tokenizer.TokCurs := 1         Tokenizer.OldSent := Sent     end if     var tmp : string := ""     var ext : boolean := false     for j : 1 .. length (token)         if Sent (Tokenizer.TokCurs) = token (j) then             Tokenizer.TokCurs += 1         end if     end for     for i : Tokenizer.TokCurs .. length (Sent)         for j : 1 .. length (token)             if Sent (i) = token (j) then                 Tokenizer.TokCurs := i + 1                 ext := true             else                 tmp += Sent (i)             end if             exit when ext         end for         exit when ext     end for     result tmp end Tokenize

What this basically does is reads a sentence and returns the next word each time you call it. So if you wanted to read a sentence and call each word, it would look like this:

Turing:

var sentence : string := "This is a test sentence" var word : array 1 .. 5 of string for i : 1 .. 5     word (i) := Tokenize (sentence, " ") end for % I don't know why someone would put two similar loops right after each other, % but this is just to show  the results for i : 1 .. 5     put word (i) + " "..     % reconstructs the sentence end for % This would output "This is a test sentence "

I have found this function quite useful for reading files for game information. You don't just have to use a space for the token either, could be anything. I do hope to expand it a bit to be more flexible like VB's, but as of the moment it is still efficient enough. It will keep drawing the next chunk of text from that sentence until it recognizes the sentence is different.

[DemonWasp]

Hmm...a good point, it may be worth implementing StringTokenizer (which is also available in the Java API) in Turing.

Word of warning: although I made an effort to code efficiently, I make no promises as to the performance of this String class. Be aware that outputting from these strings may be quite slow (to the screen, anyway). Other operations should be quick, but will still run much faster on a Turing-native string of the same size (as those operations will be run in compiled methods within Turing, rather than my Turing code.

[DemonWasp]

Addition: created a StringTokenizer class, corrected some bugs in certain String methods not tested by StringTest.t. The new StringTokenizer has methods create(^String, ^String), createBasic (^String, string), hasMoreTokens(), nextToken(), countTokens(), all of which behave more-or-less as Java's specification. I haven't done any formal testing (again!) but here's an example:

Turing:

View.Set ("text:80;120;title:StringTokenizer Test") var myStr : ^String var tokenizer : ^StringTokenizer new myStr new tokenizer myStr ->     create ("ASCII stands for American Standard Code for Information Interchange. Computers can only understand numbers, so an ASCII code is the numerical representation of a character such as 'a' or '@' or an action of some sort.") tokenizer -> createBasic (myStr, " ") put "The string is '".. myStr->output() put "'" put "#Tokens = ", tokenizer -> countTokens () var token : int := 1 loop     exit when not tokenizer -> hasMoreTokens ()     put "Token #",token," = '"..     tokenizer -> nextToken () -> output ()     put "'"     token+=1 end loop

Again, all comments / suggestions / criticisms are welcome.

[Tony]

code:

tokenizer -> createBasic (myStr, "")

Quote:

#Tokens = 1
Token #1 = 'SCII stands for American Standard Code for Information Interchange. Computers can only understand numbers, so an ASCII code is the numerical representation of a character such as 'a' or '@' or an action of some sort.'

[DemonWasp]

Hmm...not exactly a standard use-case, tokenizing a string with a blank delimiter, but you make a good point. In fact, that error turned out to be indicative of a larger issue - I had a 1 where I should have had a delim->leng(). Fixed version now outputs the full String if you delimit with a blank string, and should delimit correctly even with multi-character delimiters.

[Tony]

A blank delimiter is used to split a string into individual characters, so I would consider it a valid use (the popularity of which though, is up to one's interpretation).

[DemonWasp]

Hmmm...I'm prone to not implementing that, since you could just use charAt() to achieve the same effect (indexing characters as an array).