What is a StringTokenizer?

In java, there is a way to break up a line of text into what are called tokens. A token is a smaller piece of a string. This method of breaking up tokens are come from the StringTokenizer class.

Let's say that we wanted to break up the following line of text into smaller pieces:

"Hello and welcome to programming"

Here, the token we need is just a simple white space. So here is how to declare a StringTokeizer object:

StringTokenizer name = new StringTokenizer(variable, Token(s));

where in the above, name is an appropriate name for the tokenizer object; variable is the name of a String variable you are using or even just a simple string; and token(s) is the token you are specifying.

In the above definition, if the second argument of the constructor is not specified, the token, by default, is a white space. If the token that is declared does not exist in the String you are searching through, the entire string is returned to you.

Import statements

In order for the StringTokenizer class to be used, you must import it from a certain library; here the util library.

An import statement always appears outside the class header line. So here is how the StringTokenizer statement will look:

import java.util.*;

public class XXXXXXXXXX {
  //code here
}

The above is specifically for the StringTokenizer class. There are many other libraries that can be used in java. When the time is needed to describe these, we will.

Methods

Here are the most used functions in the StringTokenizer class.

boolean hasMoreTokens()

This method will return a boolean value that will represent if there are any more tokens left in the line you are trying to tokenize. The true return will show that there is at least 1 more token in the line.

int countTokens()

This method will simply return an integer representing the number of tokens in the String. This method does not advance the position in the String.

String nextToken()

This method is the primary method of the class as it will get you the next piece of data in the String. The position in the string is changed after this method is called.

Below is an example showing these above methods.

import java.util.StringTokenizer;
public class StringTokenizerExample{

  public static void main(String args[]){

        String s = "What on earth is going on here?";

        //by default, a white space:
	StringTokenizer st = new StringTokenizer(s);

        //when there are still more tokens, print out the
        //next one:
	while(st.hasMoreTokens())
		System.out.println(st.nextToken());
  } //main
} //class

import java.util.StringTokenizer;
public class StringTokenizerExample2{
  public static void main(String args[]){

          String s = "Five+Three=Nine-One";
	  String arr[];

          //declare it with 3 tokens as seen above:
	  StringTokenizer st = new StringTokenizer(s, "+=-");

	  //the array size is the number of tokens in the String:
	  arr = new String[st.countTokens()];

          //when there are still more tokens, place it in the array:
	  int i = 0;
          while(st.hasMoreTokens()){
		arr[i] = st.nextToken();
                i++;
          }

	  //determine the word with the largest length:
          int indexMax = 0;
          for(int i = 1; i < arr.length; i++){
             if(arr[i].length() > arr[indexMax].length())
			indexMax = i;
          }

	  System.out.println("The largest element is in index: " 
                + indexMax);

  } //main
} //class