StringTokenizer in Java

StringTokenizer class in Java provides a tokenizer String method for splitting a string into tokens based on a specified delimiter. The tokenizer String can be any string that separates the tokens, such as a comma, semicolon, or whitespace. Using the tokenizer String method of StringTokenizer class, we can split a string into its constituent parts. By calling the nextToken() method, we can retrieve each token in turn, and by using the hasMoreTokens() method, we can check if there are any more tokens left. The length() method of the StringTokenizer class can be used to get the length of each token. StringTokenizer String is a useful tool for string manipulation and can be used to parse CSV files, URLs, or other text-based data. The StringTokenizer class is part of the Java.util package and provides a simple way to split a string into tokens. The class has two constructors, one that takes a string to be tokenized and a delimiter character or string, and another that takes the same arguments as well as a Boolean flag indicating whether or not to include the delimiter as a token. Once you have created a StringTokenizer object, you can use its various methods to iterate through the tokens and perform various operations on them.

Tokenization Method

The tokenization method is the process of splitting a string into smaller parts or tokens. This process is performed using a delimiter, which can be a character or a string of characters that separates each token. For example, consider the following string:

String input = "Hello World! How are you today?";

If we want to split this string into individual words, we can use the space character as the delimiter, like so:

StringTokenizer tokenizer = new StringTokenizer(input, " ");
while (tokenizer.hasMoreTokens()) {
    String token = tokenizer.nextToken();
    System.out.println(token);
}

Output

In this example, we create a new StringTokenizer object with the input string and a space character as the delimiter. We then loop through the tokens using the hasMoreTokens() method and retrieve each token using the nextToken() method. Finally, we print each token to the console.

Token Length

In addition to the basic functionality provided by the nextToken() method, the StringTokenizer class also provides methods for retrieving the length of each token and for retrieving a specific token by index. To get the length of the current token, you can use the token.length() method. For example:

public class StringTokenizerExample {
    public static void main(String[] args) {
        String input = "Hello World! How are you today?";
        StringTokenizer tokenizer = new StringTokenizer(input, " ");
        while (tokenizer.hasMoreTokens()) {
            String token = tokenizer.nextToken();
            System.out.println("Token: " + token + " Length: " + token.length());
        }
    }
}

Output

In this example, we retrieve each token as before but also use the length() method to get the length of each token, which we then print to the console.

Example

Let's take a look at a complete example that demonstrates how to use the StringTokenizer class in Java:

public class Example {
    public static void main(String[] args) {
        String input = "John,Doe,123 Main St.,Anytown,USA";
        StringTokenizer tokenizer = new StringTokenizer(input, ",");
        while (tokenizer.hasMoreTokens()) {
            System.out.println(tokenizer.nextToken());
        }
    }
}

Output

In this example, we have a string that represents a comma-separated list of values. We create a new StringTokenizer object with this string and a comma as the delimiter. We then loop through the tokens using the hasMoreTokens() method and retrieve each token using the nextToken() method. Finally, we print each token to the console. Note that we did not use the length() method in this example, as we are only interested in the individual tokens themselves.

StreamTokenizer Class

While the StringTokenizer class provides a simple way to split a string into tokens, it does have some limitations. For example, it cannot handle multiple delimiters or different types of tokens, such as numbers and words. If you need more advanced tokenization capabilities, you can use the StreamTokenizer class. This class provides more flexibility and can handle various input types. To reinforce what you learned, we suggest you watch a video lesson from our Java Course

Conclusion

In this article, we have explored the StringTokenizer class in Java and seen how it can be used to split a string into tokens. We have also seen how to use some of the class's methods to perform various operations on the tokens, such as getting their length and retrieving specific tokens by index. While the StringTokenizer class is a simple and useful tool, it does have its limitations. If you need more advanced tokenization capabilities, consider using the StreamTokenizer class or a more powerful library such as Apache Commons Lang. Remember to always consider the specific needs of your project when deciding which tools and techniques to use. With a solid understanding of Java's string manipulation classes and methods, you can tackle even the most complex string manipulation tasks with ease.