After a few projects where I had to parse through legacy flat files I decided enough was enough and decided to write my own parser. This parser would do exactly one thing efficiently and that was convert lines from the flat file to java objects. I wanted something that was thin and did exactly what I mentioned above and no other frills. Though now that I have it working a few frills may be in order 
. I have created a project at JavaForge where this tool will reside. If you do find it useful please drop a comment in the discussions forum on the javaforge site javaforge.com/project/2066. The goal is to parse a flat file (either character separated columns or fixed length columns). The parser supports two methods of parsing a file. In the first approach you are responsible for reading the file and providing each line that needs to be transformed to the transformer. The second approach is SAX-like, in that you register a listener and the transformer will call your listener whenever it finds a record and also when it could not resolve a record. First let’s run through the first approach and at the end I will show you the SAX-line parsing approach.
Let’s create a java bean class to represent our record with space character separated columns.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
import org.aver.fft.annotations.Column; import org.aver.fft.annotations.Transform; @Transform(spaceEscapeCharacter = "_", recordIdValue = "88") public class DelimitedBean { // /..... @Column(position = 1, required = true) public int getRecordId() { return recordId; } @Column(position = 2, required = true) public String getNameOnCard() { return nameOnCard; } @Column(position = 3, required = true) public String getCardNumber() { return cardNumber; } @Column(position = 4, required = true) public int getExpMonth() { return expMonth; } @Column(position = 5, required = true) public int getExpYear() { return expYear; } @Column(position = 6, required = true) public double getAmount() { return amount; } @Column(position = 7, required = true) public String getCardSecuriyCode() { return cardSecurityCode; } @Column(position = 8, required = true, format = "MMddyyyy") public Date getTransactionDate() { return transactionDate; } // ... other methods here ... } |
As you can see we use Java 5.0 annotations to mark our record format. By default the parser sets itself up to parse character separated columns and the delimiter is space.
|
1 2 |
@Transform (spaceEscapeCharacter="_", recordIdValue="88") |
By default the parser is setup to parse character-separated columns. The attribute spaceEscapeCharacter indicates the character used to represent spaces within column data. The parser can replace that with space before loading it into your java object. The recordIdValue identifies the value of the key column. The transformer keeps an internal mapping of the key value to the java bean class that represents it. By default the first column is the key column. You can change that by passing in parameter recordIdColumn for character separated columns or using recordStartIdColumn / recordEndIdColumn for fixed length columns. By default the column separator is space for character. You can change that using columnSeparator.
That’s enough on defining the file format. Now here is how to actually read it.
|
1 2 3 4 |
Transformer spec = TransformerFactory.getTransformer(new Class[] { DelimitedBean.class }); String line = “88 Mathew_Thomas 4111111111111111 02 2008 12.89 222 10212005″; DelimitedBean bean = (DelimitedBean) spec.loadRecord(line); |
You get a transformer instance as shown above. Pass it an array of all classes that represent your various records and that uses annotations as defined above. Now you have a fully loaded bean from which to read your data. That’s all.
Now lets see how you define the same for a fixed column record format. The parsing code above stays the same. The difference is in how you annotate your result bean class.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
import org.aver.fft.annotations.Column; import org.aver.fft.annotations.Transform; @Transform(spaceEscapeCharacter = "_", columnSeparatorType = Transformer.ColumnSeparator.FIXLENGTH, recordIdStartColumn = 1, recordIdEndColumn = 2, recordIdValue=”88”) public class FixedColBean { @Column(position = 1, start = 1, end = 2, required = true) public int getRecordId() { return recordId; } @Column(position = 2, start = 3, end = 15, required = true) public String getNameOnCard() { return nameOnCard; } @Column(position = 3, start = 16, end = 31, required = true) public String getCardNumber() { return cardNumber; } @Column(position = 4, start = 32, end = 33, required = true) public int getExpMonth() { return expMonth; } @Column(position = 5, start = 34, end = 37, required = true) public int getExpYear() { return expYear; } @Column(position = 6, start = 38, end = 43, required = true) public double getAmount() { return amount; } @Column(position = 7, start = 44, end = 46, required = true) public String getCardSecurityCode() { return cardSecurityCode; } @Column(position = 8, start = 47, end = 54, required = true, format = "MMddyyyy") public Date getTransactionDate() { return transactionDate; } // … other methods here … } |
The parsing logic stays the same. Just give it the correct line of data.
Now I will show you the SAX-like parsing approach.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
package org.aver.fft; import java.io.File; import junit.framework.TestCase; public class DelimitedFullFileReaderTestCase extends TestCase { public void testFullFileReader() { Transformer spec = TransformerFactory.getTransformer(new Class[] { DelimitedBean.class }); spec.parseFlatFile(new File(“c:/multi-record-delcol-file.txt”), new Listener()); } class Listener implements RecordListener { public void foundRecord(Object o) { bean = (DelimitedBean) o; System.out.println(bean.getNameOnCard()); } public void unresolvableRecord(String rec) { } } } |
I have this project located at: www.javaforge.com/proj/summary.do?proj_id=271