你的位置:首页 > Java教程

[Java教程]Java读写avro例子


一、avro是一个数据序列化框架,可以高效得进行序列化和反序列化,支持C, C++, C#, Java, PHP, Python, 和Ruby语言。现在使用Java来读写。

二、环境搭建

  1、下载avro-1.7.7.jar and avro-tools-1.7.7.jar两个jar包,放到指定文件目录。下载地址 http://www.trieuvan.com/apache/avro/avro-1.7.7/java/

    我放到了D:\soft\avro 文件夹,在改目录下新建java文件夹,用来存放生成的Java代码

   2、该目录下新建user.avsc文件,内容是:  

{"namespace": "example.avro", "type": "record", "name": "User", "fields": [   {"name": "name", "type": "string"},   {"name": "favorite_number", "type": ["int", "null"]},   {"name": "favorite_color", "type": ["string", "null"]}
 ]}  

  3、打开cmd,进入到该目录,执行命令生成User类

java -jar avro-tools-1.7.7.jar compile schema user.avsc java .

  

  在该文件夹下的Java文件下的../example/avro/目录下就会生成User.java文件。

  4.使用eclipse新建maven项目,在pom.

<dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> <version>1.7.7</version></dependency>

三、生成的User.java文件的内容如下。

  把生成的User.java类复制到工程中,注意这个User.java里面生成的User类及其内部类的包名默认是user.avsc文件中的namespace的值,

  在本例中也就是example.avro。需要全部替换为自己的包名。

  最简单的方法就是把User.java中的example.avro全部替换为自己的包名。

/** * Autogenerated by Avro * * DO NOT EDIT DIRECTLY */package example.avro; @SuppressWarnings("all")@org.apache.avro.specific.AvroGeneratedpublic class User extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord { public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"User\",\"namespace\":\"example.avro\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"favorite_number\",\"type\":[\"int\",\"null\"]},{\"name\":\"favorite_color\",\"type\":[\"string\",\"null\"]}]}"); public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; } @Deprecated public java.lang.CharSequence name; @Deprecated public java.lang.Integer favorite_number; @Deprecated public java.lang.CharSequence favorite_color; /**  * Default constructor. Note that this does not initialize fields  * to their default values from the schema. If that is desired then  * one should use <code>newBuilder()</code>.  */ public User() {} /**  * All-args constructor.  */ public User(java.lang.CharSequence name, java.lang.Integer favorite_number, java.lang.CharSequence favorite_color) {  this.name = name;  this.favorite_number = favorite_number;  this.favorite_color = favorite_color; } public org.apache.avro.Schema getSchema() { return SCHEMA$; } // Used by DatumWriter. Applications should not call.  public java.lang.Object get(int field$) {  switch (field$) {  case 0: return name;  case 1: return favorite_number;  case 2: return favorite_color;  default: throw new org.apache.avro.AvroRuntimeException("Bad index");  } } // Used by DatumReader. Applications should not call.  @SuppressWarnings(value="unchecked") public void put(int field$, java.lang.Object value$) {  switch (field$) {  case 0: name = (java.lang.CharSequence)value$; break;  case 1: favorite_number = (java.lang.Integer)value$; break;  case 2: favorite_color = (java.lang.CharSequence)value$; break;  default: throw new org.apache.avro.AvroRuntimeException("Bad index");  } } /**  * Gets the value of the 'name' field.  */ public java.lang.CharSequence getName() {  return name; } /**  * Sets the value of the 'name' field.  * @param value the value to set.  */ public void setName(java.lang.CharSequence value) {  this.name = value; } /**  * Gets the value of the 'favorite_number' field.  */ public java.lang.Integer getFavoriteNumber() {  return favorite_number; } /**  * Sets the value of the 'favorite_number' field.  * @param value the value to set.  */ public void setFavoriteNumber(java.lang.Integer value) {  this.favorite_number = value; } /**  * Gets the value of the 'favorite_color' field.  */ public java.lang.CharSequence getFavoriteColor() {  return favorite_color; } /**  * Sets the value of the 'favorite_color' field.  * @param value the value to set.  */ public void setFavoriteColor(java.lang.CharSequence value) {  this.favorite_color = value; } /** Creates a new User RecordBuilder */ public static example.avro.User.Builder newBuilder() {  return new example.avro.User.Builder(); }  /** Creates a new User RecordBuilder by copying an existing Builder */ public static example.avro.User.Builder newBuilder(example.avro.User.Builder other) {  return new example.avro.User.Builder(other); }  /** Creates a new User RecordBuilder by copying an existing User instance */ public static example.avro.User.Builder newBuilder(example.avro.User other) {  return new example.avro.User.Builder(other); }  /**  * RecordBuilder for User instances.  */ public static class Builder extends org.apache.avro.specific.SpecificRecordBuilderBase<User>  implements org.apache.avro.data.RecordBuilder<User> {  private java.lang.CharSequence name;  private java.lang.Integer favorite_number;  private java.lang.CharSequence favorite_color;  /** Creates a new Builder */  private Builder() {   super(example.avro.User.SCHEMA$);  }    /** Creates a Builder by copying an existing Builder */  private Builder(example.avro.User.Builder other) {   super(other);   if (isValidValue(fields()[0], other.name)) {    this.name = data().deepCopy(fields()[0].schema(), other.name);    fieldSetFlags()[0] = true;   }   if (isValidValue(fields()[1], other.favorite_number)) {    this.favorite_number = data().deepCopy(fields()[1].schema(), other.favorite_number);    fieldSetFlags()[1] = true;   }   if (isValidValue(fields()[2], other.favorite_color)) {    this.favorite_color = data().deepCopy(fields()[2].schema(), other.favorite_color);    fieldSetFlags()[2] = true;   }  }    /** Creates a Builder by copying an existing User instance */  private Builder(example.avro.User other) {      super(example.avro.User.SCHEMA$);   if (isValidValue(fields()[0], other.name)) {    this.name = data().deepCopy(fields()[0].schema(), other.name);    fieldSetFlags()[0] = true;   }   if (isValidValue(fields()[1], other.favorite_number)) {    this.favorite_number = data().deepCopy(fields()[1].schema(), other.favorite_number);    fieldSetFlags()[1] = true;   }   if (isValidValue(fields()[2], other.favorite_color)) {    this.favorite_color = data().deepCopy(fields()[2].schema(), other.favorite_color);    fieldSetFlags()[2] = true;   }  }  /** Gets the value of the 'name' field */  public java.lang.CharSequence getName() {   return name;  }    /** Sets the value of the 'name' field */  public example.avro.User.Builder setName(java.lang.CharSequence value) {   validate(fields()[0], value);   this.name = value;   fieldSetFlags()[0] = true;   return this;   }    /** Checks whether the 'name' field has been set */  public boolean hasName() {   return fieldSetFlags()[0];  }    /** Clears the value of the 'name' field */  public example.avro.User.Builder clearName() {   name = null;   fieldSetFlags()[0] = false;   return this;  }  /** Gets the value of the 'favorite_number' field */  public java.lang.Integer getFavoriteNumber() {   return favorite_number;  }    /** Sets the value of the 'favorite_number' field */  public example.avro.User.Builder setFavoriteNumber(java.lang.Integer value) {   validate(fields()[1], value);   this.favorite_number = value;   fieldSetFlags()[1] = true;   return this;   }    /** Checks whether the 'favorite_number' field has been set */  public boolean hasFavoriteNumber() {   return fieldSetFlags()[1];  }    /** Clears the value of the 'favorite_number' field */  public example.avro.User.Builder clearFavoriteNumber() {   favorite_number = null;   fieldSetFlags()[1] = false;   return this;  }  /** Gets the value of the 'favorite_color' field */  public java.lang.CharSequence getFavoriteColor() {   return favorite_color;  }    /** Sets the value of the 'favorite_color' field */  public example.avro.User.Builder setFavoriteColor(java.lang.CharSequence value) {   validate(fields()[2], value);   this.favorite_color = value;   fieldSetFlags()[2] = true;   return this;   }    /** Checks whether the 'favorite_color' field has been set */  public boolean hasFavoriteColor() {   return fieldSetFlags()[2];  }    /** Clears the value of the 'favorite_color' field */  public example.avro.User.Builder clearFavoriteColor() {   favorite_color = null;   fieldSetFlags()[2] = false;   return this;  }  @Override  public User build() {   try {    User record = new User();    record.name = fieldSetFlags()[0] ? this.name : (java.lang.CharSequence) defaultValue(fields()[0]);    record.favorite_number = fieldSetFlags()[1] ? this.favorite_number : (java.lang.Integer) defaultValue(fields()[1]);    record.favorite_color = fieldSetFlags()[2] ? this.favorite_color : (java.lang.CharSequence) defaultValue(fields()[2]);    return record;   } catch (Exception e) {    throw new org.apache.avro.AvroRuntimeException(e);   }  } }}

四、用Java实现序列化,即写avro文件。

  新建一个Java类  

     public static void main(String[] args) throws IOException {
     // 声明并初始化User对象
     // 方式一
     User user1 = new User(); user1.setName("zhangsan"); user1.setFavoriteNumber(21); user1.setFavoriteColor(null);
     // 方式二 使用构造函数 // Alternate constructor User user2 = new User("Ben", 7, "red");      
    // 方式三,使用Build方式 // Construct via builder User user3 = User.newBuilder() .setName("Charlie") .setFavoriteColor("blue") .setFavoriteNumber(null) .build(); String path = "D:\\tmp\\user.avro"; // avro文件存放目录 DatumWriter<User> userDatumWriter = new SpecificDatumWriter<User>(User.class); DataFileWriter<User> dataFileWriter = new DataFileWriter<User>(userDatumWriter); dataFileWriter.create(user1.getSchema(), new File(path));
     // 把生成的user对象写入到avro文件 dataFileWriter.append(user1); dataFileWriter.append(user2); dataFileWriter.append(user3); dataFileWriter.close();
}

  run一下代码,查看指定文件目录是否生成了avro文件。

五、Java读取avro文件,即实现avro反序列化。

public static void main(String[] args) throws IOException {    DatumReader<User> reader = new SpecificDatumReader<User>(User.class);    DataFileReader<User> dataFileReader = new DataFileReader<User>(new File("D:\\tmp\\user.avro"), reader);    User user = null;    while (dataFileReader.hasNext()) {      user = dataFileReader.next();      System.out.println(user);    }   }

   运行结果:

{"name": "zhangsan", "favorite_number": 21, "favorite_color": null}{"name": "Ben", "favorite_number": 7, "favorite_color": "red"}{"name": "Charlie", "favorite_number": null, "favorite_color": "blue"}

至此,例子写完。