![]() To make it easier to use, I made it an executable with launch4j totally superfluous move but whatever. I found the avro-tools.jar tool inside Apache.Avro.I think Apache.Avro takes a long or a varint somehow, and the hex editor I used doesn't give me that info. Somehow the binary in the file gives that info somehow, but I still have no idea, I couldn't get it with a hex editor even. Well, I have no idea where a block starts and ends. I research Avro and see that it is seperated by an uncompressed Schema, followed by the name of the compression codec of the data, then the data itself, which are divided into blocks.Since Parquet.Net uses IronSnappy, I built/added out the snappy codec in Apache.Avro by basically cloning its deflate code and changing a few names.But I can get the schema as that is uncompressed apparently. I tried Apache.Avro and Confluent version as well, but their.Or how do I determine what part of the Avro is snappy compressed and pass that to Ironsnappy. Is there anything anywhere that can open Snappy compressed Avro files with C#? Just adding the snappy codec to Apache.Avro and changing the DeflateStream to Ironsnappy stream failed. I'm open to any solution, even if that means downloading the source for Apache.Avro and adding in Snappy support manually, but to be honest, I'm sort of a newbie and have no idea how compression even works let alone add support to a library. I'm starting to wonder if there is anything in the Azure HDInsight library, but I cant seem to find the nuget package that gives me a way to read Avro with support for Snappy compression. Ironsnappy is unable to decompress the file too, it says the input is using ( reader = (avro_path))Ĭonsole.WriteLine(reader.HasNext()) //trueĬonsole.WriteLine(hi.ElementAt(0).ToString()) // error Although they can get me the Schema, I cannot get at the data. This can be opened with in Spark but it seems snappy is unsupported by Apache.Avro and Confluent.Avro, they only have deflate and null. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |