Data formats for Data Science
The plain text is one of the simplest yet most intuitive format in which data could be stored. It is easy to create, human and machine readable, storage-friendly (i.e. highly compressible), and quite fast to process. Textual data can also be easily structured; in fact to date the CSV (Comma Separated Values) is the most common data format among data scientists.
However, this format is not properly suited in case data require any sort of internal hierarchical structure, or if data are too big to fit in a single disk.
In these cases other formats must be considered, according to the shape of data, and the specific constraints imposed by the context. These formats may leverage general purpose solutions, e.g. [No]SQL databases, HDFS (Hadoop File System); or may be specifically designed for scientific data, e.g. hdf5, ROOT, NetCDF.
In this talk, the strength and flaws of each solution will be discussed, focusing on their usage for scientific computations. The goal is to provide some practical guidelines for data scientists, derived from the the comparison of the different Pythonic solutions presented for the case study analysed. These will include xarray, pyROOT vs rootpy, h5py vs PyTables, bcolz, and blaze. Finally, few notes about the new trends for columnar databases (e.g. MonetDB) will be also presented, for very fast in-memory analytics.
Postdoc researcher, FBK
Valerio Maggio has a Ph.D. in Computational Science from the University of Naples "Federico II" and he is currently a Postdoc researcher in the FBK – MPBA team in Trento, Italy. His research interests are focused on Machine Learning, Deep Learning and Big Data Analysis. Valerio is very much involved in the scientific Python community, and he is an active speaker at many Python conference including EuroPython and EuroScipy. He uses Python as the mainstream language for his deep/machine learning code, making an intensive use of (especially) Numpy-based packages (e.g. scikit-learn) to crunch, munge, and analyse data. Valerio is a member of the Italian Python community, and the lead organiser of the PyData Italy conferences, held in Florence on 2015 and 2016. He also enjoys drinking tea and playing go.