Flat Data: Postprocessing Library and Examples

A collection of postprocessing helper functions and examples for Flat Data.

These examples and functions are written in Deno, a new language created by the same founders of Node.js and meant to improve on many aspects of Node.

Note: If you're noticing your scripts failing recently, try updating to 0.0.15. More info here

Usage

When writing a Flat Data Action, you can specify a path to a postprocessing Deno script that can manipulate the data downloaded by Flat even further.

- name: Fetch data 
        uses: githubocto/flat@v2
        with:
          http_url: http://api.coindesk.com/v2/bpi/currentprice.json # The endpoint to fetch
          downloaded_filename: btc-price.json # The http_url gets saved and renamed in our repository as btc-price.json
          postprocess: postprocess.ts # A postprocessing javascript or typescript file written in Deno

This is an example of a postprocessing script. Notice the use of Deno.args[0] to pass in the path of the downloaded_filename.

// The Flat Data postprocessing libraries can be found at https://deno.land/x/flat/mod.ts
// Replace 'x' with latest library version
import { readJSON, writeJSON } from 'https://deno.land/x/flat@0.0.x/mod.ts'

const filename = Deno.args[0] // equivalent to writing `const filename = 'btc-price.json'`
const data = await readJSON(filename)

// pluck a specific key off and write it out to a new file
const newfile = `postprocessed_${filename}`
await writeJSON(newfile, data.path.to.something)

Examples

Can be found in the examples folder. Once you install Deno you can run these examples with:

• deno run -A examples/csv/csv-example.ts
• deno run -A examples/csv/arquero-example.ts
• deno run -A --unstable examples/image/image-example.ts
• deno run -A examples/json/json-example.ts
• deno run -A examples/sheets/sheets-example.ts
• deno run -A examples/xlsx/xlsx-example.ts
• deno run -A --unstable examples/zip/zip-example.ts

Deno can run javascript or typescript files, so you can easily convert any of these examples to javascript and run them in the same way:

deno run -A examples/csv/csv-example.js

Using Python

While our examples use a Deno file to run postprocessing tasks, you can also use Python as specified in this example: https://github.com/pierrotsmnrd/flat_data_py_example. Thank you @pierrotsmnrd!

Using bash

You can also use bash as specified in this example: https://github.com/aborruso/flat_data_bash_example. By @aborruso

Postprocessing Library

The Flat Data postprocessing library can be found at: https://deno.land/x/flat/mod.ts

You can import and use these helper functions directly, or treat them as a starting point for writing your own postprocessing scripts.

CSV

readCSV

readCSV(path: string, options?: ParseOptions): Promise<Record<string, unknown>[]>

Args:

• path: path to a local CSV file
• options: options for parsing the CSV file

Usage:

const csv = await readCSV('./path/to/file.csv')

writeCSV

writeCSV(path: string, data: Record<string, unknown>[] | string, options?: Deno.WriteFileOptions)

Args:

• path: path to a local CSV file
• data: string or object array to store
• options: options for writing the CSV file

Usage:

const data = [ 
	{ age: 70, name: 'Rick' },
	{ age: 14, name: 'Smith' }
]
await writeCSV('./path/to/file.csv', data)

TXT

readTXT

readTXT(path: string): string

Args:

• path: path to a local TXT file

Usage:

const text = await readTXT('./path/to/file.txt')

writeTXT

writeTXT(path: string, text: string, options?: Deno.WriteFileOptions): void

Args:

• path: path to a local TXT file
• text: text to write to file
• options: options for writing the TXT file

Usage:

await writeTXT('./path/to/file.txt', 'Content for the file')

JSON

readJSON

readJSON(path: string): JSON

Args:

• path: path to a local JSON file

Usage:

const json = await readJSON('./path/to/file.json')

readJSONFromURL

readJSONFromURL(url: string): JSON

Args:

• url: URL to a json file

Usage:

const json = await readJSON('www.url.com/file.json')

writeJSON

writeJSON(path: string, data: any, replacer?: any, space?: string | number): void

Args:

• path: path to a local JSON file
• data: data to store as JSON
• replacer: replacer function that transforms the results or an array of strings and numbers that acts as an approved list for selecting the object properties that will be stringified
• space: adds indentation, white space, and line break characters to the to the JSON text to make it easier to read.

Usage:

const data = { age: 40 }
await writeJSON('./path/to/file.json', data)

await writeJSON('./path/to/file-with-indentation', data, null, 2)

XLSX

Our library relies on SheetJS, a library for parsing various spreadsheet formats. In addition to a simple readXLSX function you can access the core xlsx module by importing it directly.

import { xlsx, readXLSX } from 'https://deno.land/x/flat/mod.ts'

xlsx provides many more utility functions.

readXLSX

readXLSX(path: string): XLSX.WorkBook

Args:

• path: path to a local XLSX file

Usage:

const workbook = await readXLSX('./path/to/file.xlsx')
const sheetData = workbook.Sheets[workbook.SheetNames[0]]
const csvString = await xlsx.utils.sheet_to_csv(sheetData)

Image

We recommend using a library like imagescript for more advanced image manipulation. See an example here.

readImageFromFile

readImageFromFile(path: string): Promise<Uint8Array>

Args:

• path: path to a local image file

Usage:

const bytes = await readImageFromFile('./path/to/image.jpeg')

readImageFromURL

readImageFromURL(url: string): Promise<{ bytes: Uint8Array; name: string; }>

Args:

• url: url string to an image

Usage:

const image = await readImageFromURL('www.url.com/image.jpg')
const bytes = image.bytes
const name = image.name

writeImage

writeImage(imageBytes: Uint8Array, path: string): void

Args:

• imageBytes: a byte array
• path: path and name to write the image file

Usage:

await writeImage(bytes, './path/to/image.jpeg')

Zip

unZipFromFile

unZipFromFile(
    filePath: string,
    destinationPath: string | null = "./",
    options: any = {},
): Promise<string | false>

Args:

• filePath: a path to a local zip file
• destinationPath: a folder path to unzip the files
• options: option.includeFileName can be true or false

Usage:

const result = await unZipFromFile('./path/to/folder.zip', './unzip/path')
const output = result ? 'File unzipped successfully' : 'Error unzipping'

unZipFromURL

unZipFromURL(
    fileURL: string,
    destinationPath: string | null = "./",
    options: any = {},
): Promise<string | false>

Args:

• filePath: a path to a local zip file
• destinationPath: a folder path to unzip the files
• options: option.includeFileName can be true or false

Usage:

const result = await unZipFromURL('www.url.com/file.zip', './unzip/path')
const output = result ? 'File unzipped successfully' : 'Error unzipping'

Remove

removeFile

removeFile(path: string): void

Args:

• path: path to a local file to delete

Usage:

await removeFile('/path/to/file.x')

Testing

Run all the tests:

deno test -A --unstable tests/*

Run separate tests

deno test -A --unstable tests/csv-test.ts

License

MIT