I’ve written a demo app to show how to aggregate, store, serve & visualize data in a lightweight way, using Web::Machine, PSGI, D3.js, bower, App::Cmd, DBIC etc. It’s on GitHub.
The rest of this post is the app’s README, which tells you all about what it is and how to use it.
The purpose of this demo app is to show a way of visualizing data using lightweight web technologies.
A data set is parsed and stored in a database.
The data is served using a REST API.
The data is visualized with
The main points of interest in this demo are:
Dependencies are specified in
cpanfileand can be installed with
There is only one executable. Everything from database setup to importing the data and serving it are done with this executable, which uses
After parsing, the data is stored in a PostgreSQL database using
DBIx::Class. The DDL was written manually; the schema classes were generated by
dbicdump, also driven by the executable.
The data is served by a REST API using
There is a single HTML file which contains a basic Bootstrap-based UI and an example visualization using
All the pieces work together in a lightweight way. No dynamic HTML is generated on the server-side.
This demo uses historical S&P500 stock data. See
You can run this demo from its git working directory. There is no
Makefile.PL because it is not intended to be installed.
Dependencies are kept in
cpanfile, so you can install them with:
cpanm --installdeps .
bower.json. Install them with:
Command line interface
The app is controlled with a single executable,
./bin/stocks. It uses
App::Cmd to provide various commands. Just running the program without options will display a list of available commands.
./bin/stocks help <command> will display more detailed information about a command.
The demo is configured using an INI file. See
etc/basic.conf for an example.
Configuration keys are strictly checked using
Data::Domain. The following keys are recognized:
dbname: Name of the PostgreSQL database. Mandatory.
dbpass: Optional extra fields for the database connect string.
ddl_file: Path to the DDL file. It can be absolute or relative to the working directory root (i.e., where the
.gitdirectory is). Mandatory.
htdocs: Path to the directory that holds static Web resources like HTML files or CSS files. It can be absolute or relative to the working directory root. Mandatory.
To use a configuration file, you can either specify it as an option to a command or use an envionment variable.
./bin/stocks <command> -c path/to/config
export STOCKS_CONF=path/to/config ./bin/stocks <command>
dbsetup command drops the current database, creates it again and imports the DDL. You need to use the
-y option to make this command run.
./bin/stocks dbsetup -y
import command takes one or more paths to data files. It reads each file, parses it and inserts corresponding rows into the database.
./bin/stocks import data/sp500hst.txt
web command runs a Plack-based web server that serves both the REST API as well as static content.
As you can see in
bower are mounted on
/components, and the
stocks REST resource is mounted on
stocks resource is implemented in
Web::Machine. There are CSV and JSON representations, which you can request by setting the
Accept header in the request. If you don’t use any query parameters, you will get the full data set.
curl -H 'Accept: text/csv' "http://localhost:5000/api/v1/stocks"
However, often this is inefficient; you will want only part of the data, possibly aggregated. You can specify which fields you want by listing them in a comma-separated list in the
fields query parameter. Each field can be either just the field name or it can have a colon-prepended operation. The operation can be
sum, using the standard SQL semantics. For example:
curl -H 'Accept: text/csv' "http://localhost:5000/api/v1/stocks?fields=groupby:ticker,sum:volume"
You can also filter the data by using query parameters corresponding to the column names -
ticker. (It doesn’t really make sense to filter by
volume.) The result will then only contain those rows that have the specified value. So using those query parameters will effectively build up a
curl -H 'Accept: text/csv' "http://localhost:5000/api/v1/stocks?ticker=AAPL"
When the web server is running, you can open
http://localhost:5000/ in your browser. This will load
index.html, which contains a basic Bootstrap UI and a simple visualization using D3.js that uses the REST API to load data.
genschema command uses
dbicdump to regenerate the schema classes. It is only needed during development and is only available if the
STOCKS_DEV environment variable is set.
STOCKS_DEV=1 ./bin/stocks genschema