7. Python initialise scripts¶
Python initialise scripts provide the following features:
- Management commands
- Creating EXIOBASE numpy objects
- Building mapping coordinates files for the application
- Creating custom geojson and topojson files
Note: the scripts under point 2, 3 and 4 are not directly used by the application at runtime. If you wish to extent or develop features such as using another EEIO dataset it is recommended to investigate how these script work, alternatively these points can be skipped. Deployment management commands are used for populating the database and are needed for deployment.
7.1. Management commands¶
This application uses a sql-lite database to store mapping coordinates, user queries (Jobs) and user results (Celery results). For mapping coordinates the database needs to be populated before running the application.
The management commands uses CSV files generated by
The following command is used to populate the database
python manage.py populateHierarchies populates the database with the mapping files needed.
Aside from database population, it is advised to clean the database of user results whenever needed. This can be done
with the following command
python manage.py clear_models
Note : see project folder ramascene/management/commands
7.2. Creating EXIOBASE numpy objects¶
Originally EXIOBASE data is structured in tabulated text file format. This application uses the python numpy library to perform calculations and therefore the original EXIOBASE data is converted to numpy objects. Two versions of numpy formatted EXIOBASE data are created:
- v3: an unmodified version. Note that the unmodified version is constructed in such a way that the Rama-Scene calculation procedure including indicators works efficiently as opposed to the default EXIOBASE tabular format.
- v4: a modified version including secondary materials
The original script to create version 3 is located at
The script expects the following folder structure:
- script/create_numpy_objects_v3.py -> the actual conversion script
- data/auxiliary -> auxiliary information to determine indicators in the application
- data/clean/<year> -> a folder that contains the years reserved for output (not in the source code, because of the large amount of data)
- data/raw/<year> -> a folder that contains the original txt files per year (not in the source code, because of the large amount of data)
For version 4 please refer to https://bitbucket.org/CML-IE/pysuttoio/src/master/
Note : see project folder python_ini/devScripts/script/create_numpy_objects_v3.py and https://bitbucket.org/CML-IE/pysuttoio/src/master/
7.3. Building mapping coordinates files for the application¶
EXIOBASE v3.3 has specific classifications that needs to map to user input. For example if Europe is selected by the user as one of the parameters for analyses, the calculation procedure uses indices corresponding to all countries belonging to Europe. In turn these calculation results need to be aggregated back into a single value for Europe. For coordinating user input to calculation procedures we developed the following mapping CSV’s based on an older application called ExioVisuals :
These files are read in by the script
prepare_csv.py that in turn makes a slight modification to easily denote aggregated/disaggregated countries or product categories.
The output files of
prepare_csv.py are prefixed with mod_<filename> and is used by one of the management commands.
Note : see project folder python_ini/data & python_ini/devScripts.
7.4. Creating custom geojson and topojson files¶
For visualizations of continental data custom polygons need to be created that reflect the country mapping of EXIOBASE. A script is developed to dissolve countries that belong to a certain continent or “rest of ” classification with the library GeoPandas.
- First the mod_final_countryTree_exiovisuals.csv (modified by prepare_csv script above) is further adjusted to contain the 3-letter ISO code available in the DESIRE country list excel.The default modified file only contains 2-letter codes and this cannot be used by the script needed to create polygon files.Changing 2-letter to 3-letter codes is done manually by using the DESIRE country list, the new file is called mod_final_countryTree_ISO3166_3.csv.
- Run prepare_geomapping_ISO3166_3.py with mod_final_countryTree_ISO3166_3.csv
- After step 1 and 2, create a virtual environment and install the requirements.txt in the python_ini/geoMapbuilds to enable the use of GeoPandas.
- The script geo_dissolve_by_level.py is used to create 3 geojson files, 1) a file for the whole world, 2) a file for continents, 3) a file for countries and rest of regions. This is needed for the visualization library d3plus. Before running the script, please select which of the three files you wish to generate by adjusting the SETTING variable.
- The generated files are fairly big, in turn it is key to make these small smaller by converting them to topojson and simplify the polygons using MapShaper.org. This is achieved with 1) importing the files into mapshaper, 2) clicking on “simplify”, check “prevent shape removal” and finally set the percentage to 5% 3) finally export to topojson with command “id-field=id” “drop-table”
Note : see project folder python_ini/geomapBuilds and python_ini/devScripts