My idea here is to collect together a number of tools to help with small analysis tasks related to the JULES model. It's NOT intended to be a place for fully-developed tools/programs, but for TOOLS THAT HALF-WORK AND MAY/MAY NOT BE USEFUL. Please contribute if you have some code fragment or other useful thing that you use but haven't yet had time to work it up into a 'proper' contribution to the JULES community. Equally, if you try out one of the tools and improve it then please email me and I'll update/change what's on here (if I get a number of contributions here then I'll shift this page onto some kind of CEH bitbucket/github page / JULES TRAC page and make it more generally available).
Currently this page here is only about DATA VISUALISATION and GRID CONVERSION OPTIONS, but I've left the title as 'JULES Tools' so that I can make it more general in the future (I hope!). Also, I should stress that I'm not a Python user or an Iris user: I prefer using R as much as possible and this (obviously) influences what tools I find useful below.
Some tools available to CEH internal people only are here.
For general information about Data Visualisation, please see Emma Robinson's talk about Interpreting JULES Output at the 2016 JULES Short Course here and her examples in Python and R here.
- USING 2D NETCDF FILES OUTPUT FROM JULES: Here are some links for dealing with 'normal' 2D NetCDF files. Any links anyone else wants to contribute, please email me!
- On UNIX/Linux people seem to use GrADS (see also here and here), Xconv, Iris (see also Cube Browser and Thea) or Ncview.
- On Windows you can use Panoply (a quick, basic viewer) or a GIS package like QGIS (see also here; has a lot of advantages if you want to compare JULES output to other raster or vector data, e.g. to calculate land surface fluxes over an irregular area like a river catchment).
- You want to animate your NetCDF files, i.e. convert to a movie to include in e.g. a presentation? I think this is possible in QGIS, but otherwise I don't know (I found this example for ArcMap online, but couldn't make it work). GrADS and Ncview (at least) can display as a movie, but I don't think they can save as a movie file. Garry H showed me that there is an 'Export Movie' option on Panoply, but I haven't tried that yet (has anyone used this?). Garry also suggested that if you convert each NetCDF grid into an image then they can be pasted together into a movie in Google Maps .kml/.kmz format (presumably following something like http://stackoverflow.com/questions/10586792/display-netcdf-files-to-google-maps ) and then displayed as a movie in Google Earth.
Does anyone else know anything more about producing movies from NetCDF output? These tips are 100% of what I know at present (!).
- USING 1D NETCDF FILES OUTPUT FROM JULES (i.e. where there is a 'land axis'): Gridded runs from JULES can output data in 2D or 1D NetCDF files. Most packages for displaying gridded data can handle 2D (see above), but very few can handle 1D, so what to do with that? There are several options:
- On UNIX/Windows you can always of course write a tailor-made script to read the 1D Netcdf file into an array and then plot it. I've done this a few times now, but for me has two important disadvantages: (1) Firstly, for large files you will hit memory constraints very quickly on a script. Moving to Fortran or C removes the memory limitation, but you then have much more limited graphics for displaying the data. (2) Secondly, it's very difficult to code something that is 'general' in the sense that it can parse the input file and read in the data irrespective of how many variables you have (I usually have 50-60 in my JULES outputs), how many nonspatial dimensions it has (e.g. scpool, ntiles), what they are called (lat, latitude, latitude0, Latitude, lat_data, etc.) so you usually end up with something less than general and that means you need to recode it completely when you move on to the next application.
- On UNIX/Linux you can use GrADS. You can display a 1D NetCDF file in GrADS if you can generate a .pdef file that correctly describes the grid of the datafile you have. See here for how to construct a pdef file for gridded output. I'm also told that you can get GrADS to output a 2D NetCDF file, which would mean that it can be used as a 1D -> 2D converter as well, but I don't know how to do that (Does anyone know more?).
- On UNIX/Linux you will soon be able to use Iris as well for this. Karina Williams and Richard Gilham have been working on a set of Python libraries called jules.py for streamlining using IRIS for JULES output visualisation. These are almost ready and will include a facility for reading 1D NetCDF files. Richard G in August 2016 told me: "It works as a wrapper around Iris, meaning that once the cube is loaded it then works with normal iris commands. It has limited support for rotated pole datasets, and could be extended for more fancy projections if needed." and Karina W: "jules.py should be available to the whole JULES community very soon, as Kerry is setting up an area of the fcm repository at code.metoffice.gov.uk where we can share code. We’ll put a tutorial on how to get started with these functions there too. The most commonly used functions are jules.load and jules.load_cube, which read in the 1D list of points and converts the data to iris cubes with lat and lon dimensions.". I have been given a 'pre-release' copy of jules.py for testing, but I've been asked not to pass it around without permission (if you ask me you can have it, though I will have to emal R and K about it). Karina has now shared this here.
- LAST OPTION: During August 216 I had the idea of doing this in a completely different way: I have used the NCO tools to 'atomise' the input file (e.g. with n spatial points) into n separate .nc files, each containing only the data for a single point. The NCO tools will preserve all nonspatial dimensions automatically if you extract each point separately using ncea. My R script keeps track of the spatial locations of all these files and then remerges them into a 2D grid using ncecat. Essentially, my script this time doesn't read in any data, but produces a .sh file with a sequence of (long) NCO commands and then I sh all those commands on UNIX to generate the 2D netcdf file. No memory limitations this time and it's 'general' because the NCO tools do the donkey work. It's a bit slow (a file with 6604 land points took 3.5 hours to convert), but I can then use exactly the same script to convert another netcdf file of completely different nonspatial structure and I don't need to modify it, which is nice. Also, the conversion only needs to be done once and then I have a 2D netcdf file and I can view this in lots of different packages. THIS OPTION HAS ONLY BEEN PARTIALLY TESTED BUT YOU'RE WELCOME TO USE IT AND TELL ME HOW IT WORKS: Download here. Current limitations: only works with WFD or WFDEI data.