[Python] Cookbook

• Import file in a different/common-used folder

import sys
sys.path.insert(1, '/path/to/application/app/folder')
import file

importing-files-from-different-folder

• Run system commands without using os.system()

import subprocesssubprocess.call(["ls", "-lt"])

This command calls the shell script and returns the system output and return code attribute (e.g., 0 for no error, 1 for error). This will wait for the completed process.

[QGIS] Join csv file with shape file

Use add vector layer to link csv attribute table to an existing shapefile in QGIS by keeping the number format, without recognized as string text:

https://gis.stackexchange.com/questions/29606/changing-csv-layer-attribute-value-from-text-to-number-in-qgis

https://anitagraser.com/2011/03/07/how-to-specify-data-types-of-csv-columns-for-use-in-qgis/

[Python] Install matplotlib and basemap in Python3.7

I need to install a new matplotlib and basemap for my new anaconda python 3.7. I still remember the pain many years ago when I first installed these two guys using source...check my old blog!

One thing requires attention is that basemap is not working with the newest version proj4.6.X. You will get errors like (this thread):

epsgf = open(os.path.join(pyproj_datadir,'epsg'))
FileNotFoundError: [Errno 2] No such file or directory: '/home/user/miniconda3/envs/test/share/proj/epsg'

So need to specify the proj4 version!

$ conda create -n test python=3.7 pandas pillow numpy
$ conda activate test
$ conda install -c conda-forge matplotlib basemap proj4=5.0
$ python -c "from mpl_toolkits.basemap import Basemap"

After this, I have an issue about the QT backend support between matplotlib and anaconda, such as "Could not open XDG", "Could not connect to display" (see this post). So I add the following in ~/.bashrc:

export QT_QPA_PLATFORM='offscreen'

And it worked for me!

[Python] Removing hard-coded number with argparse

Assume you have a function as below in a python script calculate_sum.py,

def summing(a, b):

    return a+b

summing(2,3)

Now if you want to change 2 and 3 to other numbers, you can use a powerful tool argparse instead of modifying these hard-coded numbers in the script, which can be complicated in a set of complicated scripts. 

import argparse

parser = argparse.ArgumentParser(description='calculate the sum of two numbers')

# Either use positional arguments. The position matters, so a and b can not reverse in order.

parser.add_argument('a', type=int)

parser.add_argument('b', type=int)

# Or use optional arguments. The position of a and b does not matter.

parser.add_argument('--a', type=int)

parser.add_argument('--b', type=int)

# Pass the arguments to the parser python module

args = parser.parse_args()

def summing(a, b):

    return a+b

summing(args.a,args.b)

If you use positional arguments, your command should be:

$python calculate_sum.py 4 3

7

Otherwise, your command should be:

$python calculate_sum.py --a 4 --b 3

7

[Python] make python script executable

Once you wrote a python script run.py, you can run it in the terminal as below:

$ python run.py

However, you can add #!/usr/bin/env python at the top of your run.py, so that the run.py becomes executable, and can be called without the preceding language "python".

$./run.py

Please also see why using #!/usr/bin/env python but not #!/usr/bin/python at this post.

[WSL] Set up Visual Studio Code and Pycharm editors in Windows Subsystem for Linux

Last week, I've walked through the initialization of Linux subsystem in Windows (read my blog here). This week, I can't wait to share my experience about the code editors on Windows 10.

Here is some background. I mainly work with Linux and use Python and Javascript. The first question I have is whether I need to install two Python distributions for Windows and WSL. I felt that it may trigger a problem because (1) the compiling of Python in different operating systems must be different, and (2) the slashes in the two file systems are opposite. I did a quick test by launching the anaconda python.exe from the WSL linux terminal (in my case, Ubuntu). Surprisingly, I am able to open Windows Python.exe in WSL! However, it is not much meaningful because in WSL I am not able to invoke the virtual environment and all the packages installed in Windows. So the short answer to this question is that you should install separate Python distribution in Windows and WSL, in order to manage your packages and avoid conflicts. Ideally, one should use virtual environment for each Python project, no matter in Windows or in WSL.

I use two major editors, Visual Studio Code and Pycharm. I use VSC for most languages, such as Python, Javascript, Markdown, Dockerfile, R, etc. I use Pycharm when I focus myself on one single Python project. These two editors are fantastic from their interface designs to the elaborative documentation and online support.

Surprisingly, both editors support dual-system launching! By that I mean, after you install the software in Windows, you not only can open it from Windows (e.g., start menu), but also are able to launch it from the WSL linux terminal (in my case, Ubuntu). I've never seen this dual-system functionality before in Linux nor OSX, which is unbelievably amazing and smart! This feature is really important because the running/debugging environment is entirely different in Windows and WSL. With that, you can now run Python from VSC internally (with your WSL bashrc file launched).

To install and launch Pycharm in both Windows and WSL systems, please follow this guide: python-development-on-the-windows-subsystem-for-linux-wsl.

To install and launch Visual Studio Code in Windows, please follow this official guide: windows. To further launch VSC in WSL, please follow this official guide: run-in-wsl. At the bottom left of the VSC window, there is a green panel displaying which system you are now invoking VSC.

[Windows] How to set up conda path in PowerShell

In windows, I installed anaconda package from the official website, and you can open python or other conda softwares (jupyter-notebook for example) using the Anaconda Prompt.

But what if you already use PowerShell and don't want to open an extra Anaconda Prompt window? Or what if you are using visual studio to edit python scripts and run python within it?

The first time you type in "conda" in PowerShell, you may see the following error:

conda : The term 'conda' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is     
correct and try again.

The solution is to go to the anaconda directory (either PowerShell, command line, or WSL, it doesn't matter), and simply type "./conda.exe init". 

[Windows] Set up Linux subsystem on Windows 10

This is my first blog post after I left grad school! Time flies. Can't believe I started my new job for a month already. Blogging is still the best way for me to mark down and share learning experiences. So, in the next couple weeks, I plan to write about the set up of my current working environment in series.   

In my new position, I am assigned a PC work computer! At the beginning, I was not happy because I work with Linux a lot and Mac's OSX is coherent with Linux. And I haven't used Windows for many years, so it takes time to adjust to a new set of tools. To my surprise, I found myself take it pretty well so far and even love it because some programs (especially office suites) on Windows seem to have more functionalities!

I don't even know there is an embedded Linux system until I came across this great tool - Windows Subsystem for Linux. It is something close to "Terminal" in Mac but a very different concept. WSL is more like a function rather than a actual program. Once you turn on this function, you are free to install different types of Linux system as a "subsystem", while "Terminal" in Mac is essentially no different than the Mac operation system OSX.

Following this guide (Install WSL on WIN10) on the official website, I successfully enabled this function on my Windows 10 and was able to use the Microsoft store. I selected Ubuntu 16.04 because of its popularity and user-friendliness. And the reason to go for 16.04 only because it is more stable. I have used Centos, Fedora, and Ubuntu before. The major differences are the admin commands and the package management platforms. Ubuntu has a larger user community as well. If you are new to Linux, you will find more topics and discussions online.

Once Ubuntu 16.04 is installed, I opened the terminal (referring to as the command-line window, not the program in Mac) and found the default theme very unreadable. The text colors of the file system and within the scripts are fighting with each other. I tried to adjust the colors to make it like the original Ubuntu following this post: make-bash-on-ubuntu-on-windows-10-look-like-the-ubuntu-terminal. 

It looked much better, but I was not satisfied. Technically, the Ubuntu 16.04 is an app on my Win10. Every time I open a new terminal window, I have to either double-click or right-click on the Ubuntu icon. If you have 5 separate tasks, you have to open five windows and have a hard time to locate the right one immediately. I missed the gnome-terminal, where you can manage different windows in tabs. So I looked for a simple external program to do that. 

It turns out the "Windows Terminal" do a sufficient job. It is fresh and was launched on the Microsoft Store in June. I am able to open multiple "subsystem" through this one small program, and open as many tabs as I want. Finally, I customized my Windows Terminal following this guide: windows-terminal. By setting this up, I got a chance to learn the JSON format, a commonly used format in JavaScript. One thing I want to complain is the name of this program. Can Microsoft be more creative on names? "Windows Terminal", "PowerShell"... Although these names seem to make it easy to connect them to the Apple or Linux counterpart, but they also make it difficult to google their tutorials and tips online! 

So far, I am happy with the "WSL-Ubuntu" and "Windows Terminal" combination. In the next blog, I will talk about the editors I use on Windows 10.

[Python] Edit netCDF lat/lon without changing the original data

When I created a large netCDF file, I made a mistake by writing the wrong longitude range (-180 to 180) to the netCDF, which is inconsistent with the organization of the data (0 to 360). I realized the sellatlonbox command in CDO will not solve this mismatch problem. So I decided to use Python netCDF4 package to modify it (python-replacing-values-in-netcdf-file-using-netcdf4). Generally speaking, you can modify and update any variable in a netCDF file using this package. And it is very quick!

Here is the code:

dataset = netCDF4.Dataset('file.nc','r+')
dataset.variables['lon'][:] = np.linspace(0.125, 360, 1440)
dataset.close() 

忘得一干二净😉

过去两个月都在重新整理上一篇的文章项目，把十一月的工作搁置了，现在竟然完全忘记做到什么进度。只能怪当时做的时候没有好好记录和整理进展，导致现在空对着一张张的图、上百个实验结果文件夹和代码，抓破脑袋在想是为啥跑这些实验的。
加油吧，少年！花一天的时间好好捋捋思路。其实既是失落也是良机，可以趁机把一些不合逻辑的做法也刨除，整理出一条更加清晰合理的线索吧。

鸡汤好喝，无甚用处

偶然看到亲爱的客栈蒋瑶佳来的那一集，大家在旁边听她描述自己的困惑，她说想要来这里寻一个解药，不想要明天走的时候还是这种状态。她的状态真的和我挺像的，对当前状态不满意。她想的是自己能够做音乐的同时，还能赚到钱，解决生活问题。我呢，则是想找一个做研究，但是又不是压力那么大的学术岗位。