[R] install rgdal, rgeos packages dependency in Centos

For geos:
sudo yum install geos geos-devel

For gdal:
sudo yum install gdal gdal-devel proj-devel proj-nad proj-epsg

[Pycharm] New way to create project from existing sources

In the past versions, to start a new project, I need to use New Project from the File menu. 

But with the 2024 edition, I need to use Open instead. Otherwise it will create a new folder called PythonProject by default under the existing folder. Here is the guidance.

This change makes some sense, but it is quite difficult to navigate.

Learning how to feed arguments to command line - Argparse

Beginner on Argparse. The following are from the Official documentation.

The argparse module’s support for command-line interfaces is built around an instance of argparse.ArgumentParser. It is a container for argument specifications and has options that apply the parser as whole. The first descriptive message for the program is determined from sys.argv[0] or from the prog= argument. 

[GrADS] Environment Variables 环境变量设置

Beginning with GrADS version 2.0.a8, there is only one choice for GrADS_executable, a single, fully-featured build, which is good.

GADDIR Points to the directory containing the supplemental font and map files in the GrADS release package. If GADDIR is not set, GrADS will look in the default location, /usr/local/lib/grads/.

GASCRP Points to a list of directories containing GrADS utility scripts and user scripts. If more than one directory is specified, acceptable delimiters are a space, a semi-colon, colon, or a comma.

For example:
example% setenv GADDIR /ford1/local/lib/grads
example% setenv GASHP $HOME/grads/shapefiles
example% setenv GASCRP "$HOME/grads/scripts /opt/local/share/grads/library"
example% setenv GAUDFT $HOME/grads/udf/table

如何读写希腊字母？

[Python] How to shift the midpoint of the colorbar? -Normalizing colormap

给网格数据画图的时候，常常面临选择合适调色盘的难题。我们常用的matplotlib里自带的调色盘大致有三种：单向渐变，双向渐变，多色渐变。根据数据的值域以及目的，我们来选择不同的调色盘来表现。
单向渐变多用于单增的信号，且空间分布非常简单。复杂的空间分布应该采用多色渐变，区分度更大，比如地形图。
双向渐变常用于有正负值的数据，比如相关系数、敏感性等等。今天就是要来讲一讲双向渐变colorbar。
我们平时经常遇到这样的数据，就是正负值不对称，比如值域范围是-1~100。为了能够使用双向渐变colorbar，我们需要调整数据中点，不然调色盘不对称，造成读数的困难！一个暴力的方法就是通过人为设定最大、最小值来设定数据中点：
plt.imshow(data, vmin=-100, vmax=100)
下面介绍一个更加合理、不浪费空间的简便方法：

from matplotlib.colors import Normalize

class MidpointNormalize(Normalize):
    def __init__(self, vmin=None, vmax=None, midpoint=None, clip=False):
        self.midpoint = midpoint
        Normalize.__init__(self, vmin, vmax, clip)

    def __call__(self, value, clip=None):
        # I'm ignoring masked values and all kinds of edge cases to make a
        # simple example...
        x, y = [self.vmin, self.midpoint, self.vmax], [0, 0.5, 1]
        return np.ma.masked_array(np.interp(value, x, y))

data = np.random.random((10,10))
data = 10 * (data - 0.8)

fig, ax = plt.subplots()
norm = MidpointNormalize(midpoint=0)
im = ax.imshow(data, norm=norm, cmap=plt.cm.bwr, interpolation='none')
fig.colorbar(im)
plt.show()

是不是很简单呀？原作者Joe还有好几种其他方法，以后可以试验一下。

[Windows 10] Multiple displays with docking station II - enable Night Light

After I reinstalled all the drivers, I found that the external monitors didn't react to the Windows 10 Night Light adjustment through the dock. This Night Light feature is super cool and it is annoying that the dock cannot support it naturally. 

I found the instruction from this DisplayLink webpage How to enable night light or f.lux on DisplayLink displays on Windows. It worked! The only thing - you need to run the Registry Editor using the administrator credential. 

[Python] Learning a new package pbr

https://pypi.org/project/pbr/

https://michaelkuty.com/howto/build-python-with-pbr/

https://dzone.com/articles/packaging-python-software-with-pbr

[R]Reset the environment variable for R in El Capitan

The issue is I can't call R in terminal after El Capitan has been installed. This is the result of the newly banned writing permission to usr/bin. See this post: R in El Capitan public beta (new security model).

1. Find the R binary.
I have the Rstudio, where I can open R console. In Rstudio, use the following command to track where R binary is or RHOME.
If you can't track your directory, you can also look for R in this path. My system is El Capitan.

2. Modify the path profile.
Add the directory to your system path. In terminal:
$ sudo nano /etc/paths
Enter your password, when prompted.
Go to the bottom of the file, and enter the path you found in the first step.
Hit control-x to quit.
Enter “Y” to save the modified buffer.
Hit enter t confirm when prompted by "File name to write".

Major reference:
Add to the PATH on Mac OS X 10.8 Mountain Lion
Set environment variables on Mac OS X Lion

[GrADS]maskout()

Today I will discuss the usage of maskout function, which is simple but powerful.
syntax:
maskout(expression, mask) 
It basically means using a mask from 'mask' file/variable to remove the corresponding grids in 'expression' data/variable.

(1) If you already have some mask files:
ga->d maskout(data,mask(t=1))
this command will suffice.
Remember to put (t=1) behind the mask, because most of well-made mask files don't have time step.
if the mask is 0 and 1, then you either change the undef value in the mask control file into 0 (the actual missing value in the binary file); or just use maskout(data,data-0.5), then all the value below 0 will be maskout.


(2) If you need to make mask files by yourself:
for example,
if you want to mask out the value lower than 100, in other words, you want to leave the value greater than 100,
ga->define mask = const(maskout(data,data-100),1)


if you want to mask out the slope that insignificant with p>0.05,
ga->define mask = const(maskout(p,0.05-p),1)

Notice the difference here, for conditioning data > value, then the mask = data-value; for conditioning data < value, then the mask = value-data.

Besides using with const() function to produce mask, maskout can also be combined with aave and tloop to plot time series.
ga->d tloop(aave(maskout(data,mask(t=1)),g))

The idea here is to upscale by spatial averaging the masked region.

[Git] Migrating a Git Repository to a New GitHub account - Case 1

When you work on a repository, one day you may need to migrate your Git repository to a new GitHub account for various reasons, such as collaborating with new team members or changing affiliations. The process of migrating a Git repository can be challenging, especially if you want to preserve the repository's history and selectively migrate only the necessary files. I will focus on a specific scenario today and share how I migrate a Git repository to a new GitHub account. 

In this scenario, I am at a closing point of my current private project, and I need to collaborate with team members. From now on, I will work on a new repository under my team account, while keeping the original repository as an archive. I also want to selectively migrate only the main, up-do-date files, so I need to clean up the inactive scripts.

技术学习期末小结

programming...这个词有点被用的过于泛滥。
过去的一个学期，在这上面没有太用功，但多多少少更加了解了各种语言和环境。
回首看来，Justin说的东西都非常正确，我以后一定要认真听老板的话！

首先，接触两个全新的操作系统。
（11月上旬）MacOS：需要适应的主要是触摸盘，文件管理系统，多任务管理，系统整理，移动硬盘读取。
Mac让我感觉最爽的除了和Linux内核相通，任务管理的高效，还有和ipad iphone的同步。
不太适应的是，两个平台上软件不太一样，需要寻找更适合的替代工具。（这种情况下，于是发现过去大部分软件其实都是多余的…… ）

（11-1月）Linux：一开始并没有当一个系统用，现在逐渐有这个概念了。。。逐渐接触terminal, bash, shell, vi...
逻辑大概是：
》vim这种text editor就是写各种script的工具，写适用于各种软件的script/source code源代码，如R(.R)，GrADS(.gs)，matlab(.m), C(.c), Python(.py), Shell(.sh)。
》对于C和Fortran，是需要compile编译成计算机可识别的语言来执行的，所以需要make file；
》对于其他scripting language，是直译式的（比编译要慢），不需要make file，但需要通过chmod +x来使之可以执行。
》用相应的软件或者环境来运行源代码。
因此，Python有着空前绝后的优越性！！
我觉得此前最错误的就是，不理解基本概念，把各种工具混为一谈，导致自己没有选择正确的方向下手！

然后，开始上手画图工具GrADS。GrADS其实可以说是一个数据可视化的快捷工具，附有非常简便的数据读取、分析功能，可以做简单的分析。

接着，开始学R。R其实就是统计版的Matlab，界面好像，语法好像。

如今，正式开始学习Python。

短短几个月，接触的范围还是挺广的呵呵。

[Linux] Files permission

A command to change the permissions of a directory so that others can not see it.
$ Chmod -R 700 /directory

[Python] Plot

# Set the colorbar

(1)
plt.imshow(data, vmin=0, vmax=10, cmap='jet', aspect='auto')
(2)
plt.pcolor(X, Y, v, cmap=cm)
plt.clim(-4,4)
plt.show()

(3) Change the range of a colormap for a certain values
import matplotlib.colors as colors
(a)
cmap = cm.get_cmap('jet', 20)
cvals = cmap(arange(5, 20, 1))
ncmap = colors.ListerdColormap(mcolors)
(b)
mcolors = plt.cm.jet(linspace(0.3, 1, 256))
ncmap = colors.ListedColormap(mcolors)

(3)Remove the top and bottom axis
 def simpleaxis(ax):

  ax.spines['top'].set_visible(False) 

  ax.spines['right'].set_visible(False) 

  ax.get_xaxis().tick_bottom()

  ax.get_yaxis().tick_left()

(4) Draw the x=y line
plot([xmin, xmax],[ymin, ymax])

[WSL] Set up Visual Studio Code and Pycharm editors in Windows Subsystem for Linux

Last week, I've walked through the initialization of Linux subsystem in Windows (read my blog here). This week, I can't wait to share my experience about the code editors on Windows 10.

Here is some background. I mainly work with Linux and use Python and Javascript. The first question I have is whether I need to install two Python distributions for Windows and WSL. I felt that it may trigger a problem because (1) the compiling of Python in different operating systems must be different, and (2) the slashes in the two file systems are opposite. I did a quick test by launching the anaconda python.exe from the WSL linux terminal (in my case, Ubuntu). Surprisingly, I am able to open Windows Python.exe in WSL! However, it is not much meaningful because in WSL I am not able to invoke the virtual environment and all the packages installed in Windows. So the short answer to this question is that you should install separate Python distribution in Windows and WSL, in order to manage your packages and avoid conflicts. Ideally, one should use virtual environment for each Python project, no matter in Windows or in WSL.

I use two major editors, Visual Studio Code and Pycharm. I use VSC for most languages, such as Python, Javascript, Markdown, Dockerfile, R, etc. I use Pycharm when I focus myself on one single Python project. These two editors are fantastic from their interface designs to the elaborative documentation and online support.

Surprisingly, both editors support dual-system launching! By that I mean, after you install the software in Windows, you not only can open it from Windows (e.g., start menu), but also are able to launch it from the WSL linux terminal (in my case, Ubuntu). I've never seen this dual-system functionality before in Linux nor OSX, which is unbelievably amazing and smart! This feature is really important because the running/debugging environment is entirely different in Windows and WSL. With that, you can now run Python from VSC internally (with your WSL bashrc file launched).

To install and launch Pycharm in both Windows and WSL systems, please follow this guide: python-development-on-the-windows-subsystem-for-linux-wsl.

To install and launch Visual Studio Code in Windows, please follow this official guide: windows. To further launch VSC in WSL, please follow this official guide: run-in-wsl. At the bottom left of the VSC window, there is a green panel displaying which system you are now invoking VSC.

[ChatGPT] User experience so far... AMAZING

I have to say chatGPT is much more powerful that I can imagine - definitely eye-opening!

I realize the followings when using it:

1. Pay attention to the interrogatives: What/Which/If/Why/How much

2. Give as much context as possible in the question. It WILL understand!

3. It can debug your code at all different levels... This is nuts!!!

[Windows] How to set up conda path in PowerShell

In windows, I installed anaconda package from the official website, and you can open python or other conda softwares (jupyter-notebook for example) using the Anaconda Prompt.

But what if you already use PowerShell and don't want to open an extra Anaconda Prompt window? Or what if you are using visual studio to edit python scripts and run python within it?

The first time you type in "conda" in PowerShell, you may see the following error:

conda : The term 'conda' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is     
correct and try again.

The solution is to go to the anaconda directory (either PowerShell, command line, or WSL, it doesn't matter), and simply type "./conda.exe init". 

[Python] Regrid/Remap using python/cdo/gdal

Spatial resampling or sometimes we call it regridding and remapping, or even interpolation depending on whether we upscale or downscale grid cells, is something that we did quite often while dealing with large scale datasets.

If we want to use scipy, here are the functions I found relevant, but not good enough because they are disconnected from their geographic coordinates.
scipy.ndimage.map_coordinates
scipy.interpolate.RectBivariateSpline
I use the second one and found several issues. First, the latitude and longitude seems get wrong. While the other problem is more problematic. That is the boundary is wrong.

Another way to work around is using cdo remapgrid. CDO has a wrapper of SCRIP (Spherical Coordinate Remapping and Interpolation Package), which could be found on line (Los Alamos National Laboratory). I strongly recommend use this functionality of CDO, a powerful and fast tool based on Fortran. It has bilinear, bicubic, distance-weighted average, nearest neighbor, conservative (box-average), and largest area fraction interpolations.

I haven't really looked at gdal, but I guess it takes more time to figure out the commands from the unfriendly gdal manual...

[Git] Study git and github in Pycharm

Set up git in Pycharm:

• https://www.jetbrains.com/help/pycharm/set-up-a-git-repository.html#add-remote

Set up github in Pycharm:

• There are two ways of set up. One is that you create a new repo on github and then git clone to the local file. The second way, which is more convenient, is to share your local project to Github.

Set up run/debug configuration:

• https://www.jetbrains.com/help/pycharm/creating-and-editing-run-debug-configurations.html

Configure project structure in Pycharm:

• https://www.jetbrains.com/help/pycharm/configuring-project-structure.html

Unversioned a file:

• https://superuser.com/questions/898165/remove-file-from-git-version-control-without-deleting-it-from-the-filesystem

Manually delete file and then manage through git:

• https://stackoverflow.com/questions/12987907/git-how-to-commit-a-manually-deleted-file
• https://stackoverflow.com/questions/492558/removing-multiple-files-from-a-git-repo-that-have-already-been-deleted-from-disk

I just realized, after you rename a script, its previous history on github will get deleted. I still found the local log in Pycharm.

Change commit message after pushing to remote:

• https://www.educative.io/edpresso/how-to-change-a-git-commit-message-after-a-push

Compare different files in Pycharm

• Use compare with clipboard to compare current file (in editor) with the copied file

Compare different versions of the same file in Pycharm

• In  "Show history", at the left navigation pane, right click on one commit, and select compare with local. Local will show up on the right.

[Linux] server

1. mount the server
log in as root
$ su
$ mount /home/{hostname}
2. unmount the server
$ umount /home/{hostname}