73 water vapor notebook by rogerkuou · Pull Request #74 · ESMValGroup/ClimaNet

rogerkuou · 2026-07-15T11:15:15Z

Fix #73 .

Should be merged after #96, since PR #96 contains improvement for this PR.

~~Should be merged after #92~~
~~Contains changes in #92 to fix gem embedding bug.~~
(#92 already merged on Sep 9)

Changes in code
Added check for empty dataset
Init loss in predict as a tensor

Changes in notebooks:
separated notebooks to sst and watervapor folder
added daily watervapor training notebook with local execution results
added hourly watervapor traning notebook with Levante execution results

rogerkuou · 2026-07-15T11:19:03Z

Hi @SarahAlidoost , while working on #73, I got an error in the notebook. This notebooks is adapted from the SST hourly one.

It fails on average_loss.item(), which hints that average_loss is a dictionany. Do you have clue on this? I am running with the CPU mode instead of GPU.

rogerkuou · 2026-07-16T13:55:53Z

Hi @SarahAlidoost , can you review this notwbook when you have time?

The error was caused by an empty dataset. I added a check to predict, and initiated loss as an zero Tensor.

Another thing maybe we need to pay attention, is the water wapor data is downsampled by the factor of 2 comparing to SST. This requires some attention to lsm data. In the notebook I used an average window to downsample the lsm, and set <0.5 to 0, and >=0.5 to 1. We can discuss if this makes sense.

SarahAlidoost

@rogerkuou Thanks! 👍 I added one comment on the changed code. Levante is not available today and I dont have data locally. I will check the notebook later. About the spatial resolution of water vapor, I expected era5 data have the same spatial resolution; this is something to check in our next update meeting. I couldnot also check the era5 data as CDS is also on maintenance.

SarahAlidoost · 2026-07-23T11:45:59Z

Hi @SarahAlidoost , can you review this notwbook when you have time?

The error was caused by an empty dataset. I added a check to predict, and initiated loss as an zero Tensor.

Another thing maybe we need to pay attention, is the water wapor data is downsampled by the factor of 2 comparing to SST. This requires some attention to lsm data. In the notebook I used an average window to downsample the lsm, and set <0.5 to 0, and >=0.5 to 1. We can discuss if this makes sense.

Indeed there are a few things about water vapor data to check:

the spatial resolution of data on levante is 0.5. The water vapor data is available on cds with resolution 0.25
data is cut between (-80, 80) in latitude dimension
the latitudes are not increasing they are from (80, -80) while longitude are i.e. (-180, 180)

rogerkuou · 2026-08-04T14:38:51Z

Hi @SarahAlidoost , can you review this notwbook when you have time?
The error was caused by an empty dataset. I added a check to predict, and initiated loss as an zero Tensor.
Another thing maybe we need to pay attention, is the water wapor data is downsampled by the factor of 2 comparing to SST. This requires some attention to lsm data. In the notebook I used an average window to downsample the lsm, and set <0.5 to 0, and >=0.5 to 1. We can discuss if this makes sense.

Indeed there are a few things about water vapor data to check:

the spatial resolution of data on levante is 0.5. The water vapor data is available on cds with resolution 0.25

data is cut between (-80, 80) in latitude dimension

the latitudes are not increasing they are from (80, -80) while longitude are i.e. (-180, 180)

During the meeting in July 24, we discussed the issues with Axel:

The resolution is coming from the fact the satellite observations on water vapor has resolution with 0.5 degress. Hence it does not add value to go to -.25 degree resolution
The data cut and lat coordinate order comes from how regredding is set. Axel is working on generating a new version of dataset with (-90, 90) and ascending ordering.

@SarahAlidoost shall we consider merging this PR for now? My motivation is that this PR also renames example notebooks to distinguish sst and water vapor. It can set good templates for other PRs.

rogerkuou · 2026-08-14T11:41:31Z

Updated on 20260814
Ihave confirmed with Axel that the satellite data's coverage is (-80, 80), and the resolution is 0.5 deg. Axel prefers not to create a new set of data, but use the current version. This means for the order of lat we can flip it by ourselves.

About the masking, I found CDS always provide 0.25 degrees. If using esemble mean to download, the downloaded mask are floating values. I check with xel he is fine with thresholding the floating mask with 0.5 values. Maybe let's do that and create another mask file matching the Water vapor data?

What do you think @SarahAlidoost and @meiertgrootes ?

rogerkuou · 2026-09-08T11:37:41Z

Hi @SarahAlidoost, in this PR I made example notebooks for watervapor. I executed the daily one locally and the hourly one on Levante.

I observed that the validation loss does not reduce, and the training stops with early stops. This is especially significant in the hourly training. I have tried using different month (I started with Jan, then switched to April. It the same as SST setting, three identical month from 2020-2022 for training, validation and testing), and reduced the learning rate to half (just in case if we are missing optimal). The validation loss curve stays flat. I hope for a full training with multiple month can help. But I will wait till your test with #92

(Updated on SEP 10)
Reagrding the small trial of validate/test only on training data, see this GH Gist. It should the traning loss continue to drop, and the prediction results are significantly improved comparing to baseline.
The "artificial oscillation" in the daily notebook is caused by an uncleared training history, where at same epoch there are multiple loss. To avoid conflicts, I solved this in #95. By removing caches and re-run we do not have this anymore. See the updated notebook.

I merged your fix in #92 to this PR, to include your fix for the bug. So it will make sense to review this after merging #92.

rogerkuou · 2026-09-10T09:52:09Z

Hi @SarahAlidoost , I brought this branch up to date. See my previous comment for details.

Meanwhile, following our discussion today, I will make another test on the training dataset to see if the training is able to overfit on the training data. This can validate if the problem is actually training on one month

SarahAlidoost · 2026-09-11T16:33:10Z

+<<<<<<< HEAD:notebooks/sst/training_hourly_sst.ipynb
+    "monthly_data_test = xr.open_mfdataset(data_folder / f\"202001_mon_ERA5dc_full_{var_name}.nc\")\n",
+=======
    "monthly_data_test = xr.open_mfdataset(data_folder / f\"202201_mon_ERA5dc_full_{var_name}.nc\")\n",
+>>>>>>> main:notebooks/example_hourly.ipynb


these are leftover from merging. Could you fix them?

SarahAlidoost · 2026-09-11T16:33:21Z

+<<<<<<< HEAD:notebooks/sst/training_hourly_sst.ipynb
+    "spatial_patch_size = monthly_da.shape[1:]  # the whole dataset \n",
+=======
    "spatial_crop_size = monthly_da.shape[1:]  # the whole dataset \n",
+>>>>>>> main:notebooks/example_hourly.ipynb


SarahAlidoost · 2026-09-11T16:33:38Z

+        "max_concurrent_trials": args.num_nodes
+        * 2,  # less than GPUs per node (4) avoid OOM


Suggested change

"max_concurrent_trials": args.num_nodes

* 2, # less than GPUs per node (4) avoid OOM

"max_concurrent_trials": args.num_nodes * 2, # less than GPUs per node (4) avoid OOM

rogerkuou added 2 commits July 15, 2026 12:35

rename daily notebooks

67a043a

add draft example notebook for watervapor

5b3d0df

rogerkuou added 4 commits July 16, 2026 13:47

check empty dataset in prediction

5545f1f

update hourly watervapor notebook

0deab33

initiate loss as zero tensor

1dea21d

update notebook

3ac2df4

rogerkuou marked this pull request as ready for review July 16, 2026 13:52

SarahAlidoost reviewed Jul 20, 2026

View reviewed changes

Comment thread climanet/predict.py Outdated

changed loss to float

3e55c64

Merge branch 'main' into 73_water_vapor_notebook

c5a9337

SarahAlidoost added 15 commits August 26, 2026 13:01

rename patch with crop in dataset and predict module

9e289e3

fix geo_embeding in the model

9c523e5

remove spatial transfomer from the model

83a0a20

add some model improvemnets

63ba353

uncomment groupnorm in decoder

222f2e5

improve docstring

0ff075f

remove patch from dataset, adjust api in source modules

1fb8ba0

fix linters

09c55db

fix tests

814f151

fix docstring and comments

76842bd

fix nbs

e9cdd56

add model.train in train loop (a bug)

628507b

Merge branch 'main' into fix_geo_embeding

88382ce

update scripts

078121f

fix run_best_tuned_model script

b59ad45

SarahAlidoost and others added 2 commits September 2, 2026 11:29

rerun example_tuning nb

eb53bef

Merge branch 'main' into 73_water_vapor_notebook

d5000e9

rogerkuou marked this pull request as draft September 7, 2026 13:25

rogerkuou added 8 commits September 7, 2026 15:27

Merge branch 'fix_geo_embeding' into 73_water_vapor_notebook

6e3a31d

re-structure training

45ccda3

rename_notebook

6ffc51f

add example notebook for daily watervapor data

689c94b

update unit name

0b28ed6

init hourly watervapo notebook

e04c2f4

initial water vapor hourly training

6444772

add commented out code section for generating 0.5 deg lsm

f0409a7

rogerkuou marked this pull request as ready for review September 8, 2026 11:28

rogerkuou requested a review from SarahAlidoost September 8, 2026 11:37

solve confilcts

33795d1

This was referenced Sep 10, 2026

Add watervapor landmask creation to utils.py #95

Open

95 watervapor lsm util func #96

Open

SarahAlidoost reviewed Sep 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

73 water vapor notebook - #74

73 water vapor notebook#74
rogerkuou wants to merge 34 commits into
mainfrom
73_water_vapor_notebook

rogerkuou commented Jul 15, 2026 •

edited

Loading

Uh oh!

rogerkuou commented Jul 15, 2026

Uh oh!

rogerkuou commented Jul 16, 2026

Uh oh!

SarahAlidoost left a comment

Uh oh!

Uh oh!

SarahAlidoost commented Jul 23, 2026

Uh oh!

rogerkuou commented Aug 4, 2026 •

edited

Loading

Uh oh!

rogerkuou commented Aug 14, 2026 •

edited

Loading

Uh oh!

rogerkuou commented Sep 8, 2026 •

edited

Loading

Uh oh!

rogerkuou commented Sep 10, 2026

Uh oh!

SarahAlidoost Sep 11, 2026

Uh oh!

SarahAlidoost Sep 11, 2026

Uh oh!

SarahAlidoost Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		"max_concurrent_trials": args.num_nodes
		* 2, # less than GPUs per node (4) avoid OOM

	"max_concurrent_trials": args.num_nodes
	* 2, # less than GPUs per node (4) avoid OOM
	"max_concurrent_trials": args.num_nodes * 2, # less than GPUs per node (4) avoid OOM

Uh oh!

Conversation

rogerkuou commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rogerkuou commented Jul 15, 2026

Uh oh!

rogerkuou commented Jul 16, 2026

Uh oh!

SarahAlidoost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SarahAlidoost commented Jul 23, 2026

Uh oh!

rogerkuou commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rogerkuou commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rogerkuou commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rogerkuou commented Sep 10, 2026

Uh oh!

SarahAlidoost Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

SarahAlidoost Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

SarahAlidoost Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rogerkuou commented Jul 15, 2026 •

edited

Loading

rogerkuou commented Aug 4, 2026 •

edited

Loading

rogerkuou commented Aug 14, 2026 •

edited

Loading

rogerkuou commented Sep 8, 2026 •

edited

Loading