The team objective is to analyze and understand this data and then, perform a Machine Learning project with the aim of designing a model that can improve the actual predictions given by NOAA.

The work replicates a traditional ML pipeline project.

As the main data sources have been provided in csv format, not big data ingestion tools have been used.

The code has been developed using Python, and Visual Studio Code and Jupyter Notebook as IDE.

Several Python Libraries have been used in the development of the project. These libraries could be found in the References section.

1.1. EDA

EDA can be found in the Notebook “1_Initial_EDA_DSCOVR_data.ypynb" (included in the Github).

This exercise has been iterative with and executed simultaneously with the Data Cleaning and Feature Engineering ones, allowing to identify potential improvements that have been implemented in the later stages, and so on.

The final EDA has been performed with the NB that already contains the data cleaned and with the features created.

The EDA starts some statistics to later on, display several visualizations with the aim of detecting patterns and correlation between Kp and Ap and the vectors measures of PlasMAG and the different FC measurements.

1.2. Data Cleaning and Feature Engineering

At this stage, we have due several activities. First, we have given to the columns names (depending on its procedence: PlasMAG or FC)

We have minute measurements and the Kp index (indicating the expected weather) is usually predicted in 3 hours interval.

After several valuation and considering time constraints and the current intervals of Kp, team has decided to aggregate the data transforming it from minutes to 3 hour interval.

We have created, for the measurements, several statistical features covering the measurments that have taken place inside this 3-hours: mean, range, min, max, median.

As we know, and it is probably the biggest challenge we are dealing with, many of the rows are NaN but they are registered as 0. They could be an anomaly, and they can also be indicator of space weather changes, so the approach of discarding the rows, or filling the NaN with other statistical measurements has been rejected by the group.

In order to tackle this situation, we have:

1- For each 3-hour aggregation, new columns (one per each original variable) have been added in order to account for the proportion of NaN values that each feature presents during that time aggregation.

2- Additionally, and once all the aggregations have been completed, new columns (one per each aggregated variable) have been added in order to signal the ones that after aggregations contain NaN as value. This is a binary column that is equal to 1 if data is missing and 0 if not missing, so working as an indicator column for missing values. Only at this point, the NaN values have been substituted by zeros.

3- Optionally, we have added the interactions of these binary columns with the rest. However, due to storage and computational limitations, this version of the dataset with interactions could not be used for modelling phase.

Although, is it worth mentioning that some periods where presenting null values not because of anomalies happening, but because a shutdown of the DSCOVR (in instance from 27 june 2019 to 2 march 2020). This rows have been discarded, as they would damage the model.

Finally, we have prepared the Kp and Ap data and we have merged it. For Kp in time t, we have joined it with the prepared data of PlasMAG and FC at point t-1. Despite the storm can arrive to Earth in periods as brief as 15 to 60 minutes, we have consider it better to use the measurements of the previous intervals as we have thought that they would be more relevant in achieving a good prediction. Further work can be done at this point.

1.3. Model

In our work, we have tried both regression and classification models. The model presented here is a binary classification model that constitutes the first step towards building more complex models. The target variable Kp has been classified into higher or equal to 5 (1) in order to indicate an incoming solar storm, and less than 5 (0). This binary classification has increased significantly the accuracy of the model in being able to detect storms, as regression models tended to predict low values.

The data has been splitted into train and test. Train dataset consists of the first 90% of the time series data from 2016. Test dataset consists of the last 10% of the data till 2023.

The predictor variables are all the variables coming from the DSCVR and the feature engineering performed in previous stage plus the historical Kp values. The target values are future Kp values. In this case, the sequence includes 240 previous datapoints for training, and 8 future points for forecasting.

Among several architectures tried, an LSTM-based model is presented. It consists of 100 LSTM units, followed by a dense layer with dropout and a final dense layer with 8 neurons and sigmoid as activation function to predict the probabilities of the class in the next 8 time steps.

Since the output of the model are the probabilities, a user-defined threshold is set based on the training data using ROC-AUC methodology.

2. Next Steps

The team has already created a plan that wasn’t implemented on this first iteration due to the time and computational capabilities constraints. The next steps to be performed are as follow:

· Improve the feature engineering: Feature engineering is an iterative process. For us, it implies going back and forth to add/drop features and re-run models to check their impact on performance. Our plan is to add some data that can be also used by NASA (in instance, Canada Solar Flux data of the prior day: not as it is, but prepared as can be seen in notebook: ….). Also, a detailed correlation analysis, application of tree based models for detecting feature importance scores and dimensionality reduction techniques are some of the next steps planned at this point.

· Improve the models: basically on 2 different lines:

o Try more complex models that have not been tried, mainly because they required more running time, so applying them could have been an one shot exercise.

o Improve finetuning parameter of the used models.

· Improve the data-serving: Once improving our forecast for the next 3 days 3-hour intervals, the team would like to improve the visualization style of the page. Also, vinculate the forecast with a mailing list that contains the main stakeholders affected for strong space electromagnetic storms, creating a system of alerts, would be a good option to prevent and mitigate potential risks. Finally, allowing the users to access to the comparison between historical and predicted values (and even making our model open-source) could deal to future iterations that improve the final work (despite the actual deliverables are just a first sketch.

More details can be found in the Project Demo and in the Github repository.

2023 NASA Space Apps Challenge

Solar Storm Sentinels