Thanks to visit codestin.com
Credit goes to github.com

Skip to content

461 spectral window#463

Merged
dbenn merged 21 commits intomasterfrom
461-spectral-window
Jan 21, 2025
Merged

461 spectral window#463
dbenn merged 21 commits intomasterfrom
461-spectral-window

Conversation

@mpyat2
Copy link
Collaborator

@mpyat2 mpyat2 commented Dec 26, 2024

This implements the "Spectral Window" algorithm (with the results identical to the popular program Period04, http://period04.net/). The algorithm is essentially the 'classic' Discrete Fourier Transform as it is described in (Deeming 1975, bibcode 1975Ap&SS..36..137D) for a signal with unit amplitude.
Since we need DFT for this, I made the plug-in 2 in 1: DFT itself (again, identical to Period04, which uses this very algorithm) and Spectral Window. The 'Classic' DFT is the most primitive one and loses to other algorithms (for example, DC DFT) in many cases, yet why not have it for the collection?

To-do:

  1. Implement 'Create Model' functionality accessed directly from a panel with the DFT result (like in DC DFT). This button is hidden now, when I tried to use it 'as is', an error occured. It seems 'Create Model' is too tightly bound with DC DFT.
  2. Unit test
  3. Try to optimize calculations: now it is slower even as Peranso3 (where the same algorithm is implemented, too)

Side notes:

  1. This plug-in, like DC DFT and AoV ones, may open several resulting non-modal dialogs. When a new star is loaded, they lose connections between their panes. So, I decided to close all of them while loading a new star. This is implemented with the help of List<PeriodAnalysisDialog>. It is probably worth implementing such behavior in DC DFT and AoV plug-ins.
  2. Looks like there are still many places in VStar where Swing dialogs are called from threads other than EDT. This is strongly discouraged by the Java docs, strange, but it has no visible effect in the app. Nevertheless, I avoided it here with the help of SwingUtilities.invokeAndWait. Probably, this can be used in the other places in the app.

Test data: eta_Aql_2024_visual.txt
eta_Aql_2024_visual.txt

image
image

@mpyat2 mpyat2 self-assigned this Dec 26, 2024
@mpyat2 mpyat2 requested a review from dbenn December 26, 2024 18:22
@mpyat2 mpyat2 added enhancement New feature or request plug-in Plug-in Development labels Dec 26, 2024
@dbenn dbenn linked an issue Dec 27, 2024 that may be closed by this pull request
@dbenn
Copy link
Collaborator

dbenn commented Dec 27, 2024

Really cool @mpyat2! Perhaps the DFT will have issues with unevenly spaced data, which your comment may be alluding to, but this is a great addition! I am less familiar with Spectral Window, but see some references, e.g. https://encyclopediaofmath.org/wiki/Spectral_window If you have another reference you are using, could you point me to that?

As you may see from #204 I'm implementing a piecewise linear model (almost done) that will be incorporated with the AoV period search plugin for model creation as well as being standalone. Some of the other issues you see could be collaborated on if it helps.

…rovement); Math.tan(a/2) instead of sin(a), cos(a) increased performance at ~40% on my machine
@mpyat2
Copy link
Collaborator Author

mpyat2 commented Dec 27, 2024

Hi @dbenn , concerning "Spectral Window": it is implemented like in Period04, see http://period04.net/files/p04manual_v1.0.0.pdf, page 53. There is no theory (unfortunately) in the guide, but only a practical example. I'll look for a better description. Also, a very lapidary description can be found here: https://www.cbabelgium.com/peranso/UserGuideHTML/Spectralwindow.html. I also wrote a couple of words about it in the documentation for the plug-in (see DFT and Spectral Window Plug-In.pdf)

@mpyat2
Copy link
Collaborator Author

mpyat2 commented Dec 27, 2024

Side note about performance: on my mini-PC (low-performance machine, Celeron N5095A @ 2.00GHz, Windows 11) DFT calculations for the test data "eta_Aql_2024_visual.txt" (minFreq=0.0, maxFreq=2.0, resolution=0.00001) with Java 8 (1.8.0_432, last update) takes ~41s, while with Open JDK 23.0.1 it takes only ~7s (almost in 6 times faster!).

Well, recent versions of Java have done some great optimization...
image

@dbenn
Copy link
Collaborator

dbenn commented Dec 27, 2024

Recommending a new minimal version of Java of VStar may be worth considering @mpyat2.

@mpyat2
Copy link
Collaborator Author

mpyat2 commented Dec 27, 2024

Recommending a new minimal version of Java of VStar may be worth considering @mpyat2.

Well, it's probably sufficient to indicate that it it recommended to switch to the newest version available because of better optimization, performance win, etc.

@mpyat2
Copy link
Collaborator Author

mpyat2 commented Dec 27, 2024

I am less familiar with Spectral Window,

Also, the Spectral Window can be considered as a 'representation of gaps' and the finite nature of the observing interval. The Fourier spectrum of the real data is a convolution of an 'ideal' spectrum with the Fourier transform of this "gaps' function" (see this abstract https://ui.adsabs.harvard.edu/abs/1975Ap%26SS..36..137D/abstract). By the way, in the appendix to the article, a short Fortran program, which calculates both DFT and Spectral Window, is listed. Well, I noticed it now only; in fact, it is quite simple and is easy to reinvent (this is a straightforward calculation of Eq. 3 from the article).
In other words, if you calculate the spectral window (which is centered at 0), you will see all phantom structures that accompany the real spectral peaks.

@dbenn
Copy link
Collaborator

dbenn commented Dec 28, 2024

Recommending a new minimal version of Java of VStar may be worth considering @mpyat2.

Well, it's probably sufficient to indicate that it it recommended to switch to the newest version available because of better optimization, performance win, etc.

I have found issues with some of the newer versions with respect to Swing drop-down list collection widgets. Other than that, agreed. Some testing required.

…ultipleLinearRegression. See multiPeriodicFit(...)
@mpyat2
Copy link
Collaborator Author

mpyat2 commented Dec 29, 2024

Hi David, @dbenn
I implemented multiPeriodicFit for this DFT using org.apache.commons.math.stat.regression.OLSMultipleLinearRegression.
I still need more tests yet (it seems) it works identically to the current implementation of it for DC DFT / Fourier Model.
OLSMultipleLinearRegression is quite flexible, and I plan to make an extended Fourier Model plugin based on it with [algebraic] polynomial trend approximation in addition to the existing trigonometric polynomial fitting. So it will be Fourier Model + Polynomial Fit "2 in 1", but it requires additional work and I cannot promise anything now :)

…the currently existing one, it is disabled by default and is remained for experimental purposes only.
@dbenn
Copy link
Collaborator

dbenn commented Dec 30, 2024

Very cool @mpyat2. A bit slow to get to the review and trying this. Apologies. Will get to it in the next day or so. I will also have something for you to review for me soon. :)

@mpyat2
Copy link
Collaborator Author

mpyat2 commented Dec 30, 2024

No problem, David, thank you! Anyway, I still need to create a unit test for it.

…ere with each other. Local vars and unique message tags help to avoid this. This problem may occur in the other Period Analysis plug-ins.
…gorithm. It could be set to true at the previous run.
@mpyat2
Copy link
Collaborator Author

mpyat2 commented Jan 2, 2025

Hi David @dbenn ,
I've added multithreading to this plug-in and DC DFT based on OLSMultipleLinearRegression. It gives the same result as Foster's code; however, it is noticeably slower. Nevertheless, with multithreading, the calculations are faster starting from a 4-core CPU (tested under Debian (very old Dell desktop) and Windows (relatively new yet cheap MiniPC with a 4-core Celeron)). On a 12-core i7 Windows laptop, the multithreaded OLSMultipleLinearRegression-based variant is several times (~2.5-3 times) faster.
So, the plug-in combines DFT (Deeming), Spectral Window (based on Deeming's DFT), and Ferraz-Mello's DC DFT.
image

mpyat2 added 2 commits January 8, 2025 11:16
…bled: I'm not sure that the current algorithm fits this variant of the analysis.
@dbenn
Copy link
Collaborator

dbenn commented Jan 9, 2025

All looking great @mpyat2

I have not reviewed the code yet. I can start doing so now if you are ready.

Noticed one small error. For default low frequency of 0 for spectral window, there is a top hit (and no doubt, data table item) that has frequency 0, so period NaN.

@mpyat2
Copy link
Collaborator Author

mpyat2 commented Jan 9, 2025

Indeed, @dbenn , there is an error. However, it is not on the zero side. The Fourier frequency spectrum of a real-valued signal is symmetric relative to 0 (it continues to the negative frequencies), so we can determine if it is a top-hit (local maximum) or not. On the other hand, we cannot determine if the last point is a top-hit. I fixed this.
By the way, I have not restricted the number of top-hits and have included all local maxima on the list.

I think I'm ready with the plug-in. At first, I thought I would limit myself to SpectralWindow, but then I got carried away :)

…on): in rare cases, equal times may occur in the dataset.
@mpyat2
Copy link
Collaborator Author

mpyat2 commented Jan 12, 2025

Hi @dbenn
I made small changes in the plug-in, and something got broken in the unit tests for Java 9+... Could you please look at them? I cannot figure out what is wrong.

@dbenn
Copy link
Collaborator

dbenn commented Jan 13, 2025

Hi @dbenn
I made small changes in the plug-in, and something got broken in the unit tests for Java 9+... Could you please look at them? I cannot figure out what is wrong.

I recently had some non Java 8 test failures for the plugin I'm working on as well @mpyat2. Odd. Will see what I can find unless you beat me to it :).

@mpyat2
Copy link
Collaborator Author

mpyat2 commented Jan 16, 2025

Hi @dbenn

I've done local tests with

  • java-se-8u44-ri
  • openlogic-openjdk-8u432-b06-windows-x64
  • jdk-9.0.4
  • jdk-20.0.1

Plugin tests succeeded in all cases.
VStar tests failed. In all cases, VSXWebService-related functions failed; PluginManagerTest.testRetrievePluginInfo() failed for java-se-8u44-ri.

See the attached file:
_tests.zip

Also, I needed to set JAVA_TOOL_OPTIONS=-Duser.country=EN -Duser.language=en before running the tests. In my locale, a comma is used as a decimal separator, which causes problems in some tests where floating-point numbers are compared.

With my original locale, I see the errors in

  • TEST-org.aavso.tools.vstar.util.date.J2000EpochHJDConverterTest.txt
    expected:<2457501[.]86941> but was:<2457501[,]86941>
    junit.framework.ComparisonFailure: expected:<2457501[.]86941> but was:<2457501[,]86941>
    at org.aavso.tools.vstar.util.date.J2000EpochHJDConverterTest.testHJDRCar2(J2000EpochHJDConverterTest.java:254)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    <...>

  • TEST-org.aavso.tools.vstar.util.stats.anova.CommonsMathAnovaTest.txt
    expected:<9[.]26> but was:<9[,]26>
    junit.framework.ComparisonFailure: expected:<9[.]26> but was:<9[,]26>
    at org.aavso.tools.vstar.util.stats.anova.CommonsMathAnovaTest.testApacheAnova(CommonsMathAnovaTest.java:60)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)

and

  • TEST-org.aavso.tools.vstar.util.stats.anova.EpsAurVisJD2454700ToJD2455000AnovaTest.txt
    expected:<2[.]91> but was:<2[,]91>
    junit.framework.ComparisonFailure: expected:<2[.]91> but was:<2[,]91>
    at org.aavso.tools.vstar.util.stats.anova.EpsAurVisJD2454700ToJD2455000AnovaTest.testRawEpsAurAnova(EpsAurVisJD2454700ToJD2455000AnovaTest.java:539)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)

@dbenn
Copy link
Collaborator

dbenn commented Jan 16, 2025

The VSX issues could be related to #458

I'm seeing lots on failures too. Will have to return to this tomorrow.

Copy link
Collaborator

@dbenn dbenn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mpyat2 !

@dbenn dbenn merged commit 68ffc75 into master Jan 21, 2025
12 of 24 checks passed
@dbenn dbenn deleted the 461-spectral-window branch January 21, 2025 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request plug-in Plug-in Development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement "Spectral Window" analysis tool

2 participants