Efficiently processing data files

Clear["Global`*"]; DateString[]
SetDirectory[NotebookDirectory[]];

Functions

Functions are a handy way to bundle up a possibly complex calculation (or series of calculations) into a single command.  They are especially well-suited for repetitive operations.

Example:  Reading a LoggerPro data file.

First, take a quick look at the first few lines to see what you are up against:

FilePrint["TO-20250220-undamped-1.csv", 3]

This is a comma-separated file, so import it like this.  Look at the first few lines of the imported data to make sure it is fine.

rawdata = Import["TO-20250220-undamped-1.csv", "CSV"];

[[ ]] takes Part of an array.  Range[3] prints out {1, 2, 3}, so we will be taking the first 3 elements of the array.  TableForm[ ] merely prints it out in a table format.  (MatrixForm is similar.)

TableForm[rawdata[[ Range[3] ]] ]

We want to select all the lines where there are 2 elements and each element is a number.   Fiddle with just the first 3 lines until we are sure we have it right.

Select[rawdata[[ Range[3] ]],Length[#] == 2 && NumberQ[#[[1]]] && NumberQ[#[[2]]] &]

It’s not necessary here, but it turns out there’s a compact way to do those number tests on each element of a list:  VectorQ[expression, test] gives True only if test yields (RefLink[True, paclet:ref/True]) when applied to each of the elements in expression.

Select[rawdata[[ Range[3] ]],Length[#] == 2 && VectorQ[#, NumberQ] &]

So we can apply that to all the raw data.  Again, just as a visual check, we can look at the first few lines.

data = Select[rawdata,Length[#] == 2 && VectorQ[#, NumberQ] &];
TableForm[data[[Range[3]]]] (* Again, look at the first few lines. *)

Lastly, I’d like to convert the voltages to angles by applying my calibration factor:

dθdV = 0.558; (* From my voltage calibration curve *)

θvst = Table[{data[[i, 1]], dθdV * data[[i, 2]]},{i, 1, Length[data] }];
TableForm[θvst[[ Range[3] ]] ]

Bundling it all up as a function:  Think about what you ideally would like to do.  In this case, I would like to write something like:
      θvst = getFile[“TO-20250220-undamped-1.csv”, dθdV].
 The way I did it here, I used temporary arrays ‘rawdata’, and ‘data’, which I didn’t need afterwards.  I could get rid of them by squishing everything into one command, but I can also use a ‘Module’, which allows you to specify such temporary variables which are discarded after the function runs.

(* Read in Voltage vs. Time file from LoggerPro and convert the voltages to angular displacement. Items listed in green are private to this function. *)
getFile[filename_, dθdV_] := Module[
{rawdata, data}, (* List of temporary variables for this function *)
rawdata = Import[filename, "CSV"];
(* Individual commands get separated by semicolons *)
data = Select[rawdata,Length[#] == 2 && VectorQ[#, NumberQ] &];
(* The function returns the last line executed, so don't suppress that
with a semicolon. *)
Table[{data[[i, 1]], dθdV * data[[i, 2]]},{i, 1, Length[data] }]
   ]

(* Test it out *)
θvst = getFile["TO-20250220-undamped-1.csv", dθdV]; 
ListPlot[θvst]

Example:  Finding the undamped frequencies

model[θ0_, γ_, ω_, ϕ_, θoff_, t_] := θ0 Exp[- γ t/2] Sin[ω t + ϕ] + θoff

For the torsional oscillator, we want to find the frequency for each of our 5 trials.  Again, try the commands one by one, and then bundle them all up in a handy function.

fit = NonlinearModelFit[θvst, model[θ0, γ, ω, ϕ, θoff, t], {θ0, γ, ω, ϕ, θoff}, t]

fit["BestFitParameters"]

To assign one of those to a result, such as the frequency, use the /. notation:

ω /. fit["BestFitParameters"]

Bundling it up:

getωfit[data_] := Module[
{fit, ωfit},
fit = NonlinearModelFit[data, model[θ0, γ, ω, ϕ, θoff, t], {θ0, γ, ω, ϕ, θoff}, t];
ωfit = ω /. fit["BestFitParameters"]
]

(* Test it out*)
getωfit[θvst]

You can even chain the two functions together:

getωfit[getFile["TO-20250220-undamped-1.csv", dθdV]]

Operating on multiple files

Of course this is only worthwhile if you are going to do it multiple times.  In this experiment, I have 5 files.  Mathematica can list out the filenames easily, if you used an easy file name system.  The FIlenames[ ] command returns a list (in curly braces) of files matching a specification.  Using the ‘*’ wildcard, I can list out my five files:

FileNames["TO-20250220-undamped-*.csv"]

You can use that list to generate a Table of ω values:  The {f, FileNames[]} construct
assigns the variable ‘f’ to each of the elements in the FileNames list in turn.

ω0s = Table[getωfit[getFile[f, dθdV]], {f, FileNames["TO-20250220-undamped-*.csv"]}]

Mean[ω0s]

δω0 = StandardDeviation[ω0s] / Sqrt[Length[ω0s]]

Expressing the uncertainty with Around[ ]

ω0 = Around[Mean[ω0s], δω0]

Now it is very easy to repeat all that with my damped oscillation files:

ωvs = Table[getωfit[getFile[f, dθdV]], {f, FileNames["TO-20250220-damped-*.csv"]}]

Mean[ωvs]

δωv = StandardDeviation[ωvs] / Sqrt[Length[ωvs]]

ωv = Around[Mean[ωvs], δωv]

Mathematica knows how to propagate uncertainty in many straightforward calculations.

ω0 - ωv

The uncertainty is the same as we get with the standard propagation formula:

Sqrt[δω0^2 + δωv^2]