Efficiently processing data files

Clear["Global`*"]; DateString[]
SetDirectory[NotebookDirectory[]];

Functions

Functions are a handy way to bundle up a possibly complex calculation (or series of calculations) into a single command.  They are especially well-suited for repetitive operations.

Example:  Reading a LoggerPro data file.

First, take a quick look at the first few lines to see what you are up against:

FilePrint["TO-20250220-undamped-1.csv", 3]

This is a comma-separated file, so import it like this.  Look at the first few lines of the imported data to make sure it is fine.

rawdata = Import["TO-20250220-undamped-1.csv", "CSV"];

[[ ]] takes Part of an array.  Range[3] prints out {1, 2, 3}, so we will be taking the first 3 elements of the array.  TableForm[ ] merely prints it out in a table format.  (MatrixForm is similar.)

TableForm[rawdata[[ Range[3] ]] ]

We want to select all the lines where there are 2 elements and each element is a number.   Fiddle with just the first 3 lines until we are sure we have it right.

Select[rawdata[[ Range[3] ]],Length[#] == 2 && NumberQ[#[[1]]] && NumberQ[#[[2]]] &]

There is a more compact way to do those number tests on each element of a list:  VectorQ[expression, test] gives True only if test yields (RefLink[True, paclet:ref/True]) when applied to each of the elements in expression.

Select[rawdata[[ Range[3] ]],Length[#] == 2 && VectorQ[#, NumberQ] &]

So we can apply that to all the raw data.  Again, just as a visual check, we can look at the first few lines.

data = Select[rawdata,Length[#] == 2 && VectorQ[#, NumberQ] &];
TableForm[data[[Range[3]]]] (* Again, look at the first few lines. *)

Lastly, I’d like to convert the voltages to angles by applying my calibration factor.  (The voltage offset won’t be relevant for us since it has likely drifted anyway.)

dθdV = 0.558; (* From my voltage calibration curve *)

θvst = Table[{data[[i, 1]], dθdV * data[[i, 2]]},{i, 1, Length[data] }];
TableForm[θvst[[ Range[3] ]] ]

Bundling it all up as a function:  Think about what you ideally would like to do.  In this case, I would like to write something like:
      θvst = getFile[“TO-20250220-undamped-1.csv”, dθdV].
 The way I did it here, I used temporary arrays ‘rawdata’, and ‘data’, which I didn’t need afterwards.  I could get rid of them by squishing everything into one command, but I can also use a ‘Module’, which allows you to specify such temporary variables which are discarded after the function runs.

(* Read in Voltage vs. Time file from LoggerPro and convert the voltages to angular displacement. Items listed in green are private to this function. *)
getFile[filename_, dθdV_] := Module[
{rawdata, data}, (* List of temporary variables for this function *)
rawdata = Import[filename, "CSV"];
(* Individual commands get separated by semicolons *)
data = Select[rawdata,Length[#] == 2 && VectorQ[#, NumberQ] &];
(* The function returns the last line executed, so don't suppress that
with a semicolon. *)
Table[{data[[i, 1]], dθdV * data[[i, 2]]},{i, 1, Length[data] }]
   ]

(* Test it out *)
θvst = getFile["TO-20250220-undamped-1.csv", dθdV]; 
ListPlot[θvst]

Example:  Finding the undamped frequencies

model[θ0_, γ_, ω_, ϕ_, θoff_, t_] := θ0 Exp[- γ t/2] Sin[ω t + ϕ] + θoff

For the torsional oscillator, we want to find the frequency for each of our 5 trials.  Again, try the commands one by one, and then bundle them all up in a handy function.

Recall that we often had to give a guess to help NonlinearModelFit[ ] find the right solution.  For our next experiment, that will be important, so let’s include a guess here:

ωguess = 5;fit = NonlinearModelFit[θvst, model[θ0, γ, ω, ϕ, θoff, t], {θ0, γ,{ω, ωguess}, ϕ, θoff}, t]

fit["BestFitParameters"]

Check it out visually to make sure it is reasonable.  (Putting the theory first and the data second will draw the dots on top of the theory plot, making it visually clearer.  The other options here just stretch it out nicely and make clear labels.)

Show[{
Plot[fit[t], {t, 0, 30}],
ListPlot[θvst, PlotStyle -> {Red, PointSize[0.002]} ]
},
LabelStyle->Larger, AxesLabel -> {"t (s)", "θ (rad)"}, AspectRatio -> 1/2, ImageSize -> Scaled[0.8]
]

To assign one of the fit parameters to a result, such as the frequency, use the /. notation:

ω /. fit["BestFitParameters"]

Bundling it up into a function.  Note that all lines in the function (except the last) end in a ‘;’ to suppress the output.

getωfit[data_, ωguess_] := Module[
{fit, θ0, γ, ω, ϕ, θoff, t, ωfit},
fit = NonlinearModelFit[data, model[θ0, γ, ω, ϕ, θoff, t], {θ0, γ, {ω,ωguess}, ϕ, θoff}, t];
ωfit = ω /. fit["BestFitParameters"]
]

(* Test it out.  (It is not very sensitive to our guess.)*)
getωfit[θvst, 5]

You can even chain the two functions together:

getωfit[getFile["TO-20250220-undamped-1.csv", dθdV], 5]

Operating on multiple files

Of course all this work is only worthwhile if you are going to do it multiple times.  In this experiment, I have 5 files.  Mathematica can list out the filenames easily, if you used an easy file name system.  The FIlenames[ ] command returns a list (in curly braces) of files matching a specification.  Using the ‘*’ wildcard, I can list out my five files:

FileNames["TO-20250220-undamped-*.csv"]

You can use that list to generate a Table of ω values:  The {f, FileNames[]} construct
assigns the variable ‘f’ to each of the elements in the FileNames list in turn.  (I like to break things out over multiple lines to make the groupings clearer.)

ω0s = Table[
getωfit[getFile[f, dθdV], 5], 
{f, FileNames["TO-20250220-undamped-*.csv"]}
]

Mean[ω0s]

δω0 = StandardDeviation[ω0s] / Sqrt[Length[ω0s]]

Now it is very easy to repeat all that with my damped oscillation files:

ωvs = Table[
getωfit[getFile[f, dθdV], 5], 
{f, FileNames["TO-20250220-damped-*.csv"]}
]

Mean[ωvs]

δωv = StandardDeviation[ωvs] / Sqrt[Length[ωvs]] (* Uncertainty *)

Repeating to find the γ values

getγfit[data_, ωguess_] := Module[
{fit, θ0, γ, ω, ϕ, θoff, t, γfit},
fit = NonlinearModelFit[data, model[θ0, γ, ω, ϕ, θoff, t], {θ0, γ, {ω,ωguess}, ϕ, θoff}, t];
γfit = γ /. fit["BestFitParameters"]
]

γs = Table[
getγfit[getFile[f, dθdV], 5], 
{f, FileNames["TO-20250220-damped-*.csv"]}
]

Mean[γs]

δγ = StandardDeviation[γs] / Sqrt[Length[γs]] (* Uncertainty *)

Bundling both together -- returning more than one value from a function.

You can also have Mathematica return both ω and γ from the fit with a single function call.  The function should return a list (in curly braces):

getωγfit[data_, ωguess_] := Module[
{fit, θ0, γ, ω, ϕ, θoff, t, ωfit, γfit},
fit = NonlinearModelFit[data, model[θ0, γ, ω, ϕ, θoff, t], {θ0, γ, {ω,ωguess}, ϕ, θoff}, t];
ωfit = ω /. fit["BestFitParameters"];
γfit = γ /. fit["BestFitParameters"];
{ωfit, γfit} (* Return a list*)
]

Table[
getωγfit[getFile[f, dθdV], 5], 
{f, FileNames["TO-20250220-damped-*.csv"]}
]

Using Mathematica’s Around[ ] function for Uncertainties

You can use the Around function to express a number with its uncertainty.

{ω0avg, δω0} = {Mean[ω0s], StandardDeviation[ω0s]/Sqrt[Length[ω0s]]}

{γavg, δγ} = {Mean[γs], StandardDeviation[γs]/Sqrt[Length[γs]]}

ω0 = Around[ω0avg, δω0]

γ = Around[γavg, δγ]

You can use the ‘Around’ function to do calculations with uncertainty.  For example, here is the function for computing Q:

Q[ω0_, γ_] := ω0/γ

myQ = Q[ω0, γ]

Or, calculating the uncertainty the long way:

Q[ω0avg , γavg] * Sqrt[(((((δω0)/(ω0avg))))^2) + (((((δγ)/(γavg))))^2)]

This also works for messier equations,  as long as the uncertainties are all “small”.

You can extract parts of an Around object with the “Value” and “Uncertainty” keys:

myQ

myQ["Value"]

myQ["Uncertainty"]