# doc-cache created by Octave 10.1.0
# name: cache
# type: cell
# rows: 3
# columns: 14
# name: <cell-element>
# type: sq_string
# elements: 1
# length: 8
Contents


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 1077
 BIOSIG/T490 contains optimization criteria

 ROC 	        receiver operator characteristics curve
 AUC	        area under the ROC curve
 KAPPA	        kappa coefficient including significance test
 QCMAHAL	"quality" of a mahalanobis based classifier
 MUTINFO	mutual information [2,3]
 Wolpaw_Entropy [3]
 BCI3EVAL			typical bci evaluation [4]
 CRITERIA4ASYNCBCI     	criterion for asychronous BCI 
 EVALUATE_EVENT_DETECTION 	evaluates event detection
 
 see also: NaN/COVM, DECOVM


 REFERENCES: 
 [1] http://biosig.sf.net/
 [2] Schlögl A., Neuper C. Pfurtscheller G.
	Estimating the mutual information of an EEG-based Brain-Computer-Interface
	Biomedizinische Technik 47(1-2): 3-8, 2002.
 [3] A. Schlögl, C. Keinrath, R. Scherer, G. Pfurtscheller,
	Information transfer of an EEG-based Bran-computer interface.
	Proceedings of the 1st International IEEE EMBS Conference on Neural Engineering, pp.641-644, Mar 20-22, 2003. 
 [4] Evaluation of the dataset III of the BCI-competition 2003. 
	http://ida.first.fraunhofer.de/projects/bci/competition/results/TR_BCI2003_III.pdf




# name: <cell-element>
# type: sq_string
# elements: 1
# length: 43
 BIOSIG/T490 contains optimization criteria



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 17
DavisBouldinIndex


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 57
 Davis-Bouldin-Index is a Cluster separation index (CSI)



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 57
 Davis-Bouldin-Index is a Cluster separation index (CSI)




# name: <cell-element>
# type: sq_string
# elements: 1
# length: 3
auc


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 418
 AUC calculates Area-under-ROC curve

 function [AREA,TH,SEN,SPEC,ACC] = auc(d,c,color);
 d     DATA
 c     CLASS, vector with 0 and 1
 color optional, plots ROC curve

 function [AREA,TH,SEN,SPEC,ACC]=auc(d1,d0,color);
 d1    DATA of class 1
 d2    DATA of class 0
 color optional, plots ROC curve

 OUTPUT:
 AREA    area under ROC curve
 TH      Threshold
 SEN     sensitivity
 SPEC    specificity
 ACC     accuracy



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 36
 AUC calculates Area-under-ROC curve



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 8
bci3eval


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 1078
 BCI3eval evaluations a BCI-result as suggested in [1,2].
   - It returns the classification error, the signal to noise ratio,
   the mutual information, as well as mean, standard error and
   standard deviation for both classes.
   - time course of these resulting parameters are supported
   - Missing values can be encoded as NaN.


 [o] = bci3eval(x1, x2 [,DIM])

 x1 is the bci output for class 1
 x2 is the bci output for class 2
 o is a struct with various results


 see also: SUMSKIPNAN, PLOTA

 REFERENCES:
  [1] Schlögl A., Neuper C. Pfurtscheller G.
	Estimating the mutual information of an EEG-based Brain-Computer-Interface
	Biomedizinische Technik 47(1-2): 3-8, 2002.
  [2] A. Schlögl, C. Keinrath, R. Scherer, G. Pfurtscheller,
	Information transfer of an EEG-based Bran-computer interface.
	Proceedings of the 1st International IEEE EMBS Conference on Neural Engineering, pp.641-644, Mar 20-22, 2003.
  [3] Evaluation of the dataset III of the BCI-competition 2003.
	http://ida.first.gmd.de/~blanker/competition/results/TR_BCI2003_III.pdf
  [4] BIOSIG.SF.NET



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 80
 BCI3eval evaluations a BCI-result as suggested in [1,2].
   - It returns the...



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 8
bci4eval


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 2989
 BCI4eval evaluates a BCI-result for two and more classes

   Two classes are evaluated like in [1,2]:
   - It returns the classification error, the signal to noise ratio, 
   the mutual information, as well as mean, standard error, 
   within-class accuracy and standard deviation for both classes. 
   - time course of these resulting parameters are supported

   More than two classes are evaluated with 
   - Kappa coefficient including standard deviation 
   - Accuracy
   
   Missing values can be encoded as NaN.

 X = bci4eval(tsd,trig,cl,pre,post,Fs)
 INPUT:
       tsd     continous output 
               for 2 classes, tsd must have size Nx1 
               size NxM for M-classes, for each row the largest value 
               determines the assigned class 
       trig    trigger time points
       cl      classlabels
       pre     offset of trial start 
       post    offset of trial end 
       Fs      sampling rate;

 OUTPUT: 
       X is a struct with various results  
       2-classes:
               X.MEAN1, XMEAN2: mean of both classes      
               X.ERR           error rate 
               X.p_value       significance level of paired t-test 
               X.SNR           signal-to-noise ratio
               X.I             mutual information
               X.AUC           area-under-the-(ROC) curve
       N(>2)-classes:
               X.KAP00         Cohen's kappa coefficient
               X.Ksd00         standard error of kappa coefficient 
               X.ACC00         accuracy 

               X.MEAN0         average output of non-active class                
               X.MEAN1         average output of active class
               X.SNR           signal-to-noise ratio for each class
               X.I             mutual information for each class
               X.AUC           area-under-the-(ROC) curve for each class
               X.r             correlation coefficient (parametric) 
               X.rankcorrelation    rank correlation (non-parametric)
               X.I_Nykopp      Nykopp's mutual information
               X.I_Wolpaw      Wolpaws mutual information 


 see also: SUMSKIPNAN, PLOTA, BCI3EVAL

 REFERENCES: 
 [1] Schlögl A., Neuper C. Pfurtscheller G.
	Estimating the mutual information of an EEG-based Brain-Computer-Interface
	Biomedizinische Technik 47(1-2): 3-8, 2002.
 [2] A. Schlögl, C. Keinrath, R. Scherer, G. Pfurtscheller,
	Information transfer of an EEG-based Bran-computer interface.
	Proceedings of the 1st International IEEE EMBS Conference on Neural Engineering, pp.641-644, Mar 20-22, 2003. 
 [3] A. Schlögl, Evaluation of the dataset III of the BCI-competition 2003.
	http://ida.first.fraunhofer.de/projects/bci/competition/results/TR_BCI2003_III.pdf
 [4] Schlögl A, Kronegg J, Huggins JE, Mason SG;
	Evaluation criteria in BCI research.
	(Eds.) G. Dornhege, J.R. Millan, T. Hinterberger, D.J. McFarland, K.-R.Müller;
	Towards Brain-Computer Interfacing, MIT Press, p327-342, 2007



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 57
 BCI4eval evaluates a BCI-result for two and more classes



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 16
criteria2005IIIb


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 198
 criteria2005IIIb evalates the criterium according the 
 BCI competition 2005, dataset IIIb. 

  X = bci4eval(...); 
  [crit] = criteria2005IIIb(X)

 see also: SUMSKIPNAN, PLOTA, BCI3EVAL, BCI4EVAL



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 80
 criteria2005IIIb evalates the criterium according the 
 BCI competition 2005...



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 17
criteria4asyncbci


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 1513
 CRITERIA4ASYNCBCI provides an evaluation criterion of asychronous BCI. 
 based on the discussion at the BCI2005 meeting in Rensellearville.  

 X = CRITERIA4ASYNCBCI(D, TRIG, ONTIME, Fs)
 X = CRITERIA4ASYNCBCI(D, STATE, [], Fs)
   D           detector output
               can be continuous; D should contain as many traces (columns) states. 
               [ max(STATES) < size(D,2) ]
               if all values are lower than the Threshold=0, the 
               no-control (NC) state is assumed. 
               If D is discrete, it must contain only one column, but no AUC can 
               estimated. 
   TRIG        list of trigger events in seconds 
               if TRIG is a cell-array of a list of trigger times, 
                       TRIG{n} contains the list of trigger times for
                       state n. The number of detector traces (columns)
                       must be 1+number of states.
   ONTIME      on-time [in seconds] of event-driven discrete control
   Fs          sampling rate (default = 1Hz)
   STATE       "true" state 

   X.H         confusion matrix       
   X.AUC       area-under-the-curve.
 
 X = CRITERIA4ASYNCBCI(D, TRIG, ONTIME, Fs)
               TRIG and ONTIME are used to construct the STATE

   X.TPR       True Positive Ratio
   X.FPR       False Positive Ratio / False alarm rate

 X = CRITERIA4ASYNCBCI(D, STATE, [], Fs)
   D           State of detector output (discrete values, D=0 indicates no-control state) 
   STATE       Target state 
 


 




# name: <cell-element>
# type: sq_string
# elements: 1
# length: 71
 CRITERIA4ASYNCBCI provides an evaluation criterion of asychronous BCI.



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 21
criteria4momentarybci


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 1075
 CRITERIA4MOMENTARYBCI evaluates the output of momentary self-paced BCI  

 [K1,K2,K3] = criteria4momentarybci(TO,Fs,TRIG,ERW) 

 Input: 
 	TO transducer output: 0 is no control state, i>0 is i-th control state
 	Fs sampleing rate
 	TRIG 	trigger information - its a vector in case of a single IC state
		othewise TRIG{K} are the triggers for the K-th IC state
 	ERW 	expected response window in seconds, default=[-0.5,+0.5]

 output:
 	K1 	result the rule-based (heuristic) approach [1] 
 	K2 	results of the sample-by-sample approach
	K3	result by Fatourechi's Method 

 	K1.HF  		hf-difference 
 	K{123}.kappa 	Cohen's kappa coefficient
 	K{123}.H 	Confusion matrix 
 
 References: 
 [1] Jane E. Huggins, Michael T. McCann, …
 	Comparison of Evaluation Metrics for an ECoG-based Momentary BCI.
 	https://ctools.umich.edu/access/content/attachment/5fa4908e-eaba-4e67-8eee-d92169c70726/kappa_vs_HF_windowed2_jeh3.doc
 [2] M Fatourechi, R K Ward, and G E Birch
	A self-paced brain–computer interface system with a low false positive rate
	J. Neural Eng. 5: 9–23, 2008.



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 73
 CRITERIA4MOMENTARYBCI evaluates the output of momentary self-paced BCI  



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 24
evaluate_event_detection


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 484
 EVALUATE_EVENT_DETECTION quantifizes the performance of an 
  event detection algorithm producing a detection trace D. 
  The detection trace is compared to the reference trigger values, 
  
  A sample-based evaluation is used for measuring the performance
  and for optimizing unknown parameters, like delay factor (TAU),
  window length (WINLEN), and the detection threshold (TH). 

  Based on the found detection threshold, parameter of an event-based
  evaluation are reported. 



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 80
 EVALUATE_EVENT_DETECTION quantifizes the performance of an 
  event detectio...



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 5
kappa


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 1767
 KAPPA estimates Cohen's kappa coefficient
   and related statistics

 [...] = kappa(d1,d2);
	NaN's are handled as missing values and are ignored
 [...] = kappa(d1,d2,'notIgnoreNAN');
	NaN's are handled as just another Label.
 [kap,sd,H,z,ACC,sACC,MI] = kappa(...);
 X = kappa(...);

 d1    data of scorer 1
 d2    data of scorer 2

 kap	Cohen's kappa coefficient point
 se	standard error of the kappa estimate
 H	Concordance matrix, i.e. confusion matrix
 z	z-score
 ACC	overall agreement (accuracy)
 sACC	specific accuracy
 MI 	Mutual information or transfer information (in [bits])
 X 	is a struct containing all the fields above
       For two classes, a number of additional summary statistics including
         TPR, FPR, FDR, PPV, NPF, F1, dprime, Matthews Correlation coefficient (MCC) or
	Phi coefficient (PHI=MCC), Specificity and Sensitivity, Youden index (YI)
       are provided. Note, the positive category must the larger label (in d and c), otherwise
       the confusion matrix becomes transposed and the summary statistics are messed up.


 Reference(s):
 [1] Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
 [2] J Bortz, GA Lienert (1998) Kurzgefasste Statistik f|r die klassische Forschung, Springer Berlin - Heidelberg.
        Kapitel 6: Uebereinstimmungsmasze fuer subjektive Merkmalsurteile. p. 265-270.
 [3] http://www.cmis.csiro.au/Fiona.Evans/personal/msc/html/chapter3.html
 [4] Kraemer, H. C. (1982). Kappa coefficient. In S. Kotz and N. L. Johnson (Eds.),
        Encyclopedia of Statistical Sciences. New York: John Wiley & Sons.
 [5] http://ourworld.compuserve.com/homepages/jsuebersax/kappa.htm
 [6] http://en.wikipedia.org/wiki/Receiver_operating_characteristic



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 68
 KAPPA estimates Cohen's kappa coefficient
   and related statistics



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 7
mutinfo


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 318
 MUTINFO mutual information 
 [I,ERR,SNR]=mutinfo(d,c);
   d   output DATA (each column is one trial) 
   c   CLASS, vector with 0 and 1 
 
 function [I,ERR,SNR]=mutinfo(d1,d0);
   d1  DATA of class 1 
   d2  DATA of class 0
 
 OUTPUT:
   I 	mutual information
   ERR	Error rate;
   SNR	signal to noise ratio  



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 80
 MUTINFO mutual information 
 [I,ERR,SNR]=mutinfo(d,c);
   d   output DATA (...



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 7
qcmahal


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 628
 Quality check of multiple discriminator classifier
 It can be used to optimize the LDA-based classifier

 [Q,D]=qcmahal(XC);

 Q	sum of total distance Q=sum(D(:))
 D	Distance matrix (mahalanobis distance of mean to each other)
	D(k,l) gives the distance of the k-th mean to the class 'l'

 [Q,D,R,COV]=qcmahal(XC);
 optimization to reduce the number of classes
 
 Q is list of distances.  
 R is a list how the classes should be summarized
 COV gives the suggested class definition

	Copyright (c) 1999-2002 by Alois Schloegl
	alois.schloegl@gmail.com
	28.02.2001 Version 1.13
	10.10.2001 Version 1.14
	30.12.2002 Version 1.15



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 80
 Quality check of multiple discriminator classifier
 It can be used to optimi...



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 3
roc


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 3456
 ROC plots receiver operator curve and computes derived statistics.
   computes the ROC curve, and a number of derived paramaters include
   AUC, optimal threshold values, corresponding confusion matrices, etc.

 Remark: if the sample values in d are not unique, there is a certain
   ambiguity in the results; the results may vary depending on
   on the ordering of the samples. Usually, this is only an issue,
   if the number of unique data value is much smaller than the total
   number of samples.

 Tratitionally, ROC was defined in the "Biosig for Octave and matlab"
   toolbox, later an ROC function became available in Matlab's NNET
   (Deep Learning) toolbox with a different usage interface.
   Therfore, there are different usage-styles.

 Usage (traditional/biosig style):
   RES = roc(d, c);
   RES = roc(d1, d0);
   RES = roc(...);

   RES = roc(...,'flag_plot');
   RES = roc(..., s);
	plot ROC curve, including suggested thresholds
       In order to speed up the plotting, no more than 10000 data
       points are displayed. If you need more, you need to change
       the source code).

   The ROC curve can be plotted with
       plot(RES.FPR*100, RES.TPR*100);

 Usage style compatible with matlab's roc implementation:
   [TPR, FPR, THRESHOLDS] = ROC(targets, outputs)
        matlab-style interface for compatibiliy with Matlab's ROC implementation;
        Note that the input arguments are reversed;
        targets correspond to c, and outputs correspond to d.

 INPUT:
 d	DATA,
 c	CLASS, vector with 0 and 1
 d1	DATA of class 1
 d2	DATA of class 0
 s	line style (as used in plot)
 targets  DATA, when using matlab-style ROC
 outputs  CLASS when using matlab-style ROC

 OUTPUT:
   TPR   true positive rate
   FPR   false positive rate
   THRESHOLDS  corresponding Threshold values
   ACC     accuracy
   AUC     area under ROC curve
   Yi	  max(SEN+SPEC-1), Youden index
   c	  TH(c) is the threshold that maximizes Yi

   RES is a structure and provides many more results
       including optimum threshold values, correpinding confusion matrices, etc.
   RES.THRESHOLD.FPR returns the threshold value to obtain
	the given FPR rate.
   RES.THRESHOLD.{maxYI,maxACC,maxKAPPA,maxMCC,maxMI,maxF1,maxPHI} return the
	threshold obtained from maximum Youden Index (YI), Accuracy, Cohen's Kappa [3],
       Matthews correlation coefficient [2] (also known as Phi coefficient [1]),
       Mutual information, and F1 score [4], resp.
   RES.TH([RES.THRESHOLD.maxYIix, RES.THRESHOLD.maxACCix, RES.THRESHOLD.maxKAPPAix,
             RES.THRESHOLD.maxMCCix, RES.THRESHOLD.maxMIix, RES.THRESHOLD.maxF1ix])
       return the optimal threshold for the respective measure.
   RES.H_kappa: confusion matrix when Threshold of maximum Kappa is applied.
   RES.H_{yi,acc,kappa,mcc,mi,f1,phi}: confusion matrix when threshold of
       optimum {...} is applied. Its structure is [TN, FN; FP; TP].

 see also: AUC, PLOT, ROC

 References:
 [0] https://en.wikipedia.org/wiki/ROC_curve
 [1] https://en.wikipedia.org/wiki/Phi_coefficient
 [2] https://en.wikipedia.org/wiki/Matthews_correlation_coefficient
 [3] https://en.wikipedia.org/wiki/Cohen%27s_kappa
 [4] https://en.wikipedia.org/wiki/F1_score
 [5] A. Schlögl, J. Kronegg, J.E. Huggins, S. G. Mason;
     Evaluation criteria in BCI research.
     (Eds.) G. Dornhege, J.R. Millan, T. Hinterberger, D.J. McFarland, K.-R.Müller;
     Towards Brain-Computer Interfacing, MIT Press, 2007, p.327-342



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 80
 ROC plots receiver operator curve and computes derived statistics.
   comput...



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 14
wolpaw_entropy


# name: <cell-element>
# type: sq_string
# elements: 1
# length: 585
  WOLPAWENTROPY calculates the Mutual Information according to [1]
   
   B = wolpaw_entropy(acc,M)

   B = log2(M) + acc*log2(acc) + (1-acc)*log2((1-acc)/(M-1))

   acc Accuracy [0..1]  1 = 100%; 
   N   number of classes
   B	mutual information 	 	 
 	

 Reference(s): 
 [1] J.R. Wolpaw, N. Birbaumer, W.J. Heetderks, D.J. McFarland, P. Huntereckham, 
 	G. Schalk, E. Donchin, L.A. Quatrano, C-J. Robinson and T.M. Vaughan.
 	Brain-Computer Interface Technology: A Review of the First Inernational Meeting. 
 	IEEE Transactions on Rehabilitation Engineering 8(2) (Jun 2000) 164-173.



# name: <cell-element>
# type: sq_string
# elements: 1
# length: 80
  WOLPAWENTROPY calculates the Mutual Information according to [1]
   
   B =...





